Real-time Security Analytics for Visibility and Incident Detection
By Brian Smith, CTO, Click Security
Gary Loveman, CEO of Caesars Entertainment, says there are three ways to get fired from the hotel and casino company: theft, sexual harassment, and running an experiment without a control group. Analytics has transformed businesses from operating on hunch and intuition to use experimentation and data to drive decisions. Can analytics do the same thing for computer security?
Consider a hypothetical targeted attack. An attacker, let us call him Ivan, targets Big Corporation X. The attack proceeds in phases:
Phase 1: Compromise. Ivan tries multiple attacks, including direct assaults on corporate servers, social engineering, spear phishing, and using compromised credentials purchased on the black market. Most of these attacks fail, but Ivan is persistent and sophisticated, so eventually he compromises a host.
Phase 2: Installation. Ivan installs malware on the compromised host, establishes command and control, and uses the host as a base of operations.
Phase 3: Lateral Movement and Exfiltration. Armed with credentials and capabilities, he uses the permissions to explore the environment, find his target, and vacuums up whatever data might be useful along the way. He is careful to do all this activity stealthily.
When the breach is eventually discovered, a forensic team reconstructs this sequence by examining logs. They inevitably find that the compromised user or machine was behaving anomalously, but no one had discovered the anomalies or connected the dots.
And therein lies the frustration: evidence of the breach is in the data being collected, yet it goes undiscovered for months or years.
Existing security mechanisms do not prevent this attack. There are three possibilities:
• The device blocks the attack when the communication is prohibited by policy (firewall/ACL) or if the device detects the attack and the chance of a false positive is low.
• The device logs the attack if the device detects the attack and the chance of a false positive is not low.
• The device misses the attack when the device is not in a position to see the attack (e.g., encryption or mobile host), if the device is not configured to detect or log the attack (e.g., signatures turned off for performance reasons or false positive deduction), of if the device is incapable of detecting the attack (the attack is an 0-day, the vendor does not have a detection mechanism).
In Phase 1, Ivan can try different attacks until he finds one that is not blocked or noticed. And the security team is left with logs that may contain evidence of the attack, but the signal is hidden in a mountain of noise.
In Phase 2 and 3, multiple devices are producing logs that contain anomalies, but security analysts have to manually connect the dots. And there is just too many dots to connect, too many things to watch for, and connecting the dots is too slow.
Survey results reflect this reality. ESG asked 257 security professionals working at enterprise organizations (i.e. more than 1,000 employees) to rank their security team’s incident detection challenges:
• 39 percent of organizations say they are challenged by, "a lack of adequate staffing in security operations/incident response teams."
• 35 percent of organizations say they are challenged by, "too many false positive alerts."
• 29 percent of organizations say they are challenged because "incident detection involves too many manual processes."
• 29 percent of organizations say they are challenged because "incident detection depends upon too many independent tools that are not integrated together."
Real-time security analytics means to change this game. Rather than collecting data and waiting for the user to ask questions in a forensics investigation, real-time security analytics proactively analyze the data to unearth anomalous activity that is consistent with breaches then gives the user powerful tools to quickly understand whether the anomaly is important and how it fits into a larger pattern.
The approach involves five components: collecting the right data, using algorithms to identify leads, investigating the leads, automation and tuning, and measuring the results.
1.Collect the data needed for analysis. Collecting every scrap of data in your organization in the hopes that it will someday be useful is a costly exercise. A smarter approach is to identify the data needed by algorithms and investigation, and then figure out how to collect that data in your organization.
2.Identify leads in the data. Let algorithms crunch the data proactively to identify anomalous activity and correlate that activity by actors (machines, users, or groups of machines or users). The algorithms can be behavioral, rule-based, statistical, machine learning, or heuristic. For example, we use machine learning algorithms to detect suspicious DNS names (signs of domain generation algorithms in malware), statistical algorithms to detect periodic communications between hosts (signs of automation), and non-RFC compliant HTTP POSTs (a heuristic associated with command and control and malware), among others.
3.Investigate leads using context and visualization. Once you have a lead, you need to figure out whether it is valid or not. Investigators combine data from relevant databases (LDAP, DNS, Google, Robotex, Virus Total, Who is, geo location, to name a few) and visualizing the results to create a story of what happened. With the right security analytics tool, a skilled analyst can determine whether the lead is valid activity or part of a larger threat.
4.Automate what you learn. In each investigation, your team will learn a lot about the environment. The system should make it easy to encode the knowledge you gain to reduce false positives and to train the system to identify anomalies.
5.Measure the results. Measure which devices are generating data that is leading to useful information, and assess where you have blind spots. Replace devices that are not producing useful data with devices that fill the blind spots.
When implemented and combined, these elements: collecting the right data, identifying leads, investigating leads, automation, and measurement are the cornerstones of modern threat detection and security analytics. They change the game from reactive forensics to a proactive defense and make your computer security operation data driven.