You’ve decided to be proactive: Instead of sitting back passively and waiting for attackers to set off alarms, you’re going to pursue them like a cheetah in the bush hunts for its next meal. You know the attackers are out there; they’re trying to break in, and they may be succeeding. The challenge is to start hunting them to find the shreds of evidence they invariably leave behind. In this chapter, you discover what it takes to build a hunting team and start finding attackers.
People: Creating the Culture
Putting together a threat hunting team requires several different aspects. This section contains the essentials as you begin this effort.
The people on your threat hunting team should be knowledgeable about the internals of the operating systems (OS) found in your endpoints. Mainly this system will likely be Microsoft Windows, but it may also include Apple Mac OS and perhaps Linux. What I mean by OS internals expertise is this: Your threat hunters need to know how these OSes work at a detailed level, including the following:
✓ OS process tree structure
✓ Files used by the OS
✓ Registry used by the OS (Windows only)
Expertise at this level of detail is important because malware operates within these domains and makes subtle changes to the OS here. Threat hunters need to understand what to look for and what “normal” looks like (understanding what’s normal at the business‐application and human‐activity level — it’s not just about packets on the network and processes in the OS), so anomalies will be more apparent. And remember, it’s anomalies that are the primary sign that malware is lurking in endpoints.
After you know what expertise is needed, it’s time to figure out who has these expert skills and work on bringing them into the threat hunting team. Depending on the size of the organization, this team may be one person or an entire crew, and wherever you get them from, you’ll need to figure out how to reallocate roles and responsibilities so you don’t leave other teams short‐handed. For instance, you might identify one or more talented systems engineers or analysts from your security operations center (SOC) — this team is usually known for passive monitoring of security events.
Making time to threat hunt
Unless Daddy Warbucks has tossed you a new bag of money to build your threat hunting team, you probably need to carve out time from the work schedules of existing staff for threat hunting. Depending on how large or small your organization is, how many hours a week you need to spend in actual threat hunting may vary. In part, it depends a lot on your security posture and your risk tolerance.
Start with two to four man hours a week that are dedicated to hunting. When you see results from your hunts, adjust as needed. The important thing is getting results from your hunts so that they show a return on that time investment. It’s all about allocating the time and committing yourself to results.
Your threat hunters need to have passion! They must think like predators and have a hunger to hunt adversaries. After that important characteristic comes other trained skills such as the following:
✓Operating system internals: This skill is critical for threat hunters. They need to not only understand the rules and practices of process management but also the file system operation and network communication in each operating system in use.
✓Endpoint application behavior: It’s important for your threat hunters to understand how any locally used applications function on your endpoints.
✓ Threat hunting tools: Your threat hunters need to thoroughly understand how to use the tools at their disposal, so they will be effectively hunting for attackers.
✓ Incident response procedures: Your threat hunters need to understand what steps they need to take when they discover signs of intrusion in your systems and then they need to preserve that evidence for potential future legal proceedings.
It’s not enough to equip your threat hunters with the skills and tools to find their prey; they also need to know what to do when they catch them.
Put processes in place
Threat hunting needs to be a structured, long‐term effort. But first, there must be a vision for what threat hunting is about in an organization and how it works with other IT and IT security processes. An essential part of this is a means for learning several things, including the following:
✓ Endpoint baselines: You need to continuously hone your threat hunters’ knowledge of what constitutes “normal” in your endpoints, so anomalies can be more quickly recognized.
✓ Improving hunting tools, practices, and skills: You want your hunts to become better over time, and you want your new threat hunters to be able to quickly learn from the seasoned warriors on your team. In part, this is about tribal knowledge, but it also needs to include a knowledgebase, so each new threat hunter can stand on the shoulders of his or her predecessors.
✓ Improving response: Finding prey requires response that includes containment and remediation. Mainly, this means doing these things more accurately and also more quickly.
✓ Improving skills: Your threat hunters need to improve their skills and knowledge, not just from threat hunting itself, but from continuing education on ethical hacking, system and network internals, and incident response. It’s essential for your threat hunters to understand what’s “normal” in your organization so they can quickly identify anomalies that may be signs of intrusions. The local context that humans have makes all the difference in detection.
Technology: Getting the Necessary Tools in Place
Threat hunting is a man‐machine activity — you can’t do it with just people or just machines. Without the right tools in place, your threat hunters are going on a safari with nothing. Without threat hunting tools, there’s no hunt.
Endpoints are today’s battleground where intrusions into enterprises begin. Endpoints are the attackers’ crown jewels, and they’re used to make a landing into your environment. Endpoints are everything. And while the data that attackers are looking for lives on servers, access to servers starts with endpoints.
Endpoint visibility is the ability to capture, in detail, the activities going on inside of every endpoint:
✓If your organization allows Bring Your Own Device (BYOD), you have to achieve this visibility on those machines, too.
✓ Include information about every process, including its parents and children, as well as every file that’s created, read, written, and removed, plus network activity. This information needs to be able to be queried across the entire organization, so your threat hunters can quickly understand what anomalous activity is going on at anyplace and at any time.
✓ Another very important aspect of endpoint visibility is known as retrospection, which is the ability to hunt back in time. For example, you mine the data for suspicious activity that took place not just yesterday, but last week, last month, or even earlier.
Obtaining the necessary network event data
In addition to endpoint visibility, having access to network event data is essential. Sometimes the first sign of intrusion is in the command and control (C&C) network traffic from a bot that has already compromised an endpoint. Intrusion prevention systems (IPS), web filtering, firewall logs, and netflow tools are good sources for obtaining this data. Threat hunters need to be able to reference one or more of these tools from time to time to better understand what’s going on in the network.
Threat intelligence gathering
Threat intelligence feeds inform your threat hunters of the new tools and techniques that attackers are using against other organizations, as well as the domains and IP ranges they may be using. Threat intel feeds are often high volume and are delivered in structured formats such as Structured Threat Information Expression (STIX) and OpenIOC (and Cyber Observable Expression [CybOX]), all designed to be fed into your security information and event management (SIEM) system or other threat management platform.
Integrating your information
Remember that threat hunting is a man‐machine activity. In many respects, there is a high volume of information on
threats and activities in your environment. To make the most of this information, you want to understand what tools you’re using and where there may be opportunities to integrate them.
One great example is the fusion of your endpoint data, SIEM data, and threat intel feeds. By themselves, they’re useful, but when fused together, they’re far more valuable. For instance, threat intel feeds often use STIX, TAXII, or CybOX for structuring this data. APIs for these are available so that you can consume this data and get it into your other systems.
Data correlation and analytics tools
Because threat and event data is coming in from a lot of different places, you need to be able to perform event correlation and analytics to make sense of what’s going on in your environment. The tool of choice is SIEM.
SIEM systems are made for event correlation and analytics, and they do a pretty good job. They’re often used as a central repository for log and event data from network devices, firewalls, operating systems, and applications. It’s the storage for everything going on in your environment, together with the ability to make sense of it.
People and Technology: Know Your Environment
Successful threat hunters need to know as much about your environment as possible, so they can better sense what’s normal and what’s abnormal. But as they proceed in their threat hunts, in many respects they begin to have a more intimate familiarity with your environment than anyone else.
What’s normal and what’s abnormal?
The key to threat hunting is knowing what’s normal so that anything abnormal will stand out and be noticed. Because of this, threat hunters spend a good part of their time observing and becoming more familiar with normal, routing events in their environments.
However, threat hunting takes more than just observation. Threat hunters also need to be familiar with their organization’s architecture: networks, systems, tools, and applications.
Mainly, they need to understand this independently of their threat hunting, because anything they might observe in the environment may or may not be normal in the first place. What your threat hunters find and consider normal includes things that are there but aren’t allowed. Occasionally, threat hunters discover things that aren’t necessarily security incidents; instead, they’re insiders with poor judgment.
Know your high‐value targets
In goal‐oriented sports, teams defend goals against the opposing team and try to prevent them from scoring. In threat hunting, threat hunters need to know what the goals are. Depending on the attackers and their objectives, this could be information like customer or employee data, or it could be critical assets such as public facing web servers. Threat hunters need to know all these high‐value targets (HVTs) — the likely ones and those less so. And, they need to understand how attackers might go about attacking them.
Anticipate how you’ll be attacked
Just as a cheetah anticipates the next move of its prey, threat hunters need to know how attackers are likely to try to get into their environments. This is part gut feel and part knowing your environment:
✓ Architecture: Attackers are going to try and figure out the weak spots in an organization’s architecture and data flows. This helps them discover whatever valuable data they’re looking for and how to get it out unnoticed.
✓ Security posture: Attackers are going to go for an organization’s weak spots. They discover them through simple techniques like port scanning to find unpatched and vulnerable systems. Consequently, your threat hunters need to know where the organization’s weak spots are because attackers are going to find them and exploit them.
✓ People: The security culture of an organization is a great indicator of vulnerability. While attackers might not have ready access to security awareness training or other aspects of an organization’s security awareness program, attackers will be able to gauge how easy it is to lure your employees into clever social engineering, phishing, and spear phishing campaigns, whether they’re purely online or on site.
✓ Threat intel: Understanding how attackers are going after other organizations gives your threat hunters a better idea of how they may go after yours. While they will get creative and be unpredictable at times, attackers are people too — creatures of habit and apt to use tools and techniques they’re used to and what has worked for them in the past. Because organizations tend to protect themselves in similar ways, attackers are likely to attack in similar ways.
Your threat hunters need to know your environment inside and out: How does everything work, where are the gaps and weak spots, and where are the risks? They need to think like attackers, so they can better anticipate their threats and stop attacks early.