(Editor’s Note: The text appears in the free eBook: “Threat Hunting for Dummies.”)
After you’ve been threat hunting in an environment forsix months, a year, or more, you’re going to become a senior in most circles. You’re expanding your skills and knowledge, you’re building and using tools, and you’ve begun to mentor others. You’re becoming a master threat hunter. Or you want to be. Read on to discover how to grow your expertise so you, too, can get there.
Raising the Bar
✓Improved defenses: As you chase down intruders and deny their return, you’re closing down one vulnerability after another. Over time, this begins to severely limit the available techniques that can be used for successful intrusions.
✓More detection: You’ve updated your defenses based on what you’ve learned from previous incidents. Each time
you catch an intruder, you’re able to catalogue these new attack vectors to immediately gain visibility into subsequent attempts.
✓ Infrastructure familiarity: As you’ve been chasing intruders all over your organization’s environment, you’ve become intimately familiar with it — perhaps more so than its own designers and engineers. Being an expert in defense, you’ve been able to impart several useful suggestions to tighten things up from an architectural perspective. You also will have an understanding of where the organization may be weak in detection or response capabilities and be able to offer suggestions for additional tools that could be used to allow for a better overall defense.
✓ Better instincts: As you gain experience threat hunting in your environment, you begin to build an instinct for
discerning abnormal activity as well as the way in which the next intruders might attempt strike . . . and you’ll be there to catch them when they do. This continuous improvement is partly about your organization and its improved defenses, and the rest is about your growing prowess as a black belt threat hunter.
Achieving master threat hunter status doesn’t signify arrival. Rather, it represents your outlook and your discipline. You know the enemies and how they work, and you’re determined to always be learning so that you can be one step ahead of them and anticipate their next moves. It requires constant vigilance and focus.
Be Embedded in the Environment
With the hunting tools at your disposal and your ability to look deeply into any server or endpoint in the organization, you’re certainly embedded in the technical environment. The focus here needs to be about how you work with others in the organization. While threat hunting can sometimes be depicted as the activities of a solitary threat hunter surrounded by the cool glow of monitors, long hours after dark hunting for evil, more often than not, a threat hunter is a collaborator, known across IT and involved in its many varied teams.
That’s right — you need to work with teams across all of IT as they discuss the business of the day and their current projects. To defend your organization’s environment, you must work closely with these teams as they build and run the IT environment. Mainly this is because:
✓ You need to understand what they built. As you observe system operation, interaction, and data movement, you
need to work with people who understand how systems were designed, built, and implemented. This knowledge helps you better distinguish anomalies from legitimate operations.
✓ You need to understand what they’re building. Given that most IT environments grow organically, you must be
involved in this change. As you work with teams in IT and build trust with them, they’ll tell you more about their projects — the new things they’re building. There are two reasons you need to be involved:
• You need to understand how their new systems work, so your understanding of what’s normal is accurate.
• You may need to advise them to make design enhancements based on your knowledge of the current threats and adversaries facing the organization today, so those new systems will be more secure by design. What a concept, right?!
Your relationships with the teams in IT serve you well. As you work with these teams over months and years, your role as a subject matter expert will foster trust, and these teams will rely on you to provide them with accurate and reasonable guidance for improving the environment’s defenses. They’ll be more apt to take your advice and incorporate more and better security practices into the new projects they’re working on. And this is why you’re there — to help everyone in IT build and administer systems and networks that have better defenses.
One of the keys to being a master threat hunter is your insatiable desire to learn more. You want to know about the
newest exploit or that latest tool. As you dive into this field the more you know the more you want to learn, so you do
some of your own research. You need to run your own experiments to see how things work, so you build your own lab environments and test ranges. This process can include probing the malware you’ve captured to play with an exploit kit you found or reviewing experimental changes in systems to make them more resistant to attacks.
You might also be building newer and more complex queries with your threat hunting toolsets and trying to see if there are any new “hits” against a dataset containing a new batch of attack vectors. You might not have a crystal ball, but as you gain experience, you’ll constantly be thinking of new ways that intruders can try to penetrate your environment . . . and how you can stop them.
Pragmatically, your research helps you design better hunting techniques to validate your suspicions. You know where the weak points are, and it’s up to you to discover new ways to watch them. These methods include new traps, new triggers, and new filters that you can use to tighten down your environment a bit more. And sometimes, on that very rare occasion, your research might even lead to you discovering that rare holy grail of all vulnerabilities, a previously unknown zero‐day. It’s at times like this when all the late nights of wrestling with your environment and trying to probe it for security weaknesses pay off. That feeling of elation and satisfaction that you’ve found a vulnerability that no one else has ever thought of before is the greatest rush, and it’s just incredible.
A master threat hunter develops a “sixth sense” when it comes to the hunt: After enough time, he sees attack patterns
emerge out of a collection of seemingly unrelated data points. He begins to recognize reconnaissance and the intended activities behind the exploit and dropper tools that adversaries are using. At times, this can even lead to the threat hunter being able to predict what intruders might do next so they can be stopped.
Another perspective on intuition is this — the threat hunter can also put himself in the shoes of the attacker and see the environment as a potential target and anticipate the next move by the attacker given this understanding of how he sees you. Thinking like an attacker separates the master threat hunters from the rest.
Threat hunting isn’t just all about taking blind leaps; it’s also about making educated hunches — educated perhaps by new pieces of intelligence that showed up in a threat feed or something you recently read about like a new exploit in the wild. You can follow leads in other ways, as well, which include reviewing indicators from monitoring tools like an intrusion prevention system (IPS) that can alert personnel to traffic and discovering low‐reputation IPs or endpoint antimalware sandboxes firing off notifications about an application pivoting in a way that it shouldn’t.
Intuition is also about OODA. Observe, Orient, Decide, Act. This is the military’s way of responding to situations in
combat operations. You’re a threat hunter; you’re in combat as well — on the cyber battlefield. An example of OODA
applied would go something like this:
✓ Observe: Collect data from sensors on your endpoints and events in the network.
✓ Orient: Discern what this data means in context. How does this information relate to other information and what could it mean? Could command and control (C&C) traffic be occurring, or could one of your endpoints be
under attack from a ransomware variant?
✓ Decide: Make a decision about what to do. After you have a clear picture regarding an incident, the next step
is to determine a course of action. Typically, this is the containment phase in which your incident response
strategy will kick in. Only after the breach has been scoped should you proceed to the eradication and subsequent
recovery and feedback stages to prevent similar intrusions from recurring.
✓ Act: Execute the plan to shut down the intrusion, harden the organization’s security posture, and enhance detection.
While many times your hunts might return “empty” and no intrusion will be discovered that leverages that particular
vulnerability, the knowledge created is incredibly valuable because you’ve created a series of processes and detection
mechanisms that serve to harden your organization against future potential incursions.
Strong opinions, loosely held
One way to grow in knowledge about the systems and data in an environment is to mentally build a model representing how they work and interact together. The same principle holds true as you learn how an attacker might attack an organization:
You can study and develop models that represent how these actors operate. As you continue to develop your security acumen, you may notice a tendency to stand inordinately firm in certain beliefs and opinions:
✓ Operating systems always open files like this.
✓ Intruders would never attack this program.
The mental models in your subconscious are what help you understand complex topics and navigate them with ease.
However, while these constructs can be helpful to simplify certain concepts, you must never become too entrenched in a certain way of thinking because you blind yourself from new ways of thinking. This case holds doubly true in the security field where, especially with new technology, the only constant is change. You must be open to changing your understanding about things when new information comes in. This is known as strong opinions, loosely held, which is the safety valve that helps you recognize new facts that may change the way you think about things — like how operating systems and applications do what they do and how attackers do what they do.
If you cling to your time‐honored beliefs too tightly, your hunts may suffer and you may not only return with no prey, but also you could become the prey.
While many times your hunts might return “empty” and no intrusion will be discovered that leverages that particular
vulnerability, the knowledge created is incredibly valuable because you’ve created a series of processes and detection mechanisms that serve to harden your organization against future potential incursions.
Developing Your Own Tools and Custom Integrations
Master threat hunters don’t just rely on the tools and interfaces handed to them by vendors. Instead, they view these
resources as just a starting point and work to engineer ways to extend and correlate the data and capabilities of these
tools to build a system in which the whole is greater than the sum of its parts:
✓ Custom data collection scripts and analyst tools: Master threat hunters may, from time to time, need to write their own scripts to collect or analyze data. One example of this could be writing a simple WMI script to collect various instances of persistence in the Windows registry.
Another could be building a python utility to generate analytics on a set of metrics to discover anomalous data
points. Typically, master threat hunters are no strangers to leveraging powerful instruments like pivot tables and
regular expressions to twist collections of data for a specific purpose.
✓ Custom integrations: Chances are there are a lot of tools in the environment, many of which may have APIs or interfaces that can be used to acquire or distribute information.
For instance, a trigger in an endpoint detection tool could activate the creation of a new IPS or firewall rule used to
block a particular network connection. Or, information from a threat feed could be filtered and fed into a tool to
update its own rules that could then action a ticket over to the help desk or even isolate a system on the network.
Master hunters aren’t just clever operators — they’re also builders — often they’ll act as both the problem finder and
the problem solver. They must be able to not only understand how new attacks work but how to “stitch” together the various pieces of information available in the environment to enhance visibility and defenses.
A master threat hunter thinks ahead and anticipates what a known or a potential adversary might do. In this scenario,hunters can set landmines for attackers. These methods attempt to attract attackers so that an alarm can be raised to alert security that illegitimate activity may be occurring in the environment.
When using incident detection and response tools, this means setting up queries for events that might happen. This is again where it’s critical to fuel your passion to learn about new, clever attack vectors. As you continue to develop your mental cyber armory, you’ll learn how to probe sections of the environment where you previously didn’t have visibility.
In addition to your standard hunting tools, you can leverage other more advanced resources, such as honeypots, in an attempt to lure malicious actors into attacking a decoy target loaded with intrusion detection monitoring sensors. Instead of actually housing legitimate data, a honeypot is built to impersonate critical assets while having extremely sensitive monitoring and alerting configured.
In certain organizations, you might even go one step further to create honey accounts, which contains one or more honeypots, and set up user accounts that follow certain naming conventions for VIP users, and monitor for any access attempts (meanwhile, the VIP users are assigned other legitimate
SANS and Other Training
SANS uses the very best experts — the journeymen (and women) in the security world — as speakers at SANS conferences and instructors at SANS training events. Engineers, analysts, architects, and fellow hunters are among SANS speakers and instructors.
Sure, courses on threat hunting from SANS are terrific, but you shouldn’t stop there. Also consider one or more of these courses:
• SEC401: Security Essentials Bootcamp Style
• SEC504: Hacker Tools, Techniques, Exploits, and Incident Handling
•SEC511: Continuous Monitoring and Security Operations
• SEC542: Web App Penetration Testing and Ethical Hacking
• SEC503: Intrusion Detection In‐Depth
• SEC561: Immersive Hands‐On Hacking Techniques
• SEC617: Wireless Ethical Hacking, Penetration Testing, and Defenses
• SEC660: Advanced Penetration Testing, Exploit Writing, and Ethical Hacking
• FOR408: Windows Forensic Analysis
• FOR508: Advanced Digital Forensics and Incident Response
• FOR610: Reverse Engineering Malware: Malware Analysis Tools and Techniques
This list is but a small sampling of the courses available. You can find a complete list at www.sans.org/courses. I urge
you to challenge yourself and add to your skills and knowledge through continual exploration and learning, as a master threat hunter would do.
In addition to participating in academic security training, you can embed yourself within the security community. This immersion will ensure that you’re constantly being exposed to the latest defensive (and offensive) techniques in the industry.
To get you started, here is a short list of must‐attend conferences:
✓ Black Hat USA: blackhat.com
✓ DEF CON: www.defcon.org/index.html
✓ DerbyCon: www.derbycon.com
✓ InfoSec World: infosecworld.misti.com
✓ RSA Conference: www.rsaconference.com
✓ ShmooCon: shmoocon.org
✓ BSides: www.securitybsides.com
Carbon Black Community
Threat hunters benefit from training and networking with other threat hunters so they learn more about the threat hunting tradecraft. Another valuable activity is the sharing of threat activity among threat hunters.
Carbon Black established its community, serving as a forum for Carbon Black customers to share and exchange threat intel and attack information. Threat hunters can learn from each other, and their respective organizations, to ensure all‐around security. You can find more information here.