(Editor’s Note: On July 19, 2016 Carbon Black announced its acquisition of Confer. The enclosed blog was originally posted on Confer’s website on January 20, 2016.)
When conducting a proof of concept, the single best piece of advice we can offer is this: Keep it real. To the greatest extent possible, you want to replicate the conditions and behavior that are likely to prevail in an actual attack. And you want to understand what information and tools your defenders can rely on to detect and prevent an attack in progress or to investigate a past event.
- Strike a balance between the lab and the real world.
A well-designed POC will strike a balance between bench-testing live malware in a protected, virtual-lab setting and testing a subset of your real-world production environment under the conditions of an actual attack. While valuable, lab testing is a purely analytical exercise. It’s impossible to simulate a real, dynamic endpoint in the lab. For this reason, your POC should include deployment on at least 20 devices from the general population – the more the better.
- Use a representative malware source.
When it comes to malware, the enemy you know is more dangerous than the one you don’t know. Organizations are most likely to be attacked by the same actors who have attacked in the past, using methods that were previously successful. Your goal, therefore, is not to test against the most obscure or exotic malware, but rather to focus on threats you have already seen.If you maintain a repository of malware samples from past incidents, that’s a good starting point. On the other hand, if you need to rely on the vendor to provide malware test samples, ask questions about how those samples were derived. As we wrote in an earlier blog on security testing (“It’s a Circus Out There,”) vendors sometimes game POC results by repackaging known malware so it evades their competitors’ AV signature checking but is detected by their own engine.
- Avoid this huge blunder.
At the risk of stating the obvious, never test live malware in your production environment. We have seen scenarios where inexperienced security personnel have actually done this, with painful results. For live malware testing, use a segregated network consisting of virtual machines that are immediately reimaged after infection.
- Don’t test on a box that’s already known to be compromised.
When conducting a POC, you may be tempted to “kill two birds with one stone” by including real-world devices your team suspects have already been compromised. We advise against this approach because it presents an incomplete picture of any past attack that has occurred and thus dilutes the informational value that can be derived from the POC results.For example, in incidents where an attacker has already come and gone—possibly without leaving behind any on-disk tools or traces of their attack behavior—you have lost the opportunity to monitor processes that were running during the attack, which limits what you can learn about the attack in hindsight.
- Bring in the Blue Team.
This point, which was mentioned in our January 6 post bears repeating. As part of your effort to “keep it real,” we strongly encourage involving one or two key members of the Blue Team in your proof of concept. Defenders can assess what kind of visibility and functionality the product offers as an attack unfolds, analyze how your defenses react, and evaluate how the product would actually be used in your existing environment.