(Editor’s Note: This blog originally appears on the website of Bit9 + Carbon Black partner Red Canary at redcanary.co)
With endpoint security spending forecasted to grow annually by 48% through 2020, thousands of companies are including Endpoint Detection and Response (EDR) products and services in their security programs. During our discussions with many of these companies, we’ve seen many struggle to define the right questions to ask vendors.
We’ve compiled the following list of questions from both the best questions we’ve been asked and our own experience purchasing security solutions. It should help you build a framework for the features and questions that are important to your organization as you navigate through this process.
1. Why are you investing in an EDR solution?
The most important question! There are many reasons to invest in this next generation of EDR solutions. Knowing yours is the first step to narrowing the field of potential solutions. Following are a common reasons we hear:
- Existing endpoint security suites /anti-virus are failing to stop an increasing number of threats against your organization
- Need better visibility into what is happening in your organization
- A desire to drive down the time and costs of incident investigation and response
- Team does not have the capacity or expertise to engineer the solutions needed to identify and stop threats of increasing sophistication
- Compliance requirements or the threat of large fines are mandating the use of continuous endpoint monitoring
- Leadership focus on preventing a public breach and the associated headlines, brand damage, etc.
- Gartner and Forrester told me I need an EDR solution… everyone else is doing it!
2. Can you trust the vendor and do they have incentive to improve your security?
Take a hard look at the vendor’s business model to understand if they have a financial incentive to improve your security or if they make money responding to breaches their solution failed to stop.
Does the vendor provide incident response and consulting services if you are breached?
Be extremely cautious about EDR vendors that use their products as a loss leader to fuel their incident response business.
Does the solution include access to a technical account management team? If so, what is the background and expertise of that account manager? How many customers does each support?
Beware of “account managers” that are simply extensions of the sales team and do not have the security background to help you understand threats they are detecting and provide advice on improving your security posture.
Does the vendor have paying customers that found the vendor’s solution organically and will provide a reference?
3. Does the EDR solution augment or replace elements of my existing endpoint security investments?
Many of today’s EDR solutions do not directly replace common security investments and focus on strengthening a specific part of your security posture such as email security, application sandboxing, etc. Keep in mind that you may be able to augment another security investment that has a different “purpose in life” if you are only using some of the product’s features.
What does the EDR do for your existing investments in:
- File integrity monitoring?
4. What level of expertise is needed to use the solution?
EDR solutions vary dramatically in the expertise your team needs to get value from the solution. This spectrum can range from requiring your team to be trained in incident response and investigation of operating-system-level endpoint data to fully managed solutions that require your team only to respond to detected threats.
Is your staff required to…
- Manage and maintain the EDR solution’s hardware and software?
- Curate and develop sources of intelligence and techniques used to detect threats?
- Hunt through the data collected by the solution to identify threats?
- Triage alerts to separate false positives from actual threats?
- Identify indicators of compromise from actual threats?
- Respond to threats, including isolation and remediation?
5. What level of visibility does the solution provide?
An EDR solution has limited value if it cannot record and use a rich set of information collected from protected endpoints. Think of it as a physical security system: an optical security camera is vastly inferior to one that also collects X-ray, infrared, and vibration data.
What endpoint activity does the solution record and use to detect threats?
- Process starts, stops, and cross-process injection?
- Network connections?
- File modifications?
- Registry changes?
- Binary / executable / application metadata and full content?
- Memory content and structures?
What happens when the endpoint is no longer connected to your corporate network or internet?
It is important to understand how the solution performs with limited connectivity to either your corporate network where an on-premise server may reside or the internet if the solution is hosted in the vendor’s cloud.
Are tools provided to explore the endpoint data collected by the solution and view both current and historical data?
EDR solutions with true endpoint visibility should allow you to “turn back the clock” and see exactly what happened on the endpoint at a specific date. Ensure the solution’s query system and language is intuitive and not overly complex.
Can all collected endpoint data be retrieved from the solution via an API?
An often-overlooked benefit of the visibility that should come with an EDR solution is your ability to answer security questions such as: “What software in my organization is unpatched?” or IT questions such as: “How many users actually use Microsoft Outlook?” The ability to query the raw data and developed your own reports is essential to increasing your ROI.
6. Does the solution prevent threats from executing?
Many EDR solutions focus on either threat prevention or detection. Prevention solutions must have extremely high confidence that a threat is confirmed before stopping it, and thus use narrower criteria to identify threats (for example, ignoring legitimate applications that are commonly misused such as PowerShell). Detection solutions generally “cast a wider detection net” and require the operator to triage potential threats to separate false positives from actual threats.
Are potentially threatening applications prevented by runtime inspection (think AV, static binary analysis, sandboxing, etc.), or application policies (trusted publishers, installers, etc.)?
Are threats stopped before they execute? If so, how much delay is introduced in an application’s startup?
What is the process when the solution prevents a legitimate application or behavior from executing?
Is significant helpdesk training required to support users impacted by false positives?
7. How does the solution detect threats to your organization?
Understanding both the types of threats an EDR solution detects and what technologies and techniques are used should be central to your evaluation. Many solutions take a very limited approach to detection and are handicapped against solutions that provide broader coverage of threats.
What types of threats are detected?
- Malware (crimeware, ransomware, trojans, exploit kits, etc.)?
- Misuse of legitimate applications?
- File based attacks (Microsoft Office, Adobe PDF, etc)?
- Unwanted software?
- Insider threats?
- Suspicious user activity?
- Suspicious application behavior?
What technologies and techniques are used for detection?
- Behavioral analysis?
- User behavior analytics?
- Long tail analytics and anomaly detection?
- Dynamic binary analysis (“sandboxing”)?
- Static binary analysis?
- Network threat intelligence (known bad domains, IP addresses)
- Binary threat intelligence (known bad MD5s, file paths, binary signing data, YARA, etc.)
What is the solution’s false positive rate? False negative rate?
The accuracy rate of an EDR solution is heavily dependent on the technology and if expert human review is included in the solution. If you have a large and highly experienced security team, you may favor high false positive rates and low false negative rates. If your staff is limited, favor solutions that include security expert review by the vendor and have a very low false negative rate achieved by broad coverage.
Does the solution have the ability to learn and tune itself based on your environment or feedback?
8. What response capabilities does the solution offer?
Look for EDR solutions that includes actionable intelligence about threats and the capabilities to respond so your team can immediately react to the threat and stop it before it does more damage to your organization.
If the solution requires your staff to triage and investigate potential threats:
- Is a workflow provided to claim, investigate, and close potential threats?
- How are potential threats prioritized? Time of occurrence? Severity?
- Can multiple analysts triage threats simultaneously?
- Is context about the potential threat, endpoint, and user provided to the analyst?
Does the solution offer automated response features?
- Isolating an endpoint from the network?
- Killing threatening processes / applications?
- Deleting applications, files, registry keys, etc.?
- Locking a user account or forcing a password reset?
- Re-imaging to a known good state?
Does the solution allow for response regardless of the endpoint’s location/corporate network connectivity?
How quickly can you respond to a threat using the solution?
Can response actions be automated using workflows, policies, or an API?
9. Does the solution integrate with other security and enterprise tools?
Key to seeing a powerful ROI on your EDR solution is integrating it deeply into your existing security and IT tools to allow your team to be most efficient and get the most value from the solution. Ensure the vendor can show you case studies of how their solution has integrated into your, or similar, technology.
Can the solution leverage intelligence gleaned from your other security investments?
Does the solution integrate with sources of “organizational intelligence” such as Active Directory?
How are you notified about threats detected by the solution?
- Web portal?
- SIEM connectors?
- Custom platforms such as PagerDuty, VictorOps?
- Mobile app?
Is bi-directional integration with incident management / tracking solutions available?
Bi-directional integrations allow the EDR solution to create a “case” or “incident” in your incident management systems and synchronize data about your team’s progress on the incident back to the EDR solution so you have a consistent view across systems.
Does the solution provide an open and easy to use API to develop your own integrations?
10. What platforms and operating systems does the solution support?
Ensure the solution supports operating systems, platforms, and variants used by your organization. Ideally a single solution will work across your servers, workstations, laptops, and other endpoints.
Windows? OS X? Linux (RedHat, CentOS, Ubuntu, etc.)?
Legacy versions such as Windows XP, Windows Server 2003, or RedHat/CentOS 5?
Variants such as 32- and 64-bit?
11. What are the infrastructure requirements for the solution?
When evaluating the total cost of ownership of an EDR solution, ensure you include the costs of hardware, software deployment, and network infrastructure enhancements needed to support the solution without adversely affecting your business operations.
Do you need to procure and manage hardware to support the solution?
Many EDR solutions require you to procure high performance hardware hosted within your organization and manage the system’s patching and maintenance – costs that can add up very quickly and that may require expertise outside of your security or IT team. EDR solutions hosted by the vendor in a “Software-as-a-Service” model eliminate many of these costs and complexities at the expense of control.
How much network bandwidth does the solution require?
It is important to understand the load the solution will put on your network when transmitting and analyzing monitored activity.
How is the solution deployed and does it require a reboot?
When evaluating solutions that leverage a software agent or sensor, they should be easily deployed to your endpoints using GPO, SCCM, Altiris, etc. Solutions that require a reboot of the endpoint can have major business impacts and require greater organizational coordination.
12. What is the impact to your endpoints?
Most EDR solutions use a complex agent that is tightly integrated into the endpoint’s operating system, meaning it can have serious performance impacts and cause instability if not well designed and tested. The vendor should be able to show you performance data of their product tested on similar operating systems, hardware, and running similar applications.
How much CPU does the agent use? (Should be < 1%, rarely ever spiking above that)
How large is the agent’s footprint? (Lightweight sensors are ~3Mb, heavy agents ~50Mb)
Does the sensor operate at the kernel level or in userspace?
Kernel level sensors provide greater visibility but require substantially more testing by the vendor to ensure they do not impact the endpoint operating system; user-space sensors are more prone to tampering and lack the visibility into kernel-level attack payloads..
13. What security controls does the solution use to protect itself?
An EDR solution is a large security risk to your organization if the solution is not designed to detect tampering and secure the data collected from your endpoints. Ensure that both the solution and the vendor have strict security policies that are frequently tested by external parties.
Does the solution leverage role based access controls to separate administration, visibility, and response capabilities?
Are user activities auditable?
Is mandatory two-factor authentication available?
What technology and people does the vendor use to protect its internal organization?
Does the vendor undergo frequent security assessments by third parties?
Can the EDR solution detect tampering or attempts to avoid being detected by attackers?
Is data encrypted during transmission and at rest?
If hosted, what physical security does the vendor have in place?
We hope this list provides you with a solid framework for your EDR research (or at least armed you with hard questions to ask us vendors). Don’t hesitate to drop us a note if you have any comments or questions. Good hunting!