Most security teams understand the importance of log collection and building detections to provide early indicators of anomalous or potentially malicious activity. However, what is often forgotten is testing the detections to ensure they fire when the specified activity is generated. This blog post will discuss the value of actively testing detections, common pitfalls in failed detection triggering, and how purple teaming can assist in building better, higher fidelity detections.

Detection Engineering and Common Pitfalls

Detection Research Engineering (DRE) has become a more prominent subsection of cyber security work in recent years as the emphasis on high-fidelity detections has grown. In general, though, having worked with organizations of all sizes, we see many common issues across enterprise security teams, including:

• Improperly tuning/baselining detections (out of the box or otherwise)
• Relying too much on tool-based detections/not properly evaluating detection logic (Gap Analysis)
• Overreliance on Endpoint Detection and Response (EDR) for endpoint-based detections
• Incorrect audit configurations for logging
• Broken system access control lists (SACLs)
• Security Information and Event Management (SIEM) parsing issues

Tuning and Baselining Detections

Few enterprises have dedicated DRE teams that test and build detections internally.

This means that an overwhelming amount of DRE work takes place on the side of contracted Managed Security Service Providers (MSSPs) or out-of-the-box detections that are default to the organization's SIEM. While this is not a bad option, detections implemented in this fashion typically require significant tuning and adjustments that often are not performed by the parties involved. As a result, such detections are often disabled and/or over-tuned, filtering out potentially malicious activity that could occur.

Tool vs Technique-Based Detections

In addition to issues with un-tuned detections, there is often a lack of consideration on how detections are actually detecting attacker Tactics, Techniques, and Procedures (TTPs). This typically manifests in the following forms:

• Detections are implemented with a tool-based focused and NOT on detecting underlying techniques.
• Detections are implemented without reviewing the logic to identify potential gaps.

Briefly, tool-based detections are typically built by leveraging certain functionalities of a specific tool, i.e., Mimikatz or Impacket, whereas technique-based detections use the underlying behavior of the attack and the artifacts that may be generated on a system.

Both detection methodologies have drawbacks. Tool-based detections can be evaded if tool signatures or code bases are changed (e.g., renaming Mimikatz); however, technique-based detections are not always feasible and can sometimes have higher degrees of false positives depending on the attack scenario being detected (e.g., Kerberos unconstrained delegation attacks, password sprays).

A healthy DRE program should seek to balance both types of detections, placing a focus on technique-based detections and supplementing with tool-based detections (e.g., a detection for LSASS credential dumping in tandem with a Mimikatz specific detection).

Lastly, all detections should be analyzed and gaps in detection logic capabilities should be understood and documented. This will assist in identifying critical targets for future detection development, as well as building a deeper understanding of where gaps exist that require additional telemetry or may need to be compensated with other defensive operations such as Threat Hunting.

Overreliance on EDR

The next common issue we see is a critical overreliance on EDR tooling for endpoint logging and detection.

EDRs have come a long way from the AV software implementations of the past, and many now include SIEM-like functions and the ability to build custom detections based on endpoint activity. However, while they have evolved, so have attacker techniques, and new methods are being created and developed by red teamers and threat actors alike that seek to bypass these tools and render them useless or significantly impaired.

While Windows event logging is not infallible, having an additional information source that is logged to an external tool, such as a SIEM or data lake, can be a life saver when attempting to recreate an attacker’s steps to understand the extent of a potential compromise.

In addition, while EDR telemetry is overall solid and reliable, many EDR tools still require a detection to trigger an investigation and may have shorter log retention durations than logs forwarded to a SIEM.

Incorrect Audit Configurations for Logging/Broken SACLs

A less visible pitfall that we often run into is incorrectly configured logging and undefined/misconfigured SACLs for event IDs that require additional configuration, such as:

• Event ID 4662
• Event ID 5145
• Event ID 4663
• Event ID 5136
• Event ID 4688

This can be difficult to identify without performing testing that targets specific attacker activity and behavior that may reveal issues in SACL configurations. This and other audit configuration issues, if left undiscovered, can easily lead to false negative detection.

The Solution - Active Testing and Purple Teaming

Now that we’ve outlined some of the key observations and challenges that can occur within the DRE process, it’s time to discuss how we can solve the problem. In this case, Active Testing.

Actively testing detections using Purple Teaming techniques (a combination of offensive and defensive technical skillsets) is the fastest and most efficient way to identify and remediate issues within the detections and logging of your security environment.

Implementing Active Testing

While this article will not go into depth on how to build Active Testing programs, from an internal perspective, organizations who have solid blue and red teams already established should strive to collaborate in a way that ensures detections are firing as expected, and to identify issues in coverage that may be introduced by gaps in query logic or missing telemetry.

However, for security teams that may not have the internal resources to perform testing, third-party services, such as TrustedSec, can perform Purple Team exercises that are specifically designed to perform active testing to improve detection fidelity, evaluate logging/telemetry within your SIEM, and evaluate your security posture as a whole.

Conclusion

Regardless of a security program's maturity level, proactive detections and detection engineering are and will continue to be an integral part of the security ecosystem in the future. Leveraging Purple Team and Active Testing assessments to test EDR tools, SIEM logging, and detections from the perspectives of both tools will provide a better understanding of your environment, the capabilities of your tooling, and will help identify and fix detection gaps that may introduce blind spots in an organization’s defense infrastructure.