"It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts."

-Sir Arthur Conan Doyle, Sherlock Holmes, 1891

1.1      Introduction

Throughout my career, I've observed complications that arise when teams work to launch new divisions as well as when working to refine, build up, and cultivate a great cyber defense team.

This post aims to share some of my experiences with you, so that teams can be better equipped to navigate hurdles that might be slowing growth as an individual, team, or even an organization. While perfection is an elusive standard, I believe teams can effectively navigate roadblocks that keep them from reaching the next level.

1.2      Identifying and Resolving Obstacles

The obstacles keeping organizations from elevating their cyber defense team to the next level and beyond are usually a combination of multiple problems rather than a single issue. These barriers include bureaucratic delays, inexperience, groupthink, employee burnout, understaffing, and other logistical constraints.

Often, for those in a leadership position, the measurement of team growth as well as the investigative skill development of individual contributors can be difficult to evaluate. For the organization as a whole, a mindset of mutual trust is key. Also, an exceptional SOC is going to be set apart from a standard SOC in a few ways:

• Success is based on being right, not fast.
• Events, incidents, and scheduled offensive testing are viewed as learning opportunities.
• Training is provided for analysts and engineers to boost their skills.
• All groups within the defense team are in constant contact.
  • Building on this, communication between the security team and other departments should be encouraged in order to facilitate better interdepartmental communication.
• Leaders take feedback from their team in order to drive future direction.

1.3      Being Right, Not Fast

It's common to find SOCs being measured by the volume of alerts created or by the number of tickets closed. Usually there isn't much incentive to correctly define the scenario in the alert, as long as a quick triage of whether something is malicious is determined and the ticket closed. This can lead to bad habits and can cause the investigators to prioritize closing the investigation at the expense of making the correct determination.

The ideal situation is to have, or be, a defender who can begin an investigation, analyze data, correlate with extraneous data, and make a correct determination as fast as possible. Timeliness is important in order to ensure malicious activity does not go unchecked, however, a proper and detailed investigation should still be taken in order to currently identify the nature of the activity. 

Defining reasonable thresholds to escalate or close an investigation is an important and critical step to ensuring security events are addressed, but that also ensure that excessive time is not spent on any given issue if it is not warranted. As experience is gained from each investigation, final determinations should take less time to make as different techniques are encountered over time.

In my experience, the mentality of closing out issues quickly to get to the next one comes about due to a large number of incoming alerts from various tools. Measuring based on the volume of alerts is alluring to leadership because it is easy, but measuring both the quality of a detection and the team's ability is more challenging.

The immediate solution to the volume issue is a faster turn-around on alerts and increasing the number of personnel; however, this can have a negative impact. While trying to solve the "problem" of having "too many alerts”, you run the risk of reducing the team's quality output by focusing on prioritizing a clear queue over critically assessing each alert. This method also increases expenses through salary, onboarding time, and allocation of resources training the new workforce.

When critically assessing the alerts for value, you can start to find and correct issues. This is especially true when the issue identified is a result of a misconfiguration and not due to malicious activity. Often, resolving configuration issues with either tuning or deployment corrections can reduce the burden of alert fatigue for analysts. Going even further with this effort can result in a finely tuned queue, where investigators can focus on identifying and mitigating threats within the environment without the stress of a looming, ever-increasing workload.

1.4      Combating Burnout

We don't want our investigators and engineers to experience burnout. Usually, the most skilled, knowledgeable personnel will bear the burden of training the new employees as well as maintaining responsibilities to close the alerts or deploy new solutions. Often, those same employees are personally driven to ensure the security of an organization by investigating all they can. As a result, higher value personnel will often be the first to experience burnout, job dissatisfaction, and even leave a team. Matching up a team's strengths and responsibilities as well as mitigating leadership's expectations in a meaningful way can alleviate the challenges security teams face when under time constraints.

A better use of resources is to focus on tuning out false positives and disabling low-value alerts and/or alerts that have no investigative path or correlative value. This reduces volume and workload and allows for a thorough investigation of alerts that warrant closer inspection. These are all objectively quantifiable achievements. You can also bring in a third party like TrustedSec to perform a Purple Team exercise to assist with building and tuning detections to ensure they are actionable and high fidelity. 

1.5      Learning Opportunities

Ideally, members of a SOC (analysts, SMEs, engineers, etc.) should continually strive to learn and refine skills. In that spirit, they should keep an eye out for new and interesting events, incidents, and techniques that may offer an opportunity to expand their understanding and experience. For example, when an incident or odd event in the environment is noticed, all members in the SOC should view such events as opportunities to learn about attacker techniques and/or how their environments behave. Obviously, analysts do not have the time or bandwidth for every alert to become an adventure, but interesting events should be used as training opportunities for the entire team.

In addition, continuing training and/or scheduled testing of the environment are great opportunities to expand knowledge. Scheduled testing such as Penetration Tests, Red Team engagements, or Purple Team exercises are great opportunities to watch attacker tradecraft as it happens without the pressure of needing to determine malicious activity and act upon it quickly. This assumes the defenders are made aware of the testing and are not being evaluated themselves.

When we perform Purple Team exercises for our clients, we ask about recent Penetration Tests or Red Team events performed within the environment. If the Blue Team hasn't had the opportunity to see the attacks or activity at that time, we like to perform the same activity and allow the defenders to see the activity occur live. This affords local Blue Teams the time to investigate in real time, see the data, and observe how the environment responds to the attacker, tool, system, and defender points of view. Teams in the SOC should be encouraged to dive into different scenarios within the data as events occur.

1.5.1     Training Availability

There are innumerable cybersecurity training courses and certifications, varying in cost from free to tens of thousands of dollars. While quality and content vary just as widely, it is not necessarily tied to the cost of the offering. I've seen many cheap or free trainings and certifications that produce a higher value to the attendee than others that are on the higher end of the cost spectrum.

With that said, a quality cyber defense team should encourage individuals to seek training that will challenge them to improve. While each organization will have to consider what their budget allows, subsidizing part or all of the cost can go a long way to motivate members of the team to expand their skills. Also, allowing some training to be done during work hours will motivate individuals to keep up with their training, which has a positive impact on the individual to begin and maintain their progress. This can be a great opportunity for team members and leaders to collaborate on topics of interest.

As stated previously, anyone who attends trainings or develops their skillset as a result of an experience should share the knowledge with the rest of the team. It can be a nice way to not only boost everyone's skills but also to build connections, lines of communication, and camaraderie. Don't shy away from inviting others! Some trainings may even apply to and benefit teams outside of the SOC. "Lunch and learn" sessions should be held for colleagues upon completion of the training to share what was covered. This is a free or low-cost option that allows for cross-training of team members and engages the whole team.

Can you turn your investigators into great detectives? It's possible. When I was handling alerts, incidents, or even threat hunting I was able to concisely break up parts of the investigation, and when looking into some criminal investigative theory, there are many parallels to cyber investigations. A good start is using the following framework I put together based on my own experience and components of criminal investigative theory. I’ll call it CA(i)GED:

• Collect evidence
• Analyze information
• Interrogate systems
• Generate theories based on collected facts
• Execute response
• Determine most likely scenario from theories

1.6      Constant Contact

Communication is key within all teams but is especially crucial for defense teams. With layered defense strategies, team members must communicate not only with each other, but also with engineering and other support teams. This is even more important for organizations with larger remote or geographically dispersed workforces.

The following chart describes how I view an organization's cybersecurity responsibilities. While you may not have a separate team for each action, the function is most likely carried out in a single team with other responsibilities (and that's OK!). This should help illustrate the following communication requirements for these functions.

Whether each facet of defense is handled by a dedicated team or there is a single team with multiple responsibilities, there should always be regular communication opportunities structured within and between each level. SOC analysts need to communicate and coordinate with not only SMEs but also with Incident Response for training and guidance or to track an event if necessary.

Your SME level is an escalation point for your analysts and should have a close relationship with both level 1 and 2 analysts as well as Incident Response. They need to be a solid contact for engineers when requirements and recommendations for tools and workflows are presented from this and previous tiers.

The engineers need to get feedback from tool users (the analysts and SMEs) on how the workflow feels and what functionality the team needs to refine their process. This may include performance concerns, alert tuning/creation, or augmentation to available data. 

The Threat Intelligence team needs to be in communication with the entire Security branch of the organization. They should be delivering intelligence products to every level and receiving feedback on interests for topics and details. While Threat Intelligence might not be technical experts of tradecraft, they should understand trends and current Tactics, Techniques, and Procedures (TTPs) in the wild.

Once you are at a level where you can deploy proactive or preemptive teams such as Threat Hunting or a dedicated Red Team will also need to be included in communications. Threat Hunters will provide feedback to the engineers for tuning, detections, and requirements for tools. Red Teamers will need to connect with all teams in order to obtain threat intelligence, train defenders, and advise engineers of changes.

A good way to view inputs and outputs of teams is to treat each team as both a customer and a producer. What products are the teams responsible for? Who has a need for those products? What do these teams need to consume in order to operate maximally? Effective communication will make it easier to make these identifications and satisfy customers within their organization.

1.6.1     Trusted Resources

The Security group as a whole should be considered a trusted resource within an organization. I have seen Security teams being treated as enforcers or "no" people. While there may be grounds for this perspective in some contexts, it is generally not healthy to foster power differentials between groups, teams, and departments, especially when they need to work and communicate effectively to deploy and manage layered defense systems. Every negative interaction has the potential to make it exponentially more difficult to deploy security solutions or carry out investigations.

Building and maintaining trust can be difficult, but it is necessary to bring the overall maturity of the Security group to the next level. If a security decision is made for the environment, whether it's during an investigation or even planning and deployment of technology, those affected should feel like partners.

As an example, consider a developer who generates an alert in a security tool. This developer is just trying to build their next release for a new application and does not know that the process they are following can be dangerous to the security of the environment. This could be:

• a library they decided to use
• software they installed on their machine to run tests
• a web server or network share they started on their machine that is exposing sensitive data openly

Instead of approaching the user and telling them to stop that activity immediately, the responder should instead try to understand the developer's workflow or requirements and possibly bring in someone from engineering or architecture to potentially find a secure solution for the development team's needs. One choice will create a negative perception of all security teams, and the other will form a partnership. The developer may still feel as though they are being chastised for their actions, but it is a much more positive interaction.

This scenario can build and/or maintain good lines of communication and trust across departments and teams.

2    Conclusion

While not everyone can be Sherlock Holmes, the dynamics of your team should allow everyone to build upon one another's strengths and fill in areas of weakness. In tandem with building strong cross-business unit communication and relationships, this will help your team evolve into the next level of cyber defenders. In addition, ensure your team is encouraged to build their skillsets as it applies to their duties, use relevant situations as learning moments, and focus on refining the team's ability to reach the best conclusion within an appropriate amount of time.

I want to end as this opened, with the words from Sir Arthur Conan Doyle's A Scandal in Bohemia, "It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts."

Reference:

Doyle, Arthur Conan, 1859-1930. (1930). The Complete Sherlock Holmes. Garden City, N.Y.: Doubleday & Co.