Detection Engineering Series: Art or Science
Use Cases
In our previous discussion of Detection Engineering, we talked about the foundational aspects of this brewing field within cybersecurity.
As we dive deeper, it becomes evident that Detection Engineering is not just a role but a craft that blends the art of critical thinking with the science of technical analysis.
This follow-up post in the series aims to dissect the technical nuances and highlight the elements that make Detection Engineering an indispensable domain in the Cybersecurity landscape.
The Technical Backbone of Detection Engineering
At its core, Detection Engineering revolves around creating, refining, and implementing detection logic to identify potential security threats (known or unknown).
This means a good understanding of the query language, IaC, or the scripting language you are using for your detection logic. It also means you want to be agnostic as different companies utilize different mechanisms to write and store their detections.
Company A might leverage commercial tools more heavily, and have a Detection pipeline that looks like this: utilizing Cribl stream as your data pipeline, using Splunk as the SIEM, HEC to ingest logs, with Splunk SPL as the detection logic, and storing everything within the savedsearches.conf
Company B might go with a more custom open source approach: utilizing an Apache Kafka topic, Logstash as the pipeline, Elasticsearch Kiabana as the query language (KQL), storing the logic in individual SIGMA yaml files, which then ties into a CI/CD pipeline like GitHub or GitLab.
An example of an approach one company has done in the past is the following
This pipelines uses Kafka as the steaming engine, which feeds into Alertbox (automated workflows at alert time), then calls Forerunner (RPC service). and ties into Jupyter returning a notebook corresponding to the alert that triggered.
To read more about what Dropbox did at the time check out the blog post.
This process as a whole is both an art and a science, requiring a blend of creativity and flexibility as well as technical ability.
Understanding the Attack Landscape
A deep understanding of the cyber threat landscape is crucial. Detection Engineers must stay up to date of the latest attack vectors, malware types, and tactics used by threat actors.
This knowledge informs the creation of detection rules that are both relevant and effective to their organization.
This isn’t just a cursory view of MITRE framework, but an effective understanding of what attacks in their environment could look like.
Crafting Detection Logic
The essence of Detection Engineering lies in the development of detection logic. This involves writing queries in languages mentioned previously such as KQL (Kusto Query Language), SPL (Search Processing Language), or using frameworks like Sigma.
The goal is to create logic that can sift through vast amounts of data to pinpoint anomalies or patterns indicative of a threat.
Example: A Detection Engineer might write a KQL query to detect an unusual spike in failed login attempts on a network, which could signify a brute force attack.
Example snippet for potential brute force login attempts
| where TimeGenerated > ago(1d)
| summarize AttemptCount=count(), UniqueUsers=dcount(UserPrincipalName) by IPAddress
| where AttemptCount > 100 and UniqueUsers < 5
| project IPAddress, AttemptCount, UniqueUsersIn summary this detection is looking for the following
Aggregates data by IP Address
Only returning unique users
When there are > 100 login attempts AND fewer than 5 unique users
Detection Engineers often work with large datasets, analyzing logs from IDS, EDR, SaaS tools, and SIEM systems. Sifting through all this data is a skill to develop.
Doing this helps in identifying outliers or trends that deviate from the norm. This statistical approach is crucial for quantifying the effectiveness of detection logic and for making data-driven decisions.
Additionally, you want to add relevant context to your logs. This enrichment will provide analysts and engineers the data they need to make informed decisions.
Continuous Refinement
The cyber threat environment is dynamic, necessitating the continuous refinement of detection logic.
This involves regular reviews and updates to ensure that detection mechanisms remain effective against evolving threats.
What this could look like in practice is adding conditions into an existing detection to be more specific on what it's looking for, tuning something out, or at times, decommissioning an alert altogether.
It's a process of constant learning, testing, and adaptation.
Simply put, you’re never really done in Detection Engineering, (or Cybersecurity for that matter)
Collaboration and Communication
Effective Detection Engineering requires collaboration with your team and other partner teams, such as Incident Response and Threat Intelligence.
You could have a workflow where Threat Intel brings something to the attention of Incident Response or SecOps, who then tailors the findings to what could be an impact to their environment (since they’re on the ground floor), who then works with the Detection Engineering team on what effective alerts would look like. This then becomes a feedback loop.
Sharing insights and integrating feedback is essential for developing comprehensive detection strategies. This will ensure everyone is on the same page.
Furthermore, the ability to communicate complex technical details to different stakeholders is invaluable.
Within Detection Engineering itself you will most likely participate in a peer review process for Detections, filtering logic, and other project work.
Why Detection Engineering Matters
In today's digital age, the significance of Detection Engineering cannot be overstated. Here are key reasons why this field is pivotal and continuing to grow:
Proactive Defense: By identifying threats earlier in the attack chain, organizations can prevent potential breaches before they escalate.
Cost Efficiency: Effective detection reduces the financial impact associated with security incidents.
Skills Overlap: If you’re paying attention, you noticed some of the skills here can overlap to to other roles such as Incident Response, Threat Hunting, or even Data Engineering.
As we look to the future, the role of Detection Engineering is set to become even more critical.
Advancements in machine learning and artificial intelligence are opening new avenues for automating and enhancing detection capabilities.
However, the human element—creativity, intuition, and judgment remains irreplaceable.
Stay tuned for the next post on this series, and as always, I hope this helped you learn something new.