Difficult Conversations: Your Detection Coverage Map Is a Lie
TL;DR: Coverage maps derived from detection content libraries (or worse, vendor assertions) will never answer the questions stakeholders are actually asking. Adversary emulation and simulation are the answer people are looking for but aren't asking for.
An application owner comes to you. They're responsible for a critical business system, they have a compliance discussion coming up, and they want a coverage map showing which MITRE ATT&CK techniques your detection program covers for their application. They've done their homework. They're not asking for a rubber stamp. They want to know if you'll actually catch something targeting their system.
If you have a coverage map, it's already broken. If you don't, building one won't give them what they're asking for. Either way, the green squares won't answer the question.
This conversation is about to get difficult, because the language of our industry equates security monitoring maturity with coverage mapped to the MITRE ATT&CK taxonomy.
The Industry Asked for a Bingo Card and Got One
MITRE ATT&CK is a taxonomy of observed adversary behavior. It was built to give defenders a shared vocabulary for describing how attackers operate. That is what it does well. It was never designed as a test harness or a coverage certification framework.
The industry handed it to vendors as a checklist anyway.
Compliance frameworks needed a way to measure detection coverage. ATT&CK provided a structured list of techniques. The logical move, in that context, was to ask vendors to map their detections to the list. Vendors complied, because producing a heat map with green squares closed deals. Buyers rewarded vendors with good maps. Buyers presented those maps to auditors. Auditors accepted them. The vocabulary of coverage became the vocabulary of ATT&CK technique IDs.
Nobody in that chain had a strong incentive to ask whether the map measured anything real. The result is a precision instrument built to measure the wrong thing.
The Map Was Never the Territory
Binary coverage is wrong by construction, not just imprecise. Five independent failure modes explain why, and each one is sufficient on its own.
Technique variation. An ATT&CK technique ID is a label for a class of behavior, not a specification for a detection. T1078 (Valid Accounts) encompasses password spraying, credential stuffing, purchased credentials, insider misuse, and token theft across cloud, on-premises, and hybrid environments. A detection that fires on a specific authentication anomaly pattern does not cover T1078. It covers one implementation of T1078 under specific conditions. Checking the box says nothing about the others.
The black box stack. Your detection coverage does not begin and end with the rules your team wrote. Your next-generation firewall released an IPS signature update yesterday. Microsoft Defender added a behavioral detection this morning. Your email gateway quarantined something before it reached a host. A coverage map that accounts only for your in-house detections misrepresents the actual defensive posture of your environment, and worse, it creates pressure to build in-house duplicates of capabilities you already have. The coverage map devalues your own stack.
Environmental heterogeneity. Coverage is always coverage on something specific. A detection that fires on Windows endpoints with a particular EDR agent in a monitored network segment does not fire on unmanaged Linux servers, on contractor laptops outside your EDR footprint, or on assets in network segments your SIEM doesn't see. Your environment is not a monoculture. Any flat coverage map that ignores placement is describing an environment that doesn't exist.
Operational visibility degradation. Log forwarders stop running. Network changes move VLANs out of monitored scope. EDR agents fail silently on a percentage of the estate after an update. Every analyst who has worked incidents knows that visibility is never static. A coverage map is a snapshot of a thing that is always in motion. The moment it's printed, it's already partially wrong.
Taxonomy breadth. Some technique IDs are umbrellas so wide that claiming coverage requires either a 10-paragraph asterisk or a willingness to mislead.
Take T1115: Clipboard Data. The technique description reads: "Adversaries may collect data stored in the clipboard from users copying information within or between applications." If your team writes an informational alert on clip.exe or Get-Clipboard to check that box, ask what you've actually covered. Clipboard interaction happens through system APIs on Windows, macOS, and Linux, each with different logging requirements and behavior. Truly covering clipboard data exfiltration would require API-level logging across all three platforms at a fidelity that degrades performance and generates noise at a volume most organizations cannot process. The alternative is an alert that fires on the easy, explicit invocations and misses any attacker who took 20 seconds to use a different method.
From a distance, checking T1115 looks like a reasonable approximation. Up close, it's absurd. A hospital that owns a defibrillator has one capability, in one scenario, for one category of patient. "Coverage" that doesn't specify conditions is not coverage.
They Want to Know If You'll Catch the Bad Guy
The application owner in that meeting is not asking the wrong question. They want to know: is my system protected, how do you know, and will you find out if something goes wrong? Those are exactly the right questions for someone responsible for a critical business system.
The coverage map is a bad answer to a good question. It arrived with the vocabulary of assurance built in: the green squares, the percentage covered. It delivers the feeling of assurance without the substance. The stakeholder isn't naïve for accepting it. The format was designed to look like an answer.
The practitioner's job in that conversation is translation. The person across the table isn't missing knowledge. They're holding a tool that can't do what they need. Your job is to redirect toward something that can, without making them feel foolish for asking with the vocabulary they had.
A Gap Map Is Still a Map
The natural response to this problem is to make the map more honest. Instead of coloring everything green, use a heat map. Show red squares where coverage is thin. Track improvement over time. This feels like progress: you're not claiming full coverage, you're showing where the gaps are.
The heat map inherits all the same problems.
It still assumes every gap is visible. Telemetry that doesn't exist doesn't produce red squares; it produces silence. The absence of a detection doesn't appear anywhere on a technique-ID map if the data source required to build that detection was never collected.
It still assumes every gap is fillable given engineering effort. Some gaps are closable in a sprint. Some require a platform investment that hasn't been made. Some require architectural changes that are years out. The heat map cannot distinguish between them.
It still assumes gap priority is self-evident from the taxonomy. A red square on T1055 (Process Injection) looks the same as a red square on T1115 (Clipboard Data). Whether either gap matters depends on your threat profile, and the map has no mechanism for threat intelligence to inform that judgment.
Continuous improvement driven by a heat map optimizes for green squares. That is not the same as reducing risk.
Real continuous improvement is a three-way conversation. CTI identifies what matters based on the threat actors and campaigns relevant to your environment. Detection engineering assesses what's feasible given current telemetry, tooling, and operational constraints. Leadership receives an honest picture of the gaps that require strategic investment to close, not just engineering hours. When that conversation is replaced by a color on a grid, the program is optimizing for the appearance of coverage rather than the reality of it.
Continuous Emulation, Targeted by Stakeholder
Security monitoring assurance is not a state you achieve and report. It's a practice you demonstrate continuously. The replacement for a coverage map is continuous adversary emulation: automated breach and attack simulation platforms that run real-world attack scenarios against your actual control stack on an ongoing basis, confirming what blocks, what detects, and what reaches a responder. The coverage map tells you what rules exist. Emulation tells you what fires.
The right BAS platform still produces MITRE ATT&CK-aligned coverage maps: the artifact your application owner needs, your compliance officer can hand to an auditor, and your CTI team can use to track gaps against a real threat profile. What changes is how that map gets populated. Not manual assertion. Not vendor checkbox. Validated evidence from continuous simulation against your actual environment. The map still exists. It just finally reflects something real.
That platform is the foundation. How you use it depends on who needs the answer.
Incident response teams need recurring, unannounced assumed-breach exercises with real lateral movement objectives. Mixing internal and external testers matters: external vendors bring different tradecraft, and some boutique penetration testing firms operate with enough technical depth that the exercise genuinely sharpens the team. An IR team that only ever responds to internal simulations will eventually face something they haven't seen. The point of rotating vendors is to ensure they don't.
CTI stakeholders need the platform configured to simulate the specific threat actors relevant to your environment on an ongoing basis. When CTI identifies a TTP as important, it gets folded into continuous validation work first. Detection engineering then prioritizes the gaps that validation proves exist, not gaps identified by reviewing a taxonomy. The validation pipeline becomes the connection between intelligence and detection priorities.
Business stakeholders, including the application owner in that meeting, should be invited into validation dashboards directly. Showing a stakeholder that you run regular emulation exercises, that you track what fires and what doesn't, and that you have a process for addressing gaps is a more credible answer to their question than any coverage map. The conversation shifts from "are we covered" to "here's how we know, and here's what's in progress."
How to Have This Conversation
When a stakeholder asks for a coverage map, they're asking for confidence. The redirect doesn't require explaining why MITRE ATT&CK is a taxonomy and not a test harness. It requires acknowledging what they need and offering something that provides it.
Confirm you understand what they're trying to demonstrate (protection of a specific system, to a specific audience), explain briefly that a technique-ID map will raise more questions than it answers for anyone who looks closely, and offer to show them the emulation and validation data instead. Stakeholders who get a live view into ongoing adversary simulation against their environment tend to find it more compelling than a heat map.
Being an expert often means giving people what they need instead of what they asked for.
The coverage map vocabulary is entrenched. Compliance frameworks still ask for it. Vendors still produce it. The conversation isn't going away. What changes is whether you're the person handing over a map you know is misleading, or the person who can explain honestly what it shows, what it doesn't, and what you're doing instead.
References
- MITRE ATT&CK T1115: Clipboard Data — https://attack.mitre.org/techniques/T1115/
- MITRE ATT&CK Framework — https://attack.mitre.org/
- Alfred Korzybski, Science and Sanity (1933) — origin of "the map is not the territory"
- MITRE ATT&CK: Design and Philosophy — https://attack.mitre.org/docs/ATTACK_Design_and_Philosophy_March_2020.pdf
No comments:
Post a Comment