Press Center

NOTE: The views and opinions expressed in this blogpost are those of ESET and do not necessarily reflect the views or positions of MITRE Engenuity.

In this year’s Enterprise edition of the ATT&CK® Evaluations, MITRE set up three attack scenarios: a Democratic People’s Republic of Korea (DPRK) scenario to test cyberespionage activity against a macOS system, a Cl0p scenario to test a ransomware attack against a Windows system, and a LockBit scenario to test a ransomware attack against a corporate environment containing both a Linux server and Windows workstations and servers.

ESET Inspect demonstrated good visibility in each scenario, detecting every step while keeping the total number of detections (or volume) low. The detections generated from the attacks were automatically correlated into incidents by ESET Inspect’s Incident Creator, giving our security analysts a focused view of the attacks and thus a clear understanding of how they happened step-by-step.

To get a better idea of how ESET fared, we will go over some of the methodological changes introduced by MITRE and then look at how Incident Creator streamlined the view and workflow of the security analysts sitting at the ESET Inspect dashboard.

This evaluation brought several well-thought-out changes to the methodology of the Detection scenarios that we believe better reflect the security analyst’s work of dealing with real-world cyberattacks.

First, Telemetry is no longer a detection category, meaning that it is not sufficient merely to show that an event occurred. The lowest detection category is now General, requiring a detection to indicate that an event occurred and is suspicious or malicious in some way. It’s important to point out that an event occurring in a sandbox does not qualify as an event occurring in the environment under evaluation. As indicated in its description, a General detection must answer the what, where, when, and who as related to the tested environment, and these cannot be answered from external sandbox execution.

Second, some benign substeps have been baked in as a test for false positives rather than for detections. This is a welcome change because it discourages a “detect everything” approach, which can otherwise create a lot of noise and needless work for the security analyst, and drive up data storage costs to boot. Another benefit of this change is that it allows calculating a precision score, indicating how many of the detections made correspond to actually malicious or suspicious substeps.

Third, some substeps are included but not evaluated. The reason for including such substeps is to better simulate a real-world cyberattack by avoiding illogical jumps in the progression of the attack.

Finally, a volume metric was introduced that records the number of detections shown in the dashboard. This is yet another way of discouraging a “detect everything” approach, dissuading vendors from allowing their dashboards to be populated sometimes with millions of detections. We welcome this adjustment too.

The volume metric also records the severity of detections, using five levels: critical, high, medium, low, and info. Since ESET Inspect has three incident severity levels of high, medium, and low, and three detection severity levels of threat, warning, and info, we agreed with the MITRE Engenuity team on the mapping shown in Table 1.

Table 1. Mapping between ATT&CK Evaluation and ESET Inspect severity levels

It is worth pointing out why we chose a severity score of 22 to divide the low and info severity levels. As noted in our documentation:

Rules with severity 22 and below are telemetry rules. They are usually used only as additional information for investigating an incident and can often be triggered by legitimate behavior. If some of these rules generate too much traffic in your environment, you may consider turning them off.

From here on out, we will refer to severity levels only as used by ESET Inspect.

An important consequence of using this mapping is that detections not correlated to an incident are out of scope in the evaluation. This largely reflects the intended usage of ESET Inspect in the real world: incidents populated by correlated detections are the primary focus for security analysts. Additional detailed information and detections not correlated to incidents, which could be of value in some cases, are secondary.

Since dashboards inevitably have various parts that show detections and other information in detailed, summary, or graphical forms, vendors were allowed to indicate for the evaluation the standard view that security operators are expected to use for handling attacks. Only information presented via this view is eligible for consideration as a detection or false positive, and for measuring volume. In ESET Inspect, the standard view for security analysts is Incidents.

The Incidents view is the primary place that security operators should use to manage their workflow. Incidents are automatically populated into this view in two ways:

• ESET Incident Creator, which uses an AI-based engine to correlate detections into a single incident.
• ESET Inspect, which currently has over 100 rules that create an incident as a response to being triggered, aggregating detections into a single incident by affected computers, a period of time, or both.

Operators are recommended to use the following workflow:

1. Investigate each incident.
2. Investigate threat-severity detections not correlated to any incident, if time allows.

Just as last year’s evaluation, each attack scenario was run twice. Vendors were allowed to make config changes for the second run to try to increase visibility, decrease false positives, and reduce volume. Figure 1 shows the Incidents view after the config change run.