MITRE ATT&CK® Evaluations: malicious activities reported by Atos MDR

Since 2018, MITRE Engenuity has been conducting ATT&CK Evaluations focused on evaluating the potential capability of products to detect and protect against known adversary behavior. This year, MITRE Engenuity conducted its first-ever ATT&CK Evaluations for Managed Services. Fifteen managed detection and response (MDR) vendors and managed security service providers (MSSP), including Atos, participated in this year’s evaluation.

Atos MDR team detected all ten steps of the attack while ensuring that the reports contained only actionable information, thereby reducing potential false positives and information overload.

Key points about the evaluation:

• For this evaluation, MITRE Engenuity chose OilRig, a suspected Iranian threat group that has targeted Middle Eastern and international victims since at least 2014. This was a closed-book simulation, as MITRE Engenuity did not announce the adversary they would emulate.
• The evaluation scope comprised four Windows-based machines, a Windows 10 workstation, a Windows 2019 SQL server, a Windows 2019 Exchange Web server, and a Windows 2019 Domain Controller. These machines were set up on Microsoft Azure Cloud. MITRE Engenuity allowed participants to install endpoint security products on the workstation and servers and collect logs from servers. Access to the data generated on Azure cloud, such as Azure Activity, flow logs, etc., was not provided.
• The emulation was performed in 10 steps that began with the compromise of a workstation and eventually exfiltrating data from a SQL server through an Exchange server.
• One of the requirements of the evaluation was to turn off all preventive features in the products used for the evaluation. The focus of the assessment was to test detection capabilities and not prevention.
• The focus of the evaluation was to check how many ATT&CK Techniques were reported by participants as opposed to how many were detected. Preventive controls and response were excluded from the evaluation.

In MITRE Engenuity’s words:

“It is important to note that MITRE Engenuity is evaluating whether a given ATT&CK Technique was reported or not, as opposed to whether it was detected by the service provider or not. In many cases, the service provider may have detected the ATT&CK Technique under test but chose not to report it to MITRE Engenuity because they believe it is unnecessary information, or they believe it can be implied or assumed by other information provided to MITRE Engenuity.”

Atos followed our methodology of high fidelity and meaningful reporting, so chose not to report all detected findings and report only in the context of what is meaningful to customers.

Tools used by Atos in this evaluation:

Atos MDR service is built on its AI-based cyber analytics and hybrid SecOps platform – AIsaac. AIsaac is a multi-vector detection technology that supports integration with endpoint, identity, applications, data center, and cloud technologies. It is important to note that this evaluation primarily focused on the detections from endpoint security tooling. Atos chose Cybereason EDR as the endpoint security detection tool for this evaluation. Cybereason was configured in detect-only mode to meet the requirements set by MITRE Engenuity.

In the real world, customers provide access to existing security technologies across their perimeter and cloud security controls to improve threat detection. The objective is a multi-vector, layered approach for the early detection of threats. We believe that future MITRE Evaluations will enable a multi-mode architecture for high-fidelity threat detection mechanisms for MDR vendors.

How did we fare?

The Atos MDR team detected all ten steps of the attack. Our expert teams identified the adversary early in the evaluation and reported on all critical techniques of the attack. The diagram below summarizes the ten steps performed by the MITRE Engenuity red team and highlights the key findings reported by the Atos MDR team.

In this closed-book simulation of the state-sponsored threat actor, OilRIg, we believe Atos reported critical and actionable adversary activities without overburdening the customer with information.

This evaluation has helped the Atos MDR team advance its threat-informed defense capabilities aligned with the ATT&CK knowledge base. Certain adversary activities, such as download of the web shell, exfiltration of the password file to attacker and movement of the malicious file were not detected. We will incorporate learnings from this evaluation to improve our detection and reporting on similar items.

Interpreting the MITRE Engenuity ATT&CK® Evaluations for Managed Services — OilRig (2022)

• Atos reported important techniques across all ten steps of the attack simulation. All critical stages, such as initial compromise, persistence, key discovery activities, defense evasion techniques, lateral movement, credential access, privilege escalation, command and control and exfiltration, were reported with adequate artifacts to help customers take action.
• Atos MDR’s primary objective is to disrupt an attack early and rapidly respond without drowning a customer with techniques detected. In a real-world attack, this is what the team would have achieved. But, in the evaluation, although we detected most techniques, the team deliberately chose not to report a set of techniques, so we don’t overwhelm the clients with alerts. For example, we detected all the discover commands in Step 2, as shown in the screenshot below from the Cybereason platform:
  

Atos team decided to present the information more straightforwardly, as shown below:

In some cases, Atos reported techniques but did not consider them relevant to provide deeper context for simple commands, as expected by MITRE. We work with large enterprise customers who have teams that understand such context. For example, we reported the adversary’s activity to hide artifacts in Step 4 as below:

MITRE Engenuity did not attribute this finding as reported because we did not explain what “attrib” command does. We believe our customers have the expertise to understand the context without explanation.

Conclusion:

MITRE Engenuity has taken a significant step towards an industry-neutral evaluation of MDR services with this inaugural Managed Services evaluation, and Atos is proud to have participated. While the assessment was thorough, we believe there was still a significant flavor of endpoint product-centric evaluation that needs to adapt to include multi-vector analytics for services evaluation. The evaluation also did not have response orchestration in the evaluation scope, which is a critical component of real-world security in the context of MDR services. Nevertheless, it is a great start, and we expect this to evolve into a full-fledged MDR services evaluation in the future and will work with MITRE toward this.