The Challenge of Detecting Stealthy Attacks

DLL hijacking remains a popular and effective attack vector, used by both widespread malware and targeted APT groups. The technique involves a malicious library being loaded by a legitimate, trusted process, making it difficult for security solutions to detect without causing system performance issues. To address this challenge, researchers at Kaspersky turned to machine learning to distinguish malicious library loads from benign ones.

An Evolving, Multi-Generational Training Process

The project involved training three successive generations of ML models. The initial model, trained on broad file reputation data, suffered from a high rate of false positives. Researchers refined their approach in the second iteration by filtering the training data to include only malware families known for DLL hijacking behavior, which significantly improved accuracy.

The third-generation model was trained on analyst-verified detections from its predecessors. This stage introduced more sophisticated features, such as flagging potentially unwanted applications that mimic malicious behavior but are not threats. This iterative refinement was crucial, with the final model achieving a true positive rate of 80% at an extremely low false positive rate (10⁻⁵).

From Theory to Real-World Defense

The model’s success relies on analyzing complex combinations of features. Instead of just looking at file names, it uses a library name plus process signature combination to identify anomalies. It flags complex dependencies, such as a newly renamed process calling a library from a non-standard path, as highly indicative of an attack.

Today, all three model generations are active in Kaspersky’s internal systems, processing approximately 6.5 million security events daily. The technology has also been integrated into commercial products like Kaspersky SIEM and MDR, where it has already helped detect and prevent real-world incidents in client systems.

Source: https://securelist.com/building-ml-model-to-detect-dll-hijacking/117565/