Advanced Persistent Threat (APT) Detection Based on Network Traffic Analysis using Machine Learning Approach

Abstract

Advanced Persistent Threats (APTs) are stealthy and sophisticated cyberattacks targeting sensitive data and critical infrastructure. Traditional detection systems relying on signatures or anomaly-based methods face challenges in detecting evolving threats like APTs due to their stealthy, multi-stage behaviour and ability to mimic normal traffic patterns. These approaches often struggle to detect previously unseen attacks, adapt to new threat tactics, or maintain accuracy without generating high false positives. This research develops an APT detection model based on network traffic analysis using machine learning approach. The APT detection model leverages labelled datasets including Linux-APT-Dataset-2024 as Dataset A and APT Alerts as Dataset B. We analyse network traffic features such as Domain Name System (DNS) queries, traffic anomalies, and protocol deviations using Random Forest algorithm. We split into 3 sets of data based on Top 5, 10 and 15 features. The proposed APT detection model based on network traffic analysis achieved improved performance compared to existing work, with accuracy reaching 85.37% on Dataset A (Top 15 features) and 99.64% on Dataset B (Top 5 features). In addition, the model successfully reduced false positive rates, reaching as low as 44.33% on Dataset A and 0.34% on Dataset B. These findings highlight the model's robustness, offering cybersecurity professionals and organizations a reliable mechanism to detect and mitigate APT activities effectively.