Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice

Intrusion Detection Systems (IDSs) play a critical role in safeguarding networks against malicious activities. However, selecting a suitable IDS remains challenging due to variability in performance across different network environments, datasets, and detection methodologies. This paper presents a systematic evaluation of recent machine learning-based Network IDS (NIDS). Our initial curation of numerous ML-based IDS solutions revealed significant practical challenges related to dataset preprocessing, code availability, and reproducibility that complicated performance assessments. From the systems that could be successfully implemented, we thoroughly evaluated four IDSs - HELAD, AOC-IDS, NEGSC, and SLIPS - across five benchmark datasets: CICIDS2017, UNSW-NB15, Mirai, CTU13, and BoT-IoT. Our empirical analysis highlights significant performance variations, demonstrating that no single IDS universally outperforms others across all tested datasets. NEGSC exhibited the most consistent performance, achieving the highest average F1 score (0.8147), while other IDSs such as HELAD showed notable dataset-specific effectiveness (e.g., CTU13, F1=0.9902). We discuss these issues in-depth, emphasizing the critical importance of aligning IDS selection with specific network characteristics and operational needs. Our findings underline the necessity for standardized benchmarking practices and highlight practical deployment considerations, guiding users toward more informed IDS choices in real-world scenarios.