DESIGN AND EVALUATION OF HIDDEN MARKOV MODEL BASED ARCHITECTURES FOR DETECTION OF INTERLEAVED MULTI-STAGE NETWORK ATTACKS

Nowadays, the pace of coordinated cyber security crimes has become drastically more rapid, and network attacks have become more advanced and diversified. The explosive growth of network security threats poses serious challenges for building secure Cyber-based Systems (CBS). Existing studies have addressed a breadth of challenges related to detecting network attacks. However, there is still a lack of studies on the detection of sophisticated Multi-stage Attacks (MSAs).

The objective of this dissertation is to address the challenges of modeling and detecting sophisticated network attacks, such as multiple interleaved MSAs. We present the interleaving concept and investigate how interleaving multiple MSAs can deceive intrusion detection systems. Using one of the important statistical machine learning (ML) techniques, Hidden Markov Models (HMM), we develop three architectures that take into account the stealth nature of the interleaving attacks, and that can detect and track the progress of these attacks. These architectures deploy a set of HMM templates of known attacks and exhibit varying performance and complexity.

For performance evaluation, various metrics are proposed which include (1) attack risk probability, (2) detection error rate, and (3) the number of correctly detected stages. Extensive simulation experiments are conducted to demonstrate the efficacy of the proposed architecture in the presence of multiple multi-stage attack scenarios, and in the presence of false alerts with various rates.