thesis_ver2.pdf (1.42 MB)

Usage of Dynamic Analysis to Strengthen Control-Flow Analysis

Download (1.42 MB)
posted on 14.12.2020, 03:06 by Priyam Biswas
System programming languages such as C and C++ are ubiquitously used for systems software such as browsers and servers due to their flexibility and high performance. However, this flexibility comes with a price of lack of memory and type safety.

Control-Flow Hijacking (CFH), by taking advantage of the inherent lack of memory and type safety, has become one of the most common attack vectors against C/C++ programs. In such attacks, an attacker attempts to divert the normal control flow of the program to an attacker-controlled location. The most prominent defense against these kind of attacks is Control-Flow Integrity (CFI), which restricts the attack surface by limiting the set of possible targets for each indirect control-flow transfer. However, current analyses for the CFI target sets are highly conservative. Due to the ambiguity and imprecision in the analyses, CFI restricts adversaries to an over-approximation of the possible targets of individual indirect call sites. State-of-the-art CFI approaches fail to protect against special attack classes such as over-writing variadic function arguments. Furthermore, mitigation of control-flow attacks is not explored to its full potential in the context of language boundaries in current literature. Hence, we need effective solution to improve the precision of the CFI approaches as well as strong protection mechanisms against commonly abused corner cases.

We leverage the effectiveness of dynamic analysis in deriving a new approach to efficiently mitigate control-flow hijacking attacks. We present Ancile, a novel mechanism to improve the precision of the CFI mechanism by debloating any extraneous targets from the indirect control-flow transfers. We replaced the traditional static analysis approach for target discovery with seed demonstrated fuzzing. We have evaluated the effectiveness of our proposed mechanism with standard SPEC CPU benchmarks and other popular C and C++ applications.

To ensure complete security of C and C++ programs, we need to shield commonly exploited corners of C/C++ such as variadic functions. We performed extensive case studies to show the prevalence of such functions and their exploits. We also developed a sanitizer, HexVASAN, to effectively type-check and prevent any attack via variadic functions. CFH attacks, by abusing the difference of managed languages and their underlying system languages, are very frequent in client and server side programs. In order to safe-guard the control-flows in language boundaries, we propose a new mechanism, FitJit, to enforce type integrity. Finally, to understand the effectiveness of the dynamic analysis, we present Artemis, a comprehensive study of binary analysis on real world applications.


Degree Type

Doctor of Philosophy


Computer Science

Campus location

West Lafayette

Advisor/Supervisor/Committee Chair

Mathias Payer

Advisor/Supervisor/Committee co-chair

Christina Garman

Additional Committee Member 2

Sonia Fahmy

Additional Committee Member 3

Xiangyu Zhang

Additional Committee Member 4

Aniket Kate