Software Deobfuscation Techniques [HITB2021SIN]

ATTEND ONLINE: REGISTRATION CLOSED

DATE: 23-25 August 2021

TIME: 09:00 to 17:00 SGT/GMT+8

(Course timing not suitable? Let us know!)

Code obfuscation has become a vital tool to protect, for example, intellectual property against competitors. In general, it attempts to impede program understanding by making the to-be-protected program more complex. As a consequence, a human analyst who still aims to reason about the obfuscated code has to overcome this barrier by transforming it into a representation that is easier to understand.

In this training, we get to know state-of-the-art code obfuscation techniques and have a look at how these complicate reverse engineering. Afterwards, we gradually become familiar with different deobfuscation techniques and use them to break obfuscation schemes in hands-on sessions. Thereby, participants will deepen their knowledge of program analysis and learn when and how (not) to use different techniques.

First, we have a look at important code obfuscation techniques and discuss how to attack them. Afterwards, we analyze a virtual machine-based (VM-based) obfuscation scheme, learn VM hardening techniques and see how to deal with them.

In the second part, we cover SMT-based program analysis. In detail, students learn how to solve program analysis problems with SMT solvers, how to prove characteristics of code, how to deobfuscate mixed Boolean-Arithmetic and how to break weak cryptography.

Before we use symbolic execution to automate large parts of code deobfuscation, we first introduce intermediate languages and compiler optimizations to simplify industrial-grade obfuscation schemes. Following, we use symbolic execution to automate SMT-based program analysis and break opaque predicates.

The last part covers program synthesis, an approach that learns the code’s semantics based on its input-output behavior. We explore how to collect input-output pairs; then, we use program synthesis to deobfuscate mixed Boolean-Arithmetic and learn the semantics of VM instruction handlers.