An Analytical approach to Modern Binary Deobfuscation [Attend Online HITB2022SIN]

DATE: 22-25 August 2022

TIME: 09:00 to 17:00 SGT/GMT +8

Code obfuscation has become one of the most prevalent mechanisms aiming to complicate the process of software reverse engineering. It plays a major role on a wide range of domains: from malware threats to protection of intellectual property and digital rights management.

An Analytical approach to Modern Binary Deobfuscation is a curated training that provides an intensive jump-start into the field of code (de)obfuscation. Over the course of this training, students will receive a comprehensive introduction to the most relevant software obfuscation mechanisms as well as existing deobfuscation techniques to analyze, confront and defeat obfuscated code.

Students will be provided with

• Access to a VM with all tools, examples and exercises
• Access to a private chat with instructor and other students

Topics Covered

• Introduction, context and motivation

• Part 1: Code obfuscation
  • → Data-flow based obfuscation
    • Constant unfolding
    • Dead code insertion
    • Encodings
    • Pattern-based obfuscation
  • → Control-flow based obfuscation
    • Function inlining/outlining
    • Opaque predicates
    • Control-flow flattening
  • → Mixing data-flow and control-flow obfuscation
    • VM-based obfuscation
    • Hardening VM-based obfuscation
  • → Mixed Boolean-Arithmetic
    • Preliminary concepts
    • MBA rewriting
    • Insertion of identities
    • Opaque constants

• Part 2: Code deobfuscation
  • → Data-flow analysis
    • Reaching definition analysis
    • Liveness analysis
  • → Dynamic binary Instrumentation
    • Tracing code execution
    • Hooking
    • Extracting deobfuscated data
  • → SMT-based analysis
    • Semantic equivalence checking
    • Translating code conditions into SMT solver constraints
    • Proving code properties
    • Attacking simple MBA and weak cryptography
  • → Symbolic execution
    • Reasoning about code in a symbolic way
    • Working with native code
    • Working with intermediate representations
    • Plugging an SMT solver
    • Attacking opaque predicates
    • Attacking MBA obfuscation
    • Attacking VM-based obfuscation
  • → Program synthesis
    • Code syntax vs code semantics
    • Oracle-based program synthesis
    • Describing semantics through I/O behavior
    • Generating I/O pairs
    • Attacking MBA obfuscation
    • Attacking VM-based obfuscation

• Conclusions and research directions

Tools used

• Disassemblers
  • IDA Pro
  • radare2
• Obfuscation
  • Manual obfuscation
  • O-LLVM
  • Tigress
• Dynamic Binary Instrumentation
  • Frida
  • QBDI
• Symbolic execution
  • Miasm
  • Triton
  • Radius
• Program synthesis
  • syntia
  • msynth
  • qsynthesis
• Other tools
  • Z3
  • MBA-Solver
  • Custom tooling