CYB 4551 - Software Reverse Engineering

• Develop low level programs with the required complexity and sophistication to implement exploits for discovered vulnerabilities
• Write a functional, stand-alone assembly language program, such as a simple telnet client, with no assistance from external libraries
• Effectively use disassemblers, debuggers, virtualization based sandbox environments, process monitors, and network activity monitors in the laboratory environment
• De-obfuscate obfuscated programs and binaries
• Safely perform static and dynamic analysis of software of unknown origin to understand software's functionality

• Programming fundamentals
  • Proficiency in at least one high-level language (e.g., C, C++, or Java)
  • Understanding of compiled vs. interpreted languages
  • Basic data structures (arrays, pointers, linked lists)
  • Control flow (loops, conditionals, functions)
  • Concept of linking and loading
• Computer architecture
  • Binary and hexadecimal representation
  • Stack and heap concepts
  • Calling conventions (function calls, parameter passing)
• Operating systems fundamentals
  • Process management (creation, termination, scheduling)
  • Memory management (virtual memory, paging)
  • File systems and I/O
  • System calls and kernel-user space interaction
  • Linux basics (services, registry, permissions)
• Networking basics (for malware communication analysis)
  • TCP/IP fundamentals
  • Common protocols (HTTP, DNS)
  • Basics of client-server communication
• Security foundations
  • Malware types (viruses, worms, trojans)
  • Basic cryptography (hashing, encryption)
  • Sandboxing concepts
  • Static vs. dynamic analysis
• Tools familiarity
  • Command-line proficiency (Linux)
  • Basic use of debuggers (gdb)
  • Familiarity with virtualization environments (VMware, VirtualBox)

• Low-level software
• Overview of Intel assembly language
• Representation of compiled high-level language structures in assembly
• Operating systems background
  •     MS-DOS internals related to malware case studies
  •     Modern Windows execution environment
• Executable file formats
• Reverse engineering principles
• Sandboxing
• Introduction to malware and assembly language
• Why is reverse engineering necessary?
• An overview of malware
• Current and next-generation malicious software
  •     Viruses
  •     Worms
  •     Trojans
  •     Botnets
• Introduction to defensive strategies against malware
  •     Worm fingerprinting/signature generation
  •     Behavioral approaches to detection of malware
• Analysis of malicious software
  •     System monitoring tools
  •     Dynamic tracing: system calls, filesystem, and registry
  •     Compiler issues
  •     Debuggers (i.e. gdb, OllyDbg, WinDbg)
  •     Disassemblers (IDA Pro, Hopper, Hiew, etc.)
  •     Memory analysis to support reverse engineering
  •     RAM acquisition
  •     Extraction of malware
• Advanced reverse engineering techniques
  •     Encrypted/packed executables
  •     Anti-debugging techniques
  •     Anti-VM techniques
  •     Code obfuscation
• Analysis of decompiled source code
• Revelation of command-and-control functionalities 

• Developing in assembly
• In depth debugging refresher
• Software disassembly
• Malware case studies
  •     Viruses
  •     Simple obfuscation
  •     Code injection
• Rapid analysis of suspicious executables
• Introduction to encrypted/packed malware
• Unpacking encrypted/packed malware
• Malware in IoT devices
• Advanced persistent threats
• Project: peer evaluation of code of unknown origins