CYB 4551 - Software Reverse Engineering

2 lecture hours 2 lab hours 3 credits

Course Description
This course teaches students the basics of software reverse engineering. Students will be given more advanced exposure to assembly language programming and low level programming as it is necessary for cyber operations. Students will learn about the discipline of software reverse engineering. This discipline provides the ability to deduce the design of a software component, to recover the software specification for the program, discover data and / or data structures used by software, and to aid in the analysis of software via disassembly and/or decompilation. This ability to understand software of unknown origin or software for which source code is unavailable is a critical skill for analyzing malware and auditing of closed source software. Students will complete multiple lab activities in teams related to software reverse engineering in a virtualized environment.
Prereq: CSC 2210, CSC 3210 or instructor consent
Note: None
This course meets the following Raider Core CLO Requirement: None

Course Learning Outcomes
Upon successful completion of this course, the student will be able to:

Develop low level programs with the required complexity and sophistication to implement exploits for discovered vulnerabilities
Write a functional, stand-alone assembly language program, such as a simple telnet client, with no assistance from external libraries
Effectively use disassemblers, debuggers, virtualization based sandbox environments, process monitors, and network activity monitors in the laboratory environment
De-obfuscate obfuscated programs and binaries
Safely perform static and dynamic analysis of software of unknown origin to understand software's functionality

Prerequisites by Topic

C programming
Assembly language programming
Operating systems
Debugging skills using a debugger

Course Topics

Low-level software
Overview of Intel assembly language
Representation of compiled high-level language structures in assembly
Operating systems background
- MS-DOS internals related to malware case studies
- Modern Windows execution environment
Executable file formats
Reverse engineering principles
Sandboxing
Introduction to malware and assembly language
Why is reverse engineering necessary?
An overview of malware
Current and next-generation malicious software
- Viruses
- Worms
- Trojans
- Botnets
Introduction to defensive strategies against malware
- Worm fingerprinting/signature generation
- Behavioral approaches to detection of malware
Analysis of malicious software
- System monitoring tools
- Dynamic tracing: system calls, filesystem, and registry
- Compiler issues
- Debuggers (i.e. gdb, OllyDbg, WinDbg)
- Disassemblers (IDA Pro, Hopper, Hiew, etc.)
- Memory analysis to support reverse engineering
- RAM acquisition
- Extraction of malware
Advanced reverse engineering techniques
- Encrypted/packed executables
- Anti-debugging techniques
- Anti-VM techniques
- Code obfuscation
Analysis of decompiled source code
Revelation of command-and-control functionalities

Laboratory Topics

Developing in assembly
In depth debugging refresher
Software disassembly
Malware case studies
- Viruses
- Simple obfuscation
- Code injection
Rapid analysis of suspicious executables
Introduction to encrypted/packed malware
Unpacking encrypted/packed malware
Malware in IoT devices
Advanced persistent threats
Project: peer evaluation of code of unknown origins

Coordinator
Walter Schilling

Add to Portfolio.

Close Window