Another way to detect debuggers or evade them is to detect their breakpoints. Whether they are software breakpoints (like INT3), hardware breakpoints, single-step breakpoints (trap flag), or memory breakpoints, malware can detect them and possibly remove them to escape reverse engineer control.
Handling debugger breakpoints evasion
Detecting software breakpoints (INT3)
This type of breakpoint is the easiest to use, as well the easiest to detect. As we stated in Chapter 1, A Crash Course in CISC/RISC and Programming Basics, this breakpoint modifies the instruction bytes by replacing the first byte with 0xCC (the INT3 instruction), which creates an exception (an error) that gets delivered to the debugger to handle.
Since it...