In this section, we will explore different types of hooking mechanisms. In the following diagram, we can see various types of hooking techniques that rootkits use at different stages of the request process flow:
Figure 6: The hooking mechanisms of rootkits
Rootkits can install hooks at different stages of this process flow:
- User-mode hooking/API hooking: These are the user-mode API hooking mechanisms that are used for hiding malware processes, files, registries, and more. We covered this in Chapter 4, Inspecting Process Injection and API Hooking.
- SYSENTER hooking: This is the first option that's available for the kernel-mode rootkits to hook. In this case, they change the address that sysenter will transfer the execution to, and intercept all requests from the user mode to the kernel mode.
- SSDT hooking: This technique works more...
