Dangling Pointer Detection using Hardware Virtualization
|has title::Dangling Pointer Detection using Hardware Virtualization|
|Master:||project within::Computer Systems and Security|
|Student name:||student name::Gábor Kozár|
|Second reader:||has second reader::Herbert Bos|
Low-level programming languages such as C and C++ require manual management of memory: there is no automatic garbage collection available for heap memory. This is a hotbed of bugs and security vulnerabilities: memory can leak when an allocation is never cleaned up; memory may be freed twice; references for deallocated memory regions may remain and cause bugs during runtime that are very difficult to identify. Such pointers and references are said to be dangling and are a central figure in this thesis.
When accessing a dangling pointer, the best possible scenario is that the program will crash with segmentation fault, if the memory the allocation had used has since been unmapped. However, far worse behaviour can occur if the memory has since been reused and another allocation was placed in a way that partially or fully overlaps with the previous memory region that which the dangling pointer references. Dereferencing the dangling pointer in this case will usually lead to nonsensical data being accessed: type confusion will occur. The program might not even crash, but its behaviour can easily become nonsensible, as it is acting on effectively corrupted data.
Due to the temporal disconnect between when the bug occurs (the pointer is not cleaned up at the same time as the memory region is deallocated) and when the bug becomes apparent (when the dangling pointer is used), such issues can be incredibly difficult to track down. Furthermore, it poses a security problem: if an attacker is able to control how memory allocations occur and has the ability to force the program to use a dangling pointer, then he is in the position to manipulate the data structures that the program operates on, meaning that he can potentially circumvent security measures, disclose sensitive information, or even gain control of the victim process. There exist a large number of tools that are able to detect dangling pointers: examples include Polyspace, TotalView, Valgrind, Mudflap, AddressSanitizer, SoftBound, CheckPointer and DangSan. All of these tools have considerable limitations, however. Tools such as AddressSanitizer and DangSan instrument the application during the compilation process, adding extra instructions that check the validity of the pointers before they are accessed. This means that the application source code needs to be available, and it has to be possible to re-compile the application. Furthermore, these tools still impose a nontrivial performance penalty as well as memory overhead. Other solutions, such as Valgrind, can operate directly on the binaries and don’t require the application source code or re-compilation. Unfortunately, such tools effectively cripple the performance of the monitored application, leading to slow-downs as high as 100x. Similar problems plague other solutions, such as multi-variant execution implementations like MvArmor.
Ultimately, these tools can be feasibly used for debugging dangling pointer (and other memory corruption) issues during the development process, but have too great of a performance cost to allow them to be used as a security measure in production. Therefore, the security aspect of dangling pointers remains a mostly unsolved problem today. The purpose of this thesis is to explore the possibility of utilizing hardware virtualization to overcome all such limitations. We are going to be using the fact that virtual memory is plentiful, by mapping each memory object to its own virtual memory page. When the memory region is deallocated, we unmap the virtual memory page, but we do not re-use it until such time as we are reasonably certain that no references to it persist inside the application. By running the application in a virtualized environment, our code can run in ring 0, meaning that we are able to efficiently perform virtual memory management tasks (such as mapping and unmapping) without suffering the performance penalty of system calls. Our hope is that with hardware-assisted virtualization, the overall performance overhead can be negligible.