Buffer overflow and stack smashing
- Apply your understanding of how stack frames are built to the analysis of a C program. In theory, we all seem to understand procedure calling conventions and how stack frames are built. When we try to put that knowledge to practice, however, we discover that reality has its thorns. In this lab, you will work to decode the stack frame from the inside of a C function.
- Learn what makes a program vulnerable to stack smashing attacks so that you can write more secure programs in the future. One of the practical applications of the understanding of stack frames is rarely geared toward increasing the common good of society. When one understands how a stack frame is built and used by a program, it becomes possible to craft a type of security exploits called “stack smashing” that can have serious negative consequences. Working on programs that are vulnerable to stack smashing, hackers have been successful at crafting attacks that allow one to inject a harmful piece of code into a running program. The technique used puts the address of the injected program in stack frame of a function, right where the return address should be recorded. When the function terminates, it blindly jumps to that address, thinking that the flow of control will return to its caller when, in fact, it goes to the malicious code. In this lab, you will learn what exposes programs to the risk of stack smashing and, hopefully, your future C programs will not include these mistakes.
Open a shell, go into your local git repo, and do a “git pull“ to make sure you have the latest feedback from the TAs.
Next, create folder ~/csci206/Labs/Lab11 in your regular lab folder and, inside it, create file prelab.txt to store your answers to the questions below. The header of this file should have your name and lab section. Copy all the files from ~cs206/Labs/Lab11 to your Lab11 folder.
Open a browser window to http://insecure.org/stf/smashstack.html, which contains the classic paper “Smashing the Stack for Fun and Profit”. Watch the video below about the same topic and then read the Smashing the Stack paper to provide brief answers to the following questions.
(1.1) What is “stack smashing?”
(1.2) What is it that makes stack smashing possible?
(1.3) Describe what it means to disassemble machine code.
(1.4) Log in to mips.bucknell.edu and compile the file example3.c to an executable called example3. Open this file in gdb as indicated in the “Buffer Overflows” section of the paper and disassemble the main function. If you open the example3.c file in your favorite text editor, you will see that its contents are very similar to the corresponding listing in the paper. Explain why the disassembled code looks so different from what you see in the paper and the code is slightly different (buffer1 + 24 vs. buffer1 + 12).
(1.5) Describe why when you run example3 on the mips machine the output is 0 rather than 1.
When you are done with these exercises, make sure to add your prelab.txt file to git and push to gitlab.
25 points total
 for each question