Stack Layout and System Calls

Before we can design an intrusion, one of the things we have to know is the stack layout. This is a function of the hardware architecture and the language. The operating system also has some influence as well. On the x86 architecture the stack grows downwards, and the top of the stack is given by the ESP index register.

The Standard C/C++ Stack Frame

Figure 2 shows the layout of a stack frame for the C and C++ language.

Figure 2 – Starting Stack Frame

This is the stack frame just before the following code is executed:

    fd = open(“theFile”, O_RDONLY, 0744);

This is the way you call the open system call in the C language. However the C compiler does not know how to generate the code to call the kernel directly. Instead, the implementers of the operating system provide a library function with the same name as the system call. This library function is hand written and translates the function call to a system call. The assembler language generated by the compiler for the function call looks something like:

    push 0744
    push O_RDONLY
    pushd PtrToString
    call open
    mov [ebp-fd],eax
    add esp,12

The code starts to build the new stack frame by pusing the arguments in return address so that the first argment is closest to the top of the stack when the function is called (the stack grows downwards). The call to the open function pushes the address of the move instruction on the stack and loads the address of the open routine into the EIP (program counter) register. When the function returns, the result value is in the EAX register. This value is stored into the local variable fd by indexing from the ebp pointer. This is why it is called the base pointer. Because it is contains the base of the local variables. The parameters are removed from the stack by adding 12 (three 4 byte values) to the stack pointer. So lets see what the stack looks like just before the call instruction is executed.

Figure 3 – Parameters are Pushed

So the three arguments have been pushed onto the stack. When we make the call, then the return address (the address of the move statement is pushed onto the stack. This is shown in figure 4.

Figure 4 – Call is made

The beginning of the open function call is as follows:

    push ebp
    mov  ebp,esp
    sub esp,sizeOfSpaceForLocalVars

This code results in a stack that is shown in Figure 5.

Figure 5 – Second Stack Frame Complete

The stack pointer points to the top of the stack, the base register points to the beginning of the local variables for the current function. We return from the function by the code:


The leave instruction is equivalent to "mov esp,ebp+4, pop ebp" which moves the stack pointer all the way up to the saved base pointer, and gets the old base pointer from the stack. The ret instruction pops the stack into the program counter to resume the execution at the mov instruction in the calling function.

System Calls

Calls to the system use the trap mechanism. Since the stack is in user space (logical address space), system calls use registers to pass the values. An integer value identifying the system call is passed in the A register. The first argument is typically in the B register, the second in the C, register, etc. The int instruction is used to generate the trap. Linux uses the 0x80 trap for system calls. The result is usually passed back in the A register. So the code for the open function looks something like this:

    push ebp
    mov ebp,esp

    mov eax,5
    mov ebx,ebp+16
    mov ecx,ebp+20
    mov edx,ebp+24
    int 0x80

We have no local variables, so we don't have to subtract anything from the stack pointer on entry. We start by moving the id for the open system call into the A register. On linux, this is the value 5. We then copy each of the parameters from the stack into the registers. The first parameter is just past the saved base pointer and the return address, so it is 16 bytes away from the base pointer. Similarly the second and third parameters are 20 and 24 bytes away from the base pointer. After all of the parameters are in the correct registers, we generate the trap with the int instruction. Since the return value is in the A register and we are also supposed to return the value in the A register, we don't have to do anything (the value is already in the A register). So we leave the function.

The Vunlerability

Consider the following code

char * GetALine(FILE * fp){
    char * result;
    char buffer[120]
    result = malloc(strlen(buffer)+1);
    return result;


The weak spot in this code is the fact that the gets function is called on an array that is on the stack. The gets function does not check the bounds of the array. So if we send a string that is longer than 119 characters (120 with the null end of string character) we will write over memory on the stack that is not part of the array. Depending on how long the string is, this can include the contents of the result variable, the saved base pointer, the return address and the parameters. If our string contains x86 machine code and we overwrite the return address with the address of the beginning of the buffer, then when the function returns, it will start executing the code we sent. This is a simplified example. The code to read the string may be safe, but another function that copies the string may not be safe. This puts some limitations on the code and data we send. The gets (and strcpy routines) will stop if it encounters a NULL byte (i.e. the value 0x00). The gets routine reads the input until it gets a newline character (i.e. the value 0x0A). So our string may not contain any null or newline characters. If it does, then the data after that character will not be read or copied and the attack will fail.

Ethics of Security Research

When I was an undergraduate, we talked about security in my OS class too. One of the students decided he would try it out for himself. We had discussed Masquerades, and he wrote a terminal login masquerade. He was caught and subject to academic discipline. Some of you may be having similar thoughts. Hopefully, most of you are not. Let me make it clear. It is not a good idea. The purpose of learning the details about attacks is twofold:

  1. Understanding how attacks happen so that you will be better able to design and implement secure code.
  2. Use security attacks in a controlled environment to test the security of a particular program or application.

Attacking a running application is a valid security testing approach, but it must be done in a controlled environment. If you attack another machine on campus or elsewhere on the internet, you could face expulsion and possibly prison time. Foreign students caught in such an act could face deportation. In the post 9/11 world, any claim that you were just "experimenting" or just "testing" will fall on deaf ears. Such defenses were partially successful in early days of cybercrime. They do not work any more.