[Math][RevEng] Boolean algebra for noobs: stack alignment

Sometimes you see in assembly code such a function prologue:

main            proc near

                push    ebp
                mov     ebp, esp
                and     esp, 0FFFFFFF0h

What's with 'and esp, 0FFFFFFF0h'? It's a stack alignment. But why?

There is a blog post written by Altynbek Isabekov, which explains this. Good, but too long. My explanation is shorter.

A function you call from main (like printf()) may access local variables in stack. It can use SSE-instructions like MOVDQA and MOVAPS. 'A' in instruction's name means 'aligned'. Memory address must be aligned on 16-byte boundary, otherwise exception will be generated.

There are other instructions that do the same, but not that strict. For example, MOVDQU ('U' means 'unaligned'). But you have to pay some price for everything. These instructions are just slower than 'aligned' versions.

All that implies that variables passed to printf() must be aligned on 16-byte boundary. That is achieved by stack alignment at main()'s start.

Now we see that cryptic AND instruction. In fact it works as:

SP = SP - SP & 0xf

... or (logically) ...

while SP & 0xf != 0
    push ... something ...

... or ...

while SP & 0xf != 0

Stack grows to lower address. So this single AND instruction basically does alignment, by resetting low 4 bits. Two goals achieved: SP with 4 low bits cleared is aligned on 16-byte boundary. And SP decreased down to lower address (if needed).

So when you call printf() from assembly code and printf() crashing (like a problem I had before writing this post), check if you align stack in calling function.

More info at SO.

(the post first published at 20221113.)

List of my other blog posts.

Yes, I know about these lousy Disqus ads. Please use adblocker. I would consider to subscribe to 'pro' version of Disqus if the signal/noise ratio in comments would be good enough.