Friday, May 22, 2020

Thread-local stack on Linux

Threads on POSIX systems like Linux share an address space, but have their own registers and execution state like their own stacks. We also know that the stack is mapped in the same virtual memory space.

The illusion of thread-local stacks is achieved by having the stacks of the different threads start at different locations in virtual memory.

threadStorage.c prints the location of the stack.

Here are numbers from a few architectures:
x86-64: (kernel v4.15)
$ ./threadStorage
one: Address of first variable 0x7fc7bbc90edc
two: Address of first variable 0x7fc7bb48fedc
main: Address of first variable 0x7ffe79c59e1c
three: Address of first variable 0x7fc7bac8eedc

The difference between one and two, and two and three is 0x8392704

The stack of the main thread starts at 0x7ffe79c59e1c, which is much higher than the stack of the first thread. Since most processes are single-threaded, they get much larger stack frames.

Numbers from other architectures I had lying around:

arm32: (kernel v4.19.66)
$ ./threadStorage
one: Address of first variable 0x76e3ee68
two: Address of first variable 0x7663de68
main: Address of first variable 0x7e9c6c30
three: Address of first variable 0x75e3ce68

arm64: (kernel v4.15)
$ ./threadStorage 
one: Address of first variable 0xffffab10a9dc
two: Address of first variable 0xffffaa9099dc
main: Address of first variable 0xffffe26742ec
three: Address of first variable 0xffffaa1089dc

MIPS: (kernel v3.18.140)
$ ./threadStorage
main: Address of first variable 0x7fe384a4
three: Address of first variable 0x7603ef3c
two: Address of first variable 0x7683ff3c
one: Address of first variable 0x77040f3c


In all these systems, the addresses are randomized, so successive runs of the same program produce slightly different results.  The offsets between the locations in successive runs is constant.

The repository is here: https://github.com/youngelf/linux-innards