61 min read
TOC
Computer Architecture for Reverse Engineers and Malware Developers
Before you can break a program you have to know where it lives and how it runs.
We start with memory, because the CPU spends most of its life waiting on it. How storage is layered from registers down to disk. Why DRAM is slow. What cache is actually for. How virtual memory hands every process a private view of the machine and enforces it in hardware. Then the CPU itself, what it does with an instruction and which registers you need to recognise. Then the stack, where the first two meet.
By the end you should be able to open a disassembly listing and know what every sub rsp is for, why an argument sits in RCX rather than on the stack, and which of those things you can corrupt.
The Memory Hierarchy
Computer storage is built as a stack of layers. Each layer closer to the CPU is faster and smaller. Each layer further away is bigger and cheaper per byte.
Nobody can build one memory that is both huge and instant.
Fast storage cells cost transistors. Transistors cost silicon area. Area means physical distance from the core. Distance costs time, because a signal takes time to travel and a long wire is electrically slow. So a memory big enough to hold your whole program is automatically too far away to reach in one clock cycle.
This got worse over decades, not better. CPU clock speeds and instruction throughput grew far faster than DRAM latency improved. The gap has a name: the memory wall. A single trip to main memory now costs a few hundred CPU cycles. A modern core that retires four instructions per cycle could have finished around a thousand instructions in that time.
The hierarchy is the answer. If you cannot have one memory that is both fast and large, use several, and keep the data you are using right now in the fast one.
Rough figures for a current desktop x86 machine. Treat them as orders of magnitude, not spec sheet values.
| |
Look at the bottom two boxes again. An NVMe access at 50 microseconds is roughly 175,000 CPU cycles at 3.5 GHz. Storage is not “a bit slower than RAM.” It is three orders of magnitude slower.
Registers are the top of this pyramid, and they get their own topic further down.
The tiers only work because of one property of real programs, called locality. Programs do not access memory randomly. They reuse the same addresses, and they use addresses next to ones they just used. Cache covers this properly.
Data moves between tiers in fixed-size units, and the unit size changes with the tier. Between cache and DRAM the unit is a cache line, usually 64 bytes. Between DRAM and storage the unit is a page, usually 4096 bytes. Small units near the CPU, large units far from it, because the further you go the more the setup cost dominates and the more you want to amortise it.
You do not allocate cache
People assume cache is something the programmer allocates or manages. It is not. There is no API to put a variable in L2. Hardware decides, automatically, and you influence it only indirectly through your access pattern.
The other one: the pyramid is not a teaching diagram. It is a description of where things physically sit on the silicon and on the board.
Physical Memory and DRAM
Physical memory is the actual chips on the sticks you plug into your motherboard. When someone says a machine “has 16 gigs of RAM,” this is what they are counting.
It is also the furthest out the CPU can reach on its own. Anything beyond it, your SSD and your disk, has to be fetched by the operating system first and dropped into RAM before your code can touch it.
Why DRAM exists at all
A running program needs somewhere to keep its working data. That place has to do four things at the same time.
It has to be fast, answering in tens of nanoseconds.
It has to be random access, meaning you can read address 900 and then address 4 with no penalty for jumping around.
It has to be big. Gigabytes.
It has to be cheap enough that gigabytes are affordable.
Registers and cache are fast but tiny and expensive. Disks are big and cheap but thousands of times too slow. DRAM is the only technology that hits all four at once. That is the whole reason it sits where it sits, and the reason nothing has replaced it.
Why you cannot just use an SSD
Fair question, since NVMe drives are genuinely fast now. Two reasons, and only one of them is speed.
- The CPU cannot address a drive directly: a mov instruction can name a memory address, and the hardware fetches it. There is no instruction that reads a byte out of your SSD. Storage is reached in blocks, through a separate controller, and only the operating system can ask for one. The block lands in DRAM first. Then your code can read it.
- Latency: DRAM answers in 60 to 100 nanoseconds. NVMe takes 20 to 100 microseconds. Roughly a thousand times slower.
One bit of DRAM
Almost everything strange about DRAM comes from how a single bit is built. It is worth two minutes.
One DRAM bit is one capacitor plus one transistor. You will see this written as 1T1C, meaning one transistor, one capacitor.
The capacitor is the storage. A capacitor holds an electrical charge. Think of a very small, very leaky bucket. Charge in the bucket, or no charge in the bucket, is your bit.
The transistor stores nothing. It is a switch. When the memory controller wants this bit, it closes the switch, which connects the capacitor to a wire so the charge can be measured. The rest of the time the switch is open and the capacitor is left alone.
| |
That is the entire cell. One bucket, one switch.
Now compare it to cache memory in the next section, which needs six transistors for every single bit. One capacitor and one switch is far smaller and far cheaper to build. That single fact is why you can afford 16 gigabytes of DRAM but only 32 kilobytes of L1 cache.
Three things follow from this design. You will meet all three again.
The first consequence is refresh
The bucket has a hole in it. Charge drains out of the capacitor on its own, within milliseconds, whether you touch it or not.
So DRAM cannot simply be written and left alone. Every cell has to be read and written back, over and over, forever, purely to preserve data that is already there. That maintenance is called refresh. The memory controller does it in the background and your program never sees it.
This is where the D in DRAM comes from. Dynamic RAM means RAM that needs constant work to hold still.
The second consequence is destructive reads
The charge in one cell is tiny. Measuring it drains it.
So a read is never just a read. The chip connects the capacitor to a wire, a circuit called a sense amplifier catches the value before it fully disappears, and then the chip immediately writes it back. Checking wiped it, so restoring it is not optional.
Read, sense, restore. Every single time.
Refresh is not why DRAM is slow. It only consumes a few percent of available time. The real costs are this destructive read, the row timing in the next section, and the physical trip out of the CPU and across the motherboard.
In plain terms
Think of a sticky note where the ink fades the moment someone reads it. By the time they finish, the note is blank. So the rule is: every time someone reads it, they have to rewrite it immediately, or the information is gone.
The catch is that the reader cannot do the rewriting, because they need to finish reading before the ink fully disappears. So there is a robot standing next to the note whose only job is to catch the value before it vanishes and write it back.
| |
That robot is the sense amplifier. It is built into the chip for exactly this purpose. Every single read triggers it, automatically, without the CPU knowing or caring.
SRAM, the memory used in cache, works like a printed sign on the wall. Anyone can read it as many times as they want. Reading it changes nothing. That is the core reason cache is faster than DRAM, not just the distance, but the fundamental difference in how a read works.
The third consequence is that location matters
DRAM is organised as a grid, rows and columns, like a spreadsheet. Your data lives at the intersection of a specific row and column.
To read a single value, the chip cannot just grab it directly. It has to open the entire row first. Opening a row, called activating it, copies all the data in that row into a small fast buffer inside the chip. Only then can it pick out the specific column you asked for. After it is done, it closes the row again. That is called precharging.
So every read is actually three steps: activate the row, read the column, precharge to close it. That is expensive. But here is the payoff: while the row is still open, any other read targeting the same row skips the activate step entirely and answers much faster. The buffer already has it.
This is why access pattern matters. Walking through memory in order keeps hitting the same open row. Jumping around randomly forces a new activate every time. Same chip, very different performance.
One more thing. When DRAM finally answers, it does not send one byte. It sends 64 bytes in a burst, the entire neighbourhood around what you asked for. A cache line (the fixed 64-byte chunk that cache stores) is also 64 bytes. That is not a coincidence. DRAM and cache were designed together so the smallest thing DRAM wants to hand over is exactly the smallest thing cache wants to store.
The physical address space is not just your RAM
Once an address has been translated, the CPU puts it on the bus as a physical address. It is tempting to picture that address space as your installed RAM, numbered from zero upward.
It is not. The same address space also contains devices. Graphics cards, network cards, and other hardware expose registers that live at physical addresses, so writing to certain addresses talks to a device rather than to memory. Firmware reserves ranges of its own. And there are holes that map to nothing at all.
Worth knowing because the first time you look at a real memory map, it will not match the picture of a clean block of RAM.
What volatile really means
DRAM needs continuous power because the capacitors leak. Cut the power and the data decays.
It does not vanish instantly, though. Charge lingers for a window measured in seconds at room temperature, and considerably longer if the chips are cold.
Everything is plaintext in RAM eventually
Everything is plaintext in DRAM at some point. Keys, passwords, session tokens, decrypted payloads, the unpacked body of a packed executable. Credential theft and memory forensics both rest on that single fact, whether the mechanism is dumping a process, reading a hibernation file, or scraping a target process’s heap.
Freed memory is not scrubbed by hardware either. When a process frees a buffer or exits, the bytes stay where they were until something reuses the page. A secret a program carefully overwrote in one buffer may still exist in a copy it made three function calls ago.
Exiting a process does not scrub its memory
The common belief is that RAM is wiped when a process exits, and wiped instantly when the power goes out. Neither is true.
Process exit returns pages to the kernel’s free list. It does not scrub them. The reason your program does not read the previous tenant’s secrets is that both Windows and Linux zero pages before handing them to a new process. That is an operating system policy, enforced in software. If it were a hardware guarantee, half of memory forensics would not work.
Cache and SRAM
The CPU can execute billions of instructions per second. DRAM takes 200 to 350 cycles to answer. Without something in between, the CPU would spend most of its time doing nothing, waiting for data to arrive.
Cache is that something in between. It is a small amount of very fast memory built directly into the CPU die, holding copies of the data the CPU used most recently.
Why cache is fast
DRAM is slow partly because of how its cells are built. Cache uses a completely different design to get around that.
A DRAM cell is one capacitor and one transistor. A cache cell, called SRAM (Static RAM), is six transistors wired to hold one of two states and actively maintain it.
Four things make SRAM fast compared to DRAM:
- It does not leak, so it never needs refresh.
- Reading it does not destroy the value, so there is no restore step.
- It actively pushes its value out rather than waiting for a sense amplifier to detect a faint charge.
- It lives inside the CPU core, so the wires are extremely short.
Put those together and SRAM answers in 4 to 5 clock cycles. DRAM takes 200 to 350.
Why cache is small
Six transistors per bit is expensive. DRAM manages one transistor and one capacitor per bit, and is manufactured using a process specifically optimised for making cheap, dense capacitors.
That cost ratio is the entire reason L1 cache is measured in kilobytes while main memory is measured in gigabytes. Cache cannot replace RAM. It was never a candidate.
Locality
A small cache only works if the data you need is actually in it. That works in practice because of a property of real programs called locality.
Programs do not access memory randomly. They tend to reuse the same addresses repeatedly. And when they access one address, they usually access nearby addresses next. A loop counter is used thousands of times. An array is walked byte by byte.
Two names for the same idea:
- Temporal locality: an address used recently is likely to be used again soon.
- Spatial locality: an address used means nearby addresses are probably next.
Cache lines
To exploit spatial locality, the CPU does not fetch one byte at a time from DRAM. It fetches a fixed-size block called a cache line, always 64 bytes on x86-64.
Ask for one byte and you get the entire aligned 64-byte block surrounding it. Most of the time, the next few things you need are already in that block.
This is also why the DRAM burst size from the previous section is 64 bytes. DRAM and cache were co-designed: the smallest thing DRAM wants to hand over is the smallest thing cache wants to store.
The levels
The 64-byte cache line is what a single tier stores. The next question is how many tiers are needed, and why.
One cache cannot be both large and single-cycle. The larger it is, the physically bigger it gets, the further away the edges are, and the longer signals take to arrive.
The solution is multiple tiers:
- L1: smallest and fastest, inside each core. Split into L1-I for instructions and L1-D for data. The split lets the core fetch the next instruction and load data at the same time without the two requests competing for one bus.
- L2: larger, slightly slower, private to each core on most current processors.
- L3: largest, shared across all cores on the chip.
If data is not found in L1, the CPU checks L2. If not there, L3. If not there, it goes to DRAM. A miss at every level is a cache miss and costs the full DRAM penalty.
Self-modifying code and the instruction cache
This matters if you write a packer, an unpacker, or anything that generates code at runtime.
The CPU has two separate L1 caches. One for data, one for instructions. The instruction cache holds copies of the machine code the CPU is about to run.
When you write new bytes into memory and jump to them, the CPU needs to execute your new bytes, not a stale cached copy. On x86 this happens automatically. The instruction cache sees the update and your code runs correctly.
Cache stores lines, not variables
Cache does not store variables. It stores 64-byte lines of physical memory. Your four-byte int shares a line with up to fifteen other things.
A cache miss does not mean data is gone. It means a longer trip to DRAM.
Advanced: cache implementation details
Intel vs AMD cache organisation
Not all CPUs organise their cache the same way. This matters when you assume something about cache layout and you are wrong.
On Intel consumer chips, the L3 cache is one big pool shared by all cores on the chip. Core 1 and Core 8 are both pulling from the same L3.
AMD works differently. Their chips are built from smaller clusters called CCX (Core Complex), each with its own L3. A chip with two clusters has two separate L3 caches. If Core 1 on cluster A needs data that Core 5 on cluster B just cached, it cannot grab it from the local L3. It has to travel across an internal interconnect called the fabric, which is slower.
Same idea for L2. Most cores get their own private L2. But Intel’s efficiency cores share one L2 across a group of four. Do not assume L2 is always per core.
Cache coherence
Imagine two cores both have a copy of the same variable in their own L1 cache. Core 1 changes the value. How does Core 2 know?
This is the cache coherence problem. Without a solution, Core 2 would keep using its stale copy and you would have two cores disagreeing about the same memory location.
The solution is a cache coherence protocol. The CPU tracks the state of every cache line across all cores. The common one is called MESI, which stands for four possible states a cache line can be in:
- Modified: this core changed it, nobody else has the update yet
- Exclusive: only this core has it, unchanged
- Shared: multiple cores have a copy, all identical
- Invalid: this copy is stale, do not use it
When Core 1 modifies a line, the protocol automatically marks Core 2’s copy as Invalid. Next time Core 2 tries to read it, it sees Invalid and fetches the updated value.
One thing to keep separate: coherence only guarantees that all cores eventually agree on the value. It does not guarantee the order in which updates become visible. That is a different problem called the memory model, and it is why multithreaded code sometimes needs explicit synchronisation even on coherent hardware.
Cache inclusivity
When you have L1, L2, and L3, a design decision has to be made: does L3 keep a copy of everything that is also in L1 and L2, or not?
Inclusive means yes. L3 holds a copy of every line that exists in any core’s L1 or L2. If you evict something from L3, the hardware automatically evicts it from L1 and L2 too, across all cores.
Non-inclusive means no. L3 only holds lines that are not currently in L1 or L2. When L1 evicts something, it gets pushed down to L3. L3 is a spillover bin, not a mirror.
Intel used inclusive L3 before Skylake-SP and switched to non-inclusive after. AMD uses a non-inclusive design where L3 acts as a victim cache, meaning it only ever holds lines that were kicked out of L2.
Why this matters offensively: in an inclusive design, evicting a line from L3 also evicts it from another core’s L1. That cross-core eviction is exactly what some cache side channel attacks rely on.
L4 cache
Some chips add a fourth cache tier between L3 and main memory.
The reason is simple. Some workloads have a massive working set (the total amount of data they need active at once). Integrated graphics, for example, might need hundreds of megabytes of frame data. That overflows L3 completely, so every access goes all the way to DRAM.
L4 sits in between and catches that overflow. Intel put 128 MB of eDRAM (embedded DRAM, a faster denser type of DRAM built into the chip package) into their Crystal Well chips, which were Haswell and Broadwell processors with Iris Pro graphics. Some server chips use HBM (High Bandwidth Memory, a type of memory stacked directly on top of the chip) in the same role.
The main benefit is bandwidth, not latency. L4 can feed data much faster than going out to the motherboard, even if it is not as fast as SRAM cache.
One thing not to confuse: AMD’s 3D V-Cache is extra SRAM physically stacked on top of the CPU die. It acts as additional L3, not a separate L4 tier.
Virtual Memory
Every running program believes it owns the entire machine. It has its own private memory, starting from a low address and stretching into the gigabytes. No other process exists. Nothing can touch its memory.
That is a lie. A deliberate, useful lie told by the operating system.
The mechanism that makes the lie work is virtual memory.
The problem it solves
Without virtual memory, programs would work directly with physical RAM. That creates three serious problems.
The most important one for offensive security is isolation. If every program shares the same physical memory, any process can read or write any other process’s memory. Your browser’s session tokens are visible to a game running in the background. A bug in one program corrupts another. The kernel itself has no protection.
The second problem is fragmentation. As programs start and stop, physical RAM fills with gaps. Total free memory might be plentiful, but no single contiguous block is large enough to load a new program. It fails to start for no good reason.
The third is relocation. A program compiled to run at one fixed address fails if something else already occupies that address at runtime.
Virtual memory solves all three with one mechanism: indirection. No program ever touches physical RAM directly. Everything goes through a translation layer the OS controls.
The two lies every process believes
When your program runs, it believes two things:
- It has a large, private, contiguous block of memory entirely to itself.
- It is the only process running.
Neither is true. The OS creates that illusion through translation.
Pages and frames
Translation does not happen byte by byte. That would be impossibly slow to track.
Instead, memory is divided into fixed-size chunks. A chunk of virtual memory is called a page. The corresponding chunk of physical memory is called a frame. On x86-64, the standard size is 4 KB.
When a program accesses a virtual address, the hardware translates it to a physical address by looking up which frame this page maps to.
The page tables
The OS keeps the translation mapping in a data structure called a page table. Each process has its own page table. That per-process table is the isolation boundary: your process’s table describes only your pages, so you cannot reach another process’s memory even if you know the address.
On x86-64, the page table is a four-level tree. The diagram below shows how it works. The key thing to take from it: CR3 (a special CPU register that holds the address of the current process’s page table) points to the root, and swapping CR3 swaps the entire memory map.
| |
Why 4 KB pages
Every process has a 48-bit address space, meaning it can use up to 281 trillion different addresses. If the OS tracked every single one in a flat table, at 8 bytes per entry, that table would be enormous, way too big to keep in memory for every running process.
Instead, memory is divided into pages, fixed 4 KB chunks. Now the OS only needs one entry per page instead of one entry per byte. And rather than a flat table, it uses a tree that only grows branches where memory is actually used. A process that only uses a few hundred megabytes barely touches the tree. The empty parts cost nothing.
That is why the page table is a tree, not a flat list, and why pages are 4 KB and not 1 byte.
Permission bits
Each page table entry also carries flags controlling what can be done with that page.
The three that matter:
- Present: is this page currently in physical memory?
- Read/Write: can this page be written to?
- NX (no-execute): is executing code from this page forbidden?
These three flags are how all memory protections work, ASLR, DEP, W^X, guard pages, all of it. Keep that in mind for the offensive hook below.
The MMU
The hardware component that performs translation on every memory access is the MMU, the Memory Management Unit. It lives inside the CPU. Every time your code dereferences a pointer, the MMU translates the virtual address to a physical one using the page tables, or raises a fault if something is wrong.
The TLB
Walking four levels of page tables for every single memory access would add four extra memory reads to every instruction. That would be catastrophically slow.
The CPU solves this with a small cache of recently used translations called the TLB, the Translation Lookaside Buffer. A hit costs roughly one cycle. The MMU checks it first. Only on a miss does it walk the full four-level tree.
The TLB is the reason virtual memory is fast in practice, not just correct in theory.
Page faults
If the MMU cannot complete a translation, it raises a page fault.
A page fault is an exception, not an interrupt. The difference matters: an interrupt arrives from outside, asynchronously. A page fault is caused by the specific instruction currently executing, synchronously. The instruction is paused, the OS handler runs, and if the OS fixes the problem, the same instruction retries and succeeds.
Three things cause a page fault:
- The page is not in physical memory yet. The OS loads it from disk or allocates a new zeroed frame. The common case.
- The access violates the page’s permissions. Writing to a read-only page. Executing a non-executable page.
- The address is not mapped at all. The OS delivers a crash signal to the program: SIGSEGV on Linux, STATUS_ACCESS_VIOLATION on Windows.
The fact that instructions can be retried is what makes demand paging invisible to programs. Memory is not all loaded upfront. It is brought in page by page as it is touched, and the program never notices.
A worked example
| |
| |
The address space is not blank
When your program starts, the address space is not empty. The executable image, the stack, the heap, and shared libraries are already mapped.
The lowest addresses are deliberately left unmapped. Dereferencing a null pointer faults immediately instead of silently reading whatever happens to be at address zero.
Page permissions and the mitigations built on them
Nearly every technique you will write, and nearly every defence in your way, is a page table bit.
- Virtual addresses are the ones you work in: your debugger shows them. Your disassembler labels them. Crash dumps report them. Do not think of them as fake. Think of them as the real coordinates of a specific process’s world, which is why an address that means something in one process means nothing in another.
- ASLR: randomises where the executable image, the stack, the heap, and the loaded libraries get mapped. It only works because mapping is indirect. Practical consequence for tooling: you cannot hardcode an address for anything. A loader that needs a function has to resolve it at runtime, which is why import resolution and export table walking exist as techniques rather than conveniences.
- NX, called DEP on Windows: the execute-disable bit applied to writable pages. Data regions are not executable. This is the single reason you cannot allocate a buffer, copy shellcode into it, and jump, at least not without a second step. That second step is a permission change, covered a few bullets down.
- W^X, write xor execute: the policy that no page should be writable and executable at the same moment. Windows does not enforce this strictly, so you can simply allocate memory as RWX with VirtualAlloc and write shellcode directly into it. On platforms that do enforce it strictly, a loader has to get creative, mapping the same physical memory twice, once writable and once executable, and using each view for its respective purpose.
- Return-oriented programming: the workaround when NX rules out running your own bytes. Instead of supplying code, you supply a chain of addresses of code that already exists and is already executable. Mostly exploitation territory rather than tooling, but worth recognising, because ROP-shaped stacks are one of the things a stack walk is looking for.
- Page permissions are a primitive, not just a defence: the same bits that stop you are the ones you flip. A region that is writable becoming executable is the shape of nearly every shellcode loader, packer, and JIT. That is mprotect on Linux and VirtualProtect on Windows, and a heap region turning executable is precisely what detection logic watches for, because almost nothing legitimate does it.
- Per-process page tables are why injection is an API dance: you cannot write into another process by dereferencing a pointer, because your CR3 does not describe its address space. Hence the sequence of opening a handle, allocating remotely, writing remotely, and creating a remote thread. Every step exists to make the kernel cross the boundary on your behalf.
- Guard pages: pages flagged so that touching them raises a fault instead of succeeding. Used offensively, a fault you control is a place to run code, and a way to notice when something is walking your process’s memory.
- Relocation processing: a reflective loader maps an image itself, at whatever base it can get, and therefore has to walk the relocation table by hand and fix up absolute addresses.
Several of these reference the PE format and the Windows and Linux memory APIs, which are later posts. They are pointers here, not explanations.
Virtual addresses are not fake
Both they and physical addresses are real. They are different namespaces. The virtual one is the one your program, your debugger, and your exploit all operate in, and the translation between them is enforced by hardware on every single access.
Virtual memory is not swapping to disk. Swap is one thing you can build with the mechanism. It is not the mechanism. Both Windows and Linux use paging continuously on a machine with plenty of free RAM, because translation, isolation, permissions, and lazy allocation all depend on it whether or not anything is ever written to disk.
There is no one global page table. There is one tree per process. If there were one map, there would be no isolation, and nothing else here would matter.
Advanced: virtual memory details
Address space size
Your program lives in a 48-bit virtual address space. That means addresses run from 0 up to 2^48, which works out to 256 TB of addressable space.
That space is split in half. The bottom half, 128 TiB, belongs to user space. Your program lives there. The top half belongs to the kernel. The exact boundary on Linux is the address 0x00007FFFFFFFF000. Anything above that is kernel territory and your code cannot touch it.
Some newer CPUs support five-level paging, which expands this to 57-bit addresses and a much larger space. Most machines you will encounter today are still four-level.
On 64-bit Windows, user mode gets 128 TB. That has been the case since Windows 8.1. Earlier versions gave user mode only 8 TB.
TLB implementation details
The TLB is the cache that stores recent virtual-to-physical translations. Modern x86 CPUs actually have two levels of TLB. A small fast first level holds around 64 entries. Behind it sits a larger shared second level holding thousands. When the first level misses, it checks the second before walking the full page table.
Here is the problem with a cache of translations: when the OS changes a page table entry, the TLB still holds the old translation. The hardware does not notice automatically. The kernel has to explicitly tell the CPU to throw away the stale entry.
Three ways it does that:
- invlpg: invalidates the TLB entry for one specific address.
- CR3 write: reloading CR3 flushes the entire TLB for the current process.
- Interprocessor interrupts: if other CPU cores are running and have their own TLBs, the OS sends each one an interrupt telling it to flush too.
One optimisation worth knowing: TLB entries can be tagged with a process ID, called PCID on x86 and ASID on ARM. Without tagging, every context switch has to flush the entire TLB because the new process has different mappings. With tagging, the CPU keeps entries from multiple processes and just checks the tag. Context switches get much cheaper.
Page fault details
When a page fault fires on x86, three things happen automatically before your code gets involved.
First, the CPU records the address that caused the fault in a register called CR2. That is how the OS knows which address to fix.
Second, the fault is assigned number 14, written #PF, which tells the CPU which handler to call from its interrupt table.
Third, an error code is pushed onto the stack. That code is a set of flags telling the handler exactly what went wrong: was the page present or missing, was the access a read or a write, did it come from user mode or kernel mode, and was it an instruction fetch rather than a data access. The handler reads those flags to decide what to do.
Two categories of fault are worth distinguishing. A minor fault is resolved without touching disk, for example when the OS lazily allocates a new zeroed page or handles a copy-on-write situation after a process is forked. A major fault requires reading data from disk. Minor faults are the common case by a large margin. Most of the page faults happening on your machine right now are minor.
The NX bit
Every page table entry has a bit at position 63 called the NX bit, short for no-execute. When it is set, the CPU refuses to execute code from that page.
For the NX bit to work, a CPU feature called EFER.NXE has to be enabled first. EFER is a model-specific register the OS configures at boot. On any modern OS this is always on.
One thing that surprises people: x86-64 has no read-disable bit. You cannot make a page unreadable. If a page is present and your code has the right privilege level, you can always read it. The only access controls are write (the R/W bit) and execute (the absence of NX).
The lowest mapped address
The very bottom of the virtual address space is deliberately left unmapped.
The reason is null pointers. In C and C++, a null pointer is address zero. If address zero were valid memory, a bug that dereferences a null pointer would silently read whatever happened to be there. That would be a security hole and an extremely hard bug to diagnose.
By leaving the bottom region unmapped, any null pointer dereference immediately triggers a page fault and crashes the program visibly.
On Linux the exact floor is controlled by a kernel setting called mmap_min_addr. On most distributions it is set to 65536, meaning nothing can be mapped below that address.
The CPU Execution Cycle
The CPU is the logic and control circuitry that reads instructions and carries them out. Everything else in the machine exists to feed it or to store what it produced.
Hardware is fixed at manufacture. Behaviour needs to be changeable. The CPU solves that by being a machine that reads its behaviour from memory as data. Instructions are just bytes. Change the bytes and the machine does something else, without rewiring anything.
That is also the reason a compiled program, a kernel, and a PowerShell script all end up in the same place. Whatever the source language, it must be reduced to byte sequences the silicon is physically wired to decode.
The classical model is a loop. Each stage has dedicated hardware.
| |
The pieces, in the order they are actually used:
- ISA (Instruction Set Architecture). Not a component. It is the specification: which opcodes exist, how wide addresses are, which registers exist. x86_64 and ARM64 are ISAs. It is the contract the decoder implements.
- Control Unit. Reads the opcode, interprets it against the ISA, and raises the control signals that drive everything else.
- ALU (Arithmetic Logic Unit). The logic gates for integer arithmetic and bitwise operations. Add, subtract, AND, OR, XOR, shifts.
- System clock. An oscillator, measured in GHz. Stages synchronise to it. One instruction may take one cycle or many.
The correction that matters
Modern CPUs do not execute one instruction at a time in order. They are built to run as many instructions simultaneously as possible, as fast as possible.
To do that, the CPU splits into three parts:
- Front end: fetches instructions from memory, decodes them, and feeds them into the core. Still works in program order.
- Out-of-order core: executes instructions in whatever order gets results fastest, not the order you wrote them. If instruction 5 is ready before instruction 3, instruction 5 runs first.
- Back end (retirement): takes completed results and commits them in the original program order, so from the outside the CPU still looks sequential.
| |
The CPU also guesses. When it hits a branch, an if or a loop, it does not wait to find out which way it goes. It picks the most likely path and starts executing it immediately. If the guess was right, no time was wasted. If the guess was wrong, everything from that guess gets thrown away before it becomes visible.
The key idea: executed and committed are two different things. The CPU executes things speculatively and out of order. It only commits them, meaning makes them permanent, in the original order once it is sure they were correct.
It commits in order, it does not execute in order
Those are two different things, and the gap between them matters.
From your program’s perspective, instructions happen in the order you wrote them. That is the promise the CPU makes at retirement. But behind the scenes, dozens of instructions are in flight at once, executing in whatever order the hardware finds most efficient.
For reverse engineering, the sequential view the debugger shows you is accurate enough. It stops being accurate when you reason about timing, race conditions, or why two threads can disagree about the value of the same variable.
Registers
Registers are the CPU’s working memory. They hold the values the CPU is operating on right now.
Not “recently used.” Not “cached.” Right now, this instruction.
Why registers exist
Even L1 cache costs 4 to 5 clock cycles. For most arithmetic operations, waiting that long for both inputs would halve throughput.
Registers solve this by living inside the CPU core itself, connected by the shortest possible wires. Reading a register costs effectively nothing because the register name is encoded directly in the instruction. No address translation. No memory lookup.
Being few is the point, not a limitation. Sixteen names fit in 4 bits of an instruction encoding. Millions of addresses would not.
The register file
x86-64 gives you 16 general-purpose registers, each 64 bits wide.
You need the first row now. The rest show up in later posts, but are listed here for reference.
| |
Sub-register addressing
You can name a fraction of a register.
| |
One rule to memorise: writing to a 32-bit sub-register zero-extends. It clears the upper 32 bits of the full register. So mov eax, 1 clears all of RAX. Writing to an 8-bit or 16-bit sub-register does not zero-extend, and the old upper bits survive.
The instruction pointer
RIP holds the address of the next instruction to execute.
Two things that trip beginners:
- While an instruction is executing, RIP already points past it. That is why call pushes the address of the following instruction, not the current one. Position-independent shellcode depends on this.
- You cannot write RIP directly. There is no mov rip, X. It changes only through jmp, call, ret, and exceptions. A stack buffer overflow does not modify RIP. It overwrites a saved return address sitting in memory, and ret loads that value into RIP. The call stack section shows exactly where that value lives.
General purpose register roles
The 16 GPRs are interchangeable for most arithmetic. But they have implicit roles in specific instructions and in calling conventions. Those roles are what let you read disassembly without symbols.
| Reg | Historical name | Implicit roles |
|---|---|---|
| RAX | Accumulator | Operand for one-operand mul and div. Holds the syscall number before syscall. Holds function return values in both major ABIs. |
| RBX | Base | Nonvolatile in both ABIs, so a callee must restore it. |
| RCX | Counter | Loop counter for rep and loop. CL supplies the count for variable shifts. First integer argument on Windows x64, fourth on Linux SysV. |
| RDX | Data | Upper 64 bits of a 128-bit mul or div result, as the RDX:RAX pair. Port number in DX for in and out. Second integer argument in both ABIs. |
| RSI | Source Index | Source pointer for string and block moves such as movsb. Second argument on Linux, nonvolatile on Windows. |
| RDI | Destination Index | Destination pointer for the same instructions. First argument on Linux, nonvolatile on Windows. |
| RBP | Base Pointer | Historically the base of the current stack frame. See the note below, because this is the one that changed. |
| RSP | Stack Pointer | Top of the stack. Altered by push, pop, call, ret. See the call stack. |
| R8-R15 | none | Added with the 64-bit transition. R8 and R9 are the third and fourth arguments on Windows, fifth and sixth on Linux. R10 carries the first argument in Windows syscall stubs. |
RBP and frame pointer omission
Historically, RBP had one job: point to the base of the current stack frame. Every function would save the previous RBP, set RBP to the current stack position, and restore it on the way out. This created a chain of saved frame pointers you could follow backwards through every active function call.
On x64 Windows, that chain is gone. The ABI requires every function to store its stack layout in a static table inside the executable instead. When something needs to walk the call stack, it reads those tables rather than following saved RBP values. That makes the chain unnecessary.
With nothing depending on RBP as a frame pointer, the compiler is free to use it as a regular general purpose register. So on x64 Windows, seeing RBP used for arithmetic or storing a local variable in a disassembly listing is completely normal.
Stack unwinding covers how those tables work.
Reading disassembly is reading register traffic
When you open a disassembly listing with no symbols, the registers tell you what is happening. mov rcx, X right before a call on Windows means X is the first argument. RAX after the call is the return value. You do not need the function prototype.
Direct syscalls are when you call the kernel directly, bypassing the normal Windows API functions. Security tools often hook those API functions to monitor what a process is doing. A direct syscall skips the hook entirely.
To make a direct syscall, you load the registers yourself. The register layout is slightly different from a normal function call. The first argument goes in R10, not RCX. The reason: the syscall instruction itself overwrites RCX with the return address as part of how it works. If you put your argument in RCX, the instruction destroys it. So every syscall stub in ntdll starts with mov r10, rcx to move the argument out of the way first.
Two other registers worth knowing about:
- GS: on Windows x64, this register always points to a structure called the TEB (Thread Environment Block). The TEB is a block of data the OS maintains for each thread. Inside it is a pointer to the PEB (Process Environment Block), which contains a list of every DLL currently loaded. Malware uses this chain to find loaded DLLs without calling any API function that might be monitored.
- DR0 through DR7: the debug registers. Debuggers write addresses into these to set hardware breakpoints. Anti-debug code checks them to detect whether a debugger is attached.
A buffer overflow cannot touch RIP
A buffer overflow does not overwrite RIP. It cannot. It overwrites a saved return address in memory, and ret does the rest.
Advanced: register implementation details
Why 32-bit writes zero-extend
When you write to a 32-bit sub-register like EAX, the CPU automatically clears the upper 32 bits of the full 64-bit register RAX. When you write to an 8-bit or 16-bit sub-register like AL or AX, it does not. The old upper bits survive.
The reason for the difference comes down to how the out-of-order core works. If the CPU sees mov al, 1, the new value of AL depends on what was already in RAX, because the upper bits are kept. That creates a dependency the CPU has to track. Tracking unnecessary dependencies slows things down.
mov eax, 1 has no such dependency. The upper bits are cleared unconditionally, so the new RAX value is completely independent of whatever was there before. The out-of-order core can process it freely.
The 8-bit and 16-bit behaviour was kept from older x86 for compatibility with old code. It is one reason you will see compilers prefer 32-bit operations even when 8-bit would logically do.
Register renaming
The 16 GPRs are names, not actual storage locations.
Behind the scenes, the CPU has a much larger bank of physical registers, often several hundred. When the out-of-order core needs to execute multiple instructions at once that all use RAX, it maps each one to a different physical register so they do not interfere with each other.
From your perspective as a programmer or reverse engineer, there are 16 registers and you use their names. The hardware handles the rest invisibly.
The /Oy flag
On 32-bit x86, there is a compiler flag called /Oy that tells the compiler to stop using EBP as a frame pointer, freeing it up as a general register. Some older optimisation guides mention it.
It does not exist on x64. On 64-bit Windows, the compiler never needs a frame pointer in the first place, because stack layout is described in tables stored in the executable. There is nothing to suppress.
Future register extensions
x86-64 currently gives you 16 general-purpose registers. Intel has proposed an extension called APX that would add 16 more, named R16 through R31, doubling the count.
AVX-512, a SIMD extension for heavy floating-point and vector work, expands the SIMD registers from 16 to 32 and adds a new type called mask registers, K0 through K7, used to apply operations selectively to parts of a vector.
Neither is present on most machines today and you will not encounter them in typical reverse engineering work.
The Call Stack
The call stack is a region of memory each thread uses to track function calls: where to return, what the local variables are, and which register values need restoring.
Understanding the stack is what makes saved return addresses exploitable, ROP chains legible, and call stack spoofing possible.
Functions call other functions to arbitrary depth, and each call needs its own private scratch space. Registers cannot do this, there are only 16 of them and a call chain can be thousands deep. What you need is storage that grows when a function is called and shrinks when it returns, always in the reverse order. That is exactly what a stack does.
LIFO
The first part of this section covers the concept: what a stack is and why the call pattern requires it. The second part covers the mechanics of how x86-64 implements it.
Last in, first out. The book stack analogy is accurate: put down A, then B, then C, and to reach A you must lift C and then B, in that order.
| |
The two operations are named after the instructions that perform them, push to add and pop to remove.
One stack per thread
Every thread gets its own stack. Not one per process, not one shared between threads. Each thread has its own, completely separate.
When a thread enters the kernel, for example during a system call, it switches to a second separate stack called the kernel-mode stack. Your user-mode stack and the kernel stack never mix.
The stack is memory like any other. It has pages, it has permissions, and on any modern system those permissions do not include execute. Shellcode on the stack is what NX was designed to stop.
Stack sizes. Windows reserves 1 MB per thread by default, taken from the PE header, committing one page at a time behind a guard page. Linux gives the main thread 8 MB by default, adjustable with ulimit -s.
The stack pointer
RSP holds the address of the most recently pushed item. Not the next free slot, the last used one. x86-64 uses a full descending stack: it grows toward lower addresses, and RSP points at live data.
Here is the real x86-64 layout. Learn this one first. A generic upward-growing model appears at the end of the section for contrast, but do not memorise that one.
| |
The pointer arithmetic
This is where the direction trips people up.
| |
In 64-bit mode a push moves RSP by 8 for a 64-bit operand, or by 2 for a 16-bit one. There is no 32-bit push.
That “pop does not erase” line is not a footnote. The bytes stay in RAM until something overwrites them. Physical memory already made this point about DRAM generally, and the stack is where it bites most often, because the same few kilobytes get reused thousands of times per second.
Stack frames
A stack frame is the contiguous block belonging to one active call: its saved registers, its locals, and the space it allocated for its own callees. Once the frame exists, the function reaches its data by displacement, [rsp+0x20] or [rbp-0x8], with no pushing or popping needed.
Not every function gets a frame. A leaf function that calls nothing and needs no locals may allocate nothing at all. An inlined function has no frame by definition, because there was no call. You will see this directly in the debugger output under stack unwinding.
Building a frame
With four procedures calling in a chain:
| |
Each call adds a frame. Each ret removes one. The stack depth at any moment is the call depth.
Tearing a frame down
Reverse order, and it is arithmetic, not a series of pops.
| |
The generic model for contrast only
Growth direction and argument placement are architecture and convention dependent. A conceptual, upward-growing stack with arguments pushed before the return address looks like this. It is not x86-64. Labelled so you do not confuse the two.
| |
On x86-64 the return address is always pushed by call, which happens after the caller has already placed any stack arguments. So the return address ends up nearer the top, not buried under the locals.
Nobody is tracking your frames
The OS does not manage your stack frames. It reserves the memory region and that is it. Everything else, pushing, popping, tracking where you are, is done by the compiled code itself through RSP. Nothing is watching. That is exactly why corrupting a frame works.
Two more things to remember. Arguments do not always go on the stack, on x86-64 the first four or six go in registers. And pop does not delete anything. It just moves RSP.
Calling Conventions
kernel32.dll was built without knowing your program exists. Your program calls into it anyway.
That only works because both sides agreed on the rules before either was compiled: where argument 1 lives, where the return value goes, which registers the callee is allowed to overwrite, and who cleans up the stack. Without a fixed answer to each of those, calling any external code is impossible.
The rules are called a calling convention. The document that specifies them is called the ABI, or Application Binary Interface.
Two conventions matter. Most Windows-focused material teaches only the first. You need both, because you will reverse binaries from both worlds.
| Windows x64 | Linux / SysV AMD64 | |
|---|---|---|
| integer args | RCX RDX R8 R9 | RDI RSI RDX RCX R8 R9 |
| float args | XMM0-XMM3 | XMM0-XMM7 |
| further args | stack | stack |
| return value | RAX, XMM0 if float | RAX, XMM0 if float |
| shadow space | 32 bytes, caller-allocated | none |
| red zone | none | 128 bytes below RSP |
| nonvolatile | RBX RBP RDI RSI RSP, R12-R15 XMM6-XMM15 | RBX RBP RSP, R12-R15 |
| alignment | RSP % 16 == 0 at call | RSP % 16 == 0 at call |
Two terms worth knowing before reading the table.
Volatile (caller-saved): the function can overwrite this register freely. If you need the value to survive a function call, save it yourself first.
Nonvolatile (callee-saved): the function must preserve this register. If it uses it internally, it saves and restores the original value before returning. The caller can always trust it is unchanged after the call.
Note that RDI and RSI are nonvolatile on Windows and volatile on Linux. Memorise the Windows set, reverse a Linux binary, and you will be wrong.
Three things in that table trip up almost everyone: the positional slot rule, the shadow space, and the alignment requirement. Each gets its own section below.
Register slots are positional
You do not get four integer registers plus four float registers. You get four slots. Each slot uses either the integer register or the float register depending on the type of that argument.
| |
The slot number is what matters, not how many ints or floats came before it.
One more thing. When you see an argument that looks like a pointer but the function signature does not take a pointer, it is probably a struct being passed by reference. The caller puts the struct in memory and passes its address instead.
The name “general purpose” is slightly misleading. Compilers treat them interchangeably for plain arithmetic, but the ABI and specific instructions give each one a fixed role, and those roles are exactly what you read when you look at disassembly.
Shadow space on Windows only
Before making a call, the caller reserves 32 bytes on the stack for the callee’s use. Always 32 bytes, even if the function takes no arguments.
Why? The callee sometimes needs to store its register arguments in memory, for example if it needs to take the address of a parameter, or if it is a varargs function that needs all arguments laid out contiguously. The shadow space gives it a guaranteed home to do that, at a predictable location.
Why the same slot has different offsets in caller and callee
This is the single most confusing thing in the topic. The same slots have different offsets depending on whether you are looking before or after the return address gets pushed.
| |
If you read a disassembly listing that says [rsp+8] is the RCX home, that is the callee’s view. If you see the caller writing argument 5 to [rsp+32], that is the caller’s view. Both are correct. Always ask which side of the call you are on.
Alignment
The ABI requires RSP to be 16-byte aligned at the point of the call. What most people forget is what that means on the other side:
| |
That is why sub rsp, 40 appears instead of sub rsp, 32 in a function with no locals. 32 for the shadow space, 8 to fix alignment. It is not that shadow space is “32 plus 8.”
Worked example of a five-argument call
| |
Floating-point arguments use movss for 32-bit singles and movsd for 64-bit doubles. Neither takes an immediate, so a literal has to be loaded from memory or built in a register first. And the operand size follows the instruction: movss moves a dword, movsd moves a qword. Writing movss qword […] is a size mismatch.
Calling anything by hand
Calling conventions are how you read a disassembly listing without any symbols. Four mov instructions into RCX, RDX, R8, R9 followed by a call tells you the arity and, from the values, often the semantics.
They are also how you call anything by hand. Resolving a function address and jumping to it is the easy part. Getting the arguments into the right registers, reserving the shadow space, and keeping RSP aligned is what makes the call actually work. Every one of those is a place shellcode dies quietly.
The Windows syscall convention is a deliberate variant, covered under registers: R10 instead of RCX for the first argument, because syscall clobbers RCX and R11.
Shadow space belongs to the callee
Shadow space is not the caller’s scratch space. The caller allocates it, but it belongs to the callee.
The shadow space is exactly 32 bytes. The extra 8 bytes you often see in a sub rsp instruction is just alignment padding, not part of the shadow space itself.
Prologues and Epilogues
When you call a function, it takes over the CPU. It moves RSP, overwrites registers, and does its work. When it returns, the caller expects everything to be exactly as it was.
Something has to enforce that contract. That something is the prologue and epilogue: a handful of instructions the compiler inserts at the start and end of every function automatically. Because the compiler emits them the same way every time, prologues are one of the most recognisable patterns in disassembly.
Here is a real main() prologue from a Windows x64 binary:
| |
RBP and RDI are nonvolatile registers. If main wants to use them, it has to save them first and restore them before returning. That is what the two pushes do.
sub rsp, 118h does two things at once. It reserves 32 bytes of shadow space for any functions main will call, and it reserves 248 bytes for main’s own local variables.
Sometimes a function also needs to copy its incoming register arguments onto the stack. This is called spilling. A function spills its arguments when it needs to take the address of a parameter, or when it is a varargs function, or when the compiler simply decides stack storage is easier to work with. Here is what that looks like:
| |
The spills happen before the pushes. At that point call has already pushed the return address to [rsp], so arg1 sits at [rsp+8], arg2 at [rsp+10h], and so on. Once the pushes happen those offsets shift, so the spills have to come first.
Prologue patterns are how you find function boundaries in a stripped binary. Learn to spot the shape and a wall of bytes turns into a function list.
The epilogue is the prologue in reverse:
| |
The locals are not popped one by one. One add reclaims all of them at once. Only the saved registers get popped, and they must come off in the reverse order they went on.
| |
The epilogue shape is not a style choice. The x64 unwinder, covered in the next section, needs to recognise an epilogue when it sees one. The ABI restricts it to exactly: a stack adjustment, then only pops, then ret. That is why every epilogue you will ever read looks the same.
In short: the prologue is the function setting up its workspace, saving what it borrowed and carving out space for its own work. The epilogue is the function cleaning up that workspace and handing everything back exactly as it found it.
The prologue is not about the frame pointer
The prologue’s job is not to save the frame pointer. On x64 that is usually not happening at all, for the reasons given under registers. The prologue’s job is to save whichever nonvolatile registers this function intends to use, and to reserve space for locals plus the shadow space for its own callees.
Stack Unwinding
Forward dependency. This topic needs .pdata and .xdata, which are sections of the PE file format. That is a separate post. Names are used here without explanation on purpose. Come back after reading it if the file-format side matters to you.
Stack unwinding is walking back through the chain of active function calls at runtime, restoring each caller’s state as you go.
Why it exists
When an exception is thrown inside a deeply nested call chain, execution needs to jump from wherever the exception happened to the function that can handle it, which might be many frames above.
Getting there is not just a jump. Every frame in between has to be cleaned up: stack space reclaimed, saved registers restored.
The frames themselves do not describe how to do this. An instruction like sub rsp, 0x118 leaves no record in memory that 0x118 was the amount. The unwinder has no way to read that from the stack alone.
So the information is recorded somewhere else. On x64 Windows, it is stored in static tables inside the executable itself.
That is also why RBP is free for general use on this platform. If unwinding relied on a chain of saved frame pointers, every function would need one. Because it reads tables instead, no frame pointer is needed.
How the tables work
Each of the three structures answers one question the unwinder needs.
- RUNTIME_FUNCTION: where does this function start and end, and where is its unwind info?
- UNWIND_INFO: what did the prologue do? Which registers did it save? Is there an exception handler?
- UNWIND_CODE: one entry per prologue operation, describing each step so the unwinder can reverse it.
To unwind a frame, the unwinder looks up the current instruction pointer in the RUNTIME_FUNCTION table to find that function’s UNWIND_INFO, then replays the UNWIND_CODE entries in reverse to undo the prologue.
Virtual unwinding
| |
When a debugger shows a stack backtrace, it performs a virtual unwind: the same walk, but against a copy of the register state, without touching the real stack. This is what a security product does when it wants to know who called an API.
Read the chain bottom to top and you have the thread’s execution path from startup to this exact moment. Frame 03 is direct evidence that not every function has a stack frame: inlined functions vanish completely.
Both sides of the stack walk
A security product hooks an interesting API, for example VirtualProtect, and virtually unwinds the stack to see the call chain. A chain that runs through legitimate DLL code from a plausible thread entry looks ordinary. A chain that leads back to freshly allocated anonymous memory does not.
Offensively, the data being walked lives on a stack the attacker controls. A synthetic chain of return addresses pointing at legitimate code will be walked as real. This is call stack spoofing. It works in part because of leaf functions: if a function has no RUNTIME_FUNCTION entry, the unwinder assumes the return address is simply at [RSP]. An attacker who controls RSP controls what the unwinder reads.
That is not the same as undetectable. Synthetic frames leave traces: return addresses pointing at memory with no backing module, stack pointer values inconsistent with the claimed frames, chains that do not validate against the unwind tables of the modules they reference. EDR detection targets exactly those inconsistencies.
There is no chain of frames to follow
The stack does not contain a linked list of frames you can follow. Not on x64 Windows. There is no reliable chain of saved frame pointers. The walk is reconstructed from static tables in the binary plus the current register state.
A debugger backtrace is a virtual unwind, not a read of a chain. Real unwinding, during exception dispatch, does modify state. They are different operations.
Advanced: unwinding internals
Two-pass exception dispatch
When an exception fires, Windows does not immediately start cleaning up. It runs two separate passes.
Pass one: search. Windows walks up the call stack frame by frame, looking for a function that has a handler for this exception. It does not touch anything yet. It is just looking.
Pass two: unwind. Once a handler is found, Windows goes back to the beginning and walks the stack again, this time actually cleaning up each frame, running destructors, executing finally blocks, restoring registers.
The reason it does two passes instead of one: if Windows started cleaning up frames before finding a handler, and then found no handler at all, it would have already destroyed the stack state needed to produce a useful crash dump.
| |
RtlVirtualUnwind
RtlVirtualUnwind is the Windows function that does the actual work of unwinding one frame. You give it the current instruction pointer and a snapshot of the register state, called a CONTEXT structure, and it reads the unwind tables for that function and figures out what the registers looked like one frame up.
It only does one frame per call. To walk the entire stack, something has to call it in a loop, moving up one frame at a time until there are no more frames left.
One edge case worth knowing: RtlLookupFunctionEntry is the function that looks up a frame’s entry in .pdata. For a leaf function, the one with no .pdata entry, it returns nothing. When that happens, the unwinder makes an assumption: the return address is simply sitting at [RSP]. This is the case that call stack spoofing exploits, because an attacker who controls RSP controls what the unwinder reads as the return address.