Memory Management

Memory management lives in src/mm/ and src/arch/x86_64/mm/. The kernel runs in the higher half; user processes occupy isolated lower-half address spaces.

Address Space Layout

Physical Memory Manager (PMM)

src/mm/pmm.cpp is a bitmap allocator with per-frame reference counts, both allocated from the first fitting USABLE region of the boot memory map. One global spinlock (irqsave) serializes everything.

• 4 KiB frames, 1 bit per frame (set = used), plus a uint16_t refcount per frame.
• All frames start reserved; USABLE ranges are freed, then the bitmap/refcount frames and frame 0 are re-reserved.
• Frames are classified into two zones derived at init: ZONE_DMA32 (below 4 GiB) and ZONE_NORMAL (the rest). The single bitmap/refcount arrays still back both (per-zone locks and magazines are future work).
• pmm_alloc_frame() returns a zeroed physical frame from ZONE_NORMAL, falling back to ZONE_DMA32 when NORMAL is exhausted (order-0 only).
• pmm_alloc_frames(count) allocates physically contiguous frames top-down from ZONE_NORMAL; while NORMAL exists it never falls back to DMA32 (contiguous DMA32 runs are protected for 32-bit devices) and failure logs and returns null. On machines with no RAM above 4 GiB, DMA32 is the only zone and is scanned.
• pmm_alloc_frame_dma32() / pmm_alloc_frames_dma32(count) allocate strictly below 4 GiB and never fall back to high memory; 32-bit-only devices (RTL8139, AC97) must use these via vmm_alloc_dma_with_flags(..., VMM_DMA_32BIT), and attach fails when the zone cannot satisfy the request.
• Refcounts power copy-on-write and shared mappings: pmm_refcount_inc panics on a free frame; pmm_refcount_dec frees at zero; invalid frees are reported, not silent.
• Stats: pmm_get_free_memory(), pmm_get_total_memory() feed the mem shell command and SYS_GETMEMINFO.

Virtual Memory Manager (VMM)

src/mm/vmm.cpp implements classic 4-level paging (PML4 → PDPT → PD → PT). Intermediate tables are allocated with atomic compare-and-swap; a racing loser adopts the winner's frame. Encountered 1 GiB/2 MiB leaves are split on demand.

PTE flags: PRESENT, WRITABLE, USER, PWT, PCD, NX, and the software bit PTE_SHARED (bit 52: shared page, no copy-on-write). Cache types:

• PTE_MMIO = present, writable, PCD+PWT, NX (strong UC under the default PAT).
• PTE_WC = present, writable, PCD, NX with PAT index 2 programmed to WC (per-core PAT setup; APs must run pat_init or WC reads back as UC).

Key operations:

• vmm_create_address_space: fresh PML4 with kernel entries 256-511 copied.
• vmm_clone_address_space (fork): kernel half shared, user half deep-copied with leaf refcount bumps; writable non-shared pages are write-protected in both spaces (copy-on-write).
• vmm_map_mmio / vmm_alloc_dma / vmm_alloc_dma_with_flags: mappings from the MMIO/DMA window with batched invalidation; VMM_DMA_32BIT routes the frame allocation to ZONE_DMA32 for 32-bit-only devices. vmm_free_dma performs a full TLB shootdown before frames return to the pool, so no remote core can keep DMAing into a reused frame.
• Unmap/rollback ordering invariant: PTEs are cleared under the per-mm VMA lock with no flush; the lock is dropped first, then one batched vmm_invalidate_tlb_range runs, then futex waiters on freed frames are notified, then frames return to the PMM. No cross-core shootdown ever runs under the VMA lock — a sibling thread faulting on the range spins on that lock with IRQs off and could never take the shootdown IPI.
• vmm_remap_framebuffer: replaces the loader's WB mappings in place with WC+shared.
• Kernel stacks live in a dedicated region [0xFFFFFD8000000000, 0xFFFFFE0000000000) (PML4 slot 507, pre-created at init like the MMIO window so every address space sees it): each task gets 17 pages — one non-present guard page below 16 RW pages mapped from PMM frames via vmm_map_kernel_stack. A kernel-mode #PF in the region panics with the task's pid/name and fault address ("kernel stack overflow"). Unmapping flushes before frames return, preserving flush-before-free; slots recycle LIFO (capacity ~15k concurrent stacks).
• vmm_protect_kernel: .text read+execute, .rodata/.requests read-only+NX, .data through __kernel_end writable+NX.
• vmm_handle_page_fault: demand paging for VMA-covered addresses, copy-on-write resolution (copy when refcount > 1, in-place permission upgrade otherwise), shared-page flag fixup. PTE installation happens under the VMA lock, but the TLB invalidation runs after the lock is released: vmm_invalidate_tlb waits for every core to acknowledge the shootdown IPI, and a sibling core spinning on the VMA lock with interrupts disabled can never take that IPI — flushing under the lock deadlocks into the shootdown timeout panic.

The single VMA lock (process->vma_lock_ptr, the leader's embedded lock shared by all threads) is the serialization point for address-space changes. SYS_MMAP (anonymous and memfd), SYS_SHM_MAP, the framebuffer mapping, display-buffer mappings and the per-thread TLS install (tls_install) publish their VMA and install their PTEs under one continuous lock hold, with rollbacks removing the VMA again; SYS_MUNMAP, SHM/display-buffer unmap and sys_mprotect likewise clear/rewrite PTEs while still holding the lock that removed or trimmed the metadata, deferring the flush until after the unlock. This is what keeps fork's vmm_clone_address_space (which holds the same lock and does check-free pmm_refcount_inc on every present leaf) from racing an unmap into refcount_inc on free frame panics or cross-process frame reuse.

sys_mprotect applies the new protection to exactly the requested sub-range: overlapping VMAs are cut at the range boundaries first (vma_split_range in src/mm/vma.cpp), so a sub-range call — e.g. the one-page PROT_NONE guard at the base of a pthread stack mapping — leaves the rest of the mapping's flags untouched. It preserves the software PTE_SHARED bit (dropping it would make the next fork COW-downgrade a genuinely shared mapping), refuses to make a COW-shared page writable in place (leaves it read-only so the write-fault path performs the copy), and rejects lengths that would wrap the page rounding. SYS_MMAP treats PROT_EXEC as opt-in (NX unless requested), matching sys_mprotect. A PROT_NONE page keeps its PTE PRESENT — sys_mprotect recomputes the flags from prot (PRESENT|USER always, WRITABLE only on PROT_WRITE, NX unless PROT_EXEC), so on the pthread guard, carved from an already-NX RW stack mapping, the only flag dropped is WRITABLE: reads through a guard page still succeed, while writes fault to SIGSEGV — the pthread guard page stops stack overflow on write traffic only.

There is no PCID/INVPCID; local flushes are invlpg loops (> 32 pages: CR3 reload).

TLB Shootdowns

Cross-core invalidation uses an IPI with monotonic sequence acknowledgements (src/mm/vmm.cpp):

• The sender takes the shootdown lock, bumps a global sequence, enqueues {address, pages, seq} into a per-CPU FIFO ring (16 entries) on every other online CPU, sends one IPI per target, and releases the lock before waiting.
• Targets drain their ring in order, flush locally, and publish tlb_ack_sequence per request with release semantics.
• The sender polls acknowledgements for up to 40 rounds of bounded pause-loops, re-IPI-ing lagging CPUs between rounds, and panics on timeout. Enqueue waits on a full ring are also budgeted and panic rather than spin forever.
• Fresh APs copy the current sequence before publishing themselves online, so they never owe acknowledgements for earlier invalidations.
• Requests are never overwritten before acknowledgement, so late IPIs cannot be misread.

Kernel Heap

src/mm/heap.cpp is a bucketed allocator protected by one global spinlock:

• 8 buckets: 32, 64, 128, 256, 512, 1024, 2048, 4096 bytes. Free blocks live in per-bucket lists; slab pages are tracked in an open-addressed table (65536 slots). When a page's blocks are all free, the page returns to the PMM.
• Allocations carry a header with size and magic (0xC0FFEE1234567890); free and realloc validate it.
• aligned_alloc plants a 24-byte inner header {aligned_offset, base, magic} in the over-allocation; free trusts an aligned back-pointer only after a full validation chain (magic, base sanity, exact offset round-trip, live base magic, size spanning) and scrubs the base magic with a freed sentinel on release — recycled blocks can never fake aligned metadata. Any corruption aborts loudly (panic in debug, logged refusal in release), never a misdirected free.
• Requests larger than one page go straight to contiguous PMM frames at their HHDM address.
• heap_init only zeroes metadata; the heap grows on demand. Global operator new/delete forward to it.
• heap_dump_stats() walks the free lists and tracked-page table for debugging.