Repository navigation
Release a process's memory when it ends, and support mremap() - #209
Merged
Merged
Conversation
Bump the kernel to plat-hyperlight-v2 dad4e3db (James Sturtevant's process memory fixes, and three found on the way) and app-elfloader to b9abb45, and rebuild both kernels and the native kernel test fixture. What a process mapped with mmap() stayed mapped after it ended, so a guest that ran child processes again and again ran out of memory. The kernel now unmaps it when the process exits, is killed or execs; memory shared with CLONE_VM stays until the last process sharing it ends. munmap() and mprotect() of a range that began in a 2 MiB block with nothing mapped missed pages past the block's end: they kept their memory, which a mapping made there later read. With memory now reused, a JVM started after System.exit() read the old JVM's and crashed. AT_EXECFN pointed at the path in the exec caller's memory, which now goes with the image execve() replaces; the elfloader copies it to the program's stack, as Linux does. mremap() failed with ENOSYS, so musl took the main thread's stack for a single page and .NET's EnsureSufficientExecutionStack failed. A mapping now shrinks, or grows into free space after it, in place. The java driver sized a JVM after System.exit() from what was left beside the old ones and refused a restart once that ran out; it now gives each JVM the memory the first had, and the restart tests check that instead. The .NET restart and python_processes tests run in their images' memory now, not twice it, and the docs say memory comes back. test_mmap stays skipped, for MAP_SHARED file mappings now. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Three C checks, built with musl. mremap() resizes in place with Linux's errors, and musl finds the main thread's whole stack with it. munmap() and mprotect() of a range that begins in a 2 MiB block with nothing mapped reach the pages past its end. 56 children that each map 16 MiB run one after another in a 64 MiB guest, ending by exit, by a signal, or by an exec from a thread: of this program, or of no_libc, a program without libc that never calls set_tid_address() and so leaves the kernel no stale child TID to write. Each child checks AT_EXECFN still names it, though the exec caller's copy of the path is gone. Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added
the
ci/arm64
The agent image's cold-snap/hello on the Linux runner measures ~28 ms since v0.19.1 (27.86 ms on main 194db1c, 28.11 ms on the re-measure), over its 25 ms ceiling, so main's bench job fails every time. Signed-off-by: danbugs <danilochiarlone@gmail.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The kernel ownership and lifecycle changes require the complete revision delta and guest-level validation of CLONE_VM and threaded exec/exit behavior, neither of which is fully available in this review environment.
No open issues 🎉
What changed in this PR
Updates runtime behavior and regression coverage for process-memory reclamation and in-place mremap() support, allowing repeated child processes and runtime restarts within existing guest memory limits.
Changes:
- Sizes restarted JVM heaps from available memory and verifies memory reuse across Java, .NET, and Python processes.
- Adds C regression checks for mapping resizing, process cleanup, cross-boundary paging, and thread-exec cleanup.
- Updates documentation and Python conformance expectations to reflect the kernel fixes.
| File | Description |
|---|---|
tests/python.rs |
Reduces the process test’s guest memory to 256 MiB. |
tests/java.rs |
Checks heap reuse and catchable out-of-memory errors after restarts. |
tests/dotnet_jit.rs |
Tests runtime restart within the image’s 768 MiB budget. |
tests/compiled.rs |
Adds integration tests for the new C regression checks. |
justfile |
Builds the new C checks and libc-free helper. |
examples/c/process_memory.c |
Checks memory cleanup after exit, signals, and exec, plus shared-memory survival. |
examples/c/page_boundary.c |
Checks unmapping and protection across a 2 MiB boundary. |
examples/c/no_libc.c |
Provides an exec target for detecting stale child-TID references. |
examples/c/mremap.c |
Checks in-place resizing, error cases, and stack-size discovery. |
drivers/java/hl_javadriver.c |
Removes the prior-JVM memory deduction when sizing heaps. |
drivers/dotnet-jit/hl_dotnetdriver.c |
Updates comments to describe memory reuse after exit. |
docs/driver.md |
Documents runtime restart behavior and Java memory checks. |
conformance/python/known_failures.toml |
Updates the remaining mmap limitation to shared file mappings. |
CHANGELOG.md |
Records memory reclamation, paging fixes, and mremap() support. |
.gitmodules |
Updates the elfloader fork comment to include the AT_EXECFN fix. |
Review effort: Balanced
GitHub's ubuntu-latest runners moved to Ubuntu 26.04, whose glibc is built for x86-64-v3 and uses AVX2 without checking for it. The guest never set CR4.OSXSAVE or enabled AVX in XCR0, so every static program linked there (the C, C++ and Rust test binaries) died with SIGILL before main(). Bump the kernel to plat-hyperlight-v2 987cc290, which enables AVX, and AVX-512 where the CPU has it, at boot and sets XCR0 back after a snapshot restore (Hyperlight resets it to x87 only), and rebuild both kernels and the native kernel test fixture. Signed-off-by: danbugs <danilochiarlone@gmail.com>
With AVX on from boot, every VM exit costs more on a nested AMD host: on GitHub's AMD EPYC 7763 runners python-shell cold-snap went from ~19 to ~23 ms and agent's from ~33 to ~48 ms, with the same exit count. My Intel host, also nested, shows no difference. The kernel now boots with SSE only and enables AVX on the first #UD from an AVX instruction, AVX-512 on the first from an AVX-512 one, so programs that check CPUID first keep to SSE and pay nothing, and x86-64-v3 glibc programs still run. Signed-off-by: danbugs <danilochiarlone@gmail.com>
lazy_avx starts with OSXSAVE clear, runs an AVX instruction and checks that a thread blocked in read() across the switch keeps its SSE registers, and that ymm9 (and zmm16, on CPUs with AVX-512) survives another thread using it. The test then restores a snapshot from before, where AVX is off again, and one from after, in the same VM and a new one, where it stays on. Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Memory a process mapped is now freed when it exits, is killed or execs, and
mremap()resizes mappings in place. Freeing memory exposed three older bugs; each was reproduced on stock QEMU/KVM and fixed.plat-hyperlight-v2dad4e3db. It fixes skipped pages in ukpaging at a 2 MiB boundary and clears the child TID at exec. It also adds owner tagging on mappings,mremap, and freeing a process's memory when it ends.app-elfloaderbumped tob9abb45, which copies AT_EXECFN onto the stack.System.exit()gets the memory the first one had.mremap,page_boundary, andprocess_memory(56 × 16 MiB children in a 64 MiB guest).cold-snap/helloceiling raised from 25 to 30 ms; it measures ~28 ms and failed main.plat-hyperlight-v2bc3046af) enables AVX the first time a program uses it, and AVX-512 likewise, and sets XCR0 back after a snapshot restore:ubuntu-latestis now 26.04, whose x86-64-v3 glibc uses AVX2 unconditionally, so every glibc static binary died with SIGILL. It is not enabled at boot because, with AVX on, every VM exit costs more on nested AMD hosts (agent cold-snap ~33 → ~48 ms on GitHub's runners). New C check:lazy_avx.