feat: add configurable scratch region base GPA by danbugs · Pull Request #1271 · hyperlight-dev/hyperlight

danbugs · 2026-03-03T22:51:57Z

Summary

Adds a configurable scratch region base GPA to SandboxConfiguration, makes the snapshot memory region writable for non-init-paging guests, and uses compile-time #[cfg(feature = "init-paging")] gates for scratch GVA computation.

Motivation

Nanvix runs as a 32-bit (i386) guest without Hyperlight's init-paging feature. The default scratch region GPA sits at the top of the 36-bit physical address range — unreachable by a non-paging 32-bit kernel. This PR lets the caller place the scratch region at an arbitrary GPA (e.g., just below 4 GB) and ensures the snapshot region is host-writable so non-paging guests can function correctly.

Changes

Commit 1: `feat: add configurable scratch region base GPA`

config.rs: New scratch_base_gpa: u64 field (0 = use default) with getter/setter. Uses u64 with a zero sentinel rather than Option<u64> to maintain #[repr(C)] FFI safety, matching the existing heap_size_override pattern.
layout.rs: Uses custom GPA when specified
hyperlight_vm.rs: Stores and passes through scratch_base_gpa
gdb/mod.rs: Adapts to layout changes
snapshot.rs: Takes scratch_base_gpa: u64 directly instead of computing it from size

Commit 2: `fix: make snapshot region writable for non-init-paging guests`

shared_mem.rs: Makes snapshot region writable at host level so non-paging guests can write to it

Commit 3: `fix: use init-paging feature gate for scratch GVA computation`

layout.rs + hyperlight_vm.rs: Uses #[cfg(feature = "init-paging")] for scratch GVA computation. With init-paging, the GVA is the high-canonical page-table-mapped address; without it, GVA = GPA (identity mapped). This is a compile-time decision, not a runtime one.

Commit 4: `fix: use scratch base GPA instead of scratch size in page table walker`

mgr.rs: Updates read_guest_memory_by_gva (behind trace_guest/mem_profile features) to pass the correct scratch_base_gpa value instead of scratch_size to access_gpa and SharedMemoryPageTableBuffer::new, matching their updated signatures from commit 1.

Test plan

cargo clippy passes with all feature combinations (kvm,init-paging, kvm,executable_heap, kvm,mshv3,mem_profile, kvm,mshv3,trace_guest, etc.)
cargo test -p hyperlight-host --no-default-features -F "kvm,init-paging" --lib passes
Non-init-paging build: cargo build -p hyperlight-host --no-default-features -F "kvm,executable_heap" --lib succeeds
Existing CI should pass

dblnz

LGTM!
However, I am not that well acquainted with this part of the codebase. Maybe @syntactically can take a look

dblnz · 2026-03-04T09:23:22Z

src/hyperlight_host/src/hypervisor/regs/special_regs.rs

                ..Default::default()
            },
+            // CR0.ET (bit 4) is hardwired to 1 on modern x86 CPUs.
+            // MSHV rejects the vCPU state if ET is not set.


How hasn't mshv failed so far? was this scenario not tested before?

I don't believe there are many tests for the 16/32-bit guests.

dblnz · 2026-03-04T09:24:55Z

src/hyperlight_host/src/hypervisor/hyperlight_vm.rs

+        // MSHV (unlike KVM) does not provide sane defaults for the XSAVE
+        // area on a freshly created vCPU, and will reject the VP state
+        // on the first run if it is left uninitialized.
+        vm.reset_xsave().map_err(VmError::Register)?;


I'm curious how this worked before

It looks like at issue here is the difference between bringing up the vcpu in real vs long mode. Does the xsave reset call need to be there in both, or should it just be in real mode? I know we had tests of the long mode mshv stuff passing before.

I think the real vs long is the issue here. When I did some investigation in mxcsr on kvm awhile back it did work fine in long mode.

I agree that if we don't need this for some targets it would be nice to not waste cycles here

syntactically · 2026-03-04T13:12:29Z

I'm not clear on the need for a configurable region base for scratch. You can just set MAX_GPA/MAX_GVA based on target architecture + feature flags? (see src/hyperlight_common/src/arch/i686/layout.rs). Maybe the selection based on target_arch of which layout module to use is broken in the host for the i686 case where it's the guest arch that counts; maybe that needs to change to something like #[cfg_attr(any(target_arch = "x86", all(target_arch = "x86-64", not(feature = "init-paging")), path = "arch/i686/layout.rs")]. (Or, we could add cargo cfg aliases for guest arch, or something?). I would really prefer to avoid re-adding more configuration knobs.

I'm not clear on how the idea of making snapshot pages writable is going to work with the snapshot-first lifecycle (#1268). This seems like it will break a lot of assumptions and make a lot of code more complex/rife with opportunities for vulnerabilities. If you really are unable to run on amd64 natively, would teaching hyperlight about i686 paging structures be a slightly smaller break of the conceptual model?

jsturtevant · 2026-03-04T22:08:53Z

I'm not clear on how the idea of making snapshot pages writable is going to work with the snapshot-first lifecycle (#1268).

This seems like something we could flush out in that discussion? With CoW changes we've broken some downstream consumers, so it makes sense give an escape hatch (maybe behind a feature flag) while we sort through what we want to do. As far as I understand, this isn't a breaking change and doesn't introduce issues now? or does it? To be clear I think we do need to reconcile this requirement and enabling this wouldn't be a long term solution

ludfjig

does nanvix not use snapshotting? If we can do this without adding a config option, I think that would be good too.

ludfjig · 2026-03-04T23:58:05Z

src/hyperlight_host/src/hypervisor/hyperlight_vm.rs

+        // MSHV (unlike KVM) does not provide sane defaults for the XSAVE
+        // area on a freshly created vCPU, and will reject the VP state
+        // on the first run if it is left uninitialized.
+        vm.reset_xsave().map_err(VmError::Register)?;


I agree that if we don't need this for some targets it would be nice to not waste cycles here

danbugs · 2026-03-05T20:22:20Z

Addressed the config option concern. Latest push removes set_scratch_base_gpa() from SandboxConfiguration entirely. Instead, the i686 memory layout is now auto-selected at compile time:

When init-paging feature is disabled (standard hyperlight guests), the existing amd64 layout is used unchanged.
When init-paging feature is enabled (Nanvix), an i686-compatible layout is selected in hyperlight-common that places the scratch region at a lower GPA (below 4GB), compatible with 32-bit page tables.

The init-paging feature is propagated from hyperlight-host to hyperlight-common via Cargo feature unification, so no runtime configuration is needed.

Regarding the xsave commit that was on this branch — it's been dropped from the latest push since it's orthogonal to the scratch GPA changes. The writable snapshot commit remains, gated behind init-paging. Nanvix needs writable snapshots because its guest page tables and kernel state live in the snapshot EPT region and must be writable during execution.

Note that Nanvix doesn't use Hyperlight's snapshotting mechanism. Instead, it has a kernel call that, being guest-aware, can VMExit at any time to snapshot. Nanvix does this because it doesn't execute like our purpose-built guest binaries (e.g., it doesn't use guest function calls). Instead, it just calls evolve once.

Propagate the init-paging feature flag from hyperlight-host to hyperlight-common so that the layout module selects the i686 constants (32-bit MAX_GPA/MAX_GVA) when init-paging is disabled. This allows 32-bit guests (like Nanvix) to use the correct scratch region placement without needing a configurable GPA override. Also make the snapshot memory region writable for non-init-paging guests: without guest page tables there is no CoW layer, so the CPU needs direct write access (e.g. for GDT descriptor Accessed bits during segment loads). Signed-off-by: danbugs <danilochiarlone@gmail.com>

danbugs requested review from dblnz, devigned, jprendes, ludfjig, marosset, simongdavies and syntactically as code owners March 3, 2026 22:51

danbugs force-pushed the danbugs/configurable-scratch-gpa branch from 9275564 to b961eb1 Compare March 3, 2026 23:04

danbugs added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Mar 3, 2026

danbugs force-pushed the danbugs/configurable-scratch-gpa branch from ef6485a to 4b579d3 Compare March 3, 2026 23:19

danbugs mentioned this pull request Mar 4, 2026

Support Nanvix #1273

Open

dblnz reviewed Mar 4, 2026

View reviewed changes

syntactically mentioned this pull request Mar 4, 2026

feat: add hardware interrupt support (PIC, KVM IRQ chip, MSHV SynIC, WHP software timer) #1272

Open

7 tasks

ludfjig reviewed Mar 5, 2026

View reviewed changes

danbugs force-pushed the danbugs/configurable-scratch-gpa branch 5 times, most recently from cac29ff to f3b4089 Compare March 6, 2026 00:22

danbugs force-pushed the danbugs/configurable-scratch-gpa branch from f3b4089 to 92140ff Compare March 6, 2026 02:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add configurable scratch region base GPA#1271

feat: add configurable scratch region base GPA#1271
danbugs wants to merge 1 commit intohyperlight-dev:mainfrom
nanvix:danbugs/configurable-scratch-gpa

danbugs commented Mar 3, 2026 •

edited

Loading

Uh oh!

dblnz left a comment

Uh oh!

dblnz Mar 4, 2026

Uh oh!

syntactically Mar 4, 2026

Uh oh!

dblnz Mar 4, 2026

Uh oh!

syntactically Mar 4, 2026

Uh oh!

jsturtevant Mar 4, 2026

Uh oh!

ludfjig Mar 4, 2026

Uh oh!

syntactically commented Mar 4, 2026

Uh oh!

jsturtevant commented Mar 4, 2026

Uh oh!

ludfjig left a comment

Uh oh!

ludfjig Mar 4, 2026

Uh oh!

danbugs commented Mar 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

danbugs commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Commit 1: feat: add configurable scratch region base GPA

Commit 2: fix: make snapshot region writable for non-init-paging guests

Commit 3: fix: use init-paging feature gate for scratch GVA computation

Commit 4: fix: use scratch base GPA instead of scratch size in page table walker

Test plan

Uh oh!

dblnz left a comment

Choose a reason for hiding this comment

Uh oh!

dblnz Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

syntactically Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

dblnz Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

syntactically Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

jsturtevant Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

ludfjig Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

syntactically commented Mar 4, 2026

Uh oh!

jsturtevant commented Mar 4, 2026

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

ludfjig Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

danbugs commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

danbugs commented Mar 3, 2026 •

edited

Loading

Commit 1: `feat: add configurable scratch region base GPA`

Commit 2: `fix: make snapshot region writable for non-init-paging guests`

Commit 3: `fix: use init-paging feature gate for scratch GVA computation`

Commit 4: `fix: use scratch base GPA instead of scratch size in page table walker`

danbugs commented Mar 5, 2026 •

edited

Loading