Compose VM rootfs from shared layer blobs

[chruffins]

summary

replace the old OCI rootfs unpack path with explicit layer-by-layer composition from the shared OCI cache.

this keeps the guest and hypervisor contract unchanged: hypeman still produces one read-only rootfs disk for the VM.

pre-existing flow

The old path asked umoci to resolve the manifest and unpack the complete image directly from the OCI layout:

pull image into shared OCI layout
  → read manifest and config
  → umoci.UnpackRootfs(complete manifest, target directory)
  → export target directory as one rootfs disk
  → boot VM with rootfs disk

Pseudocode:

function pullAndExport(image):
    pull image into ociCache
    manifest = readManifest(ociCache)
    config = readConfig(ociCache)
    validateConfigForUnpack(manifest, config)

    umoci.unpackRootfs(
        manifest,
        ociCache,
        targetDir,
        mapContainerRootToCurrentUser = true,
    )

    exportRootfs(targetDir, rootfsDisk)

new flow

The new path reads the persisted manifest model and applies each layer blob explicitly, in manifest order:

pull image into shared OCI layout
  → extract and persist manifest model
  → validate model and layer/config pairings
  → create isolated staging directory
  → apply each layer blob in order
  → replace export directory with the composed tree
  → export tree as one rootfs disk
  → boot VM with rootfs disk

Pseudocode:

function pullAndExport(image):
    pull image into ociCache
    model = extractManifestModel(ociCache)

    composeRootfs(model, exportDir):
        validateManifestModel(model)
        stagingDir = createTempDirectory(nextTo(exportDir))

        for layer in model.layers in manifest order:
            verifyCompressedDigest(layer, ociCache)
            applyLayer(
                blob = ociCache[layer.digest],
                destination = stagingDir,
                onDiskFormat = DirRootfs,
            )
            verifyDiffID(layer, decompressedLayer)

        chmod(stagingDir, 0755)
        removePath(exportDir)
        rename(stagingDir, exportDir)

    exportRootfs(exportDir, rootfsDisk)

what changed

• Added composeRootfs, which applies shared layer blobs with umoci's DirRootfs format.
• Preserved OCI layer semantics: whiteouts, opaque directories, hardlinks, and file/directory replacement.
• Added manifest-model validation for layered rootfs configs, config media types, layer media types, and one diff ID per layer.
• Re-verify compressed blob digests and diff IDs during every composition, including cache hits.
• Applied the existing per-layer 100 GiB decompressed-size limit to image pulls.
• Preserved tar ownership when running as root instead of mapping container root to the current user.
• Kept the layer_unpack metric label and existing pull-failure message shape for observability compatibility.
• Keyed temporary build directories by resolved manifest digest so concurrent builds cannot share or delete each other's staging trees.
• Removed the old unpackLayers implementation and its manifest conversion helpers.

why one disk remains

The guest currently mounts one read-only lower disk (/dev/vda) and one writable upper disk (/dev/vdb). That contract is shared by Cloud Hypervisor, Firecracker, QEMU, and vz, so this PR composes the layers into one rootfs rather than attaching one device per layer.

Layer blobs are shared in the host-side OCI cache; the VM still receives its existing single rootfs disk. Layer-artifact reuse is a separate follow-up path.

failure and cleanup behavior

Composition happens in an isolated .compose-* directory. A successful compose replaces the destination tree. A failure or crash can leave the destination absent or strand a staging directory, so GC/reconciliation must remove stale staging trees in a follow-up.

Callers currently use fresh temporary export directories and do not read the destination concurrently.

validation

• Two-layer fixture covering whiteouts, opaque directories, file/directory replacement, ownership, and modes.
• EROFS export followed by fsck.erofs --extract.
• Captured corrupt-manifest fixture still fails cleanly with the existing diff-ID pairing error.
• Targeted image, path, build, scope, and API tests pass.
• Docker Hub pull tests are rate-limited in this environment; VM lifecycle tests require network-bridge privileges. Both failures reproduce on unmodified main.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2b465b6. Configure here.}

[sjmiller609]

• If we export a full disk per image, I think that still has the issue of duplicating storage for the case of many-VMs with mostly-the-same images or all shared base. So I think that means it doesn't have a significant benefit compared to the previous way where we just have the rootfs created for each VM. I think this would have to do something where we are sharing these blobs between different VMs without duplicating them somehow, in order to get the benefit
• doing that could potential have disk IO or other performance consequences, so I'm curious about doing benchmarking when we are swapping out the rootfs, maybe try this skill /benchmark-hypeman-fork but also try just plain fresh create, or in parallel from partially shared blobs maybe

Let me know if I'm missing something here

[chruffins]

• If we export a full disk per image, I think that still has the issue of duplicating storage for the case of many-VMs with mostly-the-same images or all shared base. So I think that means it doesn't have a significant benefit compared to the previous way where we just have the rootfs created for each VM. I think this would have to do something where we are sharing these blobs between different VMs without duplicating them somehow, in order to get the benefit

  • doing that could potential have disk IO or other performance consequences, so I'm curious about doing benchmarking when we are swapping out the rootfs, maybe try this skill /benchmark-hypeman-fork but also try just plain fresh create, or in parallel from partially shared blobs maybe

Let me know if I'm missing something here

so looking through #458 made me realize that PR doesn't add de-duplication so I need to rework that PR, but this PR is necessary for per-layer de-duplication that will be added in the follow-up PR. will benchmark resource utilization before re-requesting

update: benchmarking shows that this PR shouldn't be a regression for performance.

[sjmiller609]

We’re changing the production build path for the rootfs in this PR, but it seems like a subsequent PR would need to change it again, but we haven't worked out how that will happen yet so I think we shouldn't merge this in yet (because we would have to change how the rootfs works twice to get to the end goal of storage deduplication). I think that exporting separate disks containing overlapping layer data is the behavior we need to avoid to avoid duplication.

Looking forward into the next PR, it looks like the approach is solving for identical-prefix sharing, but not for general layer de-duplication.

#458 still merges all layers except the last into one exported base disk. your concern applies: that disk is shared only when the entire ordered base-layer prefix matches.

for example:

image A: [ubuntu, dependencies-A, app-A]
image B: [ubuntu, dependencies-B, app-B]
the exported bases are [ubuntu, dependencies-A] and [ubuntu, dependencies-B]. both contain their own copy of ubuntu, despite sharing that input layer.

In terms of how it should work, I think we would need to be extending the existing OverlayFS setup in guest init to mount multiple shared read-only layers. Here's what hypeship says:

the established pattern is immutable shared layers plus a private writable overlay—not a separately flattened copy of every image.

the straightforward design
convert each OCI layer once into a content-addressed, read-only EROFS artifact.
expose the required artifacts to each VM without copying them.
inside the guest, mount those layers and use OverlayFS to present their ordered contents as one root filesystem.
give each VM its own writable upper layer. image layers—including the final application layer—stay shared.
for your example, both VMs reference the same ubuntu artifact, even though their dependencies differ. whiteouts and overrides determine what each guest sees; they don’t require copying the base.

this is the architecture used by containerd’s EROFS snapshotter. adopting it in hypeman still requires validating device limits, guest support, snapshot/restore, and performance across hypervisors.

We need to measure disk I/O, fanout, cold boot, and physical storage before committing to the new direction.

[chruffins]

closing as superseded. the next implementation will target content-addressed artifacts for every OCI layer and compose them as multiple read-only OverlayFS lower layers, rather than introducing an intermediate flattened rootfs production path. benchmark coverage will be added before enabling the new runtime path.