Database Packaging

[isc-dchui]

Description

Resolves #986

Overview

Packaging workflow

Running package-database <module> (or publish-database <module>) forces the %IPM.Lifecycle.Database lifecycle class and routes to the Package (or Publish) phase. The full pipeline inside %Package:

1. Validate IPM is externally mapped — %IPM.Main.CLS must not live in the namespace's local routines DB, since that DB is about to be dismounted. Also validates the routines DB is not mirrored (dismounting a mirrored DB breaks the mirror).
2. Export Generated=true resources from the source namespace to temp XML files before remapping. These are classes whose compiled form exists in the source DB but whose generator does not run during standard %Reload/%Compile — for example, a class produced by an external tool. Once the old DB is dismounted, compiled generated classes are inaccessible. Resources that do regenerate during packaging (e.g. BPL/DTL produce a .1.cls subclass during compilation) are exported and re-imported harmlessly — the import is a no-op since the class already exists in the temp DB by then.
3. Create a fresh empty temp database and remap the namespace's routines to it (or use the existing routines DB as-is with -use-current-db). The remap happens before dismounting the old DB to avoid any window where the namespace has no mounted routines DB.
4. Load and compile this module's resources plus all transitive dependencies into the temp DB, so the resulting IRIS.DAT is self-contained.
5. Restore the namespace to its original routines DB (dismount temp, remap back, remount original), then import the previously exported generated resources.
6. Assemble a staging directory: IRIS.DAT, an enriched module.xml (with <Packaging>database</Packaging> and a SHA-256 <Checksum>), a deps/ directory of per-dependency IRIS export XML files, and any non-compiled resources (wheels, FileCopy, CPF). Create the .tgz from the staging directory.

If an error occurs after the remap (step 3), a safety net in %Package restores the namespace before propagating the error.

For OCI registries, database packages are tagged as <version>_database__<IRIS-major>.<IRIS-minor> and the manifest carries a com.intersystems.ipm.packaging=database annotation. Source packages use existing tag formats with no annotation (backwards compatible — absent annotation means source).

When publishing, Base.%Publish calls ..Package(.pParams, 0) first (populating ..Payload), then uploads to the registry. The lifecycle class's PACKAGING parameter determines the tag format, not the module's stored Packaging property (which reflects how the module was loaded and may be stale).

Installation workflow

From a registry (install/update): the -swap-db modifier controls which packaging type is resolved:

• Without -swap-db: only source-packaged modules are fetched. If none exist but a database package does, the user is told to re-run with -swap-db.
• With -swap-db: only database-packaged modules are fetched. Dependencies are always resolved as source packages regardless of the top-level flag, since a database package already embeds its dependencies in the IRIS.DAT.

From a local path (load -path <tgz>): no packaging filter is applied. load passes all command data (including SwapDB if -swap-db was given) through to the lifecycle. Without -swap-db, DoDatabaseInstall prompts the user for confirmation before proceeding.

IsInstallContext() returns true when module.Packaging = "database" and IRIS.DAT is present in the module root. When true, %Reload routes to DoDatabaseInstall, which:

1. Validates the package: IRIS.DAT present, module.xml declares database packaging, deps/ directory exists, SHA-256 checksum matches.
2. Checks that no other database-packaged module is already installed in the namespace (only one database package per namespace is allowed).
3. Checks that no already-installed module has code in the local routines DB that would be destroyed by the swap. Modules packaged inside this database package, modules whose code is mapped from another database, and modules with no routines-DB code are exempt. Modules the previous version carried but the incoming version drops are also exempt — their code leaves with the old DAT — and are collected for cleanup in later steps. Any other blocking module must be uninstalled first.
4. Validates IPM is externally mapped and the routines DB is not mirrored.
5. Prompts for confirmation if -swap-db was not passed. The prompt lists any modules that will be removed by the swap.
6. Unconfigures dropped modules (web applications, copied files, installer class side effects) while their code is still mounted.
7. Performs the swap: dismount existing routines DB → rename current IRIS.DAT to IRIS_<dbname>_<timestamp>.DAT as a backup → rename packaged IRIS.DAT into place → remount. Renames are atomic and avoid copying large files. On failure after the swap starts, RollbackDatabaseSwap attempts to restore the backup (best-effort, never throws — called from a catch block).
8. Registers dependency metadata from the deps/ manifests via $system.OBJ.Load. The code is already in the swapped-in DB; only IPM metadata records need registering.
9. Removes IPM records for dropped modules (now that the swap has succeeded and their code is gone).
10. Seeds all update steps as already-run on a fresh install, so a subsequent update only runs steps introduced after the installed version. Skipped when params("Update")=1.

Because all install work completes inside %Reload, %Validate, %Compile, and %Activate are no-ops during install (they check params("IsInstallContext")). Invokes fire naturally after %Reload returns — the DB is mounted by then.

Non-compiled resources (FileCopy, WebApplication, CPF) are handled by resource processor hooks at the Module level. They run via OnBeforePhase/OnAfterPhase independently of the %Activate override.

Notable details

• package-source command added. Explicit alias for the existing package behavior. Useful for clarity when both packaging types are in play.
• Lifecycle auto-derived from <Packaging>. Storage.Module calls Base.GetBaseClassForPackaging to derive the correct lifecycle class from module.xml's <Packaging> value rather than rejecting a mismatch.
• Python deps default on for package-database. Unlike package (which defaults off), package-database defaults ExportPythonDependencies=1. Both requirements.txt-based wheels and explicit <PythonWheel> resources are staged into the .tgz. A missing wheels directory only warns if a requirements.txt exists — a module with no Python requirements legitimately has no wheels directory.
• XSLT checksum injection. InjectDatabasePackagingTransform (XData) injects <Packaging> and <Checksum> into module.xml. The checksum placeholder REPLACECHECKSUM is substituted in ObjectScript before the stylesheet runs — XSLT 1.0 has no parameterized element content.
• deps/ is a directory, not a single file. Each dependency gets its own IRIS export XML, one per transitive dependency, loadable with $system.OBJ.Load.

Testing

All tests are in Test.PM.Integration.DatabasePackaging. The test infrastructure uses two shared lazy-initialized namespaces to avoid ~15 namespace create/teardown cycles per run:

• SharedNS1 (TESTDBPKGNS1): simple-db-module, dep-module, main-with-deps, module-with-tests, module-with-invokes; also has the zot ORAS registry configured
• SharedNS2 (TESTDBPKGNS2): all-resources-module, module-with-requirements, module-with-mixed-python, module-with-non-compiled-resources

Per-test install namespaces are created fresh and torn down in OnAfterOneTest.

Test	What it covers
TestSimplePackageAndInstall	Package → verify IRIS.DAT + module.xml content (Packaging, checksum, SystemRequirements version) → install with load -swap-db → classes callable, module in list and history
TestPackageAndInstallWithDependencies	Package dep-module + main-with-deps; install main-with-deps → main class exists, dep appears in list
TestPackageAndInstallAllResourceTypes	Smoke test: module with class, include file, generated class, WebApp, FileCopy, PythonWheel packages and installs; include constant accessible, generated class callable, WebApp created
TestPackageAndInstallWithTestResources	-include-test-resources includes Scope=test (TestHelper) and UnitTest (Test) classes; default packaging excludes both
TestPackagingAndInstallValidation	Valid install succeeds; tampered IRIS.DAT (checksum mismatch) and missing deps/ directory each fail validation
TestUpgradeSourceToDatabase	Source-installed module upgraded via update -path -swap-db; Packaging changes to "database"; backup confirms DB swap occurred
TestUninstallDatabasePackage	Module removed from list; backup preserved; compiled classes gone from namespace
TestMixedPackaging	Database + source package coexist in same namespace; second database package in same namespace is rejected
TestInstallRejectedWhenModulesWouldBeOrphaned	Source module with classes in the routines DB blocks install; uninstalling it clears the way; modules with no routines-DB code and packaged dependencies are exempt
TestUpdateWithChangedDependencyTree	v1→v2 upgrade with a reshaped dependency tree: dropped dep's metadata cleaned up, new dep registered, re-parented and new transitive deps handled correctly; upgrade not blocked by orphan check
TestPythonDependencyPackaging	module-with-mixed-python: explicit PythonWheel (lune) + requirements.txt wheel (pycparser) both included by default, excluded with -export-python-deps 0; lune importable after install. module-with-requirements: wheel included by default, excluded with flag
TestInvokeBehaviorDuringDatabaseInstall	<Invoke After="Compile"> skipped during database install; <Invoke After="Activate"> runs; global markers confirm each
TestNonCompiledResourcesAppliedDuringInstall	CPF file staged in package; FileCopy target deleted before install, confirmed recreated after install — proves resource processor hooks fire independently of %Activate override
TestUpdateStepsWithDatabasePackaging	v1 fresh install seeds Step001 (not run); v2 upgrade runs Step002 once; Step001 still not run; backup count > 1 confirms second swap
TestPublishDatabaseWithORAS	publish-database + publish to zot; source install (no -swap-db) → Packaging=module; database install (-swap-db) → Packaging=database; main-with-deps install resolves dep as source; after unpublishing source tag, install without -swap-db fails with hint to use -swap-db
TestSystemRequirementsOverwrite	package-database overwrites a stale IRIS version in <SystemRequirements> with the current version
TestPackagingDerivesLifecycleClass	Module with Packaging=database and no explicit LifecycleClass derives %IPM.Lifecycle.Database on validation
TestRollbackPreservesOriginalWhenNoBackup	Rollback invoked when no backup exists leaves original IRIS.DAT untouched
TestChecksumMatchesShippedDat	Checksum embedded in module.xml equals SHA-256 of the IRIS.DAT that ships in the .tgz (exercises -use-current-db copy path)
TestSHA256HexFormat	ComputeSHA256Hex produces a 64-character lowercase hex string

Not Yet Handled

• Mirroring (not supported on either side — both packaging and install reject mirrored DBs)

Dependency Lifecycle Support (Commit a8733ae and onwards)

Problem

When installing a database-packaged module, dependency code is compiled into the IRIS.DAT at packaging time and arrives already mounted after the swap. But the dependency modules' lifecycle phases (Activate, CPF merges, etc.) never ran, so dependencies with resources like CPF entries, Python wheels, or data files had their code present but none of their install-time setup applied.

Dependency Tree (example)

main-module          ← root (the package being installed)
├── parent-dep
│   ├── leaf-dep-A
│   └── leaf-dep-B

Topo (leaf-first) order: leaf-dep-A → leaf-dep-B → parent-dep → main-module.

Phase Execution: Root vs Dependencies

Each column is a module; rows are execution time (top = earlier). Dependency phases run in topological (leaf-first) order.

         │   Root               │   leaf-dep-A         │   leaf-dep-B         │   parent-dep
─────────┼──────────────────────┼──────────────────────┼──────────────────────┼──────────────────────
PRE-SWAP │   Initialize [1]     │                      │                      │
─────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────
                                    PerformDatabaseSwap (Reload)
─────────┬──────────────────────┬──────────────────────┬──────────────────────┬──────────────────────
POST-    │                      │   Initialize         │                      │
SWAP     │                      │   Validate           │                      │
         │                      │   Activate           │                      │
         │                      │   ApplyUpdateSteps[2]│                      │
         │                      │                      │   Initialize         │
         │                      │                      │   Validate           │
         │                      │                      │   Activate           │
         │                      │                      │   ApplyUpdateSteps[2]│
         │                      │                      │                      │   Initialize
         │                      │                      │                      │   Validate
         │                      │                      │                      │   Activate
         │                      │                      │                      │   ApplyUpdateSteps[2]
         │   ReassertInit [1']  │                      │                      │
         │   Initialize         │                      │                      │
         │   Validate           │                      │                      │
         │   Activate           │                      │                      │
         │   ApplyUpdateSteps[2]│                      │                      │

[1] — Root's Initialize fires pre-swap against the old DB. Dep Initialize runs post-swap because their code is already in the mounted IRIS.DAT.

[1'] — ReassertInit re-runs root's Initialize-phase resources (CPF merges, Python wheels) after all deps so root's versions land last. Deps' Initialize runs post-swap and writes into the same global state root already set pre-swap; re-running root's restores the intended ordering.

[2] — ApplyUpdateSteps runs per-module only on updates (zpm "update"). On a fresh install, Activate seeds all update steps as already-run so a future update only executes steps added after this version.

Dep phases are derived from the canonical load/install chain ending at Activate (or ApplyUpdateSteps on updates), with Reload, Compile, and * excluded because dep code is already compiled into IRIS.DAT.

Rollback Timing

All steps below execute inside %Reload via DoDatabaseInstall. The swap is the point of no return: after it commits, rollback is not attempted and failures are reported forward.

Step                          On failure
──────────────────────────────────────────────────────────────────────────────
ValidateBeforeSwap            abort, nothing changed

UnconfigureStaleModules       abort + warn: web apps, copied files, and
                              installer-class effects for stale modules are
                              already removed; IRIS.DAT not yet touched
                                    │
                                    ▼
                        ┌─── PerformDatabaseSwap ───────────────────────────┐
  failure inside swap ──┤   RollbackDatabaseSwap restores old IRIS.DAT      │
                        │   and re-registers old DB with IRIS                │
                        │   NOTE: UnconfigureStaleModules effects are NOT    │
                        │   reversed — those modules are now partially       │
                        │   installed; reinstall them to recover             │
                        └───────────────────────────────────────────────────┘
                                    │
                              swap committed
                         ══════════════════════ POINT OF NO RETURN
                                    │
RegisterDependencyMetadata    report error, stop — dep metadata may be
                              partially written; namespace has new DB mounted

ApplyDependencyResources      report error, stop — some deps may be fully
                              configured, others not

ReassertRootInitializeResources  report error, stop — root Initialize
                                 resources partially applied

RemoveStaleModuleMetadata     report error, stop — stale module records
                              may still exist in 'list' output

Why the boundary sits where it does:

• UnconfigureStaleModules side effects (web app deletions, file removals) have no clean undo, but IRIS.DAT is untouched so a re-run from scratch is still possible.
• PerformDatabaseSwap is a filesystem rename, so RollbackDatabaseSwap can reverse it. This is the last step with a clean undo path.
• Post-swap steps are not rolled back because rolling back would discard a working swap and pair old code with new configuration. Post-swap failures are typically fixable by re-running the install against the already-mounted code.
• Subtlety: module metadata is in globals inside the transaction, so a post-swap failure rolls it back automatically while the IRIS.DAT rename stays. list will then show the previous version's metadata against the newly mounted code. Re-run the install to fix this; do not uninstall first, as it would delete resources by the old version's names.

Other Notable Changes

Generated resources extended to deps. ExportGeneratedResources now iterates all transitive dependencies via GetSelfAndDependencyModules(), not just the root module. Dep generated resources become inaccessible after the namespace routines DB is remapped, so they must be exported before the remap alongside root's.

Ensemble opt-in for test namespaces. %IPM.Test.Utils.CreateNamespace previously ensemble-enabled every test namespace unconditionally, adding setup cost even when tests don't need it. The method now accepts pEnableEnsemble (default 0). When set, it passes pFromInstall=1 to suppress HS-specific global mappings that would otherwise point ^IRIS.Msg at HSLIB (read-only on IRIS for Health), then explicitly maps ^IRIS.Msg/^IRIS.MsgNames to the namespace's own database. DeleteNamespace was also fixed to skip ensemble secondary and temp databases when the namespace was never ensemble-enabled.

Warnings at packaging time. Two methods fire during %Package to surface problems early:

• WarnUnrunnableInvokes — flags <Invoke> elements on phases that never execute during a database install (root: Initialize and Reload.Before; deps: the excluded phases Reload, Compile, *).
• WarnMappingsNotApplied — flags <Mapping> elements in any module. Mappings are applied by Base.%Reload, which this path bypasses. Applying them inline would deadlock against a concurrent <CPF> merge because Config.Map* locks are held until the install transaction commits, and iris merge cannot wait that out.

Packaged-manifest guard. %Reload now rejects a source load from a package directory whose IRIS.DAT was already moved by a prior swap, rather than silently compiling source into the swapped-in database.

New tests. Integration tests added: TestDependencyResourcesAppliedDuringInstall (CPF, wheel, and data file resources on a dep are applied post-swap), TestNonCompiledResourcesAppliedDuringInstall (FileCopy and web app resources), TestInvokeBehaviorDuringDatabaseInstall (invokes fire at the right phases), and TestUpdateStepsWithDatabasePackaging (update steps run per-module on upgrade). New test fixtures dep-with-resources and main-with-resource-deps support these. Unit tests in Test.PM.Unit.LifecycleDatabase cover topological ordering, phase list correctness (excluded phases, canonical chain coverage), packaged-manifest detection, and SHA-256 format.

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.

[isc-jili]

Looks great @isc-dchui ! I left some comments!

[isc-eneil]

@isc-dchui I remember that adding package-** commands was part of your original design, but I can't remember why we went with that instead of flags on the existing package command so that it continues to be considered a module-action rather than separate commands. I'm not sure I have a strong preference either way, design-wise, but I'm noticing that the flow through the Shell is different.

[isc-dchui]

It's because of the lifecycle phases. A separate command allows it to dispatch to the %IPM.Lifecycle.Database class directly

[isc-kiyer]

@isc-dchui few small notes. I love the description of the MR! Could you add some more information for each of the "Key Details"? In particular, for the following:

• Can you explain why generated resources need special treatment? When packaging, wouldn't they be in the routine db already?
• What is the reason for exporting the module manifests and reimporting them? Could that not be accomplished by setting up a global mapping/copying over the IPM globals from the globals db to the routine db during packaging? Then its one less step at deployment time.

[isc-kiyer]

Why is this removed?

[isc-dchui]

That's just dead code. If it's generated, it gets marked as handled at the bottom of the method anyway.

[isc-kiyer]

Should indicate that this module should not be a dependent of any other module (something we should check and throw an error for). We should also probably check for DB packaging that every module other than this one has at least 1 dependent because if not, you are packaging things that aren't intended.

[isc-dchui]

Turns out this we don't need this constraint since database-packaging will iterate through the specified module's resources (and dependencies), so any module that depends on this one will simply be ignored.

[isc-kiyer]

Wouldn't it be better to have IPM metadata be put into routine db? Or I guess we achieve this by exporting the module manifests and loading them into the ns after mounting the IRIS.DAT on install from db packaged?

[isc-dchui]

Right, the export/import is the mechanism and it avoids some trickiness around namespace/database mappings and possibly conflicting IPM data

[isc-kiyer]

Nit: instead of needing a ns switch, we can use ##class(%SYS.Namespace).GetAllNSInfo()

[isc-dchui]

Right, but we only need it for one namespace so seemed unnecessary to get it for each

[isc-jlechtne]

What is the metadata value if the tag doesn't exist in the tag list? So that it's not equal to ""

[isc-dchui]

GetAllTags() won't return it so that tag won't be part of the while loop.

  set allTagsString = ..GetAllTags(..Location, name, "", client)
  set allVersionsList = ..AggregatePlatformVersions($listfromstring(allTagsString, ", "), .aggregatedPlatformVersion)
  set pointer = 0
  while $listnext(allVersionsList,pointer,moduleVersion) {
     ...
  }

[isc-dchui]

@isc-dchui few small notes. I love the description of the MR! Could you add some more information for each of the "Key Details"? In particular, for the following:

• Can you explain why generated resources need special treatment? When packaging, wouldn't they be in the routine db already?
• What is the reason for exporting the module manifests and reimporting them? Could that not be accomplished by setting up a global mapping/copying over the IPM globals from the globals db to the routine db during packaging? Then its one less step at deployment time.

@isc-kiyer Updated the description with a little more organization clarity! To explicitly answer these,

1. Standard generated resources like BPL and DTL are fine without special handling, but anything that is special and generated outside of the standard lifecycle phases won't be picked up. So this is just redundancy for that. Maybe it's a case of YAGNI though and overkill.
2. Yeah, but exporting/re-importing turns out to be simpler than setting up and managing the global mappings.

[isc-jili]

Looks good! @isc-dchui One more Q: are all the resources as part of transitive dependencies supposed to be packaged up too? Since ExportGeneratedResources() CopyNonCompiledResources() and ExportPythonDependencies() seem to be invoked on the root module only
A test case along those lines that would be nice also would be having a dependency contain a Generated="1" resource or a requirements.txt

[isc-cborbonm]

would adding it as param #6 mean that uses of this method would break?

[isc-cborbonm]

because currently param 6 is pParams and param 7 is pDedendencyGraph

[isc-kiyer]

Yes should add to the end of the method for backwards compatibility

[isc-cborbonm]

it does look like all calls to this method are accounted for however, so nothing within IPM would break

[isc-cborbonm]

and as for our uses of IPM, we never call this method directly, there are other methods that wrap around this one that tend to be more user-entry-point friendly

[isc-dchui]

Right, I initially had it at the end but @isc-jlechtne suggested I move it so I did and made sure all its callers were also updated. It seems pretty unlikely that there would be an external caller of this method directly so it seemed safe enough, but also happy to revert it.

[isc-kiyer]

Hmm taking a step back, if we are making a backwards incompatible change here, I propose just passing in the full module reference (as the method name suggests) instead of just passing specific properties. That also ensures any new stuff added to the module ref in future doesn't require changing the signature here

[isc-dchui]

Changed LoadModuleReference to take a %IPM.Storage.QualifiedModuleInfo oref as the first argument, replacing the various properties

[isc-dchui]

Fixed the issue where, on install, dependencies wouldn't run through their lifecycle phases. Details are in a new section of the description.

[isc-jlechtne]

Nit: This seems like it belongs as another another elseif clause above

[isc-jlechtne]

Nit (Inconsistent): xDir and xName have opposite order in when they were set in this clause vs the next CPF one.

[isc-jlechtne]

Yikes 🫣

[isc-jlechtne]

For my understanding, why is this section needed for DB packaging?

[isc-jlechtne]

Should the new command package-source (and maybe package-studio-project as well, though I see that's marked deprecated already) be listed here as well? Or did you decide that wasn't necessary due to their equivalence to the package command?

[isc-jlechtne]

If we're checking for an error substring here, then the call before of checking the status not okay is redundant.

[isc-jlechtne]

same as above, no longer need this line

[isc-jlechtne]

same as above, don't need to check that there was an error if we're also making a check for the error text

[isc-jlechtne]

same as above, no need to check that there was an error if we're also making a check for the error text

[isc-jlechtne]

Don't need to set a status here since it's never used

[isc-jlechtne]

Looking great! Left some comments