# Linux container test matrix

The matrix uses Ubuntu 26.04 LTS and Red Hat UBI 8.10, 9.8 and 10.2 init images, running native architecture containers with systemd as PID 1. The [UBI 8](https://catalog.redhat.com/en/software/containers/ubi8/ubi-init/5c359b97d70cc534b3a378c8) and [UBI 9](https://catalog.redhat.com/en/software/containers/ubi9/ubi-init/615bdc22075b022acc111bf6) images are published by Red Hat. UBI provides RHEL userspace through public repositories; it is not an entitled full RHEL installation, and containers share the Docker VM's kernel rather than testing each distribution's kernel or SELinux host policy. The runner records `/etc/os-release`, image metadata, architecture, Python versions, Perforce binary versions, download URLs/checksums and the source manifests for each run.

The containers run privileged to support systemd, so use a dedicated disposable Docker VM. They use private cgroup namespaces and have no host directory mounts, Docker socket, host SSH/cloud credentials, published ports or host networking. `/p4` and all test data exist only inside the containers. The images mask `systemd-binfmt.service` so container boot does not replace the VM's shared binary-format/emulation handlers. The server listens only on container loopback. `cleanup` removes only containers carrying this tooling's disposable label. It does not delete images, the Docker VM, or saved results.

## Export and run

Use the current maintained Perforce files rather than copying the entire workspace: a synced workspace can still contain stale files deleted from the depot.

```bash
python3 sdp/test/containers/export_source.py "$PWD/sdp" /tmp/sdp-clean-export
python3 sdp/test/containers/export_archive.py /path/to/deployed/sdp.tgz \
  /tmp/sdp-deployed-baseline --operator-confirmed
python3 sdp/test/containers/run_matrix.py all \
  --source /tmp/sdp-clean-export --baseline-source /tmp/sdp-deployed-baseline \
  --results /tmp/sdp-container-results
```

The exporter excludes depot deletions and pending deletions, includes pending additions, copies current local bytes (including unsubmitted edits), and writes `.export-manifest.json` with SHA-256 values. Missing maintained files or inventory errors stop the export. The destination must be new. Supply the clean export as `--source`; neither the runner nor Docker receives the rest of the workspace.

`--baseline-source` is a separate immutable pre-change export containing `.export-manifest.json` and `.baseline-info.json`. Prefer an operator-confirmed deployed archive exported through `export_archive.py`; retain that archive unchanged and record its SHA-256. A submitted Perforce export through `export_baseline.py` is also usable, but its relationship to the deployed fleet must be stated explicitly. The baseline is copied into `/inputs/baseline-sdp`, with both manifests and provenance retained in the results. The nineteen original deployed-archive regressions remain a historical reference; do not silently replace that baseline with a newer archive whose corrected behavior differs. Preserve a newer last-submitted archive separately, with its revision/hash and deployment status stated explicitly, for current-to-candidate comparisons. Comparisons exercise the explicitly listed Unix/Maintenance contracts against each version; they do not perform an existing-installation upgrade. Retired Windows files are excluded from the deployed-archive comparison, and unused retired triggers are not exercised.

The runner honors the usual `DOCKER_HOST` and `DOCKER_CONFIG` variables and leaves the selected Docker context unchanged. `--docker` can select an explicit CLI path. Names default to `sdp-assurance-matrix-ubuntu`, `sdp-assurance-matrix-rhel`, `sdp-assurance-matrix-rhel8` and `sdp-assurance-matrix-rhel9`; use `--name-prefix` for a separate run. Existing containers without the disposable label are rejected. No existing containers are automatically removed. `build` records the image ID and Dockerfile hash; `start` uses that exact image ID and rejects an existing container from another image. Keep previous evidence and containers under their previous names instead of repairing dependencies inside a running test container.

Individual stages are `build`, `start`, `bootstrap`, `test`, `collect`, and `cleanup`. `--platform all` is the default; choose `ubuntu`, `rhel` (UBI 10), `rhel8` or `rhel9` for one target. The compatibility value `both` selects Ubuntu and UBI 10 only. `bootstrap` is for a newly started container; it intentionally refuses to initialize an existing deployment. To recreate a previously marked deployment in the same isolated container, explicitly use `bootstrap --resume`; the reset guard still requires the exact path, instance, account and port marker. This removes and recreates the fixture data; it is not an upgrade test. An existing Python build can be reused only if the installer hash still matches; use a fresh container after changing it. Collect results before resetting.

`bootstrap.sh` downloads `p4`, `p4d`, `p4broker` and `p4p` from the selected Perforce release (`r26.1` by default), verifies them against that release/platform's HTTPS `SHA256SUMS`, and records their exact versions. The reusable `download_binaries.sh RELEASE BINARY_DIR RESULTS_DIR` helper also prepares binaries for isolated topology fixtures. Bootstrap derives and records `mkdirs.cfg` from the repository template, invokes `mkdirs.sh` with that exact configuration, establishes the fresh administrator password and records the configuration hash and provisioning output. The live journal is `/p4/1/journals/journal`; the results include both its sourced configuration and the running p4d argument list. Bootstrap installs `p4d_1.service` and verifies a nonzero MainPID.

The Python installer runs as `perforce`, building CPython 3.14.8 and P4Python 2026.1 under `/opt/sdp-python/baseline`. Trigger dependencies are installed and imported there. `/p4/common/python` selects this generated runtime for the primary suite. A separate `/opt/sdp-python/distro` environment uses the packaged interpreter; UBI 8 and 9 explicitly install their available Python 3.12 packages. Every declared distro variant must pass too. Runtime switching is sequential and restricted to these generated paths, and the runner restores the baseline selection afterward. Build logs, source hash, OS account, versions and import results are retained. The RHEL images change `/etc/shadow` from mode `0000` to root-readable `0400` so the Docker host's AppArmor policy permits the container PAM helper to check account state; non-root access remains denied and PAM is not bypassed.

UBI does not provide the optional `readline-devel` package through its public package set; it is not needed for these virtual-environment tests.

## Results and scope

`test --test-stage offline` selects a single stage; repeat `--test-stage` for several. Other recorded outcomes are preserved, including failures. Without a selection, `test` runs every stage.

Offline fixtures use `TMPDIR=/var/tmp`, because they execute temporary mock commands and the container's `/tmp` mount is not executable.

The primary stages are `offline`, `integration`, `runtime`, `maintenance`, `baseline-comparison` and `python-installer`. The last verifies the completed real build and its exact source hash; compilation occurs during bootstrap with `MAKEFLAGS=-j3`, using `/opt/sdp-python/build` on an executable filesystem. The first five also run as `distro-offline`, `distro-integration`, `distro-runtime`, `distro-maintenance` and `distro-baseline-comparison`. Baseline paths are supplied to offline discovery so comparison tests cannot silently skip in the matrix. `collect` saves container metadata, provisioning logs, source/binary manifests, service state and journal; host stage logs and outcomes remain authoritative over any older copies inside the container. Rebootstrapping archives previous host stage output and invalidates old container stage evidence. Containers stay available after failure. Build, provisioning, offline tests and live integration are distinct outcomes; a passing mock or parser check is not a successful live behavior test.

The runtime stage has five required scenarios: checkpoint/restore, journal rotation, send-queue cleanup, database switching and SSH host trust. Send-queue checks use actual local configuration records across global/named scopes, empty/unset values, relative/absolute paths, configured root symlinks and P4NAME/startup identity selection. The twenty-four subcases stop the actual service before calling the helper and verify that a running service is refused. A required alias inside the queue for the configured database root or queue path must cause refusal with every queue entry preserved. Dedicated queues lose regular files and symlinks while directories, FIFOs and symlink targets remain. Overlapping readonly directories, and readonly paths that depend on symlinks inside the queue even when the final storage lies elsewhere, retain every file and required alias, including actual partitioned-client have tables with queue-like filenames; have revisions are checked again after restart. The database-switch case separately seeds both physical roots, runs `recreate_db_checkpoint.sh`, and checks exact partitioned-client have state through a relative readonly path containing spaces and `=`. The gate requires these subcase assertions; a passing scenario label alone is insufficient. Live database switching uses the server-ID scope; additional P4NAME readonly-copy combinations use isolated tests.

The existing single-server harness covers environment, authentication, role guards, lifecycle, a submitted fixture, checkpoints and selected maintenance/diagnostics. It explicitly skips topology-specific behavior. The Linux offline suite excludes the retired Windows scripts and their tests. Per-script coverage and additional real-server scenarios are reported separately by the coverage tooling; do not infer complete coverage from a successful aggregate exit code.

An ordinary run uses one export for installation and validation. If only documentation or tests change afterward, preserve the installation manifest and the original logs before updating the validation source. Reuse prior live results only after comparing the manifests and proving that every production script and the corresponding live harness, fixture, installer and container definition is unchanged. Rerun changed tests, record their source hashes and retain a per-stage provenance record identifying which results were rerun or reused. Do not relabel an earlier installation as having run a newer export. This reconciliation is an explicit review step; the runner does not infer that evidence can be reused from a successful exit code or matching file count.

## Topology and historical releases

Run the six-node fixture after the main Ubuntu container has been provisioned. Copy its vendor-checked Linux binaries to a new local directory; the topology runner verifies them again before copying them into its isolated nodes. Use the drivers from the same source export as the installation. The acceptance gate compares both driver hashes and every node's source and image identity.

```bash
docker cp sdp-assurance-matrix-ubuntu:/inputs/binaries /tmp/sdp-topology-binaries
python3 /tmp/sdp-clean-export/test/containers/topology_integration.py \
  --source /tmp/sdp-clean-export --binaries /tmp/sdp-topology-binaries \
  --results /tmp/sdp-topology-results --prefix sdp-assurance-topology-review \
  --image sdp-assurance-ubuntu:26.04 --docker docker --confirm-disposable
```

Use a fresh prefix and result directory for each acceptance run. The private internal Docker network connects a commit, edge, commit standby, edge standby, broker and proxy. Each has separate storage. The fixture checks exact content, specifications, shelves and workspace state after recovery, restores every retained checkpoint generation through its journal chain, repairs a missing archive across separate hosts, injects a failed restart and process interruption, and validates promotion after fencing the old commit. Rebuilding the edge also explicitly reseeds its downstream standby from the recovered edge's fresh checkpoint; it does not assert uninterrupted replication across that history change.

Repeat the topology run with `--image sdp-assurance-rhel8:8.10` and
`--image sdp-assurance-rhel9:9.8`, each with a distinct prefix and results
directory, to exercise those production user spaces too. Supply each complete
report to the gate by repeating `--topology-results`; every supplied run must
pass and match the same source export. Keep the checked binaries on the same
CPU architecture as the selected images.

Results and stopped/fenced services remain available for inspection. `--resume` is a development aid after investigating a failure; use a new complete run for acceptance. To remove only this fixture's labeled containers and network after reviewing the saved results, repeat the topology command with `--cleanup`. It does not remove saved result files.

Follow [historical release compatibility](VERSION_COMPATIBILITY.md) to build the UBI 8/9 native and x86 images and run all fourteen release/platform cells. Use the same source export. Required emulation is confined to the disposable test VM and explicitly recorded. A passing single-node current-release test does not replace these historical checks.

## Acceptance

```bash
python3 sdp/test/containers/run_matrix.py collect \
  --source /tmp/sdp-clean-export --results /tmp/sdp-container-results
python3 sdp/test/containers/release_gate.py \
  --source /tmp/sdp-clean-export --baseline /tmp/sdp-deployed-baseline \
  --results /tmp/sdp-container-results --topology-results /tmp/sdp-topology-results/topology-results.json \
  --version-results /tmp/sdp-versions/version-compatibility.json \
  --output /tmp/sdp-release-gate.json
python3 sdp/test/containers/run_matrix.py cleanup --results /tmp/sdp-container-results
```

The topology result is produced by the separate marked commit/replica/edge/broker/proxy fixture. The version result comes from the separate historical compatibility runner: seven release lines from 2023.1 through 2026.1 on UBI 8 and 9. If older vendor binaries require x86_64 emulation on an ARM host, that architecture and emulation must be recorded rather than described as native testing. The release gate requires both results as well as both Python variants, image identity, source manifests, the configured live journal path and named behavioral assertions. It fails on missing evidence, failures or unapproved skips; merely generating a coverage report does not pass the gate. A passing gate establishes the listed fixture contracts, not exhaustive branch coverage, every patch build, or every host's configuration in the deployed fleet.

`test_container_tools.py` tests inventory filtering, current-byte export, checksums, failure reporting, container ownership guards and the absence of host mounts/published ports using fake processes. It never starts containers or contacts Perforce.