# Production validation and rollout review

The fleet described for this review contains more than 250 machines, mixed RHEL releases, and p4d releases from 2023.1 through 2026.1. Roles include commit, edge, standby, broker and proxy. Storage is independent, and the live journal is `/p4/1/journals/journal`.

The operator identified the workspace `sdp.tgz` as the deployed package. Its SHA-256 is `1bd23217356ee3808e040d39aea5be9519bda7c8f2b4924656a70aff8ad7293a`. Preserve that archive unchanged for comparison and recovery. It includes unused Windows files; those are excluded from the comparison export and the candidate package. Archive identity does not establish that every machine has identical local scripts, configuration or binaries.

## Required evidence

The [container runner](../test/containers/README.md) records the exact candidate source, baseline source, image identity, binaries, interpreters, installer command and behavioral results. Installations use the shipped `mkdirs.sh` and an explicit generated `mkdirs.cfg`.

| Layer | Required scope | What it establishes |
| --- | --- | --- |
| Current OS matrix | Ubuntu 26.04 LTS; Red Hat UBI 8.10, 9.8 and 10.2 | Script behavior in these user spaces, with the real SDP-built CPython 3.14.8 and the declared distro interpreter. |
| Paired deployed/candidate tests | Journal compression, failed compression, disabled/active retention, Maintenance selection, mutation and recovery contracts | Exact equality for preserved behavior and explicitly asserted differences for approved fixes. |
| Live single-server checks | Exact submitted and shelved bytes; service lifecycle; checkpoint, journal and offline database operations; Maintenance mutations and failure recovery | Real command behavior and data preservation for the named cases. |
| Six-node topology | Commit, edge, both standby roles, broker and proxy, independent storage | Replication, routing, content/spec preservation, retained recovery chains, missing archive repair, failed restart, interruption and standby promotion. |
| Historical release matrix | 2023.1, 2023.2, 2024.1, 2024.2, 2025.1, 2025.2 and 2026.1 on UBI 8 and 9 | Fresh installation, exact fixture, restart, checkpoint/journal restore, Maintenance queries and real service-user password initialization/authentication on each recorded binary build. |

These are required cases, not a declaration that a run has passed. The generated acceptance report is authoritative for the tested source hashes. All required matrix cells and scenarios must pass; a blocked case is not a successful skip. Test names alone, command exit codes alone, old logs and a rebuilt image that was never run cannot establish acceptance.

The historical matrix uses available vendor binaries and records each exact patch build. It does not cover every patch build used by the fleet. Where older releases require x86_64 emulation on the ARM64 test host, the report identifies that execution mode. Emulated results establish functional behavior for the fixture, not native performance. Docker's [documented QEMU setup](https://docs.docker.com/build/building/multi-platform/) is confined to the dedicated test VM.

UBI containers exercise RHEL user space on the test VM's kernel. They do not validate each production RHEL kernel, SELinux policy, filesystem, mount option, systemd configuration or resource limit. The fixtures are deliberately small; they do not establish production-scale checkpoint duration, metadata capacity, archive throughput or load tolerance. Historical single-server checks do not imply that every role combination was tested with every release.

## Proposed rollout sequence

No production deployment is performed by the test suite. Review and approve the candidate diff and acceptance report before deploying it.

1. Record each machine's OS minor release, CPU architecture, exact p4d/broker/proxy build, server role, deployed script hashes, local overrides and service configuration. Group the fleet by these properties rather than selecting arbitrary machines. Record journal and storage paths without collecting passwords or ticket contents.
2. Rehearse on production-like nonproduction machines for each distinct OS/architecture/version group. Include a representative metadata and archive size, real scheduling, file ownership, security policy and restore procedure. An existing-install upgrade rehearsal is separate from the clean-install container suite.
3. Select a small canary group with a representative role chain. Save the existing scripts/configuration and record the healthy replication, journal, verification and service state before changing it. Avoid overlapping the rollout with maintenance or failover work.
4. Observe a complete relevant operational cycle: daily checkpoint and rotation, scheduled Maintenance, verification and replication catch-up. Check exit status, durable text/structured logs, checkpoint and journal continuity, content/shelf access, and unintended configuration or ownership changes. A green deployment command is insufficient.
5. Expand in batches only after reviewing the preceding batch's evidence. Keep enough unchanged machines to compare behavior, and stop expansion on missing diagnostics, failed backups, replication delays or data differences. Do not deploy across all 250-plus machines at once.

Define the batch membership, observation period, acceptable backlog/duration thresholds and responsible operator from the actual fleet inventory. They are site decisions; the container results do not supply those thresholds.

Include spec and unload depots in the pre-deployment archive check. Corrected typed depot discovery can reveal early spec archives that an older installer wrote before setting `server.depot.root`. Preserve any former root and resolve those findings using the matching archive content; a new verifier diagnostic is not automatically a new data loss.

## Recovery boundaries

Keep the prior package and local configuration separately from the proposed package. Script rollback is distinct from database recovery or a p4d binary downgrade. Do not start an older p4d against a database upgraded by a newer release. Retain and verify the full checkpoint/journal/archive set and the recovery procedure before changes that affect database state.

Recovery failures can preserve diagnostic directories and staged database state for investigation. The shared `die()` handler attempts to release only its own checkpoint marker on failure; `load_checkpoint.sh`'s separate failure handler and interruptions that bypass cleanup can leave a marker. A released marker does not establish that recovery completed or that a retry is safe. Inspect the logs and recovery state, and establish that a remaining marker's owner has stopped before manual cleanup; follow the [recovery reference](operational_scripts.md). Intentional `server.locks` removal in the scripts remains unchanged, as requested.

Live cloud creation/restoration, external authentication, mail delivery, shared storage and production failover remain outside the local container evidence. Their documentation and isolated tests are useful, but are not substitutes for rehearsing those integrations where used.