# Session Log -- 2026-09-09 ## Summary Day after SDP 2026.1 GA shipped. Started with a routine investigation of a customer's `refresh_P4ROOT_from_offline_db.sh` log (SDP 2025.1/31674) that surfaced one real, still-live bug (SDP-1393, `verify_sdp.sh`'s spurious `grep: off: No such file or directory` with `-L off`) and confirmed a second suspected bug was already fixed. Also created a new `sdp-jira-jobs` skill capturing conventions accumulated during the release cycle, and retroactively applied its new Summary/Description convention to SDP-1390 through SDP-1393. The day's real event: the user found a genuinely critical, still-shipping bug -- `install_sdp.sh` and several related scripts hardcoded the *planned 2026.2* OS package directory name (`/opt/perforce/p4-sdp`) instead of the current, correct 2026.1 name (`/opt/perforce/helix-sdp`). What started as reviewing the user's own quick patch (change 33608) turned into a full sweep finding the bug was far more widespread than initially fixed, a second, independently-discovered upgrade bug (SDP-1395, a false "FAILED" status on successful upgrades), two full Emergency Bug Fix cycles (fix -> verify via a real install-then-upgrade test -> Release Notes -> repackage -> publish -> verify live), a full Merge Down chain (`r26.1.0` -> `main` -> `dev`) to get the fixes back into ongoing development, a post-merge doc-regen sweep across all three streams (catching a third, smaller stale-doc bug along the way), and a real upgrade-test run on the new `install_sdp.sh` test fleet to close the loop on a test-coverage gap this whole incident exposed. Also fixed a stale "SDP and P4D Version Compatibility" section in the Release Notes (should have been updated at GA, wasn't) while in the area. The new automated coverage immediately paid for itself: a full run of the Helix Install Test Suite across the whole EC2 fleet turned up a **fourth** instance of the same bug class (SDP-1396, `opt_perforce_sdp_backup.sh` hardcoding the same premature path independently of the already-fixed `SDPPackageBase` variable), reproducing identically on every platform except Rocky 8. Fixed and verified live in `dev`/`main`; not yet in the shipped `r26.1.0` tarball -- whether that warrants a third same-day Emergency Bug Fix is an open decision for the user. ## SDP-1393: verify_sdp.sh's spurious "grep: off" warning User shared a real customer's `refresh_P4ROOT_from_offline_db.sh` log (SDP 2025.1/31674, 2026-08-24) with two things that looked like bugs: `grep: off: No such file or directory`, and `mv: replace ... overriding mode 0440` prompts during the root/offline_db symlink swap. **Bug 1, confirmed still live**: `verify_sdp.sh`'s two end-of-run summary blocks (warnings-only and errors+warnings cases) unconditionally do `grep ^Warning: "$Log"` / `grep -E '^(Error|Warning):' "$Log"`. When called with `-L off` (exactly how `refresh_P4ROOT_from_offline_db.sh`'s `switch_db_files()` calls it), `$Log` retains the literal string `"off"`, so this tries to grep a nonexistent file named `off`. Confirmed via `p4 print ...@31674` that this exact unguarded line is unchanged since 2025.1 and still present in current `main`. Filed as **SDP-1393**, targeted at "2026.1 Patch 1". **Bug 2, already fixed**: the `mv` calls in `backup_functions.sh`'s `switch_db_files()` (the "Switching root and offline_db links" step) lacked `-f` at 2025.1/31674, matching the mode-override prompts in the customer log exactly. Confirmed current `main` already has `-f` on every one of these calls -- fixed at some point between 2025.1 and now, untracked but done. Nothing to file. ## New skill: sdp-jira-jobs, and a retroactive convention fix User asked for a new convention: every SDP JIRA issue's Description should start with the Summary duplicated verbatim as its first line (blank line, then the full explanation), Summary targeted at ~70 characters. Reason: in case a future JIRA -> P4Jobs integration gets built, it would likely rely on this convention (mirroring how a native P4 job's own `Description` field already works). Created a new skill, **`sdp-jira-jobs`** (mirroring `bsw-ppn-jobs`'s structure but for JIRA), documenting this plus the P4Blog/Resolution-field gotchas and fixVersion naming pattern (`"2026.1 Patch 1"` for what people call "2026.1.1") accumulated during the release cycle. Submitted as `//test-install_sdp/dev` change 33604. Also fixed a real gap while there: the new `sdp-release-process-run` skill (created the night before) had never actually been added to the master `SKILLS_INDEX.md` -- fixed alongside adding the new skill's own entry, change 33605. Retroactively applied the new Summary/Description convention to **SDP-1390 through SDP-1393** (all filed before the convention existed). While doing so, found and fixed a real encoding glitch: SDP-1391's Summary had a literal `<` HTML entity instead of a plain `<`. Fixed the Summary field directly, and added a new "stick to plain ASCII" rule to the skill (change 33606) citing this exact incident as the reason. ## SDP-1394 and SDP-1395: the premature /opt/perforce/p4-sdp path, and the false-FAILED upgrade bug User found this by inspecting their own quick emergency patch (change 33608, fixing `sdp_upgrade.sh`/`opt_perforce_sdp_backup.sh`/its systemd unit) and asked for a broader scan for any other accidental dependency on the not-yet-existing `/opt/perforce/p4-sdp` structure (the OS package directory rename planned for 2026.2, not 2026.1). **Scope turned out to be much larger than the initial patch.** Found the same `SDPPackageBase="$PerforcePackageBase/p4-sdp"` bug in three more files the initial fix missed -- critically, **`install_sdp.sh` itself**, the actual customer-facing installer, meaning every fresh package-structure install of 2026.1 was creating the wrong top-level directory. Also `mkdirs.sh` and `configure_sample_depot_for_sdp.sh`. Additionally found: `install_sdp.sh`'s `BackupScript` variable hardcoded the same wrong path directly (feeding into SELinux `fcontext`/`restorecon` calls that would silently target a nonexistent path), and `sdp_upgrade.sh` auto-set `MigrateSDPToPackageDirs=1` whenever it detected an existing `helix-sdp` structure, which would trigger an unwanted migration attempt during a routine upgrade. Filed as **SDP-1394**. Fixed all of it directly in `r26.1.0` (changes 33608 by the user, 33609 the rest), mirrored into `main` (change 33610, after confirming `main`/`dev` had never received any of it). Regenerated affected man pages in both streams (`mkdirs.sh`'s embedded `-man` text had the same stale path baked in), bumped `r26.1.0`'s `Version`, repackaged, and republished to `main`'s permanent download URL -- verified live via a fresh public-URL download. **While verifying SDP-1394 with a real install-then-upgrade test** (installed genuine SDP 2025.2 Patch 1 via `install_sdp.sh`, fetched from the frozen Classic depot at the exact GA changelist, then upgraded to the fixed 2026.1 tarball), found a **second, independent bug**: `sdp_upgrade.sh`'s STEP 9 unconditionally chowned `$SDPPackageBase/p4_binaries` (a top-level directory that only exists for `-local`-mode pre-staged binaries) -- for a normal install, the real `p4_binaries` lives nested at `$WritableSDPDir/p4_binaries`, already covered by a preceding recursive chown. Root cause: this cycle's `helix_binaries` -> `p4_binaries` rename was a literal find-and-replace of the 2025.2 line (which correctly referenced a real top-level directory back then) without accounting for the location semantics changing. Result: every normal package-structure upgrade reported a false **"SDP Upgrade FAILED"** even though the upgrade fully succeeded underneath. Filed as **SDP-1395**. Given we were still under 24 hours from GA and didn't want a bad upgrade experience to be anyone's first impression of the release, fixed this the same way, but improved the process based on the first round's mistakes (see below): fix in `r26.1.0` (change 33622, removed the redundant/broken chown line entirely, since the real directory is already covered) -> mirrored to `main` (change 33623) -> updated Release Notes (changes 33626/33627, calm "corrected the same day" wording, no alarming language) -> regenerated HTML **and PDF** (changes 33628/33629, since an EBF to a shipped release is a real release event) -> bumped `Version` to 33622 -> repackaged -> **verified with a real install-then-upgrade test using the freshly-built local tarball, before publishing** -> published -> verified live. Both SDP-1394 and SDP-1395 closed in JIRA with `resolution: Done` and `P4Blog` set to their respective r26.1.0 fix changelists. ## Process improvement: the EBF flow, learned by doing it wrong once The first round (SDP-1394) published before updating Release Notes, requiring a second, avoidable repackage-and-republish cycle once the gap was noticed. Documented the correct order of operations in `ReleaseProcessOverview.md`'s Emergency Bug Fix section (both `dev` change 33633 and `main` change 33634): fix code -> update Release Notes -> regenerate HTML **and PDF** (unlike routine dev, an EBF to a shipped release is a real release event) -> regenerate any man page whose embedded usage text changed -> bump `Version` -> repackage -> **verify with a real test before publishing** (using the local tarball directly, no need to wait for a publish/download round-trip) -> publish -> confirm live. Also documented two AI-specific gotchas hit for real: release-stream write access blocked by the same VPN-routing/IP-restricted-protections issue as `test-install_sdp` (group-based unrestricted grant is the fix, not chasing the network path), and tools that shell out to plain `p4` internally (`gen_script_man_pages.sh`, `package_downloads.sh`) needing `P4PORT`/`P4USER` exported as real environment variables, not just passed as `-p`/`-u` flags to the outer script. ## The full Merge Down chain: r26.1.0 -> main -> dev User corrected a terminology/mechanics point mid-flow: for changes originating in an already-cut release stream (the "top"/"north" of the flow, in the user's phrasing -- release streams are north of `main`, dev streams are south), only a **Merge Down** is needed (all the way to `dev`), not a Copy Up -- there's nothing in `dev` that needs to travel north, since nothing originated there. **r26.1.0 -> main**: required temporarily flipping `r26.1.0`'s `notoparent` stream Option to `toparent` (approved by the user after the auto-mode classifier blocked the piped `p4 stream -o | sed | p4 stream -i` command as risky), then `p4 merge` (not `p4 copy`, since `main` might have diverged) from `main`'s client, resolved cleanly (8 files: 7 had only RCS `$Id:`/`$Change:` keyword differences, `ReleaseNotes.adoc` had the real P4D Version Compatibility content), regenerated ReleaseNotes HTML/PDF, submitted (change 33637), flipped the Option back. **main -> dev**: found the user had a pending, unsubmitted change (33607) in `dev` -- their own earlier, partial version of the exact same SDP-1394 fix, on the same files this merge would touch, with a note "we should scan for other possible places" (exactly what turned up the additional bugs). User reverted it themselves (I lack `admin` access needed for `p4 revert -C ` on another user's client). Then hit a real mechanics gotcha: running `p4 merge -S //p4-sdp/dev` while already on a client dedicated to that exact stream produced a misleading "needs copy not merge in this direction" error -- the redundant `-S` confuses direction detection. Plain `p4 merge` (no `-S`) worked correctly. The merge swept up not just today's fixes but a **backlog of ~15 unrelated files** with only trivial `:revdate:` differences dating back to Step 11 of the GA release, never previously merged down -- confirmed each was trivial before resolving, then `-am` auto-resolved everything cleanly (0 conflicting), preserving `dev`'s own unique content (Step 22 "Announce the Release", the AWS storage mount-point fix) alongside `main`'s incoming changes. Submitted as change 33641 (23 files). Documented this as a new, explicit step 9 in the EBF order-of-operations (dev change 33642, main change 33643): "Merge Down the fix all the way to `dev`" -- flagged as the step that's genuinely easy to forget once a fix is live, plus the north/south-of-`main` directional framing and the `-S`-redundant-flag gotcha. ## Post-merge doc regen sweep, all three streams Checked for doc regen needs after the merge chain, per the user's request (PDF regen allowed in `r26.1.0`/`main`, explicitly **not** in `dev`): - **`dev`**: `make`'s timestamp-based dependency check flagged ~13 HTML files as stale (mostly a P4-sync-resets-mtimes artifact, not real content changes). Regenerated all flagged files, then checked real `p4 diff -sa` content diffs before submitting -- reverted the one genuine no-op (`README.html`), kept the 10 with real diffs (the `:revdate:`-driven backlog files plus `ReleaseNotes.html`). - **`main`**: found `SDP_Guide.Unix.html`/`.pdf` and `TheSiteDirectory.html` (rendered from `Server/Unix/p4/common/site/ReadMe.md`) were stale after the r26.1.0->main merge. Regenerated and submitted (change 33644); reverted `SDP_Legacy_Upgrades.Unix.html`/`.pdf` after confirming the HTML had zero real diff (the PDF-only "diff" was pure embedded timestamp metadata noise). - **`r26.1.0`**: same staleness check found **a third real bug still in the live shipping tarball** -- `TheSiteDirectory.html` still had the `EDITME-UpdateThisURL` placeholder from before the `ReadMe.md` HMS-link fix (submitted days ago), because it was never regenerated when that fix landed. Confirmed via the exclusion config and a live tarball extraction that this file genuinely ships. Regenerated (change 33645), repackaged (change 33646), republished to `main` (change 33647), verified live. ## Test-coverage gap, closed for real The whole SDP-1394/1395 incident traced back to a real gap: all upgrade testing this release cycle exercised the Gen6 BSW lab's `reset_sdp.sh`-based flow, never a real `install_sdp.sh` install followed by a real `sdp_upgrade.sh` upgrade. Tried the **old** `install_sdp.sh` test fleet first (`ssh-install_sdp-test-r8x86_64.sh`, word-order-swapped naming) as an opportunistic pre-existing-state fixture -- found it was a broker-only partial fixture (no real `p4d` instance), still useful as one test scenario but not sufficient alone. Did a full, controlled test on my own ad hoc EC2 VMs instead: real SDP 2025.2 Patch 1 install (fetched from the frozen Classic depot) -> real upgrade to the fixed 2026.1 tarball, for both SDP-1394 and SDP-1395 -- this is what actually caught SDP-1395 in the first place. Then closed the loop on the **new**, correct fleet (`ssh-test-install_sdp-*` wrapper scripts, machine `p4d-test-r8-01`): found it already had leftover contamination (a `p4-sdp` directory despite running 2025.2, from earlier ad hoc testing), reset it cleanly with `DANGER_CLEAN.sh`, did a real 2025.2 Patch 1 install, then upgraded to the final published 2026.1 tarball -- clean success, confirming both fixes and that this fleet is a viable, repeatable venue for real upgrade testing going forward, not just fresh-install testing. ## SDP and P4D Version Compatibility: also stale While fixing SDP-1395's Release Notes entry, user separately noticed `ReleaseNotes.adoc`'s "SDP and P4D Version Compatibility" section was stale -- still listed 2025.2 as the latest release and included P4D 2024.1 (past its End of Maintenance date). Fixed in `r26.1.0` (change 33635, doc-only, no tarball changes needed) to list 2026.1 through 2024.2, updated the "As of" date and example sentence, then real-merged into `main` as part of the larger r26.1.0->main Merge Down above. Added new guidance to `ReleaseProcessOverview.md` Step 16 (dev change 33639, main change 33640), marked "(Major Releases Only)": review/update this section each major release, using the P4 Server (P4D) End of Maintenance lifecycle table (https://portal.perforce.com/s/article/Helix-Core-Maintenance-Lifecycle-Helix-Core-Server-P4D) as the source of truth for which P4D versions to list, not an assumed fixed-length window. ## Test-coverage gap: now documented as a repeatable procedure, not just demonstrated once Followed up same day: documented the manual install-then-upgrade procedure (proven out live above) as a permanent, repeatable part of two places, so the gap doesn't quietly reopen: - **`install-sdp-test-suite-run` skill** (change 33648): new "Manual upgrade testing (not yet part of the automated suite)" section -- `DANGER_CLEAN.sh` -> install a real older release (fetched from wherever it actually shipped from) -> upgrade to the release candidate's actual tarball -> verify `Version`, directory structure, and (if applicable) that `p4d` still answers. Notes the old fleet's ad hoc-state caveat (one Rocky 8 box there is a broker-only fixture, not a full `p4d` instance) and that this is deliberately *not yet automated* into `cli_tests.cfg`. - **`ReleaseProcessOverview.md` Step 4** (dev change 33649, main change 33650): flagged that Regression Test Suite 3's automated coverage is install-only and never exercises `sdp_upgrade.sh` -- the exact gap that let SDP-1394/1395 ship -- and pointed to the new skill section, recommending it run pre-release (here) rather than only as part of Step 22's post-release smoke tests, for any release touching the package-structure install/upgrade path. ## Automating the upgrade-test into cli_tests.cfg, and a real gap that surfaced along the way Same day, user asked to actually automate this into `//test-install_sdp/dev/cli_tests.cfg` (not just document a manual procedure) and validate it on the real EC2 fleet. Confirmed via `AskUserQuestion` that this meant the fleet's own `cli_tests.cfg` (Step 4's Regression Test Suite 3), not the separate, differently-scoped `//p4-sdp/dev/test/bsw/cli_tests.cfg` used by the Gen7 lab's `qa`/`qa_build` mechanism (that file tests functional checks against an already-running instance -- `verify_sdp.sh`, `p4verify.sh`, `ccheck.sh` -- not `install_sdp.sh`/`sdp_upgrade.sh` at all). **Found something much bigger while adding the test**: `cli_tests.cfg` itself hardcoded `/opt/perforce/p4-sdp` in 56 places -- setup steps for `-local` installs staging binaries in the (now-wrong) directory, and output assertions expecting the wrong path in `OnlineInstall*`/`PreflightAbort`/`DryRun` tests. After the SDP-1394 script fix, running this suite as-is would have failed almost everywhere. Fixed all 56 (left the two genuine `.p4-sdp.home_dir_backup` filename references alone -- a fixed identifier, unrelated to package-structure naming). Change 33652. Added a new **`Upgrade`** test group: installs the real SDP 2025.2 Patch 1 as a genuine pre-upgrade baseline, then upgrades to the current `dev` tree (the release candidate) via the documented procedure, with explicit regression guards for both SDP-1394 (no premature `p4-sdp` directory) and SDP-1395 (`sdp_upgrade.sh` must report real success). **Testing it for real caught a bug in the test itself.** First run through the actual harness (`./r Upgrade` -- note: positional argument, not `-g `; `./r -g Upgrade` silently matches zero tests) failed 6 of 15, all cascading from one root cause: the Upgrade group's own setup step tried `p4 print` against the Classic depot from within `/opt/perforce/dev/sdp`, but that directory's `.p4config.local` is the DVCS clone's own local on-demand-p4d shadow config (`P4PORT=rsh:...`), not a real connection -- and the test harness's own sandboxed execution environment has no other P4 connection info available ("TCP connect to perforce:1666 failed"). Fixed by checking in a static fixture (`test_fixtures/install_sdp.2025.2-Patch1.sh`, the real fetched 2025.2 Patch 1 script) and copying it locally at test-time instead of a live cross-depot fetch -- more robust for automated testing anyway, not just a workaround. Change 33653. Re-ran through the real harness: **all 15 tests pass**, genuine end-to-end execution confirmed (real `sdp_upgrade.sh` preflight/upgrade output visible in the log, not a trivial pass). ## DANGER_CLEAN.sh compatibility for the new Upgrade test group New general rule from the user: every test group must leave the machine in a state fully cleanable by `DANGER_CLEAN.sh` (files, OS users, services) so groups can run in any order -- the one accepted exception is OS package installations, never cleaned between runs. The new `Upgrade` group (change 33653) violated this: it used `/root/install_sdp_old` (not covered by `DANGER_CLEAN.sh`'s `SDPInstallDir="/root/install_sdp"`) and let `sdp_upgrade.sh` write its own log to a bare, uncovered `/root/sdp_upgrade..log`. Fixed (change 33655) by renaming to `/root/install_sdp` (matching convention, already covered) and adding `-L /root/install_sdp/sdp_upgrade.log`. Verified via a real `DANGER_CLEAN.sh -y` run that both are now gone. This was the first test group to exercise `sdp_upgrade.sh` at all, so this particular litter case had never come up before. Also manually cleaned up pre-fix debris left over from iterative testing on `p4d-test-r8-01`. Separately flagged, not fixed (out of scope, pre-existing, unrelated to today's changes): hundreds of accumulated `gen_sudoers..log` files in `/root/` on `p4d-test-r8-01` dating back to August 13, from `install_sdp.sh`'s own `gen_sudoers.sh` call on every test run, never cleaned by `DANGER_CLEAN.sh`. Either `gen_sudoers.sh` needs to clean up after itself or `DANGER_CLEAN.sh` needs to know about this path -- user's call, later. ## Full Helix Install Test Suite run, and a fourth bug (SDP-1396) With the `Upgrade` group now added and `DANGER_CLEAN.sh`-safe, ran the full suite across the whole EC2 platform fleet (Ubuntu 22/24/26-aarch64, Rocky 8/9/10, SLES 15) to validate today's changes didn't regress anything and that the new coverage holds up broadly. **Every platform except Rocky 8 came back with the same 12 failures**, all in `BackupRecoveryMultiInstance` (Tests 234, 235, 244-253; Ubuntu 24 additionally failed Test 232, a related symptom of the same cause -- the journal window not containing the completion line because the backup never truly succeeded). Root cause: **a fourth instance of the SDP-1394 bug class**, this time in `opt_perforce_sdp_backup.sh` itself -- it independently hardcodes the literal string `"opt_perforce_p4-sdp.$ThisHost"` for its backup/staging directory names and the tar command's internal structure (lines 436, 440, 490, 1067, 1069, plus five spots in its own `-man` usage text), completely separate from the `$SDPPackageBase` variable that SDP-1394 already fixed correctly. Live evidence from the Ubuntu 22 run: `Backups are here: /mnt/p4depots/backup/opt_perforce_p4-sdp.p4d-test-u22-01` -- the wrong, not-yet-existing 2026.2 name, even on a fresh install using the already-fixed `install_sdp.sh`. A broader grep also caught 4 more stale comment-only references (not functional bugs, but same class) in `mkdirs.sh`/`install_sdp.sh`, including one that wrongly said the required `-local`-mode subdirectories were "p4-sdp and downloads" instead of "sdp and downloads". Fixed in `dev` (change 33658; plus a separate change 33659 for ~11 already-correct-at-the- -source but stale-generated HTML files found opened alongside it, catch-up regen for the already-submitted SDP-1379 Helix-to-P4 rebrand text -- confirmed safe via `p4 diff`/`p4 opened` against the source `.adoc`/`.md` files before submitting, not something I broke). Copied Up to `main` (change 33660 for the fix plus the rest of an outstanding dev->main backlog: `ReleaseProcessOverview.md`, `SampleStorageSetup-AWS.adoc`, `deprecated_files.txt`; change 33661 for the man-page/`SDP_Guide.Unix.html`+`.pdf` regen). Used `p4 integrate` (not `copy`/`merge`) for this Copy Up after both stream-aware commands refused with contradictory direction errors (`copy`: "cannot 'copy' over outstanding 'merge' changes"; `merge`: "needs 'copy' not 'merge' in this direction") -- a real mechanics gotcha, apparently triggered by today's earlier Merge Down having touched the same files. No Merge Down back to `dev` was needed afterward (`p4 istat` showed `parentChange` already caught up, since the doc-regen-only files in main's follow-up change are outside `dev`'s branch view). Verified live: re-ran just the `BackupRecoveryMultiInstance` group against the freshly- fixed `dev` tree on the Ubuntu 22 fleet machine (synced via `p4 fetch`/`p4 sync` in the DVCS clone) -- all tests pass, backup now correctly uses `/mnt/p4depots/backup/opt_perforce_helix-sdp.`. Filed as **SDP-1396**, fixVersion "2026.1 Patch 1" (matching SDP-1395's precedent -- not yet fixed in the shipped `r26.1.0` tarball), closed with `resolution: Done` and `P4Blog` pointing at the `dev` fix changelist. **Separate, minor, non-reproducing finding**: Rocky 8's run also failed Tests 12-15 (`ccheck.sh`/`opt_perforce_sdp_backup.sh`/`configure_new_server.sh -V` all reporting `show_versions: command not found`, sourced from `/p4/common/lib/utils.lib`). Manually re-ran the exact same commands on the same machine immediately after and they worked fine -- `show_versions` is defined correctly, `-V` prints the version as expected. Almost certainly a timing/race artifact from Rocky 8 being the slowest platform in the fleet (the user's own prior observation) combined with tests kicking off simultaneously across platforms; not investigated further since it didn't reproduce and is unrelated to today's fix. ## Third Emergency Bug Fix, same day: SDP-1396 into r26.1.0 User's call: "Let's done one more EBF! We're still (just barely) less than 24h from when we first shipped last night." Applied the same procedure as the SDP-1394/1395 rounds, following the now-documented order of operations: 1. **Fix** in `r26.1.0` (change 33663): same `opt_perforce_helix-sdp` correction to `opt_perforce_sdp_backup.sh` plus the 4 comment fixes in `mkdirs.sh`/`install_sdp.sh`. 2. **Release Notes** entry first (change 33664), calm wording matching the SDP-1394/1395 style ("briefly referenced... corrected the same day"). 3. **Man pages + HTML/PDF regen** (changes 33665 doc/gen + 33665 continued for `SDP_Guide.Unix.html`/`.pdf` -- man regen was needed this time, unlike SDP-1395, since `opt_perforce_sdp_backup.sh`'s `-man` usage text changed). 4. **Version bump** to 33665 (change 33666) -- matching the SDP-1394 precedent of tracking through the man-regen changelist when one was needed, rather than stopping at the bare code-fix number. 5. **Repackage** (change 33667, pending). 6. **Verify with a real test before publishing** -- and this caught a real process gotcha: the first verification attempt ran `install_sdp.sh` directly from the extracted local tarball's own tree, which does *not* use local files -- `install_sdp.sh` always fetches its own copy of the SDP tarball from `main`'s permanent download URL unless `-local` mode is used, so the first attempt actually re-verified the **still-buggy, not-yet- published** live tarball instead of the new one. Caught by noticing the backup still reported the wrong path immediately after "successful" install. Fixed by following the documented upgrade-test pattern instead: installed a same-network baseline, then staged the freshly-built local tarball directly into `/opt/perforce/helix-sdp/downloads/` and ran `sdp_upgrade.sh` against it (the same technique already proven for testing unpublished release candidates) -- confirmed live: backup correctly reports `/mnt/p4depots/backup/opt_perforce_helix-sdp.`, and the upgrade itself succeeded cleanly (re-confirming SDP-1395 still holds too). 7. **Publish**: submitted the package (33667), republished to `main`'s permanent download URL (change 33668) following `ReleaseProcessOverview.md` Step 20's documented procedure exactly (remove old version-stamped files, add new ones `-t binary+F`, edit the two fixed-name files). 8. **Confirm live**: fresh `curl` of the real public download URL confirmed `Version` 33665 and zero occurrences of the bad path. 9. **Merge Down all the way to `dev`**: `r26.1.0` -> `main` (temporarily flipped `notoparent`->`toparent` again, `p4 merge -S //p4-sdp/r26.1.0` from `main`'s client, 4 files, 0 conflicts, flipped back; change 33670), man/doc regen in `main` (change 33672), then `main` -> `dev` (plain `p4 merge`, 4 files, 0 conflicts; change 33673), plus `dev`'s own `ReleaseNotes.html` regen (HTML only, per dev policy; change 33674). Updated SDP-1396 in JIRA: description and P4Blog now point at the `r26.1.0` fix (change 33663), matching the SDP-1394/1395 convention of citing the release-stream fix rather than the `dev` change once one exists. **User caught a real gap in the above: the Release Notes fix never actually shipped.** Step 3's regen only touched `SDP_Guide.Unix.html`/`.pdf` -- a completely separate build target from `ReleaseNotes.html`/`.pdf` (built from `ReleaseNotes.adoc` via the Makefile's generic `.adoc` -> `.html`/`.pdf` rule, no relation to `SDP_Guide.Unix.adoc`). Confirmed via `tar -tzf` on the already-published tarball: `doc/ReleaseNotes.html`/`.pdf` genuinely ship inside it (nothing in `tools/excludes.downloads.cfg` excludes `doc/`), and the shipped copy had zero occurrences of "SDP-1396" -- the Change History entry from step 2 never made it into the actual generated output, let alone the published tarball. Fixed properly: first attempt tried reserving a pending changelist number (`p4 change -i`) so `ReleaseNotes.adoc`'s `Released:`/`Jobs Fixed up to` line could cite its own exact final changelist number. **User flagged this as unsafe process guidance**, even though it happened to work this time: Perforce only guarantees *submitted* changelist numbers increase monotonically in submission order, so a pending changelist can be silently renumbered at submit time if anything else submits on the server in between -- relying on a reserved pending number in documentation is fragile by design, not just this once. Correct lesson: accept the existing, already-established tolerance for a slightly-stale `Released:` number (visible elsewhere in the same file's Change History), and instead fix the actual bug -- regenerating the *right* file. Regenerated `ReleaseNotes.html`/`.pdf` for real (change 33675, bundled with the `Released:`/`up to` line correction and a matching `Version` bump, since the reserved number happened to match this time), repackaged (33676), verified in the local tarball this time (`SDP-1396` present, `Released:` line matches, zero bad-path occurrences), republished (33677), confirmed live via a fresh `curl`, then re-ran the full Merge Down chain a second time for this correction (`r26.1.0` -> `main` changes 33679/33681, `main` -> `dev` changes 33682/33683). Updated `ReleaseProcessOverview.md` (dev change 33684, Copy Up to main 33685) with the two real lessons: **(1)** `ReleaseNotes.html`/`.pdf` and `SDP_Guide.Unix.html`/`.pdf` are separate regeneration targets and don't assume `doc/*.html`/`*.pdf` are excluded from the tarball without checking `excludes.downloads.cfg`; **(2)** never reserve/reference a pending changelist number in documentation -- accept approximate numbering instead. Also documented the `install_sdp.sh`-fetches-its-own-copy-unless-`-local` verification gotcha hit while testing this same round (see previous section) as a new "AI-specific gotcha" bullet. ## Open / carried forward - **Non-JIRA task, explicitly deferred, not critical path**: ponder a policy for when an Emergency Bug Fix may update an already-shipped release in place (bump `Version`'s CL, add a Jobs Fixed entry) vs. requiring a real new patch release. User was fine doing it this way today specifically because the bugs were caught within 24 hours of GA; explicitly does not want this to become the default move for a fix discovered a month out. Candidate factors: how much changed, how severe the bug was, how long it took to be detected. Saved as a memory note (`project_emergency_fix_inplace_policy`), not yet acted on. - **Old `install_sdp.sh` test fleet**: confirmed at least one machine (`ssh-install_sdp-test-r8x86_64.sh`) is a broker-only partial fixture, not a full `p4d` instance -- one of presumably several different ad hoc states across that fleet's machines, not yet inventoried. - **`gen_sudoers.*.log` litter** on the `install_sdp.sh` test fleet (see above), not yet fixed -- user's call on `gen_sudoers.sh` self-cleanup vs. a `DANGER_CLEAN.sh` extension. - ~~SDP-1396 not yet fixed in `r26.1.0`~~ -- **resolved tonight**: user approved a third same-day EBF; fixed, verified live, and Merge Downed through `main`/`dev` (see above, including the Release Notes regen-target correction caught by the user mid-flight). - **`SDP_Guide.Unix.html`/`.pdf`, `TheSiteDirectory.html`, man-page regens**: all three streams were left in a clean, fully-submitted, mutually-consistent state as of the last commit tonight (`Released:`/`Version` all read 33675 in `r26.1.0`/`main`/`dev`) -- no known drift remaining to fix next time, but worth a quick `p4 diff -sa`-style sanity check at the start of the next session before assuming that's still true, per the usual P4-sync-mtime-reset caveat. ## Next session: starting points - **Further testing**: the SDP-1396 fix itself was verified live (real install + real upgrade using the locally-built tarball, both on the EC2 fleet and via the published `main` URL) and the targeted `BackupRecoveryMultiInstance` re-run passed cleanly on Ubuntu 22 against the fixed `dev` tree -- **no further testing is strictly required** before considering this incident closed. That said, a fresh full Helix Install Test Suite run across the whole EC2 fleet (now that `dev` carries all of today's fixes, including the Release Notes correction) would be a reasonable, low-cost way to get final confidence before moving on to Patch 1 planning, since it hasn't been re-run end-to-end since the SDP-1396 fix landed in `dev` (only the one targeted group was re-verified). The Rocky 8 `show_versions: command not found` flake (Tests 12-15, non-reproducing) is worth a passive eye if it shows up again, but isn't worth chasing proactively. - **SDP 2026.1 Patch 1 planning**: checked JIRA's fixVersion "2026.1 Patch 1" (id 46052) before writing this and found it's **not a curated list** -- it returns 15+ issues, including the 3 actually-intended-for-Patch-1 bugs from tonight (**SDP-1390** EPEL- dependent packages fail on stock Rocky/RHEL 9, **SDP-1391** empty `ExtraP4PackageList[yum]` produces a bogus yum command, **SDP-1393** `verify_sdp.sh`'s spurious `grep: off` warning with `-L off`), plus **SDP-1395 and SDP-1396** (which shouldn't still be tagged for it -- both already shipped same-day via Emergency Bug Fix, so their fixVersion is stale and should be cleared), plus a long tail of older, unrelated backlog items (SDP-1382, 1354, 1353, 1352, 1284, 1274, 1179, 1050, 1016, 889) that predate tonight and have apparently been using this fixVersion as a loose catch-all over time. **First real step next session: a genuine triage pass** to decide what actually belongs in Patch 1's scope before treating this fixVersion's contents as a plan, plus clearing the stale SDP-1395/1396 tags. - **Non-JIRA tracked tasks to review** (see full list immediately above): the EBF in-place-vs-patch-release policy question (`project_emergency_fix_inplace_policy` memory), the old `install_sdp.sh` test fleet's ad hoc/uninventoried state, and the `gen_sudoers.*.log` litter on that fleet (`DANGER_CLEAN.sh` extension vs. script self-cleanup). None are urgent; good candidates to knock out or consciously defer again while Patch 1 scope is still being decided.