# Session Log -- 2026-09-10

## Summary

Follow-up day after the SDP 2026.1 GA + three-EBF night (SDP-1393/1394/1395/1396, see
`SessionLog-2026-09-09.md`). User's colleague is doing a live customer upgrade of SDP+P4D
later today, spanning a mix of old (legacy, pre-`/opt/perforce`) and new (package-structure,
`helix-sdp`) instances -- so before that happens, checked for any remaining test gaps
specific to that scenario, added a documented EBF-vs-Patch-Release policy to the Developer's
Guide, and, prompted by the user thinking through the old-vs-new-structure scenario, found
and fixed a real (not customer-reported) safety gap in `sdp_upgrade.sh`: it could silently
proceed when run as root against a legacy-structure install, when it should refuse. Also
confirmed and removed a redundant manual doc step (`chown -R root:root sdp`) that
`sdp_upgrade.sh` has handled on its own for a long time.

## Fresh full Helix Install Test Suite run

Kicked off a fresh run across all 7 EC2 platforms (Ubuntu 22/24/26-aarch64, Rocky 8/9/10,
SLES 15) since `dev` hadn't been exercised end-to-end since last night's SDP-1396 fix
landed. **Final result: 6 of 7 platforms fully clean (646/646, zero failures)** -- Ubuntu
24, Ubuntu 26 aarch64, Rocky 8, Rocky 9, Rocky 10, and SLES 15. Only Ubuntu 22 hit the
previously-seen `show_versions: command not found` flake (Tests 12-15), and Rocky 8 (last
night's original flake location) came back completely clean this time with no recurrence --
confirming this is a genuine, occasional, platform-independent race condition rather than a
Rocky-8 quirk. Non-blocking and non-reproducing on manual retry; not investigated further
today.

## Old vs. new structure: is there a new test gap?

Before answering, traced where the package-structure (`/opt/perforce/helix-sdp`) code split
actually originates: confirmed via the Classic depot's `install_sdp.sh` history that
`install_sdp.sh` itself has *always* produced Package Structure since its very first
revision (change 30782, 2024-11-09) -- there is no version of `install_sdp.sh` that produces
Legacy Structure. Legacy-structure installs only exist as long-lived instances from before
`install_sdp.sh` existed at all (pre-Nov-2024, built via the older `mkdirs.sh`-based manual
procedure). Also confirmed every one of last night's four bugs (SDP-1393/1394/1395/1396)
lives entirely behind `UsePackageDirs`-gated code in `sdp_upgrade.sh`/`opt_perforce_sdp_backup.sh`
-- none of it touches the Legacy code path at all. Conclusion: the "old structure" half of
the colleague's upgrade carries no incremental risk from anything shipped this cycle, so no
new Legacy-specific regression test was warranted on risk grounds; the full-suite run above
(which exercises the "new structure" side where all the actual risk and fixes live) was the
higher-value use of time.

## Developer's Guide: Choosing an Emergency Bug Fix vs. a Patch Release

Picked back up the open, non-JIRA policy question from last night
(`project_emergency_fix_inplace_policy` memory) at the user's request: "we can probably put
that content in the Developer's Guide... there's going to be some natural overlap between
the Release Process and the Developer's Guide, and cross-referencing." Added a new section
to `SDP_DeveloperGuide.adoc` right after its existing Terminology block (which already
defines Emergency Bug Fix, Hot Fix, and Patch Release, with a stub cross-reference), built
around the user's three named factors: how much changed, how severe the bug was, and how
long since the release shipped -- weighed together, no fixed hour cutoff, default to a real
Patch Release when unsure. Cross-referenced both directions: the new Dev Guide section
points at `ReleaseProcessOverview.md` for the *mechanics* of an EBF, and a line was added
there pointing back at the Dev Guide for the *policy* of when to use one. Submitted to `dev`
(change 33702) and Copied Up to `main` (change 33703, which also swept up an unrelated
backlog: `ReadMe.md`, `parse_sysctl.pl`, `Changing_auth.id.md`, `SysConfig.adoc` -- all
already fully submitted by someone else, confirmed via `p4 opened -a` before including them).

Hit the known "tools bypass P4 read-only" issue again doing this: edited
`ReleaseProcessOverview.md` with the Edit tool before running `p4 edit` on it. Caught
immediately via `p4 opened` showing it not tracked as open, ran `p4 edit`, verified via
`p4 diff` that the file's only actual difference was the intended one-sentence addition
before submitting.

## SDP-1397: sdp_upgrade.sh doesn't reliably catch Legacy structure + running as root

User's framing: two sets of upgrade instructions exist (OS Package Structure requires
starting as root; Legacy/Classic requires starting as the SDP Owner/`perforce` user), there
are already some safeguards against following the wrong one, and the ask was to review the
code and make sure all four combinations result in "nothing bad happening" plus a good error
message.

**Traced the full 2x2 matrix** ({Package, Legacy} x {root, SDPOwner}) in `sdp_upgrade.sh`:

1. Package + root: correct, no error (good).
2. Package + not-root: caught early and reliably, location-independent (`-d $SDPPackageBase`
   detection) -- "run this script as root, not $ThisUser".
3. Legacy + SDPOwner: correct, no error (good).
4. **Legacy + root: not reliably caught.**

For case 4, the only existing root-related check is location-*dependent* -- it only fires if
`$PWD` exactly matches one hardcoded Classic-structure documented directory. Worse, the
later preflight ownership check (which `stat`s `/p4/common` to derive `$SDPOwner`, then
compares to the current user) treats "current user is root and doesn't own `/p4/common`" as
an unconditionally-acceptable, verified condition -- not gated on whether Package Structure
is actually in use. `ErrorCount` stays 0, `do_preflight_checks` returns success, and the
upgrade proceeds. Since Legacy mode has none of Package Structure's `chown`-based ownership
recovery steps (all gated behind `UsePackageDirs -eq 1`), a Legacy customer running the
upgrade as root from anywhere other than that one exact hardcoded path would get no warning
and could end up with newly-written files owned by root instead of the SDPOwner -- files
p4d (running as the SDPOwner) needs to write to.

Filed as **SDP-1397** before starting the code (per the user's explicit "let's immediately
file a JIRA... as we get to work on the code"). Fixed with two layers of defense: (1) a new,
location-independent early check -- Legacy structure detected (`UsePackageDirs=0`) and
running as root (and not the reserved, not-yet-implemented `-pkg` migration case) now bails
with a clear message naming the actual `$SDPOwner`; (2) hardened the preflight ownership
check to only treat "root doesn't own `/p4/common`" as acceptable when `UsePackageDirs -eq
1`, closing the same hole at a second layer (not reachable today given fix (1), but
defense-in-depth against future changes, e.g. once `-pkg`/SDP-1158 ships).

**Verified live on the EC2 fleet** (`p4d-test-u22-01`, using a lightweight fake-Legacy
scaffold -- didn't need a real install, since the new check fires very early, before any
real preflight/instance logic runs): all 4 matrix combinations now behave correctly. The two
already-good paths pass through unaffected, the already-existing Package+non-root check
still fires with its original message, and Legacy+root is now caught immediately with the
new message, correctly naming the test machine's actual SDP Owner (`perforce`). Fixed in
`dev` (change 33704), Copied Up to `main` (change 33705). Closed in JIRA with
`resolution: Done`, `P4Blog` set to the `dev` fix changelist. Not release-blocking; landed in
`dev` only, per the user's explicit call ("This is fine to go in dev -- my colleague doing
the upgrade this afternoon will follow the correct procedure reliably").

## SDP_Guide.Unix.adoc: removing a now-redundant manual chown step

While working through SDP-1397, the user asked about a related cleanup: SDP_Guide.Unix's
section 6.3.1.2 ("Sample SDP Upgrade in OS Package Structure") has a manual
`chown -R root:root sdp` step the user added roughly a year ago as a workaround for the
install not reliably getting ownership right. Question: does the SDP-1397 work make this
redundant?

Traced `sdp_upgrade.sh`'s STEP 9 ("Adjusting Ownership and Permissions"), which already runs
`chown -R root:root $ImmutableSDPDir` (the exact same target as the manual doc step)
unconditionally whenever Package Structure is in use. Via `p4 annotate`/`filelog`, confirmed
this line predates the Classic-to-Streams migration (present already in change 31574, dated
2025-05-15) -- it's been stable for well over a year, unrelated to (and untouched by) the
SDP-1397 fix.

**Verified live, not just inferred from code**: did a real install + upgrade on the EC2
fleet, deliberately skipping the manual chown step. Confirmed empirically that `tar -xzf`
run as root actually preserves the *tarball's own packaging-machine* UID/GID rather than
resetting to root -- immediately after extraction, the tree was owned by `UNKNOWN:staff`
(the tarball was built on a Mac, hence the `staff` group), not root. This is almost
certainly the original problem the manual chown workaround was added for a year ago. Then
ran `sdp_upgrade.sh -y` with the chown step *still* skipped: confirmed via a full recursive
`find`/`stat` scan afterward that every file under the `sdp` tree was `root:root` --
STEP 9 fully self-corrects the ownership with zero manual intervention needed.

Removed the redundant `chown -R root:root sdp` line from the doc (`dev` change 33706, Copy
Up to `main` 33707, `SDP_Guide.Unix.html`/`.pdf` regenerated in `main` as change 33708).

## Correction: Copy Up is reserved for coordinated release activity, not routine

User caught a real process mistake: today's Developer's Guide policy addition, the SDP-1397
fix, and the redundant-chown doc cleanup were each Copied Up to `main` the same day they
landed in `dev`, on the reasoning that they were tested and ready. That's wrong -- `main` is
meant to reflect whatever was most recently *shipped*, not a preview of upcoming `dev` work.
Copy Up should only happen via: the Release Process's own Copy Up step (part of actually
cutting a release), an Emergency Bug Fix's Merge Down from a release stream (which updates
`main` via Merge Down, not Copy Up), or a deliberate doc-only Hot Fix authored directly in
`main`. Merge Down (pulling a parent's content into `dev`) remains fine to do routinely --
only Copy Up (pushing `dev`'s content into `main`) needed the correction.

User's explicit call: don't revert the Copy Ups already done ("No need to fix it; let's just
avoid making any new changes that break the rule"). Fixed the documentation instead: expanded
`SDP_DeveloperGuide.adoc`'s "Merge Down, Copy Up" Terminology entry with the explicit cadence
policy, and added a matching note at the top of `ReleaseProcessOverview.md`'s Step 10.
Checked all skills for anything that might encourage ad hoc Copy Ups -- none do;
`sdp-release-process-run`'s existing "Copy Up is a one-time event for the release" note
already points the right direction. Submitted `dev`-only (change 33710), deliberately not
Copied Up itself, in keeping with the policy it documents. Saved as a new feedback memory
(`feedback_copy_up_is_release_only`) so this isn't relearned the hard way again.

## Follow-up: Copy Up scope, not just cadence -- En-Masse vs. Selective

User distinguished *when* a Copy Up happens (the correction above) from a separate question:
*how much* of `dev` it should include when it does happen. Framed two common models:
*Selective* (push one change, or a hand-picked set, at a time, explicitly marking what's
ready vs. still in progress) vs. *En-Masse*/"Copy Up Everything" (push all of `dev` in one
operation, no picking and choosing). Neither is an absolute best practice -- it depends on
the team. Selective can be made reasonably safe with strong, current understanding of
cross-file dependencies and the ability to QA each change in isolation, but even then the
resulting `main` tree is a specific file combination that, as a whole, was never itself
tested (it didn't exist as a real tree until the selective copy assembled it). En-Masse
trades that away for rigidity: it only makes sense with comprehensive testing
(integration/regression/unit/performance) done against `dev` as a whole immediately before
the Copy Up -- but then the copied tree is exactly the tested tree, with no untested
recombination possible.

**Decision for SDP: En-Masse.** Documented as a new "Copy Up Scope: En-Masse vs. Selective"
section in `SDP_DeveloperGuide.adoc`, right after the Terminology section (change 33711,
`dev`-only, per the same day's cadence policy). Ties together neatly with material already
in the docs: `dev`'s "must always be release-ready" requirement (Working in Streams table)
and the Release Process's existing "this is an all-or-nothing operation for the whole tree"
note on Step 10 both already implied En-Masse; this makes the decision and its rationale
explicit. Updated the `feedback_copy_up_is_release_only` memory with this follow-up.

## SDP-1399: opt_perforce_sdp_backup.sh and sdp_upgrade.sh could hang for hours on large depots

User relayed a real customer report: an SDP upgrade hung at STEP 10 (starting the
`opt_perforce_sdp_backup` service). `systemctl status` showed a `find /p4/1/depots/ -type d`
subprocess running. Happened on 3 machines; completed on 2 after about an hour each (far
longer than a script backing up a few MB should ever take); still running with no error on
the 3rd (largest depot) when reported. Asked me to review the 2026.1 version of the script
for what could cause this.

**Root cause, found immediately**: both `opt_perforce_sdp_backup.sh` and `sdp_upgrade.sh`
determine `P4Depots` via `find "$Root"/*/depots/ -type d | head -1` -- no `-maxdepth`, so
instead of checking whether `depots` exists, `find` recursively walks the *entire* depot
tree before `head -1` can see a line and trigger SIGPIPE. How long that takes is
proportional to depot size and storage speed relative to `find`'s stdout buffer filling --
exactly matching the reported symptom. Every other `find` call in the same script correctly
uses `-maxdepth`; this was a one-off oversight, duplicated byte-for-byte in both scripts.

**Regression status, checked precisely**: the `opt_perforce_sdp_backup.sh` instance is a
genuine 2026.1 regression -- traced via `p4 annotate` to change 31691 (2025-06-08) in the
`dev_rebrand` branch, which only merged into mainline `dev` on 2026-09-02 and first shipped
in 2026.1 GA; the Classic depot's version of this script never had this logic at all. The
`sdp_upgrade.sh` instance is *not* a regression -- confirmed present already in the real SDP
2025.2 Patch 1 GA tarball (Classic depot change 33059), so it's been shipping for about a
year, presumably usually finishing fast enough not to draw attention.

**Fix, refined through real testing, not just reasoning from code**: `find -L "$Root"
-maxdepth 2 -type d -name depots -print -quit`. The `-L` turned out to be essential, not
optional -- discovered by actually testing on the EC2 fleet: `depots` is a symlink in the
standard SDP layout, and `find`'s default (`-P`, physical) mode does not follow symlinks for
`-type` matching, so the first version of the fix (without `-L`) returned nothing at all,
which would have been a *worse* regression (breaking `P4Depots` detection entirely) than the
hang it was meant to fix. Caught this before it went anywhere near a real script.

Filed as **SDP-1399**. Also independently checked `opt_perforce_sdp_backup.sh`'s separate
`find_p4depots_probe_path` function (a different depots-probing mechanism, used for backup
mount-point detection) for the same bug pattern -- confirmed clean, already correctly
bounded with no recursion.

**Two more small fixes squeezed in during the same review**, per the user's requests:
- `sdp_upgrade.sh` had two blocks labeled "STEP 10" (the backup-service install, and the
  custom post-upgrade hook). Renumbered the second to STEP 11.
- `sdp_upgrade.sh`'s STEP 9 now chowns `~SDPOwner/.p4-sdp.home_dir_backup` to the SDPOwner if
  it exists and is still root-owned from an install predating that file's ownership fix in
  `install_sdp.sh` (SDP-1376) -- runs regardless of Package vs. Classic Structure, since the
  file lives in the SDPOwner's home directory either way. User explicitly said this one
  doesn't need its own JIRA (too minor).

**EBF decision**: user's call -- "This is EBF territory since there's an element of
regression... this seems like a safe thing anyway. Let's make it happen" -- explicitly past
the "within 24h" guideline from the Developer Guide's own EBF-vs-Patch policy, but judged
still the right call given the small, well-understood fix and severe user-facing impact.
Still called "r26.1.0 GA" (2026.1), not a new patch version.

**Also folded in this cycle: catching up r26.1.0 with main's accumulated backlog.** Before
this EBF, `main` had 10 files' worth of content `r26.1.0` didn't (from the prior day's now-
corrected ad hoc Copy Up habit, plus older legitimate backlog going back to GA week):
SDP-1397's fix, its Release Notes/doc entries, a `deprecated_files.txt` addition, a
`SampleStorageSetup-AWS.adoc` mount-point fix, minor `ReadMe.md`/`Changing_auth.id.md`
wording tweaks, a new `SDP_DeveloperGuide.adoc`/`ReleaseProcessOverview.md` policy content,
a brand-new `SysConfig.adoc` guide, and a real Perl bug-fix pass on `parse_sysctl.pl` by a
different engineer (jhalbig) -- fixed `=~` used for string equality, missing `my`,
unescaped regex metacharacters, a duplicate line, plus updated tuning values. Reviewed every
file's actual diff content before recommending action; assessed all of it as safe to ship
(doc-only, additive-data-only, already-tested-by-me, or low-blast-radius since
`parse_sysctl.pl` is a human-reviewed advisory tool outside the install/upgrade critical
path). User's call: "Proceed!" -- Copied `main`'s full content Down into `r26.1.0` first
(temporarily flipping `nofromparent`->`fromparent`, `p4 copy` since P4 requires copy not
merge for mainline->release, change 33716), *then* applied the SDP-1399 fix on top.

**Full EBF cycle, verified live twice**: fix (change 33719) -> Release Notes entries for
both SDP-1399 and SDP-1397, now shipping in `r26.1.0` for the first time (change 33720,
man-page regen included since the version stamp changed) -> `Version`/`Released:` line
aligned to the same exact changelist (33720) since both were already known post-submit, no
guessing needed this time -> repackaged -> **verified against the real final package** on
the EC2 fleet with a genuinely large synthetic depot (~42,000 directories) and a simulated
stale root-owned `home_dir_backup` file: real install, real upgrade, completed in under 17
seconds (previously could have hung indefinitely), backup completed correctly, ownership
fix applied correctly -> published to `main`'s permanent URL (change 33723) -> confirmed
live via fresh `curl` (`Version` 33720, zero occurrences of the old bug pattern, both JIRA
numbers present in the live `ReleaseNotes.html`) -> Merge Down chain `r26.1.0` -> `main`
(33725, doc/man regen 33727) -> `main` -> `dev` (33728, `dev`'s own `ReleaseNotes.html`
regen 33729). All three streams confirmed clean afterward.

Closed **SDP-1399** in JIRA (`resolution: Done`, `P4Blog` -> change 33719). Updated
**SDP-1397**'s `P4Blog` to note it now also ships in `r26.1.0`/2026.1 GA via this EBF, not
just `dev`.

## Post-EBF Release Notes Hot Fix: helix_binaries deprecation callout + section cleanup

After shipping, user asked for two more web-only Release Notes tweaks in `r26.1.0`, treated
as a Hot Fix (doc-only, no JIRA, no `Version` bump or repackage needed):

1. In "What's New in SDP 2026.1 GA", the existing "Rebranding: 'Helix' -> P4" bullet didn't
   call out its most immediate, user-visible consequence. Added: `/p4/sdp/helix_binaries`
   and `get_helix_binaries.sh` are deprecated; use `/p4/sdp/p4_binaries` and
   `get_p4_binaries.sh` instead.
2. User spotted a real structural issue while reviewing: two separate "What's Coming Next in
   SDP 2026.2?" sections existed (one near the top of "What's New?", one near the bottom,
   the latter apparently orphaned from the former given its "Also in SDP 2026.2..." phrasing
   that no longer had anything to be "also" with). Consolidated into one section, merging the
   orphaned `sdp_upgrade.sh` Legacy->Package migration note into the existing OS Package
   install/upgrade bullet. Also reordered per the user's request: "What's New in SDP 2026.1
   GA (This Release)" now comes first, the consolidated "What's Coming Next in SDP 2026.2?"
   second, with the rest of the release history unchanged in relative order.

Submitted in `r26.1.0` (change 33731), Merge Downed through `main` (33733, regen 33735) and
`dev` (33736, `dev`'s own HTML-only regen 33737). All three streams confirmed clean.
Confirmed live via the actual published web page
(`https://workshop.perforce.com/view/p4-sdp/main/doc/ReleaseNotes.html`): exactly one
"Coming Next" section, `helix_binaries` deprecation note present.

## End of day

User posted to the team's `#p4-sdp` channel (in the 2026.1 GA announcement thread) about
today's work. Hoping today's SDP-1399 EBF was the last one needed for a while, with tomorrow
free to focus on new work rather than firefighting.

## Late-night JIRA review + SDP 2026.1 Patch 1 Release Notes skeleton

User asked (after saying good night the first time, then continuing with two more Hot Fix
tweaks above) for one more pass before actually turning in: review all SDP JIRA issues
created in the past two weeks against this project's session logs, true up `P4Blog` and
status/resolution where the logs show more recent reality than JIRA does, and start a new
"SDP 2026.1 Patch 1" skeleton section in `ReleaseNotes.adoc`.

Queried all 15 issues created in the last 14 days (`SDP-1383` through `SDP-1399`) and
cross-checked each against session-log grep hits:

* **SDP-1398** (mkrep.sh `FromServerID`/preflight redesign): transitioned **Open ->
  In Progress** (transition id 4). Change 33709 is real, completed work, but still shelved
  awaiting Swarm review, not yet submitted -- so "In Progress" fits better than "Open" or
  "Closed." Deliberately left `P4Blog` unset: no final integrated changelist exists yet.
* **SDP-1395** and **SDP-1396**: both had a stale `fixVersion = 2026.1 Patch 1` left over
  from before today's EBF -- both actually shipped directly into `r26.1.0`/2026.1 GA today.
  Cleared `fixVersion` to empty on both so Patch 1 planning doesn't double-count them.
* All other 12 issues (SDP-1383/1384/1386/1387/1388/1389/1390/1391/1392/1393/1394/1397/1399)
  checked out as already correctly stated in JIRA -- no changes needed. Notably SDP-1390,
  SDP-1391, and SDP-1393 remain the only issues still genuinely tagged and open for 2026.1
  Patch 1.

**Policy update from the user mid-task**: maintain the `P4Blog` field on JIRA issues as we
go, not just at closing time -- set it as soon as a representative changelist exists (even
mid-development), and update it again if a more complete/final changelist later supersedes
it. Documented this in `P4Info.md` (new "JIRA P4Blog Field: Maintain As You Go" section,
change 33740 in `test-install_sdp/dev`) and updated the corresponding memory file.

Started the new `=== SDP 2026.1 Patch 1:` section in `dev`'s `ReleaseNotes.adoc`, placed
immediately above `=== SDP 2026.1 GA:` (most-recent-first ordering), with `EDITME`
placeholders for the eventual version/date and upper changelist bound, and *no* Jobs Fixed
bullets yet -- after tonight's fixVersion cleanup, nothing currently qualifies as
done-and-not-deferred for Patch 1. Regenerated `dev`'s `ReleaseNotes.html` (HTML only, no
PDF, per `dev` policy) and submitted both files together in `dev` only (change 33739) --
per today's earlier Copy Up cadence policy, this is unreleased forward-looking prep, so it
stays in `dev` and does *not* get Copied Up to `main` yet.

User is off to bed; no further action expected tonight beyond this log update and submit.