# SDP Operational Scripts (fork additions)

This catalog documents the scripts that are **specific to this SDP fork** (they
are not part of upstream Perforce SDP): the operational scripts in
`/p4/common/bin`, the server triggers in `/p4/common/bin/triggers`, and the
fork-added setup scripts in `Server/Unix/setup`. It is the counterpart to
[`maintenance_scripts.md`](maintenance_scripts.md), which covers the Python
administration scripts under `/p4/sdp/Maintenance`.

Unless noted otherwise, these scripts:

- are run as the **`perforce`** OS user (many are scheduled from the perforce
  user's crontab — see the install tooling for the per-server-type cron sets);
- source the SDP environment via `/p4/common/bin/p4_vars <instance>`;
- take the **SDP instance** (e.g. `1`) as their first argument, falling back to
  `$SDP_INSTANCE` when omitted;
- log under `/p4/<instance>/logs`.

**Service control:** this fork is systemd/systemctl-managed. The p4d and
p4broker services run as systemd units that wrap the generated `*_init` scripts,
so where a script stops/starts the server it does so through the normal SDP
mechanism; operators start and stop services with
`sudo systemctl {start|stop|status} p4d_<instance>`. See
[`../Server/Unix/setup/systemd/README.md`](../Server/Unix/setup/systemd/README.md).

## Contents

- [Replication & Topology](#replication--topology)
- [Checkpoints & Database](#checkpoints--database)
- [Verification & Integrity](#verification--integrity)
- [Replica File Repair](#replica-file-repair)
- [Cache & Storage Cleanup](#cache--storage-cleanup)
- [Log Analysis & Tuning](#log-analysis--tuning)
- [Server Triggers](#server-triggers)
- [Setup & Provisioning](#setup--provisioning)

---

## Replication & Topology

Scripts that create or convert replica/edge/standby servers. They reconfigure
server specs, create the service user, and set the replication startup commands.

### mkstandby.sh

**Purpose:** Configures a server as a forwarding-standby replica that replicates
metadata and files from a master while being able to forward client requests.

**Applies to:** standby

**Synopsis:**
```
mkstandby.sh SDP_INSTANCE REPLICA_NAME SVCUSER_PASSWORD MASTERPORT REPLICATING_FROM_SERVER_NAME [mandatory]
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.
- `REPLICA_NAME` — Name of the replica server being configured.
- `SVCUSER_PASSWORD` — Password for the service user (`svc_<REPLICA_NAME>`).
- `MASTERPORT` — Connection string to the master (e.g. `master.example.com:1666`).
- `REPLICATING_FROM_SERVER_NAME` — Server ID this standby replicates from.
- `mandatory` — Optional; `mandatory` marks replication mandatory (default `nomandatory`).

**Example:**
```
mkstandby.sh 1 p4d_standby mypassword master.example.com:1666 master mandatory
```

**What it does / Output:**
- Changes the server spec `Services` from `standard` to `forwarding-standby`.
- Sets `P4TARGET`, `serviceUser`, and replication configurables (read-only DB and library).
- Configures journalcopy + pull startup commands with progressive batch sizes.
- Creates/updates the service user, adds it to `service.g`, and grants `super` to `service.g` if absent.

**Notes:** For high-availability deployments where the standby shares the full
workload. Requires the master to be running and reachable.

### mkstandby_shared.sh

**Purpose:** Configures a forwarding-standby replica with **shared** file-library
replication, allowing multiple standbys to share cached files.

**Applies to:** standby

**Synopsis:**
```
mkstandby_shared.sh SDP_INSTANCE REPLICA_NAME SVCUSER_PASSWORD MASTERPORT [mandatory]
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.
- `REPLICA_NAME` — Name of the replica server being configured.
- `SVCUSER_PASSWORD` — Password for the service user (`svc_<REPLICA_NAME>`).
- `MASTERPORT` — Connection string to the master.
- `mandatory` — Optional; `mandatory` marks replication mandatory (default `nomandatory`).

**Example:**
```
mkstandby_shared.sh 1 p4d_standby2 mypassword master.example.com:1666 mandatory
```

**What it does / Output:**
- Changes `Services` to `forwarding-standby`; derives the replicating-from server ID from `P4MASTERPORT`.
- Sets `P4TARGET`; configures read-only DB replication with **shared** file libraries.
- Creates the service user; sets minimal startup tasks (journalcopy and `pull -L`).
- Adds the service user to `service.g` and grants `super` if needed.

**Notes:** Differs from `mkstandby.sh` by using shared (not read-only) library
replication and fewer startup pull operations. Suitable when multiple standbys
coordinate file caching.

### mkedge.sh

**Purpose:** Configures a server as an edge server that replicates metadata from
the master and caches files locally.

**Applies to:** edge

**Synopsis:**
```
mkedge.sh SDP_INSTANCE REPLICA_NAME SVCUSER_PASSWORD MASTERPORT
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.
- `REPLICA_NAME` — Name of the edge server being configured.
- `SVCUSER_PASSWORD` — Password for the service user (`svc_<REPLICA_NAME>`).
- `MASTERPORT` — Connection string to the master.

**Example:**
```
mkedge.sh 1 edge1 edgepassword master.example.com:1666
```

**What it does / Output:**
- Changes `Services` from `standard` to `edge-server`; sets `P4TARGET`.
- Configures read-only DB and cache-based file-library replication.
- Creates the service user; sets pull operations beginning immediately at 1-second intervals.
- Adds the service user to `service.g` and grants `super` if needed.

**Notes:** Edge servers are read-only replicas useful for geographic
distribution. No `mandatory` flag.

### mkedge-standby.sh

**Purpose:** Configures an edge standby that replicates from another edge server
and caches files locally, providing HA for edge deployments.

**Applies to:** edge

**Synopsis:**
```
mkedge-standby.sh SDP_INSTANCE REPLICA_NAME SVCUSER_PASSWORD MASTERPORT REPLICATING_FROM_SERVER_NAME [mandatory]
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.
- `REPLICA_NAME` — Name of the edge standby being configured.
- `SVCUSER_PASSWORD` — Password for the service user (`svc_<REPLICA_NAME>`).
- `MASTERPORT` — Connection string to the master.
- `REPLICATING_FROM_SERVER_NAME` — Server ID (typically another edge) to replicate from.
- `mandatory` — Optional; `mandatory` marks replication mandatory (default `nomandatory`).

**Example:**
```
mkedge-standby.sh 1 edge_standby1 password master.example.com:1666 edge1 mandatory
```

**What it does / Output:**
- Changes `Services` to `forwarding-standby`; sets `P4TARGET`; configures cache-based file-library replication.
- Creates the service user; sets journalcopy + `pull -L` startup tasks.
- Adds the service user to `service.g` and grants `super` if needed.

**Notes:** Combines edge-server functionality with the forwarding-standby role,
typically replicating from another edge in the same region.

### convert_old_master_to_standby.sh

**Purpose:** Converts an existing master to a standby — used when transitioning
to a new master during failover.

**Applies to:** master (being demoted)

**Synopsis:**
```
convert_old_master_to_standby.sh SDP_INSTANCE STANDBY_ID
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.
- `STANDBY_ID` — Server ID to assign to the converted standby.

**Example:**
```
convert_old_master_to_standby.sh 1 old_master_as_standby
```

**What it does / Output:**
- Stops p4d; replaces `server.id` with `STANDBY_ID`; sets `sdp_server_type.txt` to `p4d_standby`.
- Removes the journal, state files, and `rdb.lbr` to reset replication tracking.
- Restarts p4d as a standby; logs to `$LOGS/convert_old_master_to_standby.log`.

**Notes:** Destructive (clears replication state). Run only after the new master
is configured and running.

### convert_standby_to_master.sh

**Purpose:** Promotes a standby to master during a controlled failover.

**Applies to:** standby (being promoted)

**Synopsis:**
```
convert_standby_to_master.sh SDP_INSTANCE
```

**Arguments / Options:**
- `SDP_INSTANCE` — SDP instance identifier.

**Example:**
```
convert_standby_to_master.sh 1
```

**What it does / Output:**
- Stops p4d; backs up `server.id` as `server.id.replica`; sets `server.id` to `P4MASTER_ID`; sets `sdp_server_type.txt` to `p4d_master`.
- Recovers the latest committed journal from `statejcopy`, moves it into place as `P4JOURNAL`, and runs `p4d -jr`.
- Removes state files and `rdb.lbr`; restarts p4d as the new master; logs to `$LOGS/convert_standby_to_master.log`.
- Reminds the operator to update DNS/address configuration.

**Notes:** Critical failover step — run only after verifying replication is in
sync and the old master is stopped. `server.id.replica` can be reused to rebuild
the former master as a standby.

---

## Checkpoints & Database

### recreate_db_checkpoint.sh

**Purpose:** Recreates the active database from the offline database (built from
the latest checkpoint + journals), swapping DB files during a brief outage.

**Applies to:** master, edge

**Synopsis:**
```
recreate_db_checkpoint.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance (defaults to `$SDP_INSTANCE`).

**Example:**
```
recreate_db_checkpoint.sh 1
```

**What it does / Output:**
- Replays journals into `offline_db`; swaps broker config to maintenance before stopping p4d and back after restart.
- Stops the server, switches active DB files with the offline copies, then restarts.
- For edge servers, replays the active journal (edges keep running after rotation).
- Removes old save files, recreates offline_db, compresses journals, removes old checkpoints/logs. Logs to `$LOGS/checkpoint.log`.

**Notes:** Requires `offline_db` to contain a restored copy of the most recent
checkpoint. Rotates previous-run logs before executing.

### recreate_db_sync_replica.sh

**Purpose:** Resyncs a read-only/forwarding replica from the master by pulling
checkpoints and journals and replaying them, resetting replication to the
current master journal position.

**Applies to:** replica

**Synopsis:**
```
recreate_db_sync_replica.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance (defaults to `$SDP_INSTANCE`).

**Example:**
```
recreate_db_sync_replica.sh 1
```

**What it does / Output:**
- Rsyncs checkpoints and journals from the master (retries once on failure before aborting).
- Recreates offline_db and replays rotated journals; stops the replica; switches offline DB to active; clears `state*` and `rdb.lbr`.
- Restarts and issues `p4 pull -lj` to confirm replication; removes old checkpoints/journals/logs. Logs to `$LOGS/sync_replica.log`.

**Notes:** Requires SSH key setup for passwordless rsync (or shared data). Exits
if `P4REPLICA` is `FALSE`. Companion to `recreate_db_checkpoint.sh` for replicas.

---

## Verification & Integrity

### p4verify.py

**Purpose:** Windowed depot verification with adaptive concurrency, pull-queue
gating, replica journal-lag gating, and shelves verification; emits
prometheus-style metrics and detailed logs.

**Applies to:** master, standby, replica (skips edge/edgerep)

**Synopsis:**
```
p4verify.py <instance> [options]
```

**Arguments / Options:**
- `instance` — SDP instance (or `$SDP_INSTANCE`).
- `-P, --concurrency N` — Initial verify concurrency (default 120).
- `-D, --debug` — Debug logging.
- `-s, --shelves-only` — Verify shelves only.
- `-c, --min-changelist NUM` — Lower-bound changelist for windowed verify.
- `--no-transfer` — Verify without `-t` (transfer disabled).
- `--depot DEPOTNAME` — Verify only this depot.
- `--window-size N` — Initial window size (default 10).
- `--no-adaptive-window` — Disable adaptive window sizing.
- `--target-seconds N` — Adaptive target seconds per job (default 180).
- `--min-window N` / `--max-window N` — Adaptive window bounds (default 10 / 24).
- `--auto-threads` / `--no-auto-threads` — Toggle concurrency auto-tuning (default on).
- `--max-threads N` — Max tuner concurrency (default 200).
- `--pull-queue-limit N` — Pull gating limit (default 1000; `0` disables).
- `--pull-check-interval N` — Pull gating re-check seconds (default 60).
- `--journal-behind-max N` — Pause if replica is more than N journals behind (default 1; `-1` disables).
- `--journal-check-interval N` — Journal-lag re-check seconds (default 30).

**Example:**
```
p4verify.py 1 --concurrency 120 --window-size 10 --pull-queue-limit 1000
p4verify.py 1 --shelves-only
p4verify.py 1 --depot my_depot
```

**What it does / Output:** Verifies shelves first, then depots/server in windowed
changelist batches with parallel jobs; auto-tunes window size to a target runtime
and pauses launches when the pull queue or replica journal lag exceed thresholds.
Outputs `verify_output.log`, `p4verify.log`, `p4verify_errors.log`,
`p4verify-shelves.log`, and `p4verify_autotune.csv`.

**Notes:** Requires the SDP environment and `backup_functions.sh`. Journal-lag
gating needs `p4 pull -ljv`; pull-queue gating needs `p4 pull -ls`.

### verify_shelves.sh

**Purpose:** Verifies the integrity of recent shelved changes, reporting missing
files.

**Applies to:** any

**Synopsis:**
```
verify_shelves.sh <instance> <num_shelves>
```

**Arguments / Options:**
- `<instance>` — SDP instance.
- `<num_shelves>` — Number of most-recent shelved changelists to verify.

**Example:**
```
verify_shelves.sh 1 100
```

**What it does / Output:**
- Lists the most recent N shelved changes (`p4 changes -s shelved`).
- For each, runs `p4 verify -qSt --only MISSING`. Logs to `$LOGS/verify_shelves.log` (cleared at start).

**Notes:** Useful for catching shelf corruption/missing depot files before they
affect users. Commonly scheduled on the master.

### cron_verify.sh

**Purpose:** Lightweight, cron-safe verification of specified depot paths to find
missing head revisions and queue retransfers, guarded by a lockfile.

**Applies to:** any

**Synopsis:**
```
cron_verify.sh <instance> <path_list_file>
```

**Arguments / Options:**
- `<instance>` — SDP instance.
- `<path_list_file>` — File of depot paths, one per line (`//depot/path/...`).

**Example:**
```
cron_verify.sh 1 /p4/common/config/verify_paths.txt
```

**What it does / Output:**
- Tracks the last submitted changelist in `$LOGS/lastcl.txt`; exits silently if `$LOGS/p4_verify.lock` exists.
- For each path, runs `p4 verify -qt --only MISSING` for changes since the last run.
- Manages the lockfile via an EXIT/INT/TERM trap; logs to `$LOGS/p4_verify.log`.

**Notes:** Built for cron — exits cleanly (no error) if another run is active.
Maintains `lastcl.txt` to avoid reverifying.

### check_pull_status.sh

**Purpose:** Reports progress of a long-running batch-verify job by comparing
processed files against a manifest.

**Applies to:** any

**Synopsis:**
```
check_pull_status.sh [-f input_file] [-h]
```

**Arguments / Options:**
- `-f <file>` — Manifest of total files (default `files_to_fstat.txt`).
- `-h` — Help.

**Example:**
```
check_pull_status.sh -f files_to_fstat.txt
```

**What it does / Output:**
- Finds the latest `batch_verify_*.log` in the current directory; counts completed batches against the manifest total.
- Prints percentage complete, processed/total/remaining counts, and the last 5 log lines.

**Notes:** Errors if no `batch_verify_*.log` exists or the manifest is missing. A
monitoring helper for verification jobs.

---

## Replica File Repair

### copy_missing_shelf_files.sh

**Purpose:** Copies shelf library files from the local depot to a remote commit
server via rsync, to replicate missing shelf data.

**Applies to:** master, standby

**Synopsis:**
```
copy_missing_shelf_files.sh <changelist>
```

**Arguments / Options:**
- `<changelist>` — Shelved changelist number (or `$CHANGELIST`).

**Example:**
```
copy_missing_shelf_files.sh 12345
```

**What it does / Output:**
- Lists the shelf's files (`p4 files @=<changelist>`); resolves each library path with `p4 fstat -Ob`.
- Rsyncs each library file to the remote commit server at the matching depot path.

**Notes:** Requires `RSYNCUSER` and `COMMITSERVER`. Source-side companion to
`rsync_missing.sh`.

### rsync_missing.sh

**Purpose:** Pulls missing files to a replica by parsing verify errors and
generating rsync commands from the master.

**Applies to:** replica

**Synopsis:**
```
rsync_missing.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance (or `$SDP_INSTANCE`).

**Example:**
```
rsync_missing.sh 1
```

**What it does / Output:**
- Parses `$LOGS/p4verify.log` for `MISSING` entries; builds per-file rsync commands from `${RSYNCUSER}@${P4MASTERHOST}`.
- Lowercases paths for case-insensitive servers; writes/executes/removes `${P4TMP}/temp_rsync.sh`.

**Notes:** Requires `RSYNCUSER` and `P4MASTERHOST`, and a prior `p4verify` run.
Replica-side companion to `copy_missing_shelf_files.sh`.

---

## Cache & Storage Cleanup

### cacheclean.sh

**Purpose:** Reclaims storage on `lbr.autocompress=1` servers by deleting
depot-cache files older than a configurable age when free space drops below a
threshold.

**Applies to:** master, standby, edge, replica

**Synopsis:**
```
cacheclean.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance.

**Example:**
```
cacheclean.sh 1
```

**What it does / Output:**
- Exits if free space is above the required threshold (default ~1 TB).
- Otherwise deletes files under `/p4/<instance>/depots/` not accessed within N days (default 2), skipping unload and spec depots, using parallel background deletes.

**Notes:** Tunables: `cache_clean_reqavail.txt` (required bytes free),
`cache_clean_atime.txt` (age in days), `skip_cache_clean.txt` (presence disables)
under `/p4/<instance>/bin`. Skipped if replication is `shared`. Requires global
shelves promotion to avoid deleting shelved work.

### proxyclean.sh

**Purpose:** Removes aged files from a proxy cache to reclaim space.

**Applies to:** proxy

**Synopsis:**
```
proxyclean.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance.

**Example:**
```
proxyclean.sh 1
```

**What it does / Output:**
- Deletes files under `/p4/<instance>/cache/` not accessed within 10 days (skips `pdb.monitor`), using background processes.

**Notes:** Skipped if `/p4/<instance>/bin/skip_cache_clean.txt` exists. Proxy
counterpart to `cacheclean.sh`.

### p4cachepurge.sh

**Purpose:** Runs `p4 cachepurge` to purge old file revisions from cache, wired
into SDP logging/monitoring.

**Applies to:** master, standby, edge, replica, proxy

**Synopsis:**
```
p4cachepurge.sh <instance>
```

**Arguments / Options:**
- `<instance>` — SDP instance.

**Example:**
```
p4cachepurge.sh 1
```

**What it does / Output:**
- Initializes `$LOGS/p4cachepurge.log` (rotating prior logs); logs in via `p4login`.
- Runs `p4 cachepurge` (template targets a depot path with a size threshold); emails on failure.

**Notes:** **Edit the `p4 cachepurge` command** to specify your depot paths
before production use.

### sizes.sh

**Purpose:** Reports the data size of recent changes to find large submissions.

**Applies to:** any

**Synopsis:**
```
sizes.sh
```

**Arguments / Options:** None.

**Example:**
```
sizes.sh
```

**What it does / Output:**
- For the last 200 submitted changes, runs `p4 sizes` and prints gigabyte-scale entries to stdout.

**Notes:** Read-only. Adjust the `-m 200` limit / `//...` filter as needed.

---

## Log Analysis & Tuning

### clean_log.sh

**Purpose:** Preprocessing filter that strips administrative log entries before
analysis.

**Applies to:** any

**Synopsis:**
```
clean_log.sh < <logfile>
```

**Arguments / Options:** None (reads stdin, writes stdout).

**Example:**
```
clean_log.sh < server.log > server_cleaned.log
```

**What it does / Output:**
- Removes lines for `clients`, `changes`, `groups`, `meta`, `storageup`, and peek-count entries; writes the remainder to stdout.

**Notes:** Run before feeding logs to a log analyzer for cleaner results.

### log_analyzer2.py

**Purpose:** Comprehensive P4LOG analyzer — performance metrics, slow commands,
lock contention (culprits/victims), blocking chains, and reports.

**Applies to:** any

**Synopsis:**
```
python log_analyzer2.py <logfile> [options]
```

**Arguments / Options:**
- `logfile` — P4LOG file to analyze.
- `--top N` — Show top N slowest commands (default 20).
- `--user USER` / `--command CMD` — Filter (partial match).
- `--min-lapse SEC` — Only commands slower than SEC.
- `--export CSV` — Export per-command data to CSV.
- `--summary` — Summary statistics only.
- `--locks` — Lock-contention analysis.
- `--lock-threshold MS` — Culprit/victim threshold (default 1000).
- `--block-chains` — Blocking-chain narrative reports.
- `--chain-threshold MS` — Min lock wait for chain analysis (default 10000).
- `--max-chains N` — Max chains to show (default 10).
- `--pid PID` — Analyze a specific PID's chain.
- `--hourly` — Hourly command distribution.
- `--verbose` — Detailed parsing info.
- `--keep-threshold SEC` / `--keep-lock-threshold MS` — Memory-reduction filters for huge logs.

**Example:**
```
python log_analyzer2.py server.log --top 20
python log_analyzer2.py server.log --locks --chain-threshold 5000
python log_analyzer2.py server.log --export results.csv --keep-threshold 1.0
```

**What it does / Output:** Parses P4LOG by PID (lapse, lock waits/held per table,
RPC, memory, rows scanned) and reports summary stats, slowest commands, lock
culprits/victims, blocking chains, and hourly distribution; can export per-command
CSV. Uses MAX (worst single) lock times for contention analysis.

**Notes:** Transparently handles gzip/bzip2/xz logs. Needs `server=3` tracking
detail. Keep-threshold filtering keeps summary stats accurate while reducing
memory.

### find_blocker.py

**Purpose:** Given a slow command's PID/time, identifies the command that held the
blocking locks, traces the blocking chain, detects pile-ups, and writes a
narrative analysis.

**Applies to:** any

**Synopsis:**
```
python find_blocker.py <logfile> --pid <PID> [--timestamp "YYYY/MM/DD HH:MM:SS"]
python find_blocker.py <logfile> --completed "<completed log line>"
```

**Arguments / Options:**
- `logfile` — Perforce server log.
- `--pid PID` — PID of the slow command (required unless `--completed`).
- `--timestamp "..."` — Completion timestamp (auto-selects longest lapse if omitted).
- `--start-time "..."` — Start timestamp (alternative to `--timestamp`).
- `--completed "..."` — Full "completed" line from the log.
- `--no-chain` — Skip blocking-chain tracing.
- `--quiet` — Suppress progress messages.

**Example:**
```
python find_blocker.py server.log --pid 7432
python find_blocker.py server.log --pid 7432 --timestamp "2026/04/01 11:44:28"
python find_blocker.py server.log --completed "2026/04/01 11:44:28 pid 7432 completed 454s"
```

**What it does / Output:** Locates the target command, finds candidate lock
holders during the overlap, ranks them, recursively traces the blocking chain,
and counts the pile-up of commands released when the root blocker finished.
Produces a narrative report with root cause, lock details, a chain diagram, time
breakdown, impact, and recommendations. Handles upstream (commit/edge) forwarding
delays.

**Notes:** Needs `server=3` tracking detail. Two-pass: anchors the command by
timestamp, then collects tracking lines in its window; handles PID reuse.

### recommend_limits.py

**Purpose:** Analyzes a P4LOG to recommend group limits (MaxResults,
MaxScanRows, MaxLockTime, MaxMemory) that permit normal use while blocking
resource-intensive outliers.

**Applies to:** any

**Synopsis:**
```
python recommend_limits.py <logfile> [options]
```

**Arguments / Options:**
- `logfile` — Perforce server log.
- `--top-pct N` — Consider the slowest N% of commands (default 20).
- `--top N` — Top N blocked commands per metric (default 15).
- `--csv FILE` — Export per-command data to CSV.
- `--sample N` — Parse only the first N lines (`0` = all).
- `--quiet` — Suppress progress messages.
- `--limits-only` — Print only the recommended limits.

**Example:**
```
python recommend_limits.py server.log
python recommend_limits.py server.log --top-pct 20 --csv data.csv
python recommend_limits.py server.log --limits-only
```

**What it does / Output:** Builds percentile distributions of scan rows, max lock
time, memory, and lapse; flags outlier users; and recommends multi-tier limits
(p99 … p99.99, suggested p99.95) rounded to "nice" numbers, plus group
configuration recommendations (a `limits` group, `no_limits` for service
accounts, `build_limits` for heavy CI accounts). Exports a per-command CSV.

**Notes:** Handles gzip/bzip2/xz logs. MaxLockTime capped at 90,000 ms. Ignores
commands faster than 1 s by default; running aggregates keep summary accuracy
independent of `--sample`.

---

## Server Triggers

These fork-added triggers live in `/p4/common/bin/triggers` and are invoked by
p4d through the trigger table (`p4 triggers`). Each entry below shows a
representative trigger-table line; adjust the trigger name, depot path, and
arguments to your environment. The Python triggers require **P4Python**; the
OTP and RADIUS triggers additionally require `pyotp` and `pyrad` respectively.

### P4Triggers.py

**Purpose:** Base framework/library for building Python triggers. Provides a
`P4Trigger` class (P4 connection lifecycle, change parsing, logging, error
reporting) and a `P4Change` class (wraps `p4 describe` output).

**Type:** shared library (not a trigger)

**Triggers-table entry (example):**
```
N/A (library, imported by other triggers such as CheckCaseTrigger.py)
```

**Arguments / Inputs:** Imported, not invoked. `P4Trigger` is subclassed with `setUp()`/`validate()` overrides; `parseChange(changeNo)` drives connect → getChange → validate and returns 0/1.

**What it does / Output:** Handles the P4 connection, exception handling, and error logging (to `$LOGS/p4triggers.log` by default, override with `log=`); exposes `message()`, `reportError()`, `reportP4Errors()` helpers.

**Notes:** Requires P4Python. Intended for subclassing.

### otpauthcheck.py

**Purpose:** OTP authentication trigger with tiered checks: local password file, then LDAP / P4 auth server, then a 6-digit TOTP (Google Authenticator) token. Service users authenticate by password only.

**Type:** auth-check

**Triggers-table entry (example):**
```
authtrigger auth-check auth "/p4/common/bin/triggers/otpauthcheck.py %user% %serverport%"
```

**Arguments / Inputs:** `%user%`, `%serverport%`; password on stdin.

**What it does / Output:** Validates against `local.passwd`, then (for users in `serviceusers.txt`) LDAP/auth server, then for regular users the auth server plus the trailing 6-digit TOTP. Returns 0 on success, 1 on failure; can block listed users.

**Notes:** Requires `pyotp`. Config: `local.passwd`, `serviceusers.txt`. `P4USER` must be an admin with unlimited login timeout.

### otpkeygen.py

**Purpose:** Generates a TOTP base32 key for a user, stores it on the server, and prints the provisioning URI/QR for Google Authenticator.

**Type:** standalone utility

**Triggers-table entry (example):**
```
N/A (utility -- run by an admin to provision a user's OTP key)
```

**Arguments / Inputs:** `username` (argv).

**What it does / Output:** Creates a random base32 key (`pyotp.random_base32()`), stores it via `p4 key`, and prints a QR-code URL plus the raw key.

**Notes:** Requires `pyotp`. `P4USER` must be an admin. Run before a user can use OTP auth. Companion to `otpauthcheck.py`.

### rad_authcheck.py

**Purpose:** RADIUS/RSA SecurID authentication trigger with tiered checks: local password file, LDAP for service users, then RADIUS (PIN + token) for regular users.

**Type:** auth-check

**Triggers-table entry (example):**
```
authtrigger auth-check auth "/p4/common/bin/triggers/rad_authcheck.py %user% %serverport% %clientip%"
```

**Arguments / Inputs:** `%user%`, `%serverport%`, `%clientip%`; password on stdin.

**What it does / Output:** Checks `local.passwd`, then LDAP (for `serviceusers.txt` users), then sends a RADIUS AccessRequest to the configured servers. Honors blocked-user/blocked-IP lists; logs RADIUS errors to `rad_errors.log`. Returns 0/1.

**Notes:** Requires `pyrad` and `python-six`, plus a RADIUS `dictionary` file alongside the script. Config: `RAD_SERVERS`, `RAD_SHARED_SECRET`, AD hosts/domain, `local.passwd`, `serviceusers.txt`.

### radtest.py

**Purpose:** Standalone utility to test RADIUS connectivity and credentials independently of Perforce.

**Type:** standalone utility

**Triggers-table entry (example):**
```
N/A (utility -- run manually to debug RADIUS auth)
```

**Arguments / Inputs:** `username` (argv); password on stdin.

**What it does / Output:** Sends an AccessRequest to each server in `radsvrs`; prints "access accepted" (exit 0) on the first accept, otherwise "access denied" (exit 1), dumping returned attributes.

**Notes:** Requires `pyrad`, `python-six`, and the `dictionary` file. Configure `radsvrs` and `sharedsecret`. Pure debugging tool.

### CheckCaseTrigger.py

**Purpose:** Prevents submitting files/directories that differ only in case from existing depot entries, avoiding cross-platform (Windows/Unix) conflicts.

**Type:** change-submit

**Triggers-table entry (example):**
```
checkcase change-submit //... "python /p4/common/bin/triggers/CheckCaseTrigger.py %changelist% port=%serverport% user=perforce"
```

**Arguments / Inputs:** `%changelist%`; optional `port=`, `user=`, `charset=`, `api_level=`, `filefilter=` (mapping file). Excludes `git-fusion-user`.

**What it does / Output:** For new (add/branch) files, compares canonical (lowercased) names/paths against existing depot entries; blocks the submit listing each case-only conflict.

**Notes:** Requires `P4Triggers.py` and P4Python. Optional `filefilter` limits the checked paths. Git Fusion aware.

### submit_form_1_in.py

**Purpose:** Requires submissions from designated users to include a valid `Incident:` number in the change description.

**Type:** change-submit

**Triggers-table entry (example):**
```
submitform1_in change-submit //depot/somepath/... "/p4/common/bin/triggers/submit_form_1_in.py %changelist% %user%"
```

**Arguments / Inputs:** `%changelist%` (and `%user%`).

**What it does / Output:** Reads the change description (`p4 describe`) and requires an `Incident:\s+(\d+)` match; exits 0 if present, 1 (blocking) otherwise.

**Notes:** Pairs with `submit_form_1.py` (the form-out side) to enforce incident tracking.

### TFSJobCheck.py

**Purpose:** Requires changes that have associated jobs to reference a TFS work-item id (e.g. `TFS123`) in the description.

**Type:** change-submit

**Triggers-table entry (example):**
```
TFSJobCheck change-submit //depot/path/... "/p4/common/bin/triggers/TFSJobCheck.py %changelist%"
```

**Arguments / Inputs:** `%changelist%`.

**What it does / Output:** Reads `p4 describe -s`; if a "Jobs fixed" section exists, requires a `TFS\d+` pattern. Exits 0 if satisfied, 1 (blocking) if jobs exist without a TFS id (or no jobs at all).

**Notes:** Assumes jobs are associated with changes and follow TFS naming.

### submit_form_1.py

**Purpose:** Customizes the change form for users in `submit_form_1_users` by inserting structured Incident / Description / Reviewed-by fields.

**Type:** form-out

**Triggers-table entry (example):**
```
submitform1 form-out change "/p4/common/bin/triggers/submit_form_1.py %formfile% %user%"
```

**Arguments / Inputs:** `%formfile%`, `%user%`.

**What it does / Output:** If the user is in `submit_form_1_users`, replaces the default `<enter description here>` placeholder with the structured template, written back in place.

**Notes:** Pairs with `submit_form_1_in.py` (the change-submit validator).

### keep_group_unset.py

**Purpose:** Prevents non-admin groups from being given unlimited timeout / Max* values by resetting them to `unset`.

**Type:** form-in

**Triggers-table entry (example):**
```
keep_group_unset form-in group "/p4/common/bin/triggers/keep_group_unset.py %formfile% %user%"
```

**Arguments / Inputs:** `%formfile%`, `%user%`.

**What it does / Output:** Admin users (`p4admin`) bypass it. Otherwise, for non-admin groups, rewrites any `Max*:`/`Timeout*:` values to `unset`. Admin groups (p4-admins, p4-replicas, p4-unlimited) are left untouched. Always exits 0.

**Notes:** Modifies the form in place before save.

### PreventWsNonAscii.py

**Purpose:** Rejects workspace/label forms whose name contains non-ASCII or reserved special characters.

**Type:** form-save

**Triggers-table entry (example):**
```
PreventWSNonASCII form-save client "/p4/common/bin/triggers/PreventWsNonAscii.py %formfile%"
```

**Arguments / Inputs:** `%formfile%`.

**What it does / Output:** Rejects (exit 1) if the Client/Label field contains any of ``\!$%^&()<>+=`` or any line has non-ASCII characters; otherwise accepts (exit 0).

**Notes:** Also treats unicode-decode failures as violations.

### SetLabelOptions.py

**Purpose:** Enforces the `autoreload` option on labels by rewriting `noautoreload` to `autoreload`.

**Type:** form-in / form-out

**Triggers-table entry (example):**
```
setlabelopts form-out label "/p4/common/bin/triggers/SetLabelOptions.py %formfile%"
```

**Arguments / Inputs:** `%formfile%`.

**What it does / Output:** Replaces all `noautoreload` with `autoreload` and writes the form back; always exits 0.

**Notes:** Usable as form-in (on open) or form-out (on save). Users cannot override.

### command_block.py

**Purpose:** Restricts a server command to designated users, allowing read-only (`-o`) use by others.

**Type:** command

**Triggers-table entry (example):**
```
command-block command pre-user-obliterate "/p4/common/bin/triggers/command_block.py %user% %args%"
```

**Arguments / Inputs:** `%user%`, `%args%`.

**What it does / Output:** Allows users in the `passusers` list (default `perforce`); allows anyone when `-o` is present; otherwise rejects (exit 1).

**Notes:** Edit `passusers`. Typically used to gate `obliterate`/`protect`.

### block_checkpoint.sh

**Purpose:** Blocks `p4 admin checkpoint` while allowing other `p4 admin` subcommands.

**Type:** command

**Triggers-table entry (example):**
```
block-checkpoint command pre-user-admin "/p4/common/bin/triggers/block_checkpoint.sh %user% %args%"
```

**Arguments / Inputs:** `%user%`, `%args%`.

**What it does / Output:** Rejects (exit 1) if the args contain `checkpoint` (case-insensitive); otherwise allows (exit 0).

**Notes:** Prevents ad-hoc checkpoints that would bypass the SDP checkpoint workflow.

### JobsCmdFilter.py

**Purpose:** Blocks `p4 jobs` searches with multiple wildcards that can overload the server.

**Type:** command

**Triggers-table entry (example):**
```
jobs-cmd-filter command pre-user-jobs "/p4/common/bin/triggers/JobsCmdFilter.py %args%"
```

**Arguments / Inputs:** `%args%`.

**What it does / Output:** Rejects (exit 1) when the args match problematic wildcard patterns (bare `*`, `**`, `=*`, etc.); otherwise allows (exit 0).

**Notes:** None.

---

## Setup & Provisioning

### configure_new_server.sh

**Purpose:** Applies best-practice configurables to a new production server — creating the super user, automation group, depots, protections, and counters.

**Applies to:** Once on a new master at install time, after `P4PORT`/`P4USER` are set and you have run `p4 login`.

**Synopsis:**
```
configure_new_server.sh <instance> [-checkpoint]
```

**Arguments / Options:**
- `<instance>` — SDP instance.
- `-checkpoint` — Also take a live checkpoint after configuration completes.

**Example:**
```
configure_new_server.sh 1 -checkpoint
```

**What it does / Output:**
- Starts p4d (and trusts `P4PORT` if SSL); creates the `P4USER` super user from `SDP_ADMIN_PASSWORD_FILE` (or the `.enc` variant); creates the Automation group.
- Sets the server as a commit-server, loads the Protections table from `protect.p4t`, and creates the `spec` and `unload` depots.
- Sets `SDP_DATE`/`SDP_VERSION` counters, applies `configurables.cfg` via `ccheck.sh`, and restarts. Logs to a timestamped file.

**Notes:** Requires `p4_vars`, `backup_functions.sh`, `ccheck.sh`, and the template files (`protect.p4t`, `spec.depot.p4s`, `unload.depot.p4s`) alongside the script. Skips users/groups/protections that already exist.

### install_sdp_python.sh

**Purpose:** Downloads, builds, and installs Python, the Perforce C/C++ API, and P4Python on a host that needs P4 scripting.

**Applies to:** Any supported Unix host (RHEL/CentOS or macOS, x86_64) needing P4Python.

**Synopsis:**
```
install_sdp_python.sh [-R rel] [-P python_rel] [-r python_root] [-f] [-no_ssl] [-w work_dir] [-L log] [-si] [-v<n>] [-n] [-D]
install_sdp_python.sh [-h|-man|-V]
```

**Arguments / Options:**
- `-R rel` — Perforce release (default `r17.1`).
- `-P python_rel` — Python release (default `3.6.2`).
- `-r python_root` — Install root (default `/p4/common/python`).
- `-f` — Force reinstall (removes the existing root and downloads).
- `-no_ssl` — Build without OpenSSL (limits SSL connections).
- `-w work_dir` — Build working directory (default a temp dir).
- `-L log` — Log file path, or `off`.
- `-si` — Silent (log only).
- `-v<n>` — Verbosity 1–5.
- `-n` — No-op (print commands only).
- `-D` — Extreme debug (`set -x`).
- `-h` / `-man` / `-V` — Help / man page / version.

**Example:**
```
install_sdp_python.sh -R r18.1 -P 3.7.0 -f -v4
```

**What it does / Output:** Verifies the SDP environment, OS user, architecture, and build tools (gcc, g++, wget, openssl); downloads the Python source, Perforce API, and P4Python; builds and installs Python at the chosen prefix; then builds P4Python against the API (with SSL unless `-no_ssl`). Writes `build.log`/`install.log` and suggests fixes for common build errors.

**Notes:** Must run as the SDP `OSUSER`. The script is self-logging — do not add shell redirection or `tee`. x86_64 only; RHEL/CentOS and macOS only.