p4seedreplica

p4seedreplica

p4seedreplica feeds missing archive revisions to a Perforce replica without overwhelming its archive pull queue. Its workflow has three phases:

  1. discover generates bounded depot path specifications.
  2. verify finds missing archive revisions in those paths.
  3. seed submits the missing revisions in controlled batches and can resume from a checkpoint.

Requirements

Run the tool as a Perforce user that can run p4 sizes and p4 verify against the source server and p4 pull -lsh against the replica. The p4 CLI must be available on PATH, unless its location is provided with --p4bin.

Set P4PORT and P4USER in the environment, or pass -p/--p4port and -u/--p4user to each command. Point discover and verify at the source server, and point seed at the replica.

Every P4 command includes -Ztrack. With --debug, the tool logs each command’s elapsed time, user/system CPU time, and database lock totals. At the end of each phase, it writes command and database-table totals, plus archive library opens, closes, checkins, and exists counts, to standard output.

Downloads

Build release binaries with:

make dist

The resulting files are written to bin/:

To build a binary for the current platform instead, run make or make build.

Discover Depot Paths

Generate a bounded set of depot paths from the source server:

p4seedreplica discover -o depot-paths.txt -p source:1666

The generator considers only top-level local and stream depots. For local depots, it runs p4 sizes -sh with a five minute timeout. When a path exceeds that timeout, the command is stopped, its immediate directories are obtained with p4 dirs -D, and each directory is checked in the same way.

For stream depots, the generator reads the depot’s StreamDepth, then lists stream roots with one direct p4 dirs request at that depth. It randomly samples stream roots with p4 sizes -sh to estimate file density, grouping them by their shared path prefixes. Groups estimated to exceed --max-stream-files are split into the next path level. This avoids issuing a size command for every stream, so the file limit is an estimate rather than a strict cap. A stream depot with no StreamDepth metadata or no stream roots is ignored. Independent depots are processed concurrently; use --path-workers to change the default of 10 workers.

Use --path-timeout to change the per-path timeout, --max-stream-files to set the estimated stream-group limit (default: 1000000), and --stream-size-samples to set the number of random samples (default: 10). Generation refuses to overwrite an existing output file. Add --debug to log every p4 command to standard error:

p4seedreplica discover -o depot-paths.txt --debug

Verify Missing Revisions

Verify the generated depot paths against the source server and write missing revisions to an input file for seeding:

p4seedreplica verify \
  --paths depot-paths.txt \
  -o missing-revisions.txt \
  -p source:1666

verify runs p4 verify -q --only MISSING concurrently, once for each line in the depot path file. A line can contain multiple space-separated filespecs, which are sent in one p4 verify command. Use --workers to change the default of 10 workers. It will not replace an existing output file unless --replace is specified.

Seed a Replica

The input file contains one revision per line, in the form //depot/path/file#revision. Run the seeding step with an appropriate batch size and pull-queue limit:

p4seedreplica seed \
  --input missing-revisions.txt \
  --batch-size 10000 \
  --max-pull-queue 100000 \
  -p replica:1666

For each batch, the tool submits p4 -x <batch-file> verify -qt --only MISSING, then waits until p4 pull -lsh reports no more than --max-pull-queue transfers. It stops if the pull queue output cannot be parsed, rather than submitting unbounded work.

Each completed batch is recorded in missing-revisions.txt.seedreplica.json. Resume an interrupted run with:

p4seedreplica seed --input missing-revisions.txt --resume -p replica:1666

Use --follow to keep waiting for appended revisions after the end of the file, or --max-batches to stop after a controlled number of batches.

Command Options

discover
  -o, --output FILE           Generated depot path file; required
  -p, --p4port P4PORT         Source server override
  -u, --p4user P4USER         Perforce user override
    --path-timeout DURATION Per-path p4 sizes timeout (default: 5m)
    --max-stream-files N    Estimated maximum files per stream path (default: 1000000)
    --stream-size-samples N Random stream paths sampled for file estimates (default: 10)
    --path-workers N        Parallel depot path workers (default: 10)

verify
    --paths FILE            Depot path specifications; required
  -o, --output FILE           Missing revision output file; required
  -p, --p4port P4PORT         Source server override
  -u, --p4user P4USER         Perforce user override
    --workers N             Parallel verification workers (default: 10)
    --replace               Replace an existing output file

seed
    --input FILE            Missing revision input file; required
  -p, --p4port P4PORT         Replica server override
  -u, --p4user P4USER         Perforce user override
    --batch-size N          Revisions per verify command (default: 10000)
    --max-pull-queue N      Queue limit before waiting (default: 100000)
    --poll-interval DURATION Queue polling interval (default: 30s)
    --max-batches N         Stop after this many batches; zero is unlimited
    --checkpoint FILE       Checkpoint file (default: input.seedreplica.json)
    --resume                Resume from the checkpoint
    --follow                Wait for additional input after reaching EOF
    --dry-run               Validate and checkpoint batches without running p4

All commands accept --p4bin PATH, --debug, -V/--version, and -h/--help.

HTML Documentation

Generate a standalone HTML version of this document with:

make html