USAGE for load_checkpoint.sh v3.0.2: load_checkpoint.sh { [ ...] | -latest | -latest_jnls | -jo [ ...] | -jo_latest } [-R|-F ] [-i ] [-s ] [-t ] [-no_start | [-no_xu] [-verify {default|"Verify Options"} [-delay ]]] [-c] [-l] [-r] [-b] [-y] [-L ] [-si] [-d|-D] or load_checkpoint.sh [-h|-man] DESCRIPTION: This script can load a specified checkpoint and/or numbered journals into P4ROOT (/p4/N/root) and/or /p4/N/offline_db (where 'N' is the SDP instance name). It supports a variety of use cases for replaying checkpoints and journals, including: * Seeding or Reseeding a replica or edge server. * Loading a checkpoint on the commit, e.g. in a recovery scenario. Checkpoints and/or journals can be specified in one of two ways: they can be specified as parameters to this script, or they can be determined by this script if they appear in the SDP standard location according to the journalPrefix standard. They key methods are: * Specify the path to the checkpoint to replay. The checkpoint can be in the form of a compressed .gz file, an uncompressed checkpoint file, or a directory (for parallel checkpoints). * Use '-latest' to have this script find the latest checkpoint available. For a commit server, /p4/N/checkpoints/p4_N is searched. For other servers, their journalPrefix is used. The timestamp on the latest available *.md5 file is used to determine what checkpoint is the latest available, regardless of checkpoint form (compressed or uncompressed file, or a directory for parallel checkpoints). * Use '-latest_jnls' to find the latest checkpoint as with '-latest', and then also find and replay any available subsequent numbered journals. * Use '-jo' ("journal only") to specify path(s) to one or more numbered journals to be supplied as parameters to this script. Journal files provided may be compressed or uncompressed. * Use '-jo_latest' to find any numbered journals available to be replayed based on the journal counter of the data set. At the start of processing, preflight checks are done. Preflight checks include: * The specified checkpoint and corresponding *.md5 file must exist. * The specified checkpoint can be a compressed or uncompressed file or a directory (for parallel checkpoints). * All journal files to replay (if any are specified) must exist. * The $P4ROOT/server.id file must exist, unless '-s' is specified. * If the $P4ROOT/server.id file exists and '-s' is specified, the values must match. * The $P4ROOT/license file must exist, unless '-l' is specified or if the replica type does not require a license (such as an edge server). * The SDP structure and key files must exist. * Disk space checks are done to attempt to determine if sufficient space is available to replay the checkpoint. If the preflight passes, the p4d_N service is shutdown. The p4broker_N service is shutdown if it is configured. If a P4LOG file exists, it is moved aside so there is a fresh p4d server log corresponding to operation after the checkpoint load. If a P4JOURNAL file exists, it is moved aside as the old journal data is no longer relevant after a checkpoint replay. (Exception: If the P4JOURNAL is speciffed in a list of journals to reply, then it is not moved aside). Next, any existing state* files in P4ROOT are removed. Next, any existing database it is files in P4ROOT are preserved and moved aside, unless '-R' is specified to remove them. Next, the specified checkpoint is loaded. Upon successful completion, 'p4d -xu' is executed (by default) to help ensure the service can be started with the p4d binary used to replay the checkpoint. Then the Helix Core service is started with the current p4d binary. If the server to be started is a replica, the serviceUser configured for the replica is logged into the P4TARGET server. Any needed 'p4 trust' and 'p4 login' commands are done to enable replication. Note that this part of the processing will fail if the correct super user password is not stored in the standard SDP password file, /p4/common/config/.p4passwd.p4_N.admin After starting the server, a local 'p4 trust' is done if needed, and then a 'p4login -service -v' and 'p4login -v'. By default, the p4d_N service is started, but the p4broker_N service is not. Specify '-b' to restart both services. Finally, the offline_db is rebuilt using the same specified checkpoint and journals. ARGUMENTS AND OPTIONS: Specify the path to the checkpoint file or directory to load. Exactly one checkpoint must be specified. If a checkpoint file is specified, a serial checkpoint replay will be done. If a checkpoint directory is specified, a parallel replay will be done using the individual files in the directory. For checkpoint files: The file may be a compressed or uncompressed checkpoint, and it may be a case sensitive or case-insensitive checkpoint. The checkpoint file must have a corresponding *.md5 checksum file in the same directory, with one of two name variations: If the checkpoint file is /somewhere/foo.gz, the checksum file may be named /somewhere/foo.gz.md5 or /somewhere/foo.md5. For checkpoint directories: This option is required unless the '-latest' option is used. [ ...] Specify the path to the one or more journal files to replay after the checkpoint, in the correct sequence order. -latest Specify this as an alternative to providing a specific checkpoint file or directory. The script will then search for the latest *.md5 file in the standard checkpoints directory and use that to replay. The most recent *.md5 file determines which checkpoint to load, be it a file or directory (for parallel checkpoints). This option is mutually exclusive with '-latest_jnls'. -latest_jnls This option is similar to '-latest'. However, with '-latest_jnls', in addition to replaying the latesst checkpoint, any subsequent numbered journals available in the standard checkpoints directory are also replayed. This option will only replay numbered journals, not the live P4JOURNAL file. However, if the $P4JOURNAL is provided, then it will be replayed after all available numbered journals are replayed. This option is mutually exclusive with '-latest'. If used with '-jo', where the checkpoint and possibly some numbered journals will already have been replayed into P4ROOT, then the meaning of this option changes. It will replay needed numbered journal up to the latest available, so long as those journals appear in the standard checkpoints directory with the usual naming convention. With this option, the journal needed are calculated based on the journal counter stored in database in the P4ROOT dir. -R Specify '-R' to remove db.* files in P4ROOT rather than moving them aside. By default, databases are preserved for possible future for investigation. A folder named 'MovedDBs.' is created under the P4ROOT directory, and databases are moved there. Keeping an extra set of databases requires sufficient disk space to hold the extra set of db.* files. If -R specified, old databases in P4ROOT are removed, along with state* and other files, and the server.locks directory. -F When replacing an existing set of db.* files, a safety factor is used. This is simply the factor by which the size of pre-existing databases is multiplied when comparing against available disk space. Specify '-F 0' to disable the safety factor check. The disk space safety check is only meaningful if P4ROOT was previously populated with a full set of data. Specifying a nubmer greater than 1, say 1.2 (the default) gives more breathing room. Specifying a value lower than 1, say 0.95, may be OK if you are certain the expanded-from-a-checkpoint db.* files are significantly smaller than size the prior set of db.* files. This option is mutually exclusive with '-R'. If '-R' is used, databases are removed, and there is no need to calculate disk space. -i Specify the SDP instance. This can be omitted if SDP_INSTANCE is already defined. -s Specify the ServerID. This value is written into $P4ROOT/server.id file. If no $P4ROOT/server.id file exists, this flag is required. If the $P4ROOT/server.id file exists, this argument is not needed. If this '-s ' is given and a $P4ROOT/server.id file exists, the value in the file must match the value specified with this argument. -t Specify the replica type tag if the checkpoint to be loaded is for an edge server or replica. The set of valid values for the replica type tag are defined in the documentation for mkrep.sh. See: mkrep.sh -man If the type is specified, the '-s ' is required. If the SDP Server Spec Naming Standard is followed, the ServerID specified with '-s' will start with 'p4d_'. In that case, the value for '-t edge' value is inferred, and '-t' is not required. If the type is specified or inferred, certain behaviors change based on the type: * If the type is edge, only the correct edge-specific subset of database tables are loaded. * The P4ROOT/license file check is suppressed unless the type is ha, ham, fs, for fsm (standby replicas usable with 'p4 failover'). Do not use this '-t ' option if the checkpoint being loaded is for a commit server. For an edge server, an edge seed checkpoint created with edge_dump.sh must be used if the edge is filtered, e.g. if any of the *DataFilter fields in the server spec are used. If the edge server is not filtered by means other than being an edge server (for which certain tables are filtered by nature), a standard full checkpoint from the commit can be used. For a filtered forwarding replica, a proper seed checkpoint must be loaded. This can be created on the commit using key options to p4d, including '-P -jd ] -verify "Verify Options" [-delay ] Specify '-verify' to initiate a call to 'p4verify.sh' after the server is online. On a replica, this can be useful to cause the server to pull missing archive files from its P4TARGET server. If this load_checkpoint.sh script is used in a recovery situation for a commit server, this '-verify' option can be used to discover if archive files are missing after the metadata is recovered. The 'p4verify.sh' script has a rich set of options. See 'p4verify.sh -man' for more info. The options to pass to p4verify.sh can be passed in a quoted list, or '-verify default' can be used to indicate these default options: -o MISSING By default, a fast verify is used if the p4d version is new enough (2021.1+). See 'p4verify.sh -man' for more information, specifically the description of the '-o MISSING' option. In all cases, p4verify.sh is invoked as a background process; this load_checkpoint.sh script does not wait for it to complete. The p4verify.sh script will email as per normal when it completes. The optional delay option specifies how long to wait until kicking off the p4verify.sh command, in seconds. The default is 600 seconds. This is intended to give the replica time get get caught up with metadata before the archive pulls are scheduled. The delay is a workaround for job079842. This option is cannot be used with '-no_start'. -c Specify that SSL certificates are required, and not to be generated with 'p4d_N -Gc'. By default, if '-c' is not supplied and SSL certs are not available, certs are generated automatically with 'p4d_N -Gc'. -l Specify that the server is to start without a license file. By default, if there is no $P4ROOT/license file, this script will abort. Note that if '-l' is specified and a license file is actually needed, the attempt this script makes to start the server after loading the checkpoint will fail. If '-t ' is specified, the license check is skipped unless the type is 'ha', 'ham', 'fs,' or 'fsm'. Replicas that are potential targets for a 'p4 failover' need a license file for a failover to work. -r Specify '-r' to replay only to P4ROOT. By default, this script replays both to P4ROOT and the offline_db. -no_start Specify '-no_start' to avoid starting the p4d service after loading the checkpoint. This option is cannot be used with '-verify'. -no_xu Specify '-no_xu' to skip the 'p4d -xu' step that upgrade the database schema. By default, a 'p4d -xu' is done to help ensure the service can be started with the current p4d binary after the checkpoint is replayed. If the p4d binary used to replay the checkpoint is a newer major version than the one used to create the checkpoint, the service will not start after the replay until the 'p4d -xu' step is done. If this '-no_xu' option is used and the p4d binary is a newer major version, have a plan to get the 'p4d -xu' done before the service is started. In EXAMPLES below, see the example titled "Multi Pass Replay of Checkpoints and Journals" for an example of using this option as part of a migration procedure. -jo [ ...] Specify '-jo' to replay only one or more numbered journals without first replaying a full checkpoint. With this option, the cleanup that normally occurs before the replay is disabled. The db.* and state* files in P4ROOT, as well as P4LOG and P4JOURNAL files, etc. are left in place. With '-jo', the paths to journal files must be specified. This option is mutually exclusive to the similar option '-jo_latest'. This option implies '-r'. -jo_latest Specify '-jo_latest' to replay only one or more numbered journals without first replaying a full checkpoint. With this option, the cleanup that normally occurs before the replay is disabled. The db.* and state* files in P4ROOT, as well as P4LOG and P4JOURNAL files, etc. are left in place. With '-jo_latest', numbered journals to replay are calculated and determined, not specified as parameters. This option is mutually exclusive to the similar option '-jo'. This option implies '-r'. -b Specify '-b' to start the a p4broker process (if configured). By default the p4d process is started after loading the checkpoint, but the p4broker process is not. This can be useful to ensure the human administrator has an opportunity to do sanity checks before enabling the broker to allow access by end users (if the broker is deployed for this usage). -y Use the '-y' flag to bypass an interactive warning and confirmation prompt. -L Specify the path to a log file. By default, all output (stdout and stderr) goes to: /p4//logs/load_checkpoint..log NOTE: This script is self-logging. That is, output displayed on the screen is simultaneously captured in the log file. Do not run this script with redirection operators like '> log' or '2>&1', and do not use 'tee.' -si Operate silently. All output (stdout and stderr) is redirected to the log only; no output appears on the terminal. -d Set debugging verbosity. -D Extreme debugging verbosity using bash 'set -x' mode. HELP OPTIONS: -h Display short help message -man Display man-style help message USAGE TIP: All the non-interactive examples below illustrate the practice of using redirects to create an extra log file named 'load.log' in the $LOGS directory for the instance. This load.log file is identical to, and in addition to, the standard timestamped log generated by this script. The intent of this practice is to make it easier to find the log for the last checkpoint loaded on any given server machine. This convention is only useful if used consistently. Several examples below illustrate the instance option, '-i' option to specify the SDP instance. This is optional and can safely be omitted in an environment where the standard SDP shell environment is sourced on login, and where there is only a single instance on the server machine. EXAMPLES: EXAMPLE 1: Non-interactive Usage Non-interactive usage (bash syntax) to load a checkpoint: nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -y < /dev/null > /p4/1/logs/load.log 2>&1 & Then, monitor with: tail -f $(ls -t $LOGS/load_checkpoint.*.log|head -1) EXAMPLE 2: Checkpoint Load then Verify, for the SDP Instance alpha. Non-interactive usage (bash syntax) to load a checkpoint followed by a full verify of recent archives files only with other options passed to verify.sh: nohup /load_checkpoint.sh /p4/alpha/checkpoints/p4_alpha.ckp.95442.gz -i alpha -verify -recent -nu -ns -y < /dev/null > /p4/alpha/logs/load.log 2>&1 & EXAMPLE 3: Load Checkpoint and Journals Non-interactive usage (bash syntax) to loading a checkpoint and subsequent journals: nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz /p4/1/checkpoints/p4_1.jnl.4025 /p4/1/checkpoints/p4_1.jnl.4026 -i 1 -y < /dev/null > /p4/1/logs/load.log 2>&1 & Then, monitor with: tail -f $(ls -t $LOGS/load_checkpoint.*.log|head -1) EXAMPLE 4: Interactive usage. Interactive usage to load a checkpoint with no license file. /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -l With interactive usage, logging still occurs; all output to the screen is captured. Note that non-interactive usage with nohup is recommended for checkpoints with a long replay duration, to make operation more reliable in event of a shell session disconnect. Alternately, running interactively in a 'screen' session (if 'screen' is available) provides similar protection against shell session disconnects. EXAMPLE 5: Seed New Edge Seeding a new edge server. nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -s p4d_edge_syd < /dev/null > /p4/1/logs/load.log 2>&1 & WARNING: While this script is useful for seeding a new edge server, this script is NOT to be used for recovering or reseeding an existing edge server, because all edge-local database tables (mostly workspace data) would be lost. To recover an existing edge server, see the recover_edge.sh script. EXAMPLE 6: Seed New Edge and Verify Seeding a new edge server and then do a verify with default options. nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -s p4d_edge_syd -verify default < /dev/null > /p4/1/logs/load.log 2>&1 & EXAMPLE 7: Load a Parallel Checkpoint on an Edge and Verify Recent This non-interactive example loads a parallel checkpoint directory. The usage difference is that the checkpoint path provided is a parallel checkpoint directory rather than a single checkpoint file. This example loads the checkpoint for a new edge server, and verifes only the most recent 3 changes in each depot. The delay before calling p4verify.sh, 10 minutes (600) by default, is shortened to 5 seconds in this example. nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025 -i 1 -s p4d_edge_syd -verify "-o MISSING -recent=3 -ns -L /p4/1/logs/p4verify.fast_and_recent.log" -delay 5 -y < /dev/null > /p4/1/logs/load.log 2>&1 & EXAMPLE 8: Multi Pass Replay of Checkpoints and Journals In this example, we want to use a multi-pass procedure involving replay of a checkpoint at one point in time, and then later replay subsequent numbered journals later. This method can be useful to reduce downtime required for migration procedures involving a checkpoint replay if the checkpoint replay takes a while, e.g. a few hours or more. The gist of the approach is to replay the checkpoint a day or so a head of the scheduled maintenance. Then replay subsequent numbered journals each day after. Then in the maintenance window, replay just the last numbered journal from the old environment in the new environment. This approach involves a few options: * When the checkpoint is replayed, '-no_start' and '-no_xu'. Either specify the path to the checkpoint, or use '-latest'. * When the numbered journals are replayed in days leading up to the maintenance window, use the '-jo_latest' option to replay only a numbered journal. * During the maintenance window, load any final numbered journals, then start the service. Pass 1, 3 days before maintenance: nohup load_checkpoint.sh -latest -no_start -no_xu -r -y < /dev/null > /p4/1/logs/load.log 2>&1 & Pass 2, 2 days before maintenance: nohup load_checkpoint.sh -jo_latest -no_start -no_xu -y < /dev/null > /p4/1/logs/load.log 2>&1 & Pass 3, 1 day before maintenance: nohup load_checkpoint.sh -jo_latest -no_start -no_xu -y < /dev/null > /p4/1/logs/load.log 2>&1 & Pass 4, during the maintenance window: nohup load_checkpoint.sh -jo_latest -y < /dev/null > /p4/1/logs/load.log 2>&1 &