SDP Legacy Upgrade Guide (for Unix)

The SDP should be upgraded prior to the upgrade of Helix Core (P4D). If you are planning to upgrade P4D to or beyond P4D 2019.1 from a prior version of P4D, you must upgrade the SDP first. If you run multiple instances of P4D on a given machine (potentially each running different versions of P4D), upgrade the SDP first before upgrading any of the instances.

The SDP should also be upgraded before upgrading other Helix software on machines using the SDP, including P4D, P4P, P4Broker, and the 'p4' command line client on the server machine. Even if not strictly required, upgrading the SDP first is strongly recommended, as SDP keeps pace with changes in the p4d upgrade process, and can ensure a smooth upgrade.

See the SDP Guide (Unix) for instructions on how to upgrade Helix binaries in the SDP structure after the SDP has been upgraded to 2020.1 or later.

The SDP is often forward- and backward-compatible with P4D versions. However, for best results they should be kept in sync by upgrading SDP before P4D. This is partly because the SDP contains logic to upgrade P4D, which can change as P4D evolves.

The SDP is aware of the P4D version(s) it manages, and has backward-compatibility logic to support older versions of P4D. This is guaranteed for supported versions of P4D. Backward compatibility of SDP with older versions of P4D may extend farther back then officially supported versions, though without the "officially supported" guarantee.

There are several methods for upgrading to a new version of the SDP:

Presumably if you are reading this, you are intending to upgrade the SDP. During planning, you’ll want to decide if you want to upgrade only the SDP. Or you may want to upgrade Helix Core software in the same maintenance window.

Tactically, the SDP upgrade is done first. However, upgrading SDP and Helix Core can be done in the same maintenance window, so both upgrade tasks can be done in one upgrade session. Alternately, Helix Core can be upgraded at a later date after the SDP upgrade.

Key questions to answer during upgrade planning are:

Most upgrades to SDP to 2020.1 will require downtime for the Helix Core (p4d) server, even if p4d is not being upgraded. The only exception to requiring downtime is if the current SDP is r2019.x and your SDP is not configured to use systemd, i.e. does not have /etc/systemd/system/p4*.service files. (Most 2019-2021 era SDP deployments use systemd, so this is a narrow exception.)

If the current SDP and P4D versions are both r2019.1 or later, the downtime can be brief. The scripts can be upgraded while the server is live, stopping p4d only long enough to change the systemd *.service files with new ones.

If your SDP is older than 2019.1, other changes will be needed depending on your SDP version, which will extend the downtime for p4d needed to upgrade the SDP. Read on to get a sense for what steps are required, which vary based on your SDP version.

If you are upgrading P4D along with the SDP, and your P4D version is older than 2019.1, live checkpoints are recommended after the upgrade is complete. This can significantly extend downtime required. The "to-or-thru P4D 2019.1" upgrades involve significant Helix Core database structural changes. If your Helix topology includes edge servers, you’ll want to account for taking checkpoints of the edge servers in "to-or-thru 2019.1" upgrade planning; edge checkpoints can occur in parallel with the checkpoint on the master server to reduce overall upgrade process duration.

Early in your planning, you’ll want to take stock of all server machines on which the SDP is to be upgraded.

For each machine, you’ll need to be aware of what SDP instances need to be upgraded on that machine.

For each instance on any given machine, determine what servers or services are in place. For example, a given machine might be a master p4d server for one instance, a replica for another, and a simple proxy for a third instance.

You will need to be aware of the three standard SDP mount points. While referred to as "mount points" in this document, in any given installation, any or all of the three SDP mount points may be simple directories on the root storage volume, or symlinks to some other storage volume. In some installations, fewer than three volumes were used, and in some cases 4 were used (2 for metadata). In some cases the operating system root volume was used as one of the volumes. Investigate and be aware of how your installation was configured. Comparing the output of pwd and pwd -P in the same directory can be informative.

The mount points do not necessarily need to be changed during the SDP upgrade process, as the SDP structural design has always and remains flexible with respect to mount point names. However, understanding whether the "mount points" are actual mount points, regular directories or symlinks is something to be aware of for detailed planning.

In the examples below, the modern SDP mount point names are used:

Depending on the version of the SDP, the above values may be used, or earlier defaults such as:

In some cases, custom values were used like /p4depots, /p4db, /p4jnl, etc. In these cases, it is important to know what standard names are referred to by the local names.

In the sample steps in this document, adapt the steps to use your local values for mount point names to the new values.

If your site uses two volumes for metadata, /hxmetadata1 and /hxmetadata2, continue using those same names.

If your SDP is 2019.1 or newer, skip this section.

Become familiar with the Pre-SDP 2019.1 and SDP 2019.1+ structures. (Note: This is not related to the P4D version).

A topology option with SDP deployments is to share the /hxdepots mount point across machines, e.g. with NFS. SDP upgrade procedures involve updating the /p4/common directory that is physically on the /hxdepots volume. When updating the /p4/common on one machine, be aware that any changes will be immediately visible on all other machines that share from the same NFS network location.

If your SDP is 2019.1 or newer, skip this section.

In current and all legacy SDP installations, including all topology variations, the top-level /p4 directory is always a regular directory on the local machine. However, the instance directory, e.g. the 1 in /p4/1/root, might be a directory or a symlink depending on your SDP version.

In the modern SDP, the /p4/N directory (where N is the SDP instance, e.g. 1), is also a regular directory on the local machine. This /p4/N directory is referred to as the P4HOME directory for the instance.

In older versions of the SDP, the N was a symlink rather than a local directory. Check with a command like ls -l /p4/N to determine if the P4HOME dir is a symlink or a directory. If it is a symlink, the upgrade procedure will need to change the symlink to a directory.

If your SDP is 2019.1 or newer, skip this section.

In the modern SDP, the "Instance bin" directory, /p4/N/bin, is a directory. As with P4HOME, on older versions of the SDP, /p4/N/bin may be a directory or symlink. Check with a command like ls -l /p4/N/bin to determine if that is a directory or symlink, depending on how old the SDP is.

You will need to be aware of your operating system user (OSUSER) that p4d runs as in your environment.

The sample upgrade steps below assume that Perforce runs as the perforce operating system user, which is typical. You do not need to change it, but you will need to adapt the samples below if your OSUSER is something other than perforce.

The OUSER user should dedicated to operating the Perforce Helix Core and related Helix services.

In modern installations, the default home directory is /home/perforce, though in some older installations the home directory is /p4. In either case, this does not need to be changed during the upgrade process.

Depending on how old the SDP in place is, the structure will have either Fixed or Variable Metadata Symlinks. Determine which you have.

To determine this, login as the OSUSER (e.g. perforce), and run a command like this sample (for instance 1):

The root and offline_db will be always symlinks in all versions of the SDP. However, they might be fixed or variable.

Variable Metadata Symlink References

If one of the symlinks points to a directory ending in db1, and the other in db2 (it doesn’t matter which is pointing to which), you have variable metadata symlinks.

Fixed Metadata Symlink References

If the target of the root and offline_db symlinks points to directories ending in the same names, i.e. root and offline_db, then you have fixed metadata symlinks.

You will need to determine whether you are running with the Systemd or SysV init mechanism.

Generally, newer operating systems like RHEL/CentOS 7 & 8, Ubuntu 18.04 and 20.04, and SuSE 12 & 15 will run with the systemd init mechanism. Older ones likely use the SysV init scripts (with some exceptions). If the command systemctl exists in your path, then you are running a system that supports systemd.

During planning, for each SDP instance, determine if depot spec Map: fields are used, and if the triggers table references the script SetDefaultDepotSpecMapField.py.

The modern SDP uses the server.depot.root configurable to define where depots are stored. In older versions of the SDP, the Map: field of the depot spec was used to track which depots were stored physically on which volumes. Starting in 2014, p4d introduced the server.depot.root configurable to define a standard location for all depots. The SDP took advantage of this, and stopped using the depot spec Map: fields. The Map: field was managed manually, or with the add of the (now obsolete) SetDefaultDepotSpecMapField.py.

For each instance, use the p4 depots command, and examine the Map: field of each depot spec to see if it has the default value or not. The default value is the name of the depot followed by a trailing /…​. If any Map: fields have values other than the default, we want to be aware of that during planning, as we’ll need to deal with them during the upgrade.

Following is a way to perform the analysis quickly:

If the text NONDEFAULT_MAP appears in the output, there are depots with non-default Map: fields that need to be addressed.

Use the p4 triggers -o command and see if your triggers table contains a reference to the SetDefaultDepotSpecMapField.py. If so, plan to remove that trigger during the upgrade.

Modern SDP uses and requires server.id files in the \$P4ROOT directory. If your SDP does not have server.id files in the \$P4ROOT directory, you’ll need to add them before starting p4d after upgrading.

When upgrading the SDP, you may want to redo replica, replacing any existing replicas with new ones generated with the SPD mkrep.sh script, and ensuring the ServerID values follow the SDP Server Spec Naming Standard and replication best practices. See the SDP Guide (Unix) to learn about using mkrep.sh.

If you do plan to do this, you may want to plan your overall upgrade procedure as follows:

Tackling the replication redo increases the benefit of the overall upgrade. However, if checkpoints take a long time in your environment, it also may significantly increase the required duration of the downtime in the maintenance window. This is due to the need for new checkpoints as needed during the replication recreation process. It is possible to defer replication upgrades to a separate maintenance operation.

Preparation steps are detailed below.

This section outlines sample steps for executing an actual upgrade after planning and preparations identified in Section 3.1, “Preparation” have been completed. The following typically performed in a scheduled maintenance window.

Execution steps detailed below.

Set the SDP_VERSION and SDP_DATE counters.

The SDP_VERSION counter is the SDP version.

Based on the planning done in Section 2.13, “Depot Spec Map Fields”, instances should be identified that require clearing of depot spec Map: fields and/or removal of the SetDefaultDepotSpecMapField.py trigger. For each instance, run the script as in the following example, replacing N with your actual instance name(s):

If you have more than one instance, run the command for each instance, changing the N value for each instance.

If changes to configuration changes were identified in Section 3.1.16, “Plan Changes to Configurables” and a configure_THIS_server.sh script was crafted, execute it now as the perforce user:

Re-enable crontabs as perforce on each machine with commands like these:

The crontab -l before and after is done for situational awareness and to confirm the crontab was loaded correctly.

At this point, do any sanity testing you desire to have confidence everything is OK.

Then, allow users back in by undoing whatever you did to lock users out in Section 3.2.1, “Lockout Users”.

Cleanup steps can occur after the upgrade. In some cases cleanup is done immediately following the upgrade; in other cases it may be deferred by days or weeks.

If the Helix Management System (HMS) is used to manage this installation, you should have custom site-specific documentation for upgrading the SDP that supersedes this documentation. If the file /p4/common/bin/hms exists at your site, you have an HMS-managed site. Contact Perforce Consulting for more information.

Note that HMS solutions are inherently custom and not officially supported, but can be fully automated for global Helix Core topologies.

If you desire to use DFM brokers for the legacy upgrade to SDP 2020.1 as outlined in this document, you cannot use the standard init scripts and procedures for managing the p4broker process, as those scripts and procedures will be disrupted during the upgrade process. Also, using a DFM broker will tie up the port number, creating a bit of complexity and interfering with scripts or procedures that start/stop the services. Care must be taken to avoid attempting to run the DFM broker during certain parts of the upgrade process.

With those caveats noted, if care is taken, it is possible to use a specially configured DFM broker for the legacy upgrade to SDP 2020.1. It can be done by making a copy of the new p4broker binary to some non-SDP-managed location (such as /home/perforce) and generating a disposable configuration file.

Here is a sample procedure to use a DFM broker. You will need to modify the sample configuration file to use the correct ports for your environment.

Here is a similar sample using a DFM broker on SSL:

Then, create a broker config file /home/perforce/dfm/p4broker.cfg with contents like this sample:

The DFM broker can then be started with a simple command line like:

It can be stopped with a command like:

This stop command will stop only the DFM broker. (Note: The -f in that command is not "force", but rather "full regex matching"; see man pgrep for more information.)

In your planning, account for starting and stopping the DFM broker at times to avoid interfering with operations that start/stop services.

If you have multiple machines in your environment, copy the /home/perforce/dfm directory to all machines, and account for starting/stopping the DFM broker on all machines in your planning.

If you are upgrading SDP from a prior patch of 2020.1, the procedure is the same as above, though most steps can be skipped (per details above). When upgrading from the 2020.1 GA release or any subsequent patch to the current latest patch, no Helix Core service downtime is required.

Note that customers upgrading from development DEV-PRE-RELEASE versions of SDP 2020.1 will require a small amount of downtime if the the systemd init mechanim is used, as the systemd *.services files will need to be updated.