SDP Windows to Linux Migration Guide

The migration can be nearly seamless. Typical impacts to users (humans and automation/bots) for a Windows to Linux migration include:

Other than being aware of the above, users to not need to do any special preparation for the cutover. For example, users do not need to be concerned about the state of files in their workspaces. Whatever state files in workspaces are in at the time of cutover — checked out to default or numbered pending changelists, shelved or not, etc. — is not affected by the cutover.

This document focuses on the failover style strategy. This entails creating a server spec (ServerID) for a standby of the commit server that we’ll call p4d_fs_linux, that will operate for a time as a Linux standby replica of the current production Windows commit server. Depending on various factors such as data scale, project priority and complexity, etc. this Linux replica of the Windows commit server may operate for days, weeks or even months before it is ready for the planned and scheduled failover that will promote the Linux standby server to become the new commit server.

This Failover strategy has several benefits:

While planning and preparation will take time and effort, the disruption to end users can be minimal.

The Failover strategy requires that the Windows Helix Core P4D service be at version 2019.1 (latest patch) or later. If it is not already at the latest patch available of 2019.1 or a later major version, see [_combining_upgrade_with_migration].

The largest single variable affecting effort for a Windows to Linux migration is effort dealing with custom automation, such as triggers or extensions. This can be literally zero effort if there are no custom triggers or extensions (or none that need to survive the migration). If porting and/or testing is required, that becomes a software development and testing project on its own that folds into the larger migration project.

Any custom triggers or extensions will need to be reviewed. Any that can’t be discarded will need to evaluated for porting and testing needs.

Triggers written in a native Windows langage such as batch or PowerShell, or operated as compiled .exe files, will need to be ported. Even triggers written in more portably languages such as Python or Perl will need testing and may need adjustment to operate in the Linux environment.

Extensions are written in Lua, the interpreter for which is entirely containdd in the Helix Core p4d binary itself. As such, custom extensions are less likely to require porting. However, they should still be evaluated and/or tested to be sure they have no Windows OS depenencies in their implementation.

Extensions provided by Perforce Software, such as those assoicated with Helix Swarm and the Helix Authentication Service, are inherently cross-platform and do not need to be ported.

Determine whether you have any custom softawre that runs directly on the Windows commit server machine. Custom automation that executes directly on the Windows server machine itself needs to be evaluated for porting and testing needs.

Any automation that merely connects as a client to the Windows p4d server, such as build server farms, need not be considered (other than possibly needing to login or trust p4d again and/or possibly change the P4PORT, as noted in Section 2.1, “Plan for User Impact”). The Linux server speaks the same protocol as Windows.

This section desribes a change that must be done on the Windows commit server early in the process, before a Linux standby replica can be setup.

Perforce Helix depot specs have a field named Map: that, if used, must be eliminated prior to the deployment of Linux standby servers. This applies to depots that contained archive files to be replicated, including depots of type local, stream, spec, extension, unload, tangent, and trait.

To list all of your depots and their types, run this command:

For a Linux server to replicate archives from a Windows commit server, the server.depot.root configurable must be set on the Windows commit server, and the Map: fields of all depots must be changed to the default value.

Check to see if the server.depot.root configurable is set with:

If that displays a value, then it is set. Otherwise it is not.

Next, check the Map: field of all depots with this command:

Ignore depots of type remote or archive.

The key to making the adjustment to depot spec Map: fields and the server.depot.root configurable non-disruptive is to understand that the p4d server will use the Map: field value if it is set to anything other than the default, and otherwise will fall back to the server.depot.root configurable as a default location, to find archive files for the given depot. If the value of the Map: field of any given depot is TheDepotName/…​, that means the Map: field value is not explicitly set, and thus it will fall back to using the server.depot.root configurable for that depot. If the value of the Map: field is anything other then TheDepotName/…​, then the Map: field value will be used to find archive files for that depot.

Your goal in making the change is to make it so that, immediately as/after the depot spec is updated to set the Map: field to its default value, the server.depot.root will cause the server to look in the exact same physical location for versioned files. Thus, there is no point in time during the procedure where the p4d server cannot locate its archive files, not even a nanosecond.

Before making changes, the singular server.depot.root value must be made to work for all existing depots. Make the single server.depot.root path work without actually moving any versioned files by using Windows directory symlinks. If individual depots are on different drives, put symlinks to all depots in the directory pointed to by the server.depot.root configurable so that p4d can find all depot files from that path. You may also find the Map fields use Windows UNC paths or if Windows junctions.

The Windows commit server must have the journalPrefix value set in order to set up the Linux replica. It can be set to any value that works to enable the p4d service to find numbered journals that have already been rotated. However, if no value is set at all, a value must be set before the Linux replica can be setup.

In some cases, the journalPrefix is undefined as a configurable, but a journalPrefix is provided as an argument to custom scripts that create checkpoints or rotate journals. In these cases, the journalPrefix for the Windows commit server should be set to the same used in custom checkpoint scripts.

In other cases, the journalPrefix is undefined, and so numbered/rotated journals (if there are any) appear in the default location in the P4ROOT directory. In these cases, an appropriate journalPrefix value should be set immediately, with a value set so that journal land in an appropriate directory.

A sample command to set the journalPrefix is:

Using the p4 configure command to interact with db.config is a good way, and in many cases the only way, to set various configuration items with a Helix Core server. However, there are certain settings that must not be defined with p4 configure, as they conflict with settings the SDP defines with shell environment variables on Linux.

Review the output of the command p4 configure show allservers and see if any of the following are set for the any config (the global defaults):

If any of these are set with p4 configure, the migration plan will need to deal with unsetting them after first ensuring they are set in some other way on the Windows service. Following is an example of how to replace how P4LOG is set displays in the output of p4 configure show all servers. Note that changing this requires a brief service restart to take effect.

Examine how checkpoints and journals are currently taken in the Windows environment (or if they are taken at all).

If journals on the Windows service are compressed, replication will not work. Replication requires uncompressed journals.

One thing to check for is whether scripts call the p4d -jc or p4d -jd command with the -z option (lowercase 'z'). If so, the '-z' should be changed to -Z (uppercase 'Z'). The lowerase -z compresses both checkpoints and numbered journal files, and thus is not suitable for replication. The uppercase -Z compresses the checkpoint file, but not the numbered journal files, and is intended for replication. Other changes to custom scripts that manage checkpoints in the Windows environment may be warranted.

Define the desired target P4D version. For illustration in this document, we will assume a target P4D version of 2023.1; 2023.2 or later will also be appropriate.

Since this document is about Windows to Linux migrations, the data set will naturally and necessarily be case-insensitive at the start of the project. This document does not discuss case sensitivity change, as it is unnecessary for a Windows to Linux migration. If there is a desire to become case-sensitive (for example, to support Linux clients), we advise deferring that as a separate project to be done after the Windows to Linux migration is complete.

A Windows to Linux migration that preserves the original case-insensitive behavior, as described in this document, is minimally disruptive. A case sensitivity conversion is best to defer until the Windows to Linux migration, for several reasons:

Generally speaking a case sensitivity conversion is more complex than a Windows to Linux conversion, sufficiently so that we advise relegating case sensitivity conversion to a separate project from the Windows to Linux migration. The case sensitivity conversion, if done at all, can be started after the Windows to Linux migration is complete. The case sensitivity involves doing neurosurgery on your Helix Core data set using the p4migrate utility.

Further discussion on case sensitivity conversions is outside the scope of this document.

In some cases, Linux proxies will have existed with a Windows commit server all along, as running proxies on Linux is advisable even in a topology with a Windows servers (for some of the same reasons that a Windows to Linux migration is popular, such as much faster native filesystems).

Any Windows should be migrated to Linux as well. However, while strongly discouraged, a Windows p4p (proxy) can remain in place with a Linux p4d server topology (so long as it operates in case-insensitive mode, which we assume in this document).

Helix Brokers should be migrated to Linux as well. However, while strongly discouraged, a Windows p4broker can remain in place with a Linux p4d server topology.

If brokers are configured with any custom software (broker "filter" scripts), porting this software to Linux should be accounted for in planning.

Every Helix Core topology will have exactly one commit server. If there is only a single server in the topology, it is the commit server. It may have additional p4d servers that extend the topology. Following are types of p4d servers (various typess of replicas) and their implications for a Windows to Linux migration:

See the definition of Instance in SDP parlance.

Set the intended SDP instance name. For purposes of this document, we’ll use the SDP default instance name of 1.

Reminder, setting a value for journalPrefix on the Windows commit server and dealing with the server.depot.root must be done before setting up the Linux replica. See:

Address the above items before moving forward.

On the Windows commit server, create a server spec to represent p4d on Linux. Call it p4d_fs_linux. Run this command:

Set these field values:

Next, determine a P4PORT value that can be used from the Linux replica server machine to reference the Windows commit server. Getting this to work may entail opening a firewall rule on the Windows commit server to ensure that the Linux server can reach it on whatever port p4d runs on (often 1666).

Then define other configurables with commands like these, to be run on your Windows commit server:

Next, create the replication servie user, and set a password for it (generate or think of a password first):

Store the password securely, however you store passwords (e.g. in some kind of password vault).

Next, add the svc_p4d_fs_linux user to a group name ServiceUsers, and ensure that group has a Timeout value set to unlimited. Then add this line near the bottom of the Protections table:

Once the above metadata change are complete, the Linux replica is defined. The next routine checkpoint to be taken in the Windows environment will have the definition of the Linux replica "baked in", and thus it can be used to seed the Linux replica.

EDITME - Add content here.

Once the Linux replica are setup, a variety of strategies can be used to transfer archive files.

Plan to execute about 3 iterations of p4verify.sh on Linux, to get p4d to pull the archives. The first pass, starting with no archive files, is to start a bulk pull. That could take hours, days or weeks depending on data scale. Also, it may need some nudging, clearing and resetting the replication "pull queue."

The second to fill in gaps, and the 3rd pass should be clean.

Depending on scale of data, you may want to consider using outside-p4d mechanisms for transferring some archives (especially the .gz files, ,v files should be transferred with p4 pull ideally).

There are lots of variations on how to get the archives files there. This document focuses primarily on using replication (i.e. replica startup.N threads calling p4 pull commands) for these reasons:

The above noted, for initial, first-time bulk pulls of Terabytes of data, a Windows port of rsync might be considered for pulling .gz archives files (and only those). It may well pull bulk archives faster than replication. However, rsync entails extra setup effort (not covered in this document), and also has greater risk of impacting the production Windows environment.

There are many options here; somehow or other the goal is to get the archive files in place so p4verify.sh is happy.

During the migration to Linux and the Server Deployment Package, a script should be crafted that defines best practice configurables to be set during the cutover, immediately after the failover and upgrade. The Linux server testing should be done with these best practices in place.

The SDP contains a configure_new_server.sh that defines best practices for a new server, mostly by running many p4 configure set commands to set various configurables. This should not be used exactly as it is because, as the name implies, it is intended for an entirely new server. However, it should be used as a guide to develop a custom script specific to this migration. Typically the procedure starts by creating a script named something like configure_THIS_server.sh, initially copied from the stock SDP script configure_new_server.sh.

Then the configure_THIS_server.sh script is edited in light of the output of p4 configure show on the commit server. Generally the editing involves:

Ultimately, the goal of developing this script is to capture the results of the process of reviewing and applying best practice configurables to the current data set. The script typically takes a few hours to develop and review, but can be executed in mere seconds during the cutover. This script should be exercised during the dry run and again for the Production Cutover.

At least one Dry Run is required to confidently execute a migration. Plan to have at least one.

In the dry run, the p4 failover command is NOT used, nor is user traffic directed to the Linux server. Instead, the Linux service is stopped, and the $P4ROOT/server.id file is simply hand-edited to be the ServerId the the commit server. Then the service is restarted.

At that point, the Linux commit server will believe itself to be the new commit server, even though users will still be using the Windows server for real work. Then the Linux server can be tested in various ways:

In addition to the value of exercising a procedure that is largely similar to the Production Cutover procedure, the dry run procedure is useful in gathering timing info related to various steps in the process. Endeavour to capture timing of key steps in the process.

After all Dry Runs are declared complete, the Linux server environment can be reset to be a replica of the Windows environment once again, and the archive pull process repeated (which should be much faster this time around, as most archives are in place).

Craft the Production Cutover procedure with learnings from the dry run(s).

The following is a sample cutover procedure for a topology with a commit server and an edge server, with custom triggers that have been ported to Linux.

The sample instructions assume the perforce OS user on the Linux servers has been setup with the proper shell environment, specifically that the ~/.bashrc has sourced the /p4/common/bin/p4_vars file with the appropriate SDP instance parameter.

The preparation for this sample cutover scenario would have included:

STEP 1: Verify Replication

Verify that replication is healthy on the Linux replica, the Windows edge, and the Linux replica of the Windows edge server.

STEP 1: Verify Replication

Verify that replication is healthy on the Linux replica, the Windows edge, and the Linux replica of the Windows edge server.

STEP 2: Checkpoint Linux Replicas

On the Linux standby of the Windows commit server, and separately and in parallel on any Linux standbys of Windows edge servers, request a checkpoint:

Next, on the Windows commit server, execute a journal rotation:

Once this command has been run, it will trigger the Linux server to start taking a checkpoint. On the Linux server, a checkpoint should immediately appear in the checkpoints directory.

Monitor the checkpoints directory and await the appearance of a *.md5 with the same number as the checkpoint. The existence of the MD5 file indicates the successful completion of the checkpoint process.

Use the watch utility and wait until the *.md5 file appears on the Linux standby of the Windows commit:

In parallel, use the watch utility and wait until the *.md5 file appears on the Linux standby of the Windows edge:

STEP 3: Replay Checkpoint to offline_db.

On the Linux commit and edge servers, replay their local checkpoint files created in the prior step into to the offline_db. This can be done regardless of whether the local p4d service is replicating or even online at all. The replay to the offline_db is nether affected by nor disruptive to the p4d service. It can be done on the commit and the standby in parallel, though can only be one on each machine only after the checkpoint completes and the local *.md5 file exists on the given machine.

Monitor until completion with:

This preparation of the offline_db allows the Linux service to start operation with daily checkpoints with a reasonably current offline_db. It may a day or so behind by the time the cutover occurs. (If for some reason the cutover is postponed by more than a few days, repeat this procedure of creating and replaying checkpoints on Linux to keep the offline_db reasonably current. Repeating the procedure will replace the offline_db with a more recent checkpoint).

STEP 1: Verify Replication

Verify that replication is healthy on the Linux replica, the Windows edge, and the Linux replica of the Windows edge server.

STEP 2: Disabled Scheduled Tasks

On the old Windows commit and edge server machines, disable any Scheduled Tasks related to backups or checkpoints. Also ensure no long-running checkpoint or backup operations are in progress that won’t be complete by the time of the intended cutover.

STEP 3: Disable Crontabs

On the new Linux commit and edge server machines, save and then disable all crontabs intended for routine production operation (and they may have been left on during dry runs).

STEP 4: Lockout Users with Protections

STEP 5: Stop Services

STEP 6: Start Services

STEP 7: Rotate Journal

STEP 8: Verify Replication

STEP 9: Failover Edge Server

STEP 10: Apply Metadata Changes for Linux

Apply metadata changes required for operation on Linux and commit server now being on SDP.

STEP 11: Failover Commit Server

STEP 12: Do Sanity Tests

STEP 13: Decide: GO/NO GO

STEP 14: Restore Default Protections

STEP 15: Direct User Traffic to Linux

STEP 16: Enable crontabs