[DRAFT] Perforce P4D Sample Storage Setup

Todo Items

Test script operation

Preface

This document illustrates a basic set of commands to setup data storage using Linux Volume Manager (LVM).

LVM is the preferred method of setup for Perforce volumes. Both Physical servers and Cloud/VM servers benefit from features of LVM covered in this document.

The goal in these examples is to configure three LVM storage volumes separate from the OS volume on a server destined to become a Helix Core server. At the start of this procedure, empty volumes with no data are formatted.

Console sessions

When console sessions are shown in the text:

Normal user shown with $ prompt

$ ls -lR

Root user shown with # prompt

# systemctl daemon-reload

Please Give Us Feedback

Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to [email protected].

1. Why use LVM ?

Some of the benefits of using LVM over standard disk partitions are:

Expansion of physical volumes is not limited to the size remaining on the disk. Additional disks can be added to the Volume Group as needed for expansion.
Volume name/UUID are embedded into the media. This helps identify snapshots and cloned volumes used for migrations.
LVM avoids need for UUID’s in /etc/fstab, as volumes are always assigned unique device names.

LVM also supports a volume snapshot and revert feature. Howver, LVM snapshots incur a significant performance penalty due to the Copy-On-Write (COW) method utilized. For this reason LVM snapshots are not recommended for use on Perforce volumes.

2. Sample Storage Setup for SDP - Mounts and Storage

2.1. Starting State

This procedure assumes the following start state:

Destination Server has a basic install of Ubuntu 24.04. ( RHEL/Rocky9 TBD ) with following installed:
- LVM2, if missing run apt install lvm2
- XFS, if missing run apt install xfsprogs
Three separate storage volumes are attached Server / VM. (in addition to the OS root volume), intended to become /p4depots, /p4logs, and /p4metadta:
- /p4depots - Give this as much space as you think you’ll need. This is highly variable. You may want 30G source code projects, or 1T or more for virtual production or game development. Size can be increased easily and non-disruptively later, so you don’t need to overprovision (and overpay) for storage.
- /p4metadata - Use 25G to start. This needs to hold (2) copies of the database.
- /p4logs - Use 20G to start. Typically low usage, but large enough to contain large journals during any occasional purge/obliterate operations.
Volumes may be Physical disks, Direct-Attached-Storage (DAS), EBS volumes on AWS, VMFS disks on a VM or other block storage.

There is no easy method of matching the device being attached to the assigned kernel name such as: nvme1n1 or nvme1n2. However if we choose disks of different sizes they can easliy be matched up with kernal assigned names.

For instance, in the example above, the 1000G volume nvme2n1 is for /p4depots, the 25G volume nvme1n1 is for /p4metadata and the 20G volume is for /p4logs

See Also: https://www.reddit.com/r/linuxadmin/comments/8cg1t4/benefits_of_lvm/

2.2. Storage Formatting and Mounting Procedure

First, become root with the sudo su - command, and then list the mounted storage before we mount the new volumes:

$ sudo su -

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.8G     0  1.8G   0% /dev/shm
tmpfs           1.8G   17M  1.8G   1% /run
tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   10G  1.6G  8.5G  16% /
tmpfs           356M     0  356M   0% /run/user/0

You don’t yet see the newly attached volumes, as they’re not yet mounted. But you can list them with lsblk:

# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   10G  0 disk
└─nvme0n1p1 259:1    0   10G  0 part /
nvme1n1     259:2    0   25G  0 disk
nvme2n1     259:3    0   20G  0 disk
nvme3n1     259:4    0  100G  0 disk

2.2.1. LVM

The Linux Volume Manager (LVM) is used in place of old style disk partitions, and is much more flexible. Volumes may be expanded while the system is running, and volumes may span more than one disk.

Ubuntu 24.04 installer includes LVM.

see: https://documentation.ubuntu.com/server/explanation/storage/about-lvm/#

This document provides a summary of only the basic commands to setup LVM following best practices.

LVM components

PV’s: Physical Volumes. PV volumes provide storage to their assigned VG Marking a disk as a PV makes it available to be assigned to a VG
VG’s: Volume Groups. VG are comprised of one or more PVs that supply storage to the VG The VG storage is comprised of all disks attached to it and is not limited to the size of any single disk.
LV’s: Logical Volumes LVs are composed by allocating space from a VG Disks with different performance characteristics should be separated into different VG’s so they may be assigned to appropriate LVs

LVM marks PV’s, VG’s and LV’s with unique UUIDs and includes the name of the VG or LV on the media.

2.2.2. Physical Volumes(PVs)

Normally the entire disk is marked as a PV, however in the past folks have also marked disk partitions as PVs.

The reason full disks are used without parititions , is that any later disk expansion can then avoid expanding that partition.

First mark each of the 3 new blank volumes as an PV.

[root@~]# pvcreate /dev/nvme1n1
Writing physical volume data to disk "/dev/nvme1n1"
Physical volume "/dev/nvme1n1" successfully created.

[root@~]# pvcreate /dev/nvme2n1
Writing physical volume data to disk "/dev/nvme2n1"
Physical volume "/dev/nvme2n1" successfully created.

[root@~]# pvcreate /dev/nvme3n1
Writing physical volume data to disk "/dev/nvme3n1"
Physical volume "/dev/nvme3n1" successfully created.

Display PV’s

You can display the new PV’s in short or long mode.

Short

# pvs
PV            VG   Fmt     Attr   PSize  PFree
/dev/nvme1n1       lvm2    ---   <25G    <25G
/dev/nvme2n1       lvm2    ---   <20G    <20G
/dev/nvme3n1       lvm2    ---   <1000G  <1000G

Long

# pvdisplay
... < many lines showing Name, size, UUID >

2.2.3. Volume Groups (VGs)

Next create VGs for each of the three types of storage.

We will name the 3 VGs as:

vg_p4metadata
vg_p4logs
vg_p4depots

Create the new 'VGs' and assign the appropriate PV with:

# vgcreate vg_p4metadata /dev/nvme1n1      (1)
Volume group "vg-p4metadata" successfully created

# vgcreate vg_p4logs /dev/nvme2n1          (2)
Volume group "vg-p4logs" successfully created

# vgcreate vg_p4depots /dev/nvme3n1        (3)
Volume group "vg-p4depots" successfully created

1	p4metdata was determined to use `nvme1n1` by matching disk size of 25GB
2	p4logs was determined to use `nvme2n1` by matching disk size of 20GB
3	p4depots was determined to use `nvme3n1` by matching disk size of 1000GB

Display VG’s

You can display the new VG’s in short or long mode.

Short

# vgs
VG              #PV  #LV  #SN  Attr   VSize   VFree
vg_p4metadata   x    0     0  wz--n-  <250G
vg_p4logs       x    0     0  wz--n-  <200G
vg_p4depots     x    0     0  wz--n-  <1000G

Long

# vgdisplay
... < many lines showing Name, Size, PV UUID >

2.2.4. Logical Volumes (LVs)

Finally create logical volumes LVs inside each VG, using all of the available storage.

# lvcreate -n lv_p4metadata -L 100%FREE  vg_p4metadata
Logical volume "lv_p4metadata" created.

# lvcreate -n lv_p4logs -L 100%FREE  vg_p4logs
Logical volume "lv_p4logs" created.

# lvcreate -n lv_p4depots -L 100%FREE  vg_p4depots
Logical volume "lv_p4depots" created.

Display LV’s

You can display the new LV’s in short or long mode.

Short

# lvs
LV              VG              Attr     LSize  Pool  Origin  Data% Meta%
lv_p4metadata   vg_p4metadata  -wi-a-    25G
lv_p4logs       vg_p4logs      -wi-a- 0  20G
lv_p4depots     vg_p4depots    -wi-a- 0  1000G

Long

# lvdisplay
... < many lines showing Name, Size, Status, LV UUID >

2.2.5. Formating LVM volumes

LVM physical block devices are mapped to Linux virtual block devices by the Device Mapper.

The device mapper will automatically create devices for the above LVM disks as follows:

/dev/mapper/vg_p4metadata-lv_p4metadata
/dev/mapper/vg_p4logs-lv_p4logs
/dev/mapper/vg_p4depots-lv_p4depots

The explicit LVM device names assigned makes formatting these volumes much safer.

Next, Format these new LVs as XFS.

# mkfs.xfs /dev/mapper/vg_p4metadata-lv_p4metadata
meta-data=/dev/mapper/vg_p4metadata-lv_p4metadata           isize=512    agcount=16, agsize=1638400 blks
           =                       sectsz=512   attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=1, rmapbt=0
           =                       reflink=1
  data     =                       bsize=4096   blocks=26214400, imaxpct=25
           =                       sunit=1      swidth=1 blks
  naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
  log      =internal log           bsize=4096   blocks=12800, version=2
           =                       sectsz=512   sunit=1 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0

# mkfs.xfs /dev/mapper/vg_p4logs-lv_p4logs
meta-data=/dev/mapper/vg_p4logs-lv_p4logs           isize=512    agcount=16, agsize=1638400 blks
           =                       sectsz=512   attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=1, rmapbt=0
           =                       reflink=1
  data     =                       bsize=4096   blocks=26214400, imaxpct=25
           =                       sunit=1      swidth=1 blks
  naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
  log      =internal log           bsize=4096   blocks=12800, version=2
           =                       sectsz=512   sunit=1 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0

# mkfs.xfs /dev/mapper/vg_p4depots-lv_p4depots
meta-data=/dev/mapper/vg_p4depots-lv_p4depots           isize=512    agcount=16, agsize=1638400 blks
           =                       sectsz=512   attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=1, rmapbt=0
           =                       reflink=1
  data     =                       bsize=4096   blocks=26214400, imaxpct=25
           =                       sunit=1      swidth=1 blks
  naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
  log      =internal log           bsize=4096   blocks=12800, version=2
           =                       sectsz=512   sunit=1 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0

Formatting the wrong device may destroy data !

2.2.6. Update /etc/fstab mount table

Make a backup copy of the /etc/fstab file, and then modify that file to create new volumes.

# cd /etc

# ls -l fstab*
-rw-r--r--. 1 root root 394 Nov 15 04:43 fstab

# cp -p fstab fstab.bak.2022-03-08

Edit fstab

Next add entries to /etc/fstab for the mount points.

# echo "/dev/mapper/vg_p4metadata-lv_p4metadata /p4metadata xfs defaults 0 0" >> /etc/fstab
# echo "/dev/mapper/vg_p4logs-lv_p4logs /p4logs xfs defaults 0 0" >> /etc/fstab
# echo "/dev/mapper/vg_p4depots-lv_p4depots /p4depots xfs defaults 0 0" >> /etc/fstab

The previous method of mounting Volumes in /etc/fstab` by their UUID values is not used with LVM. LVM uses UUID’s internally, and the Linux kernel Device mapper always maps UUID’s to the same device.

Update mounts

After changing /etc/fstab run this command up update mount info.

# systemctl daemon-reload

Run the systemctl daemon-reload command after each change to /etc/fstab or bad things might happen. It is also a good idea to mount new filesystems the first time with mount -a which should report any errors in /etc/fstab before rebooting the machine.

Proceed with creating empty directories that will be the "mount points" for the volume to be mounted.

# mkdir /p4depots /p4logs /p4metadata

Next, use the mount -a command. This will now assocate the mount points you just created with the storage device information that is now in that /etc/fstab file, and mount the volumes.

# mount -a

Then see if they are mounted. This is what victory looks like, with the /p4* volumes all mounted with desired sizes:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.8G     0  1.8G   0% /dev/shm
tmpfs           1.8G   17M  1.8G   1% /run
tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   10G  1.6G  8.5G  16% /
tmpfs           356M     0  356M   0% /run/user/0
/dev/mapper/vg_p4metadata-lv_p4metadata  120G  890M  120G   1% /p4metadata
/dev/mapper/vg_p4logs-lv_p4logs          100G  747M  100G   1% /p4logs
/dev/mapper/vg_p4depots-lv_p4depots      500G  3.6G  497G   1% /p4depots

At this point, you are ready to install the Server Deployment Package (SDP) software.

3. Volume Expansion

An LVM partition can usually be expanded while the system is running without any outage.

The procedure involves the following steps:

Expand the associated VG by one of following
1. Expand the underlying PV disk ( such as a virtual disk )
  1. Rescan the storage buss
  2. Resize the PV
  3. Extend the LV -or-
2. Add additonal disks to the VG ( typically physical disks )
  1. Rescan the storage buss
  2. Extend the VG
  3. Extend the LV

3.1. Expand VG by expanding underlying disk

In a virtual environment it is typically easy to expand the size of the underlying disk.

Even though this procedure can be run without downtime, to be safe, ensure you have a recent backup of the system.

Expand the virtual block storage device, by changing its size on AWS, VMware, or other environment.
Rescan the VM’s scsi-buss to pick up the change

Note replace nvme1n1 with your block device

$ sudo su
# echo 1 > /sys/block/*nvme1n1*/device/rescan_controller
# exit

//  Alternate procedure
$ sudo apt install scsi-tools
$ sudo rescan-scsi-bus -s

Resize the PV to use the new expaned size

### Note replace *nvme1n1* with your block device
$ sudo pvresize /dev/*nvme1n1*

Resize the LV to use the new storage

Note replace /dev/mapper/vg_p4depots-lv_p4depot with your device Run one of the following

$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4depots-lv_p4depots
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4logs-lv_p4logs
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4metadata-lv_p4metadata

Grow the Filesystem

Note replace /dev/mapper/vg_p4depots-lv_p4depots with your device

# xfs_growfs /dev/mapper/vg_p4depots-lv_p4depots     # Full Size

3.2. Expand VG by adding more disks

Even though this procedure can be run without downtime, to be safe, ensure you have a recent backup of the system.

Expand the VG by adding additional disks to the VG.
Hot-pluggable server disks are required to add without downtime. After adding the disk, rescan the VM’s scsi-buss to pick up the change

To determine the exact device name of the added disk, look at the kernel ring buffer for the expected device name immediatly after adding the disk.

$ sudo dmesg | grep nvme     # If device is expected to be nvme*

Note replace nvme5n1 with your block device shown with dmesg sudo dmesg # Look for new block device signature, e.g nvme5n1

$ sudo su
# echo 1 > /sys/block/*nvme5n1*/device/rescan_controller
# exit

#Alternate procedure
$ sudo apt install scsi-tools
$ sudo rescan-scsi-bus

Mark new Disk as a PV

Note replace nvme5n1 with your new device

$ sudo pvcreate /dev/*nvm5n1*

Extend the VG

Replace nvme5n1 with your new disk

$ sudo vgextend vg_p4metadata-lv_p4metadata /dev/*nvme5n1*
  Volume group "vg_p4metadata" sucessfully extended

Extend the LV

Replace the LV with your desired LV

$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4metadata-lv_p4metadata
  Size of logical volume "lv_p4metadata" changed fromm 205GB to 300GB
  Logical volume "lv_p4metadata" sucessfully resized.

3.3. LVM Storage setup script

For reference if device names are known in advance, the LVM setup procedure may be scripted as follows:

#!/bin/bash

#NOTE DRAFT,  error checking not fully included

# Devices to format  ( root on nvme1n1 )
DEV_META="/dev/nvme2n1"
DEV_LOGS="/dev/nvme3n1"
DEV_DEPOTS="/dev/nvme4n1"

# Mount points
MP_META="/p4metadata"
MP_LOGS="/p4logs"
MP_DEPOTS="/p4depots"

# VGs
VG_META=vg_p4metadata
VG_LOGS=vg_p4logs
VG_DEPOTS=vg_p4depots

# LVs
LV_META=lv_p4metadata
LV_LOGS=lv_p4logs
LV_DEPOTS=lv_p4depots


# Backup /etc/fstab
cp /etc/fstab /etc/fstab.bak.$(date +%F)

# If device unused, create PV's
for dev in $DEV_META $DEV_LOGS $DEV_DEPOTS; do
    if blkid "$dev" &> /dev/null; then
        echo "Warning: $dev already has a file system. Skipping mkfs.xfs."
    else
        echo "Creating PV on $dev"
        pvcreate "$dev"
    fi
done

# Create VG's
vgcreate $VG_META $DEV_META
vgcreate $VG_LOGS $DEV_LOGS
vgcreate $VG_DEPOTS $DEV_DEPOTS

#Create LV's
lvcreate -n $LV_META   -L 100%FREE $VG_META
lvcreate -n $LV_LOGS   -L 100%FREE $VG_LOGS
lvcreate -n $LV_DEPOTS -L 100%FREE $VG_DEPOTS


# Create mount points if they don't exist
[ ! -d "$MP_META" ] && mkdir -p "$MP_META"
[ ! -d "$MP_LOGS" ] && mkdir -p "$MP_LOGS"
[ ! -d "$MP_DEPOTS" ] && mkdir -p "$MP_DEPOTS"


# Add entries to /etc/fstab
echo "/dev/mapper/${VG_META}-${LV_META}  ${MP_META} xfs defaults 0 0" >> /etc/fstab
echo "/dev/mapper/${VG_LOGS}-${LV_LOGS}  ${MP_LOGS} xfs defaults 0 0" >> /etc/fstab
echo "/dev/mapper/${VG_DEPOTS}-${LV_DEPOTS}  ${MP_DEPOTS} xfs defaults 0 0" >> /etc/fstab

# Reload systemd daemon
echo "Reloading systemd to apply changes to /etc/fstab."
systemctl daemon-reload

# Mount all file systems
mount -a

# Verify mounts
mountpoint "$MP_DEPOTS" || echo "Error: $MP_DEPOTS is not mounted."
mountpoint "$MP_LOGS" || echo "Error: $MP_LOGS is not mounted."
mountpoint "$MP_META" || echo "Error: $MP_METADATA is not mounted."

# Display current mounts
df -h

4. Notes and References

A few possible issues that may come up are:

Mounting duplicate named Disks
- If you clone a disk and mount it to the same machine you will get an error because the UUID’s and LVM assigned name are the same.
- https://unix.stackexchange.com/questions/495669/how-to-mount-lvm-partitions-with-duplicate-names
Snapshot and revert before dangerous ops.

[DRAFT] Perforce P4D Sample Storage Setup - LVM

Todo Items

Preface

Console sessions

1. Why use LVM ?

2. Sample Storage Setup for SDP - Mounts and Storage

2.1. Starting State

2.2. Storage Formatting and Mounting Procedure

2.2.1. LVM

2.2.2. Physical Volumes(PVs)

Display PV’s

2.2.3. Volume Groups (VGs)

Display VG’s

2.2.4. Logical Volumes (LVs)

Display LV’s

2.2.5. Formating LVM volumes

2.2.6. Update /etc/fstab mount table

Edit fstab

Update mounts

3. Volume Expansion

3.1. Expand VG by expanding underlying disk

3.2. Expand VG by adding more disks

3.3. LVM Storage setup script

4. Notes and References