# Coordinated AWS and Azure snapshots The SDP snapshot job creates one coordinated backup set and confirms completion before expiring old backups. It records the current journal number as metadata. It does not stop Perforce or claim application consistency. Keep normal SDP checkpoint/journal backups and test recovery from a complete cloud set. Deploy the entire `Server/Unix/p4/common/cloud` tree. Python 3 and the relevant provider CLI must be available to the Perforce service account. The existing `aws/bin/snapshot.sh` initializes `p4_vars`, logging, login and the journal-counter read before invoking the shared `bin/sdp_snapshot.py` controller. Existing cron entries default to AWS: ```bash /p4/common/cloud/aws/bin/snapshot.sh 1 /p4/common/cloud/aws/bin/snapshot.sh 1 azure # Equivalent Azure entry point: /p4/common/cloud/azure/bin/snapshot.sh 1 ``` `SDP_CLOUD_PROVIDER=azure` also selects Azure when no second argument is supplied. `SDP_SNAPSHOT_RETENTION_DAYS` is a positive integer, defaulting to 90. A failed creation, validation or wait prevents expiration entirely. Deletion failures return failure and remain visible in `snapshot.log` and structured run output. No new strict shell modes are added. ## AWS setup and ownership Configure the service account's normal AWS profile/region or instance role. Give the selected EBS volumes exact `Name` tags `-root`, `-hxdepots` and `-hxlogs`, using the host name before its first dot. Each tag must select exactly one unique volume. All three must have a single active attachment to the same EC2 instance. Optional `SDP_AWS_INSTANCE_ID=i-...` pins that instance explicitly. The controller inspects the EC2 block-device mapping, excludes every unselected data volume, and handles boot-volume exclusion separately. More than 40 excluded data volumes fails before creation. One `create-snapshots` request produces the selected coordinated set, with ownership tags applied in that same request. AWS documents this operation as crash consistent across the selected volumes. [CreateSnapshots reference](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-snapshots.html) The tags are `SDPManagedBy=sdp-cloud-snapshot`, `SDPSchema=1`, `SDPAccount`, `SDPHost`, `SDPInstance`, `SDPSourceInstance`, `SDPRun`, `SDPJournal` and `SDPSetSize`, plus descriptive `Contents`, `Host` and `Name` tags. Every member shares the run ID and journal number; original `VolumeId` identifies the source disk. Ensure all Perforce data, journals and checkpoints to be recovered are on the selected disks. Network storage and other attached disks are not backed up. The job waits for every returned snapshot, then verifies the completed volume set, account and tags. The AWS waiter polls every 15 seconds and fails after 40 unsuccessful checks. If a large set exceeds that interval, inspect its logged run ID and completion state; the job will leave older backups intact. [Waiter reference](https://docs.aws.amazon.com/cli/latest/reference/ec2/wait/snapshot-completed.html) Expiration queries use `--owner-ids self` and all stable ownership tags. The controller verifies those fields again locally, groups snapshots by run ID, and only deletes old sets whose declared member count is complete and whose members are all completed. Legacy snapshots with only the old `Contents` tag, snapshots from other hosts/instances/accounts/source EC2 instances, and incomplete/failed sets remain untouched. Inspect retained incomplete sets manually. A deletion failure can leave part of an expired set, which is retained for review rather than treated as a complete backup on subsequent runs. The backup identity needs `ec2:DescribeVolumes`, `ec2:DescribeInstances`, `ec2:DescribeSnapshots`, `ec2:CreateSnapshots`, `ec2:CreateTags` and `ec2:DeleteSnapshot`; identity discovery calls STS `GetCallerIdentity`. Constrain create/delete permissions with the intended resources and ownership tags in IAM. Encrypted volumes also require the key permissions applicable to those volumes. Restore permissions should be granted separately to operators. ## Azure setup and ownership Authenticate the service account using the normal Azure CLI managed-identity or service-principal workflow. Configure these values in the environment sourced by the job: ```bash export SDP_AZURE_SUBSCRIPTION="" export SDP_AZURE_RESOURCE_GROUP="" export SDP_AZURE_VM="" # Optional; otherwise a deterministic dedicated name is derived: export SDP_AZURE_COLLECTION="" # Optional positive seconds for each provisioning/disk-copy wait: export SDP_AZURE_WAIT_SECONDS=3600 ``` The selected subscription is resolved to its UUID, then every Azure CLI call is pinned to it. The controller validates the VM's resource ID and unique VM ID. It includes the OS disk and all attached data disks in one `CrashConsistent` VM restore point. It rejects unmanaged, ephemeral, write-accelerated, shared, Ultra Disk and Premium SSD v2 configurations before creating a collection or point. This entry point supports Linux VMs outside scale sets. Azure Boost also does not support crash-consistent restore points; the provider error stops the job with no expiration and no independent-disk fallback. Confirm platform support before scheduling. [Azure support and limitations](https://learn.microsoft.com/en-us/azure/virtual-machines/virtual-machines-create-restore-points) A dedicated restore point collection is tagged with the same stable SDP scope fields plus the source VM UUID. An existing collection must already have the exact ownership tags and source VM ID; the script does not adopt or retag an unrelated collection. Missing collections are created with that ownership. Individual point names use `sdp-v1-j-`, because restore-point creation does not expose per-point tags. [Collection CLI reference](https://learn.microsoft.com/en-us/cli/azure/restore-point/collection?view=azure-cli-latest) After creation the job waits for successful provisioning and 100% background copy of every disk. It checks source VM identity, the exact source disk set and all disk restore point statuses. It then lists the owned collection with `--restore-points` and considers only its managed name format, matching VM UUID, crash-consistent successful state and age. Expired points are inspected again for completed disk copies before deletion. Other names, sources, collections and unfinished points are retained. The script never deletes a collection. [Restore-point CLI reference](https://learn.microsoft.com/en-us/cli/azure/restore-point?view=azure-cli-latest) Use a role scoped to the VM, disks and dedicated collection/resource group with read access to `Microsoft.Compute/virtualMachines` and `Microsoft.Compute/disks`, read/write on `Microsoft.Compute/restorePointCollections`, and read/write/delete on its `restorePoints` children, including disk restore point status reads. Collection discovery needs read access at resource-group scope. The scheduled job does not need permission to delete collections or VMs. Encryption-key access must remain valid for backup and restoration. Azure limits restore point count, creation rate and concurrent creation for a VM; provider failures preserve the previous backup sets. ## Recovery procedure Use an isolated recovery VM and disks. Select one complete run, restore every member of that run, and keep the original backup until verification succeeds. The journal number is metadata from before creation, not proof of a clean Perforce database shutdown. Validate filesystem and Perforce recovery using the site's checkpoint/journal procedure before directing clients to the replacement. For AWS, list the selected run's snapshots and verify account, source instance, volume IDs and `completed` state: ```bash aws ec2 describe-snapshots --owner-ids self \ --filters Name=tag:SDPRun,Values= --output json aws ec2 create-volume --availability-zone \ --snapshot-id --volume-type gp3 ``` Repeat volume creation for every member, selecting the required disk type, capacity, performance and encryption settings for the recovery environment. Attach them according to the original volume/mount mapping; never mix members from different runs. The AWS snapshot model supports restoring new volumes from individual members of a coordinated set. [EBS snapshot restoration model](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html) For Azure, inspect one point's saved VM configuration and all disk restore point IDs, then create new managed disks from those IDs: ```bash az restore-point show --subscription --resource-group \ --collection-name --name --instance-view --output json az disk create --subscription --resource-group \ --name --source \ --sku ``` The OS disk reference is `sourceMetadata.storageProfile.osDisk.diskRestorePoint.id`; data disk references are under `sourceMetadata.storageProfile.dataDisks`. Restore all disk members and use the saved VM configuration to attach them to a new recovery VM. Grant restore operators disk/VM creation and the needed network and encryption permissions separately. [Azure restore workflow](https://learn.microsoft.com/en-us/azure/virtual-machines/virtual-machines-create-restore-points-cli) ## Verification scope Run `PYTHONPATH=sdp/test python3 -m unittest test_cloud_improvements test_core_review`. The provider tests mock every cloud call, including successful sets, ownership mismatches, partial sets, failed waits, malformed dates and scoped expiration. Wrapper tests use fake AWS and Perforce commands while checking real process status and retained logs. No live cloud credentials, cloud resources, Perforce mutations, or disaster-recovery restore were exercised by these tests. Validate a complete create-and-restore cycle in the target account/subscription before relying on scheduled backups.