# Coordinated AWS and Azure snapshots

The SDP snapshot job creates one coordinated backup set and confirms completion
before expiring old backups. It records the current journal number as metadata.
It does not stop Perforce or claim application consistency. Keep normal SDP
checkpoint/journal backups and test recovery from a complete cloud set.

Deploy the entire `Server/Unix/p4/common/cloud` tree. Python 3 and the relevant
provider CLI must be available to the Perforce service account. The existing
`aws/bin/snapshot.sh` initializes `p4_vars`, logging, login and the journal-counter
read before invoking the shared `bin/sdp_snapshot.py` controller. Existing cron
entries default to AWS:

```bash
/p4/common/cloud/aws/bin/snapshot.sh 1
/p4/common/cloud/aws/bin/snapshot.sh 1 azure
# Equivalent Azure entry point:
/p4/common/cloud/azure/bin/snapshot.sh 1
```

`SDP_CLOUD_PROVIDER=azure` also selects Azure when no second argument is supplied.
`SDP_SNAPSHOT_RETENTION_DAYS` is a positive integer, defaulting to 90. A failed
creation, validation or wait prevents expiration entirely. Deletion failures
return failure and remain visible in `snapshot.log` and structured run output.
No new strict shell modes are added.

## AWS setup and ownership

Configure the service account's normal AWS profile/region or instance role. Give
the selected EBS volumes exact `Name` tags `<short-host>-root`,
`<short-host>-hxdepots` and `<short-host>-hxlogs`, using the host name before its
first dot. Each tag must select exactly one unique volume. All three must have a
single active attachment to the same EC2 instance. Optional
`SDP_AWS_INSTANCE_ID=i-...` pins that instance explicitly.

The controller inspects the EC2 block-device mapping, excludes every unselected
data volume, and handles boot-volume exclusion separately. More than 40 excluded
data volumes fails before creation. One `create-snapshots` request produces the
selected coordinated set, with ownership tags applied in that same request.
AWS documents this operation as crash consistent across the selected volumes.
[CreateSnapshots reference](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-snapshots.html)

The tags are `SDPManagedBy=sdp-cloud-snapshot`, `SDPSchema=1`, `SDPAccount`,
`SDPHost`, `SDPInstance`, `SDPSourceInstance`, `SDPRun`, `SDPJournal` and
`SDPSetSize`, plus descriptive `Contents`, `Host` and `Name` tags. Every member
shares the run ID and journal number; original `VolumeId` identifies the source
disk. Ensure all Perforce data, journals and checkpoints to be recovered are on
the selected disks. Network storage and other attached disks are not backed up.

The job waits for every returned snapshot, then verifies the completed volume
set, account and tags. The AWS waiter polls every 15 seconds and fails after 40
unsuccessful checks. If a large set exceeds that interval, inspect its logged run
ID and completion state; the job will leave older backups intact.
[Waiter reference](https://docs.aws.amazon.com/cli/latest/reference/ec2/wait/snapshot-completed.html)

Expiration queries use `--owner-ids self` and all stable ownership tags. The
controller verifies those fields again locally, groups snapshots by run ID, and
only deletes old sets whose declared member count is complete and whose members
are all completed. Legacy snapshots with only the old `Contents` tag, snapshots
from other hosts/instances/accounts/source EC2 instances, and incomplete/failed
sets remain untouched. Inspect retained incomplete sets manually. A deletion
failure can leave part of an expired set, which is retained for review rather
than treated as a complete backup on subsequent runs.

The backup identity needs `ec2:DescribeVolumes`, `ec2:DescribeInstances`,
`ec2:DescribeSnapshots`, `ec2:CreateSnapshots`, `ec2:CreateTags` and
`ec2:DeleteSnapshot`; identity discovery calls STS `GetCallerIdentity`.
Constrain create/delete permissions with the intended resources and ownership
tags in IAM. Encrypted volumes also require the key permissions applicable to
those volumes. Restore permissions should be granted separately to operators.

## Azure setup and ownership

Authenticate the service account using the normal Azure CLI managed-identity or
service-principal workflow. Configure these values in the environment sourced by
the job:

```bash
export SDP_AZURE_SUBSCRIPTION="<subscription UUID or name>"
export SDP_AZURE_RESOURCE_GROUP="<VM resource group>"
export SDP_AZURE_VM="<Linux VM name>"
# Optional; otherwise a deterministic dedicated name is derived:
export SDP_AZURE_COLLECTION="<dedicated restore point collection>"
# Optional positive seconds for each provisioning/disk-copy wait:
export SDP_AZURE_WAIT_SECONDS=3600
```

The selected subscription is resolved to its UUID, then every Azure CLI call is
pinned to it. The controller validates the VM's resource ID and unique VM ID. It
includes the OS disk and all attached data disks in one `CrashConsistent` VM
restore point. It rejects unmanaged, ephemeral, write-accelerated, shared,
Ultra Disk and Premium SSD v2 configurations before creating a collection or
point. This entry point supports Linux VMs outside scale sets. Azure Boost also
does not support crash-consistent restore points; the provider error stops the
job with no expiration and no independent-disk fallback. Confirm platform
support before scheduling.
[Azure support and limitations](https://learn.microsoft.com/en-us/azure/virtual-machines/virtual-machines-create-restore-points)

A dedicated restore point collection is tagged with the same stable SDP scope
fields plus the source VM UUID. An existing collection must already have the
exact ownership tags and source VM ID; the script does not adopt or retag an
unrelated collection. Missing collections are created with that ownership.
Individual point names use `sdp-v1-j<JOURNAL>-<UTC_RUN_ID>`, because restore-point
creation does not expose per-point tags.
[Collection CLI reference](https://learn.microsoft.com/en-us/cli/azure/restore-point/collection?view=azure-cli-latest)

After creation the job waits for successful provisioning and 100% background
copy of every disk. It checks source VM identity, the exact source disk set and
all disk restore point statuses. It then lists the owned collection with
`--restore-points` and considers only its managed name format, matching VM UUID,
crash-consistent successful state and age. Expired points are inspected again
for completed disk copies before deletion. Other names, sources, collections
and unfinished points are retained. The script never deletes a collection.
[Restore-point CLI reference](https://learn.microsoft.com/en-us/cli/azure/restore-point?view=azure-cli-latest)

Use a role scoped to the VM, disks and dedicated collection/resource group with
read access to `Microsoft.Compute/virtualMachines` and `Microsoft.Compute/disks`,
read/write on `Microsoft.Compute/restorePointCollections`, and read/write/delete
on its `restorePoints` children, including disk restore point status reads.
Collection discovery needs read access at resource-group scope. The scheduled
job does not need permission to delete collections or VMs. Encryption-key access
must remain valid for backup and restoration. Azure limits restore point count,
creation rate and concurrent creation for a VM; provider failures preserve the
previous backup sets.

## Recovery procedure

Use an isolated recovery VM and disks. Select one complete run, restore every
member of that run, and keep the original backup until verification succeeds.
The journal number is metadata from before creation, not proof of a clean
Perforce database shutdown. Validate filesystem and Perforce recovery using the
site's checkpoint/journal procedure before directing clients to the replacement.

For AWS, list the selected run's snapshots and verify account, source instance,
volume IDs and `completed` state:

```bash
aws ec2 describe-snapshots --owner-ids self \
  --filters Name=tag:SDPRun,Values=<run-id> --output json
aws ec2 create-volume --availability-zone <recovery-zone> \
  --snapshot-id <one-member-snapshot-id> --volume-type gp3
```

Repeat volume creation for every member, selecting the required disk type,
capacity, performance and encryption settings for the recovery environment.
Attach them according to the original volume/mount mapping; never mix members
from different runs. The AWS snapshot model supports restoring new volumes from
individual members of a coordinated set.
[EBS snapshot restoration model](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html)

For Azure, inspect one point's saved VM configuration and all disk restore point
IDs, then create new managed disks from those IDs:

```bash
az restore-point show --subscription <subscription> --resource-group <group> \
  --collection-name <collection> --name <point> --instance-view --output json
az disk create --subscription <subscription> --resource-group <recovery-group> \
  --name <recovered-disk-name> --source <diskRestorePoint-resource-id> \
  --sku <appropriate-disk-sku>
```

The OS disk reference is `sourceMetadata.storageProfile.osDisk.diskRestorePoint.id`;
data disk references are under `sourceMetadata.storageProfile.dataDisks`.
Restore all disk members and use the saved VM configuration to attach them to a
new recovery VM. Grant restore operators disk/VM creation and the needed network
and encryption permissions separately.
[Azure restore workflow](https://learn.microsoft.com/en-us/azure/virtual-machines/virtual-machines-create-restore-points-cli)

## Verification scope

Run `PYTHONPATH=sdp/test python3 -m unittest test_cloud_improvements test_core_review`.
The provider tests mock every cloud call, including successful sets, ownership
mismatches, partial sets, failed waits, malformed dates and scoped expiration.
Wrapper tests use fake AWS and Perforce commands while checking real process
status and retained logs. No live cloud credentials, cloud resources, Perforce
mutations, or disaster-recovery restore were exercised by these tests. Validate
a complete create-and-restore cycle in the target account/subscription before
relying on scheduled backups.