SDP-701
Open
Have upgrade.sh preserve fast rollback until entire upgrade is done.
The key idea: "Don't touch the offline_db during the entire upgrade procedure."
Then only update the offline_db with some a manual process after the
need for rollback is determined not to be necessary.
PREP: Ensure offline_db is current and health on all servers, master
and all replicas/edges. Make use of 'p4 topology'.
SAMPLE PROCEDURE
1. Lockout users (Protections, Brokers, and/or Firewall Rules).
2. Do a journal rotation on the master to give a rollback point,
triggering journal rotation on all replicas.
3. Replay that to the offline_db on master and all replicas.
4. Upgrade all edges/replicas, outer-to-inner, with p4d -xu.
5. Do another journal rotation on master. DO NOT replay to offline_db.
6. Upgrade master with p4d -xu.
7. Do 'p4 upgrades' on master, wait until it is done.
8. Do 'p4 upgrades' on all replicas, wait until it is done.
9. Do one more journal rotation on master, so we have a journal with
just upgrades.
10. Open access for sanity testing (e.g. swarm Swarm and a few "test pilot" human users.
11. Determine rollback is not needed.
12. Update the offline_db for the first time since Step 4 above.
13. Open flood gates to all users when sanity testing is done.
EXPECTATIONS MANAGMENT:
In any case, rollback after the users start using the system is not
supported. The above gives a clean and fast rollback up until the
point where users are allowed back in. If a problem is detected after,
a rollback would involve data loss of any changes made since the start
of the procedure. The only option then is to fix and roll forward.
The key idea: "Don't touch the offline_db during the entire upgrade procedure."
Then only update the offline_db with some a manual process after the
need for rollback is determined not to be necessary.
PREP: Ensure offline_db is current and health on all servers, master
and all replicas/edges. Make use of 'p4 topology'.
SAMPLE PROCEDURE
1. Lockout users (Protections, Brokers, and/or Firewall Rules).
2. Do a journal rotation on the master to give a rollback point,
triggering journal rotation on all replicas.
3. Replay that to the offline_db on master and all replicas.
4. Upgrade all edges/replicas, outer-to-inner, with p4d -xu.
5. Do another journal rotation on master. DO NOT replay to offline_db.
6. Upgrade master with p4d -xu.
7. Do 'p4 upgrades' on master, wait until it is done.
8. Do 'p4 upgrades' on all replicas, wait until it is done.
9. Do one more journal rotation on master, so we have a journal with
just upgrades.
10. Open access for sanity testing (e.g. swarm Swarm and a few "test pilot" human users.
11. Determine rollback is not needed.
12. Update the offline_db for the first time since Step 4 above.
13. Open flood gates to all users when sanity testing is done.
EXPECTATIONS MANAGMENT:
In any case, rollback after the users start using the system is not
supported. The above gives a clean and fast rollback up until the
point where users are allowed back in. If a problem is detected after,
a rollback would involve data loss of any changes made since the start
of the procedure. The only option then is to fix and roll forward.
- Status
- Open
- Project
- perforce-software-sdp
- Severity
- C
- Reported By
- C. Thomas Tyler
- Reported Date
- Modified By
- C. Thomas Tyler
- Modified Date
- Owned By
- tom_tyler
- Component
- core-unix
- Type
- Feature