| 4 years agoashaikh requested review 27379 for perforce-software-sdp:devFix mkdirs.sh install errors on replica servers
When you run the mkdirs.sh script on an edge server, you get a symlink error for the checkpoints dire...ctory:
Running: ln -s /tmp/hxmounts/hxdepots/p4/1/checkpoints.edge.1 /tmp/p4/1/checkpoints.edge.1
ln: failed to create symbolic link ‘/tmp/p4/1/checkpoints.edge.1’: No such file or directory
Error: (line: 735) Failed to create symlink.
You also get an error for the chown commands on the checkpoints directory because there is an extra / slash in the path.
SDP Job - https://swarm.workshop.perforce.com/jobs/SDP-596 « | |
|
|
| 4 years agoashaikh created SDP-596 forInstall errors when running mkdirs.sh on replica servers
When you run the mkdirs.sh script on an edge server, you get a symlink error for the checkpo...ints directory:
Running: ln -s /tmp/hxmounts/hxdepots/p4/1/checkpoints.edge.1 /tmp/p4/1/checkpoints.edge.1
ln: failed to create symbolic link ‘/tmp/p4/1/checkpoints.edge.1’: No such file or directory
Error: (line: 735) Failed to create symlink.
You also get an error for the chown commands on the checkpoint directories because there is an extra / slash in the path. « | |
|
|
| 4 years agoashaikh modified SDP-585 forImprove error message for edge_dump.sh when seed file already exists
When running the edge_dump.sh script, you would get a generic error message if a... seed file already existed making you refer to logs to figure out what happened.
Current error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0449.log
Edge seed checkpoint dump failed!
Improved error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0456.log
/p4/1/checkpoints/p4_1.edge.1.seed.6.gz already exists.
Review: https://swarm.workshop.perforce.com/reviews/27262/ « | |
|
|
| 4 years agoashaikh requested review 27262 for perforce-software-sdp:devImprove error message for edge_dump.sh when seed file already exists
When running the edge_dump.sh script, you would get a generic error message if a... seed file already existed making you refer to logs to figure out what happened.
Current error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0449.log
Edge seed checkpoint dump failed!
Improved error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0456.log
/p4/1/checkpoints/p4_1.edge.1.seed.6.gz already exists.
Refer to job SDP-585. « | |
|
|
| 4 years agoashaikh created SDP-585 forImprove error message for edge_dump.sh when seed file already exists
When running the edge_dump.sh script, you would get a generic error message if a... seed file already existed making you refer to logs to figure out what happened.
Current error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0449.log
Edge seed checkpoint dump failed!
Improved error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0456.log
/p4/1/checkpoints/p4_1.edge.1.seed.6.gz already exists. « | |
|
|
| 4 years agoashaikh requested review 27173 for perforce-software-sdp:devThe recover_edge.sh SDP script errors out because a variable is accessed before it is declared.
In this case, the following line throws an error:
...
declare EdgeCheckpointsDir="${CHECKPOINTS}.${SERVERID#p4d_}"
Error:
/p4/common/bin/recover_edge.sh: line 27: CHECKPOINTS: unbound variable
Opened a job regarding this error: https://swarm.workshop.perforce.com/jobs/SDP-579 « | |
|
|
| 4 years agoashaikh created SDP-579 forEdge recover script fails to run
The recover_edge.sh SDP script errors out because a variable is accessed before it is declared.
In this case, the... following line throws an error:
declare EdgeCheckpointsDir="${CHECKPOINTS}.${SERVERID#p4d_}"
Error:
/p4/common/bin/recover_edge.sh: line 27: CHECKPOINTS: unbound variable « | |
|
|
| 4 years agoashaikh commented on SDP-563 forSounds good. I went ahead and updated the backup_functions.sh to support replicas. This has been tested on several servers and works as expected.
htt ... | |
|
|
| 4 years agoashaikh requested further review of 26875 for perforce-software-sdp:dev After speaking more with Tom and him understanding the use case for this, I'm re-opening this pull request. | |
|
|
| 4 years agoashaikh commented on SDP-563 forThanks for the detailed explanation.
So would the checkpoint created by using the request_replica_checkpoint.sh be different than the checkpoint gene ...Thanks for the detailed explanation.
So would the checkpoint created by using the request_replica_checkpoint.sh be different than the checkpoint generated by adding support to the SDP script to replay the filtered replica journals to it's offline db and creating a checkpoint off of that?
The drawback I see from requesting a checkpoint using 'p4 admin checkpoint -Z' is that if we have a large filtered replica server, replaying journals and creating a checkpoint could take a significant amount of time. During this time users would not be able to use the server until the process finishes right? « | |
|
|
| 4 years agoashaikh archived review 26875 for perforce-software-sdp:dev Per Tom, checkpoints are not meant to be run on forwarding replicas. | |
|
|
| 4 years agoashaikh requested review 26875 for perforce-software-sdp:devFix checkpoint error on forwarding replicas
When running the daily_checkpoint.sh script on forwarding replicas, you get the following error:
Offli...ne journal replay failed. Abort!
This is because the daily_checkpoint.sh is trying to replay a journal that doesn't exist (hasn't been rotated yet).
I've created SDP-563 job for this issue.
https://swarm.workshop.perforce.com/jobs/SDP-563
These changes have been tested on several forwarding replicas. « | |
|
|
| 4 years agoashaikh modified SDP-563 forSDP daily_checkpoint.sh script fails on forwarding replicas
The daily_checkpoint.sh script fails on forwarding replicas. It tries to replay journals... to the offline_db that do not exist.
$ /p4/common/bin/daily_checkpoint.sh 1
Offline journal replay failed. Abort!
$ cat /p4/1/logs/checkpoint.log
Fri Oct 30 05:07:00 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Start p4_1 Checkpoint
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Offline journal number is: 22
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Skipping call to truncate_journal() on edge or replica server.
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay any unreplayed journals to the offline database.
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.22 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.22...
real 0m0.027s
user 0m0.004s
sys 0m0.013s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.23 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.23...
real 0m0.023s
user 0m0.004s
sys 0m0.012s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24...
Perforce server error:
open for read: /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24: No such file or directory
open for read: /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24: No such file or directory
real 0m0.007s
user 0m0.003s
sys 0m0.004s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: ERROR!!! - replica p4_1 /p4/common/bin/daily_checkpoint.sh: Offline journal replay failed. Abort!
$ ls -al /p4/1/checkpoints.replica.1/
total 40
drwx------. 2 p4 p4 267 Oct 30 05:08 .
drwx------. 6 p4 p4 81 Oct 30 03:44 ..
-r--r-----. 1 p4 p4 2838 Oct 30 04:54 p4_1.replica.1.jnl.15
-r--r-----. 1 p4 p4 4126 Oct 30 04:55 p4_1.replica.1.jnl.16
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.17
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.18
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.19
-r--r-----. 1 p4 p4 2838 Oct 30 04:56 p4_1.replica.1.jnl.20
-r--r-----. 1 p4 p4 2838 Oct 30 04:57 p4_1.replica.1.jnl.21
-r--r-----. 1 p4 p4 3228 Oct 30 05:06 p4_1.replica.1.jnl.22
-r--r-----. 1 p4 p4 1940 Oct 30 05:06 p4_1.replica.1.jnl.23 « | |
|
|
| 4 years agoashaikh modified SDP-563 forSDP daily_checkpoint.sh script fails on forwarding replicas.
The daily_checkpoint.sh script fails on forwarding replicas. It tries to replay journal...s to the offline_db that do not exist.
$ /p4/common/bin/daily_checkpoint.sh 1
Offline journal replay failed. Abort!
$ cat /p4/1/logs/checkpoint.log
Fri Oct 30 05:07:00 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Start p4_1 Checkpoint
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Offline journal number is: 22
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Skipping call to truncate_journal() on edge or replica server.
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay any unreplayed journals to the offline database.
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.22 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.22...
real 0m0.027s
user 0m0.004s
sys 0m0.013s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.23 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.23...
real 0m0.023s
user 0m0.004s
sys 0m0.012s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: Replay journal /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24 to offline db.
Recovering from /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24...
Perforce server error:
open for read: /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24: No such file or directory
open for read: /depotdata/p4/1/checkpoints.replica.1/p4_1.replica.1.jnl.24: No such file or directory
real 0m0.007s
user 0m0.003s
sys 0m0.004s
Fri Oct 30 05:07:01 UTC 2020 /p4/common/bin/daily_checkpoint.sh: ERROR!!! - replica p4_1 /p4/common/bin/daily_checkpoint.sh: Offline journal replay failed. Abort!
$ ls -al /p4/1/checkpoints.replica.1/
total 40
drwx------. 2 p4 p4 267 Oct 30 05:08 .
drwx------. 6 p4 p4 81 Oct 30 03:44 ..
-r--r-----. 1 p4 p4 2838 Oct 30 04:54 p4_1.replica.1.jnl.15
-r--r-----. 1 p4 p4 4126 Oct 30 04:55 p4_1.replica.1.jnl.16
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.17
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.18
-r--r-----. 1 p4 p4 1940 Oct 30 04:55 p4_1.replica.1.jnl.19
-r--r-----. 1 p4 p4 2838 Oct 30 04:56 p4_1.replica.1.jnl.20
-r--r-----. 1 p4 p4 2838 Oct 30 04:57 p4_1.replica.1.jnl.21
-r--r-----. 1 p4 p4 3228 Oct 30 05:06 p4_1.replica.1.jnl.22
-r--r-----. 1 p4 p4 1940 Oct 30 05:06 p4_1.replica.1.jnl.23 « | |
|
|
| 4 years agoashaikh modified SDP-563 for The SDP daily_checkpoint.sh script fails to run on forwarding replicas.
It tries to replay journals to the offline_db that do not exist. | |
|
|
| 4 years agoashaikh created SDP-563 for The SDP daily_checkpoint.sh script fails to run on forwarding replicas.
It's tries to replay journals to the offline_db that do not exist. | |
|
|
| 4 years agoashaikh commented on review 26709 for perforce-software-sdp:dev@tom_tyler are any changes needed for this or can we get this merged it? | |
|
|
| 5 years agoashaikh commented on review 26709 for perforce-software-sdp:dev | |
|
|
| 5 years agoashaikh requested review 26709 for perforce-software-sdp:dev | |
|
|
| 5 years agoashaikh created SDP-554 forIncorrect Server ID when deploying an edge server
When using SDP to deploy a server type p4d_edge, the server id is incorrectly set to the master ser...ver id.
Based on this discussion, the mkdirs.sh needs to be updated so that the edge server does not use the $MASTER_ID var when creating the server.id file.
https://swarm.workshop.perforce.com/reviews/26530/ « | |
|
|
| 5 years agoashaikh commented on review 26530 for perforce-software-sdp:dev | |
|
|
| 5 years agoashaikh commented on review 26530 for perforce-software-sdp:devThanks. So I should go back and update all the edge servers that have all this set to their server.id and update it to the be the actual commit master ...Thanks. So I should go back and update all the edge servers that have all this set to their server.id and update it to the be the actual commit master server.id? « | |
|
|
| 5 years agoashaikh commented on review 26473 (backup_functions.sh, line 488) for perforce-software-sdp:devWould this be an issue for edge servers since their server id is set as the master id?
Per the mkdirs.cfg:
The server.id of the master server. Or, i...Would this be an issue for edge servers since their server id is set as the master id?
Per the mkdirs.cfg:
The server.id of the master server. Or, if configuring
an edge server or replica of an edge server, set this
to the server.id of the edge server.
MASTER_ID=master.${SDP_INSTANCE} « | |
|
|
| 5 years agoashaikh archived review 26336 for perforce-software-sdp:dev Fixed in another patch | |
|
|
| 5 years agoashaikh requested review 26450 for perforce-software-sdp:dev Fix incorrect backup functions import statement. | |
|
|
| 5 years agoashaikh commented on review 26336 for perforce-software-sdp:devShould we just remove that if statement? | |
|
|
| 5 years agoashaikh commented on review 26336 for perforce-software-sdp:devThank you. So what does that mean? Is there a fix in the works or is it an open issue that needs to be addressed? | |
|
|
| 5 years agoashaikh commented on review 26336 for perforce-software-sdp:devCan you link me to SDP-446? I'm not sure what you are referring to. Thanks! | |
|
|
| 5 years agoashaikh requested review 26336 for perforce-software-sdp:devUpdate logic in backup_functions.sh to take into account checkpoint directory paths for various types of Helix servers.
Currently the daily_check...point.sh script fails to run on the edge servers because it looks in the wrong folder for checkpoints. It's looking in the ${CHECKPOINTS} folder instead of ${CheckpointsDir}. Also update the file prefix var for the checkpoint file name. « | |
|
|
| 6 years agoashaikh updated files in review 25810 for perforce-software-sdp:devAdd support for opting out of information email messages
Currently, we're trying to filter our email to only contain actionable messages. By default,... the SDP scripts will always send an email if it's configured when a job has completed. I'd like to add a parameter to suppress informational messages and only get notified if there is an error.
I kept the default behavior of always sending an email. « | |
|
|
| 6 years agoashaikh requested review 25810 for perforce-software-sdp:devAdd support for opting out of information email messages
Currently, we're trying to filter our email to only contain actionable messages. By default,... the SDP scripts will always send an email if it's configured when a job has completed. I'd like to add a parameter to suppress informational messages and only get notified if there is an error.
I kept the default behavior of always sending an email. « | |
|
|
| 6 years agoashaikh commented on review 25577 for perforce-software-sdp:devThere's another review for an update to the purge_revisions.sh script which adds support for wildcards. I've documented the script functionality withi ...There's another review for an update to the purge_revisions.sh script which adds support for wildcards. I've documented the script functionality within the scripts themselves. Is there another document that needs to be updated?
Also, where are the tests located? Are there any docs on requirements for tests? This is something I may need help with. We can take this offline.
Thanks! « | |
|
|
| 6 years agoashaikh requested review 25399 for perforce-software-sdp:devAdd support for wildcards/globbing to purge_revisions.sh script
This change allows you to specify wildcard paths in the purge revisions config file i...n case you want to apply the same revisions policy to multiple files. « | |
|
|
| 6 years agoashaikh requested review 25304 for perforce-software-sdp:devFix syntax error in mkdirs.sh script
Currently the mkdirs.sh script will not create the /p4/common/bin or the /p4/common/config folders due to a synt...ax error and failure in the conditional check. This leads to a cascading failure in other parts of the script.
From my limited testing, the CN value is never an empty string because if mkdirs.sh is not run in test mode, the value is imported from the mkdirs.cfg file. « | |
|
|
| 6 years agoashaikh requested review 25271 for perforce-software-sdp:devFix syntax errors for SDP_ADMIN_PASSWORD_FILE
The current backup_functions.sh script isn't working correctly due to syntax errors. When I try to run... 'run_if_master.sh 1 echo a' on a master server, nothing happens. After I made the following fixes, it worked as expected. « | |
|
|
| 6 years agoashaikh commented on review 25070 for perforce-software-sdp:devI added another enhancement so the journal watch script only checks to see if a checkpoint is running only if the script attempts to truncate the jour ...I added another enhancement so the journal watch script only checks to see if a checkpoint is running only if the script attempts to truncate the journal. This will prevent errors if there is a checkpoint already in progress and journal watch happens to run at the same time but no journal rotation is necessary because of space issues. « | |
|
|
| 6 years agoashaikh updated files in review 25070 for perforce-software-sdp:devChange log rotation to only rotate journal_watch log
Currently this script rotates all logs everytime it runs. However, if you run the journal_watch... script on a frequent basis, you may not have much of a history of other logs files. « | |
|
|
| 6 years agoashaikh requested review 25070 for perforce-software-sdp:devChange log rotation to only rotate journal_watch log
Currently this script rotates all logs everytime it runs. However, if you run the journal_watch... script on a frequent basis, you may not have much of a history of other logs files. « | |
|
|
| 6 years agoashaikh commented on review 24877 for perforce-software-sdp:devThank you! This has been giving me issues recently! | |
|
|
| 7 years agoashaikh updated files in review 24327 for perforce-software-sdp:devAdd a script to archive/purge revisions based on number of days.
This script will allow you to archive files and optionally purge files based on a co...nfigurable number of days and minimum revisions that you want to keep. This is useful if you want to keep a certain number of days worth of files instead of a specific number of revisions. This script currently only accepts paths to specific files, it does not support globbing or wildcards. « | |
|
|
| 7 years agoashaikh requested review 24327 for perforce-software-sdp:devAdd a script to archive/purge revisions based on number of days.
This script will allow you to archive files and optionally purge files based on a co...nfigurable number of days and minimum revisions that you want to keep. This is useful if you want to keep a certain number of days worth of files instead of a specific number of revisions. This script currently only accepts paths to specific files, it does not support globbing or wildcards. « | |
|
|
| 7 years agoashaikh requested review 24106 for perforce-software-sdp:devAdd a new SDP script to warn and/or rotate live journal due to configurable low disk space condition
A new journal_watch.sh script is being introdu...ced with alerting and journal truncate functionality for enviroments with high Perforce activity. Two separate thresholds can be configured, one for a warn alert (notify via email - optional) and another to trigger a journal rotation.
Values for the threshold can be defined via K, M, G or a percentage of disk space. These values represent the available disk space on the journal volume (free space minus filesys.P4Journal.min) before the script takes aption.
Another optional feature is to enable KEEPJNL override which dynamically calculates the number of journals to keep based on the oldest checkpoint on disk. When this is enabled, KEEPJNL is temporarily overridden during the journal_watch.sh run to prevent removing any journals that may be required to recover from a checkpoint on disk and all the journals required to bring the server back to a current state.
This script can be run on a master/commit, edge and replica server. If the edge or replica server is running low on journal space, it will trigger a journal rotation on the master/commit server (based on P4TARGET value), which then will cause the journals to rotate on the edge/replica servers.
<code>#review</code> « | |
|
|
| 7 years agoashaikh requested review 23953 for perforce-software-sdp:dev Added support for offline checkpointing on standby servers using the same maintenance workflow as commit servers
<code>#review</code> | |
|
|
