SDP-302
Closed
Parallelized checkpoint processing to reduce duration.
Enable parallel checkpoints, and include test suite coverage for
same.
Excerpt of email from Alan Kwan:
---
I've framed out a pseudo code implementation of how it could behave
as backup_functions in SDP:
dump_checkpoint_parallel()
- get list of db files (this can be optimized to sort by largest or
smallest to keep work queues as saturated as possible)
- get p4_var variable set to # of worker threads, else use logic to
determine a just in time value:
- figure out cpu core count
- check system active load value
- define # of threads = to core count minus active load, minus 1 (if
result is 0 or less than 1, set to 1 - not parallel)
- define work queue (ls -1 /p4/1/offline_db/)
- insert code to execute against work queue based on (
http://hackthology.com/a-job-queue-in-bash.html ), and while limit,
keep working until end - checkpoint files are named
/p4/1/checkpoints/p4_1.ckp.db.have.number.gz (along with their MD5)
- rewrite the offline_
restore parallel would implement something similar - get the list of
compressed checkpoint files, throw in a work queue, and jr -z on each
one into the same offline_db folder until they're all done.
augment remove_old_checkpoints_and_journals to incorporate these sort
of checkpoints
Excerpt of email from Robert Cowham:
---
An alternative step along the way also is to use pigz or similar for
parallel compression which is where a lot of time is spent.
Typically the focus should be on the 3-7 or so files which comprise
the vast majority of the data (db.have/db.rev and friends/db.integed/
db.label depending)
I would also be tempted to tar the result into one file rather after
zipping/before unzipping for ease of management.
Enable parallel checkpoints, and include test suite coverage for
same.
Excerpt of email from Alan Kwan:
---
I've framed out a pseudo code implementation of how it could behave
as backup_functions in SDP:
dump_checkpoint_parallel()
- get list of db files (this can be optimized to sort by largest or
smallest to keep work queues as saturated as possible)
- get p4_var variable set to # of worker threads, else use logic to
determine a just in time value:
- figure out cpu core count
- check system active load value
- define # of threads = to core count minus active load, minus 1 (if
result is 0 or less than 1, set to 1 - not parallel)
- define work queue (ls -1 /p4/1/offline_db/)
- insert code to execute against work queue based on (
http://hackthology.com/a-job-queue-in-bash.html ), and while limit,
keep working until end - checkpoint files are named
/p4/1/checkpoints/p4_1.ckp.db.have.number.gz (along with their MD5)
- rewrite the offline_
restore parallel would implement something similar - get the list of
compressed checkpoint files, throw in a work queue, and jr -z on each
one into the same offline_db folder until they're all done.
augment remove_old_checkpoints_and_journals to incorporate these sort
of checkpoints
Excerpt of email from Robert Cowham:
---
An alternative step along the way also is to use pigz or similar for
parallel compression which is where a lot of time is spent.
Typically the focus should be on the 3-7 or so files which comprise
the vast majority of the data (db.have/db.rev and friends/db.integed/
db.label depending)
I would also be tempted to tar the result into one file rather after
zipping/before unzipping for ease of management.
- Status
- Closed
- Project
- perforce-software-sdp
- Severity
- C
- Reported By
- akwan
- Reported Date
- 7 years ago
- Modified By
- C. Thomas Tyler
- Modified Date
- 2 years ago
- Owned By
- tom_tyler
- Dev Notes
- [2023/04/13 tom_tyler]: This job is now closed.
Parallel checkpoints
are now fully supported. The needed p4d features to ensure reliable
processing have been released, and the SDP now takes advantage of
them. See notes about DO_PARALLEL_CHECKPOINTS in the Instance Vars
file (e.g. /p4/common/config/p4_1.vars) for more info.
[2021/07/06 tom_tyler]: This job has been suspended. Turns out some
needed p4d support (a command to get a list of checkpointed tables)
isn't available. Also, there is hope that a future release of p4d will
provide this capability without the need for scripting.
While there are implementations of the parallel checkpoint mechanism
that have been made to work (by checkpointing all tables whether they
need it or not), this is the sort of thing that can never fail. We
decided this feature, while it would be valuable, is best done as a
p4d feature rather than an SDP feature. When the needed functionality
is added to p4d, this job will be re-opened.
[2020/08/18 tom_tyler]: Re-opening this job to re-add this feature,
with full test suite coverage.
Older Notes:
This can be done reliably, but will be sophisticated. We may want
to add an optional new setting in instance_vars.template, e.g.
PARALLEL_CHECKPOINTS with a default value of 0.
Then either dump_checkpoint() or dump_checkpoint_parallel() would be
called depending on whether that new var is set to 1 or not. So
by default it would still do single-threaded checkpoints, and
would do parallel checkpoints if explicitly enabled. - Component
- core-unix
- Type
- Feature
Testing Robert's suggestion of using
pigz. Pretty neat utility. Trying that approach where I do a normaljdand pipe it throughpigzfor the large database files, and bundle the remaining ones. (whattttttt this Swarm doesn't support markdown in comments? 馃槕 )I can't recommend this solution since checkpoints are run on the master, and you will hog too many of the server's CPUs.
Good point. The proposed logic includes determining the CPU to engage and using as many as possible, essentially assuming that "getting it over with" is the highest priority compared to "not significantly impacting performance while checkpoints occur." Both are reasonable priorities. We'll want to add throttle control set the minimum number of CPUs to keep available for other processing, to avoid slamming the CPU availability too hard. Given the high value of getting checkpoint duration reduced, I think most admins will tolerate some impact (compared to the current single-threaded behavior), so long as it's not blocking/locking or exhausting resources. But we'll need to make it super easy to "go back to the old way." Or perhaps make it so a configuration change from the default is needed to enable parallel checkpoint processing? (I'd be inclined to make it the default, though, since we're only going to release it once fully proven.
I'm going to bump here to show some serious interest in this (on windows). Our checkpoints just take far too long to complete. I'd like to note that we take our checkpoints on read only replicas. In a federated environment I care a little less about my server being hammered hard than my ability to take frequent backups that let me restore to a closer point in time to any failure we have.
Last time I worked on this during a hackathon, I discovered that a checkpointing process invokes a lock on the database itself, so other checkpoint processes couldn't run, even if you specified a single individual db.file. The only way I'm aware of working around this would involve symlinking the database files from discrete locations as described by google: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/39983.pdf
With the right kind of hardware, doing a jd without -z piped through pigz did make checkpointing individual databases quicker and allowed the system to take advantage of spare resource (noted that even with a 16core box I never saw pigz reliably gobble more than 5-6 cores). pigz is capable of constraining itself on how much its allowed to consume. While parallel checkpoint creation wasn't particularly quick, having multiple checkpoint files would allow for a way to parallel restore said checkpoint.
For us personally, we've moved full checkpointing to a replica, process journal rotations on the commit, and do database rebalancing on commit on the weekends.
That is correct Alan, and that is the reason we use the offline database for doing the checkpoint.