P4XFER-6
Closed
P4Source.missingChanges() is downloading unnecessary data, then throws away most of it and repeats the whole process for every batch of changelists.
This is problematic for depots that have a long history. In my case, the source depot has over 2 million changelists going back almost 20 years.
missingChanges then ends up discarding most of the results if self.options.change_batch_size or self.options.maximum are set. If I have a batch size of 500 (because my network is flaky or I want the log files to stay small, let's say), and the changes command is retrieving 1,000,000 changelists, 999,500 changelists' worth of data are thrown away and re-retrieved next batch. I expect that the call to "reverse()" in missingChanges is also pretty slow when changelist counts get this high.
My workaround was to modify the code to download only the changes that are needed, using the -r and -m flags to constrain the query. The revRange string remained the same, but I modified the options and values being passed into the 'changes' command.
I used self.options.change_batch_size or self.options.maximum (whichever is set) as the value for the -m parameter. Even with 20000 changes, the command returns relatively quickly.
Few random thoughts:
- Interestingly enough, without the -r, the command took much longer to run.
- When I tried changing the revRange string to use "counter + batch size" (i.e. counter + 20000) as the end of the revision range (instead of #head), the command took much longer to run.
- bonus: adding the -r flag meant that the call to reverse() was no longer necessary since the changelists come down in the appropriate order
This is problematic for depots that have a long history. In my case, the source depot has over 2 million changelists going back almost 20 years.
missingChanges then ends up discarding most of the results if self.options.change_batch_size or self.options.maximum are set. If I have a batch size of 500 (because my network is flaky or I want the log files to stay small, let's say), and the changes command is retrieving 1,000,000 changelists, 999,500 changelists' worth of data are thrown away and re-retrieved next batch. I expect that the call to "reverse()" in missingChanges is also pretty slow when changelist counts get this high.
My workaround was to modify the code to download only the changes that are needed, using the -r and -m flags to constrain the query. The revRange string remained the same, but I modified the options and values being passed into the 'changes' command.
I used self.options.change_batch_size or self.options.maximum (whichever is set) as the value for the -m parameter. Even with 20000 changes, the command returns relatively quickly.
Few random thoughts:
- Interestingly enough, without the -r, the command took much longer to run.
- When I tried changing the revRange string to use "counter + batch size" (i.e. counter + 20000) as the end of the revision range (instead of #head), the command took much longer to run.
- bonus: adding the -r flag meant that the call to reverse() was no longer necessary since the changelists come down in the appropriate order
- Status
- Closed
- Project
- perforce-software-p4transfer
- Severity
- B
- Reported By
- ronprestenback
- Reported Date
- 6 years ago
- Modified By
- Robert Cowham
- Modified Date
- 4 years ago
- Owned By
- ronprestenback
- Dev Notes
- This has been implemented in github version.
It checks for p4d >= 17.2 and then uses the -r flag and -m as suggested - Type
- Bug
Can you branch the project and shelve or submit your change?