# The form data below was edited by robert_cowham # Perforce Workshop Jobs # # Job: The job name. 'new' generates a sequenced job number. # # Status: Job status; required field. There is no enforced or # promoted workflow for transition of jobs from one # status to another, just a set of job status values # for users to apply as they see fit. Possible values: # # open - Issue is available to be worked on. # # inprogress - Active development is in progress. # # blocked - Issue cannot be implemented for some reason. # # fixed - Fixed, optional status to use before closed. # # closed - Issue has been dealt with definitively. # # punted - Decision made not to address the issue, # possibly not ever. # # suspended - Decision made not to address the issue # in the immediate future, but noting that it may # have some merit and may be revisited later. # # duplicate - Duplicate of another issue that. # # obsolete - The need behind the request has become # overcome by events. # # Project: The project this job is for. Required. # # Severity: [A/B/C] (A is highest) Required. # # ReportedBy The user who created the job. Can be changed. # # ReportedDate: The date the job was created. Automatic. # # ModifiedBy: The user who last modified this job. Automatic. # # ModifiedDate: The date this job was last modified. Automatic. # # OwnedBy: The owner, responsible for doing the job. Optional. # # Description: Description of the job. Required. # # DevNotes: Developer's comments. Optional. Can be used to # explain a status, e.g. for blocked, punted, # obsolete or duplicate jobs. May also provide # additional information such as the earliest release # in which a bug is known to exist. # # Component: Projects may use this optional field to indicate # which component of the project a givenjob is associated # with. # # For the SDP, the list of components is defined in: # //guest/perforce_software/sdp/tools/components.txt # # Type: Type of job [Bug/Feature]. Required. # # Release: Release in which job is intended to be fixed. Job: P4XFER-6 Status: closed Project: perforce-software-p4transfer Severity: B ReportedBy: ronprestenback ReportedDate: 2019/04/18 17:10:28 ModifiedBy: robert_cowham ModifiedDate: 2021/04/06 04:47:09 OwnedBy: ronprestenback Description: P4Source.missingChanges() is downloading unnecessary data, then throws away most of it and repeats the whole process for every batch of changelists. This is problematic for depots that have a long history. In my case, the source depot has over 2 million changelists going back almost 20 years. missingChanges then ends up discarding most of the results if self.options.change_batch_size or self.options.maximum are set. If I have a batch size of 500 (because my network is flaky or I want the log files to stay small, let's say), and the changes command is retrieving 1,000,000 changelists, 999,500 changelists' worth of data are thrown away and re-retrieved next batch. I expect that the call to "reverse()" in missingChanges is also pretty slow when changelist counts get this high. My workaround was to modify the code to download only the changes that are needed, using the -r and -m flags to constrain the query. The revRange string remained the same, but I modified the options and values being passed into the 'changes' command. I used self.options.change_batch_size or self.options.maximum (whichever is set) as the value for the -m parameter. Even with 20000 changes, the command returns relatively quickly. Few random thoughts: - Interestingly enough, without the -r, the command took much longer to run. - When I tried changing the revRange string to use "counter + batch size" (i.e. counter + 20000) as the end of the revision range (instead of #head), the command took much longer to run. - bonus: adding the -r flag meant that the call to reverse() was no longer necessary since the changelists come down in the appropriate order DevNotes: This has been implemented in github version. It checks for p4d >= 17.2 and then uses the -r flag and -m as suggested Type: Bug