| 6 years agoronprestenback created P4XFER-9 for perforce-software-p4transfer:Script doesn't handle cases where archive file is missing from the p4 server
If an archive file is missing from the p4 server, calls to 'p4 sync' wil...l throw an exception similar to:
[Error]: 'Librarian checkout f:\\depot-main\\dev/sound/banks/foley/stone.bank failed.\nopen for read: f:\\depot-main\\dev/sound/banks/foley/stone.bank,d\\1.472161.gz: The system cannot find the file specified. '
Consider providing an option to enable handling this case similar to purged files (create dummy file to use in place of revision) « | |
|
|
| 6 years agoronprestenback created P4XFER-8 for perforce-software-p4transfer:Encoding issues
I encountered a file that contained an accented character in the filename: Melée.jpg. The code point for that character was E9,... but the UTF-8 code point is C3 A9. The file that was downloaded from the source server had the correct characters in the filename. But the file name that was being tracked by the script had incorrect characters (usually the UTF-8 replacement char, uFFF0). When it tries to open that file for add on the target server, it fails because the filename it has in memory no longer matches the name of the file that's on disk. Additionally, the logging library threw exceptions when trying to log the filename, which led to some additional "fun".
I still not sure exactly which encoding is the "right" one for this (since python uses different names for some encodings than what I could find elsewhere online), but I was able to work around the issue by setting self.p4.encoding = '1252' in P4Base.connect.
This worked great for the filename, but introduced a new issue. Now that the p4 connection was running in 1252 encoding, it threw exceptions when it encountered a changelist description (different changelist) that contained those microsoft "smart quotes", or as I like to call them "screw-up-perforce-encoding quotes" (as they also cause p4's encoding detection algorithm when to fail when adding a text file, if the unicode characters don't occur in the first X number of bytes of an otherwise ANSI compatible file....and they frequently don't).
Note: copy/pasting that é character will often change the code page being used for encoding that character, so you'll need to type it manually if you're trying to repro the issue. The problematic code point is E9, which you can type with a US keyboard by holding ALT while typing 0233 on the numeric keypad (has to be the numpad; the numbers at the top of the keyboard don't work as those are accelerator/shortcut keys). I had to use HexEdit4 to verify the character was still encoded as E9, because copy/pasting the filename would "helpfully" re-encode that character using UTF-8 « | |
|
|
| 6 years agoronprestenback modified P4XFER-7 for perforce-software-p4transfer:If you have both self.options.change_batch_size and self.options.maximum set, missingChanges does unnecessary work.
It first creates a copy of... the 'changes' list by indexing into it with self.options.change_batch_size. Then, if you also have maximum set, it repeats that step with self.options.change_batch_size.maximum. It appears that the intention is for self.options.maximum to act as an override; if so, we could avoid the second list copy by restructuring that code to be an else-if block, where it checks for self.options.maximum first, e.g.
maxChanges = 0
if self.options.maximum:
maxChanges = self.options.maximum
elif self.options.change_batch_size:
maxChanges = self.options.change_batch_size
if maxChanges > 0:
changes = changes[:maxChanges]
Err, not sure why some parts of this are bolded... « | |
|
|
| 6 years agoronprestenback modified P4XFER-7 for perforce-software-p4transfer:If you have both self.options.change_batch_size and self.options.maximum set, missingChanges does unnecessary work.
It first creates a copy of... the 'changes' list by indexing into it with self.options.change_batch_size. Then, if you also have maximum set, it repeats that step with self.options.change_batch_size.maximum. It appears that the intention is for self.options.maximum to act as an override; if so, we could avoid the second list copy by restructuring that code to be an else-if block, where it checks for self.options.maximum first, e.g.
maxChanges = 0
if self.options.maximum:
maxChanges = self.options.maximum
elif self.options.change_batch_size:
maxChanges = self.options.change_batch_size
if maxChanges > 0:
changes = changes[:maxChanges] « | |
|
|
| 6 years agoronprestenback created P4XFER-7 for perforce-software-p4transfer:If you have both self.options.change_batch_size and self.options.maximum set, missingChanges does unnecessary work.
It first creates a copy of... the 'changes' list by indexing into it with self.options.change_batch_size. Then, if you also have maximum set, it repeats that step with self.options.change_batch_size.maximum. It appears that the intention is for self.options.maximum to act as an override; if so, we could avoid the second list copy by restructuring that code to be an else-if block, where it checks for self.options.maximum first, e.g.
maxChanges = 0
if self.options.maximum:
maxChanges = self.options.maximum
elif self.options.change_batch_size:
maxChanges = self.options.change_batch_size
if maxChanges > 0:
changes = changes[:maxChanges] « | |
|
|
| 6 years agoronprestenback created P4XFER-6 for perforce-software-p4transfer:P4Source.missingChanges() is downloading unnecessary data, then throws away most of it and repeats the whole process for every batch of changelists....
This is problematic for depots that have a long history. In my case, the source depot has over 2 million changelists going back almost 20 years.
missingChanges then ends up discarding most of the results if self.options.change_batch_size or self.options.maximum are set. If I have a batch size of 500 (because my network is flaky or I want the log files to stay small, let's say), and the changes command is retrieving 1,000,000 changelists, 999,500 changelists' worth of data are thrown away and re-retrieved next batch. I expect that the call to "reverse()" in missingChanges is also pretty slow when changelist counts get this high.
My workaround was to modify the code to download only the changes that are needed, using the -r and -m flags to constrain the query. The revRange string remained the same, but I modified the options and values being passed into the 'changes' command.
I used self.options.change_batch_size or self.options.maximum (whichever is set) as the value for the -m parameter. Even with 20000 changes, the command returns relatively quickly.
Few random thoughts:
- Interestingly enough, without the -r, the command took much longer to run.
- When I tried changing the revRange string to use "counter + batch size" (i.e. counter + 20000) as the end of the revision range (instead of #head), the command took much longer to run.
- bonus: adding the -r flag meant that the call to reverse() was no longer necessary since the changelists come down in the appropriate order « | |
|
|
| 6 years agoronprestenback created P4XFER-5 for perforce-software-p4transfer:Exception is thrown by getIntegration in certain cases.
I don't remember the exact details of the changelist that caused this, but it was somet...hing like "file was brached from a depot that was obliterated" or something like that.
I only hit exceptions on two calls to getIntegration(): the call from the bottom of P4Source.getChange() and the first call at the top of P4Target.moveAdd().
For the case in getChange(), I changed that line to read:
if chRev.action == 'move/add' and chRev.hasIntegrations()
For the case in moveAdd(), I changed the line to call file.hasIntegrations instead of file.getIntegration() when checking whether it has any:
if file.getIntegration() and file.getIntegration().localFile: « | |
|
|
| 6 years agoronprestenback commented on P4XFER-4 forSorry, this is a dupe of P4XFER-3 - please delete! | |
|
|
| 6 years agoronprestenback created P4XFER-4 for perforce-software-p4transfer:If transfer is stopped while in some states, it cannot be resumed without manual intervention and cleanup.
Case: If all files in a changelist were... opened for edit/add/integrate/whatever on the target server, but not yet submitted (e.g. script was submitting when it was killed manually or threw an exception), the transfer cannot be resumed until the files are reverted.
Reason: After resuming the transfer, when the script goes to checkout the files, it will get an error from the p4 server (don't recall the exact errors, but they're different depending on why the file was checked out: edit/integrate/add, etc.) In cases where all files were checked out in the previous run, the submit will fail because the new changelist has nothing in it. In cases where the script was still checking out files when it was killed, the new changelist will not contain all of the files that were in the source changelist, causing validation to fail.
Case: If the script gets killed/excepted after submit completed, but before the counter was updated, the script will attempt to process that changelist again.
Reason: There is only one counter that is set at the end of the changelist replication process. We probably need another counter that is set when we get to a point in the changelist replication process where recovery steps are now necessary (basically everything after replication process has started talking to the destination server). Recovery code could use that counter to determine whether recovery actions are necessary (if p4transfer_started < p4transfer_completed). The two most common issues I ran into was files left checked out, and changelists not updated with the origin changelist's user and time.
My solution was to create a batch file that performed three steps before I could restart the script, if it was killed or threw an exception while process a changelist:
1. revert everything on the target server
2. sync target workspace to #0
3. sync source workspace to #0
The downside to this (and the only reason I think it might be worth attempting to add support for recovery) is if the changelist in question requires a long sync, it really slows down the process. I was willing to live with this drawback, because adding recovery steps can be really complicated (maybe impossible) to implement reliably. You could probably get away with not syncing source workspace to #0 (which would mitigate the sync time drawback), as long as the files have not been modified such that they're now different from the source server's version. « | |
|
|
| 6 years agoronprestenback created P4XFER-3 for perforce-software-p4transfer:If transfer is stopped while in some states, it cannot be resumed without manual intervention and cleanup.
Case: If all files in a changelist were... opened for edit/add/integrate/whatever on the target server, but not yet submitted (e.g. script was submitting when it was killed manually or threw an exception), the transfer cannot be resumed until the files are reverted.
Reason: After resuming the transfer, when the script goes to checkout the files, it will get an error from the p4 server (don't recall the exact errors, but they're different depending on why the file was checked out: edit/integrate/add, etc.) In cases where all files were checked out in the previous run, the submit will fail because the new changelist has nothing in it. In cases where the script was still checking out files when it was killed, the new changelist will not contain all of the files that were in the source changelist, causing validation to fail.
Case: If the script gets killed/excepted after submit completed, but before the counter was updated, the script will attempt to process that changelist again.
Reason: There is only one counter that is set at the end of the changelist replication process. We probably need another counter that is set when we get to a point in the changelist replication process where recovery steps are now necessary (basically everything after replication process has started talking to the destination server). Recovery code could use that counter to determine whether recovery actions are necessary (if p4transfer_started < p4transfer_completed). The two most common issues I ran into was files left checked out, and changelists not updated with the origin changelist's user and time.
My solution was to create a batch file that performed three steps before I could restart the script, if it was killed or threw an exception while process a changelist:
1. revert everything on the target server
2. sync target workspace to #0
3. sync source workspace to #0
The downside to this (and the only reason I think it might be worth attempting to add support for recovery) is if the changelist in question requires a long sync, it really slows down the process. I was willing to live with this drawback, because adding recovery steps can be really complicated (maybe impossible) to implement reliably. You could probably get away with not syncing source workspace to #0 (which would mitigate the sync time drawback), as long as the files have not been modified such that they're now different from the source server's version. « | |
|
|
| 6 years agoronprestenback created P4XFER-2 for perforce-software-p4transfer:P4Target.updateChange calls utcfromtimestamp to convert the source changelist's timestamp to UTC, but it doesn't convert the UTC time back into local... time when updating the changelist in the destination server.
This causes the changelist submit time (when viewed in e.g. P4V) to not match between the source and destination servers. « | |
|
|
