cvs2p4 1.3 December 26, 2001 ==== INTRODUCTION This small set of tools provides a means for importing a CVS module into Perforce. It was developed for our use at Network Appliance, to convert our product source code revision history from CVS history into Perforce. As such it has sprouted some NetApp-specific features suited to our special needs, but I have made an attempt to make these unobtrusive to the general user. Basically, it is patterned at a high level after the PVCS to Perforce converter available on the Perforce web site, doing the following steps during a conversion: - Scans the CVS repository to generate a metadata file; - Scans the metadata file to identify groups of RCS revisions that comprise Perforce changes; - Imports the revisions/log history into a Perforce depot, by running p4 client commands against a Perforce server (this is driven by the output of the previous phase); - Finally, (and optionally), generates a map of RCS revisions and the Perforce changes they belong to. cvs2p4 tries make the rsultant Perforce depot look as if the work in CVS had been going on in Perforce. In particular, it attempts to create changes corresponding to the whole creatin of new branches a la p4 integrate //depot/branchA/... //depot/branchB/... This is in contrast to rcstoperf.sh, which, as best I can tell, scattered the "integrates" corresponding of the creation of files on new branches into many changes (basically, according to when the file was actually file changed in the new branch). cvs2p4 also allows one to import only selected branches, and/or to map some branch other than the the CVS trunk to become the new "main" branch in Perforce. See the notes in the template config file ("test/config") for more information. As of version 1.3, cvs2p4 will also import CVS symbolic version tags. (Amazing what some insomnia on the night _after_ the night before Christmas can do!) Note: A CVS tagged revision will make it into Perforce labels ONLY when the revision is in fact present in the converted depot, subject to the branches selected for import. (See the notes for the "WANTLINES" variable in the config file). ==== MANIFEST After unpacking the distribution archive, use the MANIFEST script to verify that you have all of the pieces. The output should go something like this: $ MANIFEST MANIFEST Artistic README NEWS bin/genmetadata bin/genchanges bin/dochanges bin/dolabels bin/revmap lib/util.pl test/file,v test/dollar$file,v test/space file,v test/config test/runtest test/norm test/metadata.good test/lines.good test/changes.good test/p4_changes_-l.good test/p4_describe.good test/p4_describe.good-r97.3 test/p4_filesat.good test/p4_labels.good All ok ==== REQUIREMENTS This stuff should work on any Unix host that supports: - Perl 5.x, with working dbm support (i.e., dbmopen()/dbmclose() work). The scripts assume that perl will be found via $PATH. It must be a perl5! Some people have reported problems that seem to be related to dbm limitations with some perls when converting very large repositories. I like implementations based on Berkeley-DB. - RCS (actually, only the "co" command is used). Tested with RCS version 5.7. - Any Perforce release sin 97.3 - though it's possible that incompatibilities with older Perforces' may have crept in. The tests work fine with 2001.1 *NOTE*: if you are using 97.3 or earlier, you should rename mv test/p4_describe.good test/p4_describe.good-r98.2 mv test/p4_describe.good-r97.3 test/p4_describe.good to avoid spurious reports of failure during the final diff when running the test cases provided with this distribution. The ability to spoof the actual checkin times and users depends on using Perforce version 97.3 or later. (Note: I've tended to run with both the server and the converter/client sides on the same host, but I am unaware of any requirement to do so; it should work if you want to use different hosts for each side). ==== WHAT IT DOES This converter will import a CVS module into Perforce, preserving the branching structure seen in the RCS ,v file in the CVS repository, and translating them into Perforce branches within the depot. As it stands, it will only import RCS branches up through the highest numbered revisions on branches that have branch tags referring to them; thus, it will not necessarily bring *every* revision in the CVS module into Perforce, but *will* bring in every revision leading up to the current revision for every branch it imports. I think this is what most people will want; if not, hack away. Like the RCS -> Perforce converter available on the Perforce web site, it applies heuristics to try and identify multiple changes in CVS that are highly likely to comprise what would be seen as a single change in Perforce, and does commit them as a single Perforce change. (The heuristics are: checked in by the same user, proximal in time, and bearing an identical log message). It deals OK with files that are dead on the CVS trunk (I.e., where the RCS ,v files are in the "Attic/". The "UI" for the converter is not very slick, but for most people it's a one-time kind of tool anyway. Feel free to improve it if you are so inclined. In general, I would call this "hackware"; I'm doing this minimal stab at documenting it, and then giving it away, in hopes that somebody will find it useful (Or perhaps only entertaining :-). ==== TESTING I have included a *very* rudimentary automated test "suite", in the test/ directory. You can use this to verify that it seems to work in your environment. To run it: 1. Set up a virgin Perforce server; 2. If you're going to run the client side of the converter as some other user than the one you used to set up the server in step (1)., use "p4 protect", and grant p4 superuser privileges to the user who will be running the client side of the converter. 3. Edit test/config, and change the lines # p4 command location (If other than "/usr/local/bin/p4") # $P4 = "/u/p4/VERS/bin.solaris/p4"; # Perforce server we're using. # $P4PORT = "cranford:1668"; to reflect the actual location of your "p4" command and the server hostname and port that you are using. 4. Run the tests with test/runtest This should run all of the conversion scripts on the test CVS module (well, file - it's a one-file module!), and then verify a few things by querying the Perforce server after the conversion is complete. If everything goes well, the end of the output should be something like: runtest> diff /home/rmg/web/richard_geiger/guest/richard_geiger/utils/cvs2p4/test_conv_dir/p4_labels /home/rmg/web/richard_geiger/guest/richard_geiger/utils/cvs2p4/test/p4_labels.good runtest> diff /home/rmg/web/richard_geiger/guest/richard_geiger/utils/cvs2p4/test_conv_dir/p4_filesat /home/rmg/web/richard_geiger/guest/richard_geiger/utils/cvs2p4/test/p4_filesat.good runtest: ok In this version, the converted CVS "module" consists of a very few files, but it does have a carefully constructed branching structure, intended to verify that the converter does the right stuff with respect to branching. ==== USAGE 1. Make a directory to hold all the glop for the importation, and create a config file, starting with test/config as a template: $ mkdir convdir; cp test/config convdir Edit the convdir/config file to reflect your locale and intent. (See the comments in the config file). 2. Set up a virgin Perforce server (well, it doesn't have to be a virgin, I guess, and if there's no volcano, then why bother?) Make sure P4PORT in the config file is set to the hostname:port for this server. As described in the prep for running the tests, above, use "p4 protect" if required, to grant the user who will be running the conversion scripts Perforce superuser privileges. 3. Run bin/genmetadata: It takes a single argument - the name of the directory where the "config" file resides. (It will create all intermediate, temp, and working files under this directory. By the end of the entire process, this will include a tree with the same topology as what you end up with in the imported depot, so, if you're converting a big repository, you might need considerable space in this directory). $ bin/genmetadata convdir genmetadata: rm -rf convdir/logmsgs.dir convdir/logmsgs.pag ... . . (filenames of each file in the CVS module, as they are scanned) . ===== Lines referenced: chupa curly ha <- a list of branch tags encountered in the scan; larry also saved to convdir/lines. shemp xxx This reads cvsdir/config to get its marching orders, then scans the CVS module for all ,v and Attic/,v files, creating: convdir/metadata <- the extracted RCS/CVS metadata convdir/logmsgs.pag <- An ndbm database convdir/logmsgs.dir <- of the log messages convdir/lines <- A list of "codelines" (== branch tags) At this point, you may want to look at the list of branch tags encountered, (which was written to convdir/lines), edit the config file, setting WANTLINES to 1, and filling in the "<<LINES" here file with the names of the branches you care to import to Perforce; then, rerunning bin/genmetadata to rescan and pick up only those revisions you care about. (Things should be modified so genchanges honors the %WANTLINES hash, and then you wouldn't need to rerun genmetadata, sigh). 4. Run bin/genchages: Again, this takes a single argument - the dir name of the "conversion directory": rmg $ bin/genchanges convdir 16354 <- This counter spins as it's running, showing the number of changes processed. It will count up to the number of lines in the metadata file. This reads convdir/config and convdir/metadata, and writes convdir/changes. 5. Run bin/dochanges Verify that the Perforce server to receive the conversion is up, and that the user who will run "bin/dochanges? has superuser privileges. Also, if you are using the checkpointing feature (I.e., you have set a nonzero CHECKPOINT_INTERVAL in the config file), you must be able to do "rsh <p4dhost> -l <p4duser> <somecmd>" as the user, and from the host, which is going to run "dochanges". E.g., Let's say I'm running the p4 server on host "cranford", under login name "p4", and running "dochanges" as "rmg" on host "turok". Then user "rmg" on turok must be able to run "rsh cranford -l p4 <cmd>". Typically, this will require an entry in ~p4/.rhosts on "cranford", with a line of the form "turok rmg". You can avoid all of this rigamarole by running with CHECKPOINT_INTERVAL set to 0, or by runnig the converter and the p4d on the same host as the same user. [At this time running with CHECKPOINT_INTERVAL *other* than 0 is useless anyway, as the "recover from checkpoint" stuff hasn't been implemented yet!] Finally, you might want to save a copy of the output with "tee". The output will look something like: $ bin/dochanges T 2>&1 | tee bin/dochanges.out dochanges: /bin/rm -rf /n/makita/users/rmg/proj/cvs2p4/convdir/p4 dochanges: /u/p4/VERS/bin.solaris/p4 client -d dochanges Client 'dochanges' doesn't exist. dochanges: *** "/u/p4/VERS/bin.solaris/p4 client -d dochanges" ... Client dochanges saved. ========== change group 0 rmg 971112190556 Tmodule/convdir/1.1 Exp shemp - === deletes ===adds/edits=== dochanges: /u/p4/VERS/bin.solaris/p4 files /n/makita/users/rmg/... dochanges: /usr/local/bin/co -p1.1 Tmodule/T,v >/n/makita/users... Tmodule/T,v --> standard output revision 1.1 dochanges: /u/p4/VERS/bin.solaris/p4 add -t ktext /n/makita/use... //depoconvdir/ontap/trunk/T#1 - opened for add dochanges: ...| /u/p4/VERS/bin.solaris/p4 submit -i >/tmp/p4_su... SUBMITR->:Change 1 created with 1 open file(s). SUBMITR->:Submitting change 1. SUBMITR->:Locking 1 files ... SUBMITR->:add //depoconvdir/ontap/trunk/T#1 SUBMITR->://depoconvdir/ontap/trunk/T#1 - refreshing SUBMITR->:Change 1 submitted. dochanges: ...| /u/p4/VERS/bin.solaris/p4 change -i -f >/tmp/p4... SUBMITR->:Change 1 updated. ===branches=== ========== change group 1 . . . (Extremely voluminous output. You may want to tweak the . scripts to recduce it if you're tight on disk space . and tee'ing the output! . Basically, that's it. When this command finishes, your CVS module has been imported to Perforce. It make take a while. Converting our product source (=~ 4400 ,v files, and 16000 individual Perforce changes) takes something like 16 hours. But hey, that represents over 5 years of checkins to CVS!... 6. If you want to import labels from CVS tags, run $ bin/dolabels convdir //depot/Test/main/file#1 test/file/1.1 //depot/Test/main/file#3 test/file/3.1 //depot/Test/yyy/file#1 test/file/3.2 //depot/Test/main/file#5 test/file/3.3 //depot/Test/curly/file#1 test/file/3.3 . . . 7. If you want the RCS revision-to-Perforce change map, run: You can also get the RCS -> file#rev map with, e.g.: rmg $ bin/revmap -map rrevmap test_conv_dir test/file/1.1 //depot/Test/main/file#1 test/file/3.1 //depot/Test/main/file#3 test/file/3.3 //depot/Test/curly2/file#1 test/file/1.2.2.1 //depot/Test/chupa/file#1 test/file/1.2.2.1.2.2 //depot/Test/chupa/file#3 . . . ==== SUPPORT I originally wrote and contributed this tool while working for Network Appliance in 1997. I now work for Perforce, and, while I _am_ chartered with supporting Open Source software (such as this) as part of my job, it must be understood that Perforce Software still does not officially support it. I (and Perforce Software) can make absolutely no warranty that this will be helpful or even nontoxic for you, nor make any guarantee that I will be able to provide support. On the other hand, I have been able to be help in supporting many users in the past, so it's worth a try! If you use this and find it helpful, I'd love to hear from you. If there's enough of a stream of people using this, I have some ideas for major improvements that might be worth the time... - Richard Geiger Open Source Engineer at Perforce rmg@perforce.com (revised December 26, 2001, release 1.3)
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#35 | 5639 | Richard Geiger |
Changes to clarify what to use in WANTLINES to get the cvs trunk (Perforce "main", usually). |
||
#34 | 5605 | Richard Geiger | Final tweaks for 3.0b1 | ||
#33 | 5599 | Richard Geiger | About to unleash 3.0 upon an unsuspecting world... | ||
#32 | 5532 | Richard Geiger | get these in... | ||
#31 | 5510 | Richard Geiger | Add the $Depotmap{"somecvstopdir"} = "//somep4depot"; facility | ||
#30 | 5494 | Richard Geiger |
Heck, it's turning into 3.0! This is a watershed commit - switch to rlog-based ,v parsing... |
||
#29 | 5482 | Richard Geiger | More for 2.6.0... | ||
#28 | 5449 | Richard Geiger | gearing up for 2.6.0 release... | ||
#27 | 5442 | Richard Geiger |
A checkpoint commit on the way to a 2.6.0 release with the new IronPort inspired improvements. |
||
#26 | 4730 | Richard Geiger | Update about my employment, etc. | ||
#25 | 4272 | Richard Geiger | Integrate from the main release line. | ||
#24 | 3709 | Richard Geiger | Add the "how to make mods" section. | ||
#23 | 3594 | Richard Geiger | Changes for 2.3.5 | ||
#22 | 2988 | Richard Geiger |
Add the "WHY DOES dolabels LABEL FILES IN MANY BRANCHES?" section. |
||
#21 | 2586 | Richard Geiger |
Update the README to reflect my wandering ways. Lord I was born a travellin man Also, fix a few typos in the config file. Not planning on releasing just for these. |
||
#20 | 2547 | Richard Geiger | README updates about perforce metadata versions. | ||
#19 | 2447 | Richard Geiger | Set up to package 2.3.4 | ||
#18 | 2446 | Richard Geiger |
Change the link location to the old version. I'm going to publish this "a-la-carte"; we'll pick it up in the next tarball. |
||
#17 | 2360 | Richard Geiger | Typos fixed. | ||
#16 | 1987 | Richard Geiger | Changes for 2.3.1 | ||
#15 | 1966 | Richard Geiger |
Add a note about incremental conversions (and perfmerge2.pl) to the README. |
||
#14 | 1964 | Richard Geiger | Add $TEXTTYPE and surrounding accoutrements. | ||
#13 | 1782 | Richard Geiger | Changes to call this 2.0 | ||
#12 | 1781 | Richard Geiger |
This change reintegrates cvs2p4 2.0 developement work (through 2.0b6) back into my mainline development. |
||
#11 | 1357 | Richard Geiger |
Attempt to channel support requests to the opensource@perforce.com address, so as to take advantage of Calltrack. |
||
#10 | 1329 | Richard Geiger |
Fix the dochanges example in README to say "convdir" like the other examples; Remove extraneous ALLTHEWAYBACK=1; the slipped into the config file. Probably from listening to the WayBacks too much. http://www.waybacks.com |
||
#9 | 1185 | Richard Geiger |
Changes for 1.3 (Labels!) |
||
#8 | 1031 | Richard Geiger | Changes for 1.2.17; fix one-letter id internal error bug. | ||
#7 | 985 | Richard Geiger | Fix a couple of typos. | ||
#6 | 791 | Richard Geiger |
Update for a new release; my change-of-venue noted; desensitized to conflicting P4CONFIG. |
||
#5 | 331 | Richard Geiger |
1.2.8 - handle cr/lf in ,v's ... Yuck |
||
#4 | 251 | Richard Geiger | Changes for 1.2.6 - metacharacters in filenames. | ||
#3 | 247 | Richard Geiger |
Just correct the version number at the top of the README, so this doesn't rool the version number! |
||
#2 | 228 | Richard Geiger |
Changes for 1.2.4 - recognize alternative "p4 files" message for files not present in the depot. |
||
#1 | 130 | Richard Geiger |
CVS-to-Perforce converter. This is release 1.2.2 (first submit to the Perforce Public Depot) |