README.txt #1

$Id: //guest/hari_krishna_dara/perforce/utils/pvcstop4/main/README.txt#1 $

PVCS to Perforce converter

Latest status - R Cowham 2003/10/09: The scripts have been tweaked
for a client, but the client didn't have branching in place. Thus be a
little careful! However, the new stuff with PCLI and detecting moved
files is known to work.

This version requires PCLI to be installed - see notes at project home page:

http://public.perforce.com/public/perforce/utils/pvcstop4/index.html


Summary

- Obtain and install a version of Perl for your machine.  If you are running
  a version of UNIX and for some reason don't already have Perl available,
  download http://www.perl.com/CPAN/src/latest.tar.gz and get your sysadmin
  to compile and install it.  If you are running Windows you can
  download a precompiled binary from http://www.activestate.com/
  FOR BEST RESULTS - run on Windows. Tested with version 5.6

- Ensure both Perforce and PVCS can execute correctly.  It is best if PVCS
  has a simple or no configuration file.  Note that archives with access
  lists may require you to run as the PVCS super user; either that or disable
  the access lists before running the converter.  Test Perforce by
  running "p4 info".
  The Perforce server need not run on the same machine you are running
  the converter on, however it may be wise to do so to avoid network
  traffic.

- Be aware that if you are using PVCS through the GUI (or via the web
  interface, e.g. v6.7) then you can have some issues with moved or
  deleted files. For example, the web interface allows you to move
  files. The scripts now handle this, but require PCLI (PVCS Command Line
  Interface) to be installed as this is the only thing which knows about such
  info.
  Note that the PCLI may not be installed by default - it is an
  option and also requires at least PVCS 6.5 or greater (by 6.7.10
  all PCLI commands were introduced).
  Manual is PCLI.pdf, this is located at: <PVCSInstallDir>\books\vm\vmbooks\*

- Untar or Unzip the converter on a disk large enough to hold all files
  in your PVCS archives, plus the space required for the "metadata".
  The directory you put the converter in must not be under the Perforce
  server root or the PVCS archive root!

- Ensure your PVCS archives are organized in a sensible tree structure with
  a single root.  It is better to have directories "c:\archive\projecta" and
  "c:\archive\projectb" than "c:\projecta\archive" and "c:\projectb\archive";
  in the former case your root is "c:\archive", in the latter it is
  "c:\" and your depot will have useless "archive" directories.
  As well, it is important that branches are labelled, and files which you
  wish to be marked as deleted are labelled (see below for more details).

- Change directory to the directory the converter was installed into and
  edit the file "config.ini", then run "perl convert.pl"

- Note that it can take a significant amount of time to run.

- Deleted files are currently handled by specifying them in a label.
  It should be possible to use PCLI to discover this information, but
  this ability has yet to be scripted.

- There are also issues with "linked" files - these are not copied in
  both new locations.

Performance
===========
The script can take a long time (many hours) to run if you have a
large PVCS repository. The script needs to parse the history of PVCS
and then get individual versions of each file out and check them in
to Perforce. This by its nature takes time. To ensure the fastest
possible conversion:

- Copy PVCS repository locally (and use a fast machine!)

- Install P4Perl from
  http://public.perforce.com//guest/tony_smith/perforce/API/Perl/index.html.

  The latest versions of P4Perl may work fine, but the following are known to work:

  http://public.perforce.com/guest/tony_smith/perforce/API/Perl/released/p4perl56-setup-2.4320.exe
  or
  http://public.perforce.com/guest/tony_smith/perforce/API/Perl/released/p4perl58-setup-2.4320.exe

  This is the Perl interface to Perforce and is significantly faster
  than executing p4.exe at the command line which is the default.
  This can reduce conversion times by 25 - 50%.

It is highly recommended, however, that you read the following before
attempting to do the conversion.


Differences between PVCS and Perforce
-------------------------------------

PVCS uses the same model as RCS: each "workfile" has associated with it an
"archive file" which contains all revisions of the "workfile" as well as
all information about those revisions - who checked each one in and when,
and so on.  We will refer to the latter information as the "metadata".

PVCS archives may be located anywhere in the local filesystem; Perforce
keeps all archive files in a centralized depot.  Local files may be
mapped from depot files to any desired location in the local filesystem.

Perforce uses RCS format files to store revisions, but all metadata is stored
in a single, central database.  Perforce stores useful information that
PVCS does not.

Perforce uses the notion of an atomic change which may consist of many
add, edit, and/or delete operations.  It is guaranteed that either all
of a change is visible, or none of it is.  PVCS does not provide atomic
change transactions;  when checking in a number of files one is given
the option of using the same change description for each, but the fact
that the revisions were all logically part of one change is lost.

PVCS archives contain all branched revisions in one archive file, with
branch revisions being numbered based on the branching point (e.g. 1.2.1.0
is a branch from 1.2, 1.4.2.3.1.2 is a branch from 1.4.2.3, which itself
is the second branch from 1.4).  Perforce handles branches differently.

In Perforce when you wish to create a "branch", you simply copy the files,
typically to another directory.  The recommended convention, and the convention
used by the converter, is to make the second component of the depot pathname
the branchname.  Thus, when it is time for the main line of development
to be released as v7, you would typically do a
"p4 integrate //depot/main/... //depot/v7/..."
which (since there aren't any files in //depot/v7 currently) would create
copies of all files in //depot/main in //depot/v7.  To refer to the mainline
version of foo.c use //depot/main/foo.c; for the v7 branched copy, use
//depot/v7/foo.c

This works well because Perforce implements a lazy copy.  Archive files
are never duplicated; only the metadata changes.

Merging operations are recorded in Perforce - if, for example, patches
were made to v7 code these patches could be merged back into the main line
of development with "p4 integrate //depot/v7/... //depot/main/..."
Only files which have changes that have not been previously integrated
into main will be integrated, and the integration will be recorded so
that next time those changes will not have to be revisited.

In PVCS there is no way of knowing that a revision was the result of a
merge aside from noting that in the change description.

Labels are commonly used in PVCS; they are available in Perforce but not
usually needed.  Perforce uses atomic change transactions, and thus the
state of the depot or any part of it is easy to specify.  Perforce keeps
track of deleted revisions (obviating the need for a special "DELETED"
label of some sort) and branches are simply differently named files
(labelling branches is optional in PVCS).


Possible Missing/Incorrect Information in PVCS metadata

- incorrect timestamps due to improperly set clocks or clocks which have
  drifted relative to each other (time of PVCS timestamp is that of the
  machine on which it is run, not of a central machine)

- no identification of checkins which are logically all part of the same change
  (identify by similar time + same author and change description)

- no identification of branches which are logically all part of the same branch
  (hopefully identify by label, "floating" label is a branch label)

- no identification of merged branches

- no consistent method for identifying deleted files
  (hopefully identify by a label such as "OBSOLETE" or "DELETED")

- incorrect file type information ("binary" files with GENERATEDELTA or
  EXPANDKEYWORDS)


Restrictions and limitations

Filenames may not contain slashes (/) or backslashes (\) (they would
be misinterpreted as pathnames), ASCII unprintable characters, at signs
(@) or hashes (#).  These characters will be mapped to underscores.

Userids and labels may not contain spaces or ASCII unprintable characters.
Again, these characters will be mapped to underscores.

The date format used by PVCS must be one of the following:
dd mon yyyy hh:mm:ss
mon dd yyyy hh:mm:ss
where mon is the first three letters of the English month name.

Change descriptions with the 30-dash separator line in them will be truncated.
(you could get around this by running vlog -br<revno> for each revision
in each archive, but that is a high price to pay..)

The converter is not regularly tested against old server versions, and thus
it is not guaranteed to always work against all older versions.  Unless
you have a good reason to use an old server version, you should run the
converter against the latest version of Perforce.


How does the converter work?

The converter works in four phases:
Phase I:   extract PVCS metadata
Phase II:  improve the metadata (compact changes using heuristics)
Phase III: execute appropriate PVCS and p4 commands
           to construct Perforce database
Phase IV:  niceties after the fact

PVCS archives are a proprietary binary format, however the metadata is
provided in a human readable form by the PVCS vlog command, and individual
revisions can be extracted with the PVCS get command.

The converter relies on PVCS branches being labelled with floating
labels; if they are not labelled then a name is made up for the branch
to hold them.  If the unlabelled branches are not actually desired
then they should be removed from the PVCS archive before running the
converter (use "vdel").

The converter marks files as deleted immediately after the checkin of
a revision which is marked in PVCS with a label indicating it should
be deleted.

When the converter runs it reads configuration values from the file "config".
Only root must be supplied; the others have defaults.  See the comments
in "config" for a description of each option.


How do I check the end result, and what does it all mean?

If you want to be certain that the revisions stored in Perforce are
correct, answer yes to the question "Do you wish to verify the result?"
(If you say no and then change your mind, just run "perl verify.pl").
The verification step isn't done by default because it takes a long
time, and because there is no portable way to determine whether earlier
phases of the converter succeeded.

NOTE: the verification step does *not* verify that all files were
submitted with the correct file type.  On Windows/NT submitting a
"binary" file as "text" will cause either a verification failure
(due to CRLF -> LF translation) or *no error* despite a truncated
file.  The latter error can occur because files opened as "text"
files on NT consider ^Z (ASCII 26) to be an end of file marker.
Thus, when the file is submitted, only bytes up to the first
occurence of ^Z will be submitted.  The solution for this is to
force the file to be submitted as type binary by adding the filename
extension to the "type_binary:" line in the config file.
You can verify the file type the converter chose by examining the
file "metadata/files".

You can verify that all changes were successfully submitted by running
"p4 changes -s pending"; it should produce no output.  Look in
the directory "metadata" for files containing a mapping of all PVCS
archives and revision numbers to the associated Perforce filename and
change number.  The sorted, more human readable version is in the file
"mapping"; the tabular version is in "mapping.ns" (ns for "not sorted").

Change numbers should not be confused with version numbers.  To see the
files and versions affected by a particular change, type

p4 describe -s <change_number>

(Without the -s all differences between files involved in the change are
displayed as well).

To see the list of all changes (you may want to pipe this through more):
p4 changes

To see the list of labels:
p4 labels

To see the list of changes affecting a particular file:
p4 filelog filename

To list revisions associated with the label "labelname":
p4 files @labelname


A note about re-running the converter

If you want to re-run the converter it would be best to erase the directory
tree "data", so that there cannot be any conflicts between runs.  If you
don't want to keep the results of the previous conversion, delete all
files in the server root and directories below it.


James Strickland (james@perforce.com)
Robert Cowham (robert@vaccaperna.co.uk)