TODO #1

  • //
  • guest/
  • timothee_besset/
  • TODO
  • View
  • Commits
  • Open Download .zip Download (16 KB)
TODO for VCP

*NOTE* Doing a `grep -r TODO lib bin` will find lots of small things and
future feature ideas.  This file will grow to include the more important
of those.

- Bugfixes

    - if vcp::source::p4 fails to create temporary p4 client,
      don't try to delete it, or at least, don't complain when
      the delete fails.

    - deduce_rev_root should be called after all metadata is extracted
      (or perhaps as it is being extracted) and deduce the root from
      all of the revisions extracted.  This will eliminate the need for
      --rev-root to be used with VCP::Source::p4's --follow-branch-into,
      for instance.

    - VCP::Source::cvs should either ignore a leading "/" before the
      module name or complain about it.

    - Clean up VCP::Dest::p4 vcp_#### integrate spec.

    - Make it so that a dest of cvs:blah:foo is either an error or is just
      like cvs:blah:foo/... (if it looks like a directory given the source
      revs or the target repository topology)

    - Prevent keyword expansion on all checkouts.  Found by
      Thomas Quinot <[email protected]>

    * Carry executable bit through (Nick Ing-Simmons)

    * Make <rev_root>, <name>, etc. use binary escapes when needed

    * VCP::Dest::cvs needs to set the binary mode properly on files it creates
      and checks in [[check status]]

    * VCP::Source::cvs needs to deduce binary modes correctly [[check status]]

    - there should be an filter to detect collisions between
      underscorified names (users, files, tage) in VCP::Dest::*.

    - use the upcoming -t option for p4 filelog (expected in 2002.2) to
      extract times with dates.  Issue a warning recommending an upgrade
      or enabling a metadata reading mode if not present.  Warn about
      metadata loss when using older p4.

    - get p4 forms parsing to lex *exactly* like p4 does it.

    - fix docs to give .vcp file examples before command line examples

- Testing

    - finish t/95vss2p4.t (all tests should pass, need to clean up RevML).

    - automate VCP-tests/

    - fully test scan/filter/transfer in test suite

    - t/99foo2bar.t "real world corner conditions" tests.

        - Test VCP::Source::cvs branching corner conditions, perhaps all
          corner conditions for this sort of thing.
            - when the first rev on a branch is a delete
            - when the first rev on a branch occurs as the first change in an
              incremental export
            - when a file is added on a branch in an incremental conversion

    - Test importing two bootstrap imports one on top of the other
        - should VCP warn when it detects this?

    - Test CVS vendor tags in cvs->p4 and cvs->cvs; not sure how to do this
      best; there is no VCP internal designation of a vendor branch.
      Such a flag would also have to be visible in RevML and would be
      needed for testing purposes.  Right now, CVS seems to make vendor
      branches appear as branches of rev 1.1, which is OK, I think, but
      it means that the first p4 integrate looks backwards.

    - Test aliased CVS tags

    - Test unlabelled CVS branches

- Feature Adds

    - Implement --ignore-deleted-files

    - Rename "previous_id" to "parent_id"

    - Deprecate <placeholder /> and implement even <action>...</action>

    - Implement a MetaDataFudge filter to fudge missing user_id, times
      and comments.  This should perhaps be implemented as a VCP::Dest
      feature so that it could work on the aggregated comment.

    - Have VCP::Source::cvs set limits on the $r->time field on all
      branch revs to be identical within the limits possible
        - Must   be >= parent's time and <= .1 rev's time
        - Should be >  parent's time and <= .1 rev's time
        - May need to have multiple times in the case of disjoint
          timespans between parents and children on the same branch.
            - Example:
                - A#1.1     time is 2000-01-01
                - A#1.1.2.1 time is 2000-01-02
                - B#1.1     time is 2000-01-03
                - B#1.1.2.1 time is 2000-01-04
            - In this case, it's not possible for A#1.1.2.0 and
              B#1.1.2.0 to have the same timestamps.

    - Improve metadb support
        - Single flat file, named in spec
        - add RevML and SOP formats?

    - create .rep files to define repositories
        - p4 spec format

    - Prompt for repo_id in VCP.stml

    - Pay attention to case-sensitive() throughout
        - needs a code audit
            - look for compile_shellish qr, lc, eq, cmp, and uc
        - pass it in the header to all filters
        - store it in revml

    - Add more default filter wizards to DefaultFilters.pm,
      particularly 'vss2p4_default_filters' and maybe
      'p42vss_default_filters'

    - Add the following hints for VSS-> conversions (from [email protected])

        |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
        Subject: RE: [p4] Migration hints from Visual SourceSafe
        Date: Sun, 13 Jul 2003 09:45:37 +0200
        From: "Yariv Sheizaf" <[email protected]>
        To: <[email protected]>, <[email protected]>

        Hi Thierry,

        We just finished to convert to big (3GB each) VSS databases
        to Perforce using the VSS2P4 script.

        The script itself running coorrectly, and its installation is easy.
        Pay attention to two elements:

        1. Locate all components required for the conversion process:
           VSS DB, Perforce server+DB, Vss2P4 script -
           on the same machine.
           It reduces the conversion time very much (6 hours instead of 6 days).

        2. Run "analyze" and "analyze -f" if needed on the VSS DB until it fixed -
           no error messages should be appeared.
           In any case that the VSS DB is not consistent (e.g. - does not
           passing "analyze" correctly) - the conversion process will fail.

        Regards,

        Yariv
        |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

    - Finish Interactive UI
        - test suites

        - [email protected] says:
            - Better examples needed for things like CVSROOT
            - Clean up the docs

    - Bundle HTML docs with the app, don't require HTML::Links as a
      prerequisite in the CPAN dist or vcp.pl file.

    - Allow the called executable name (cvs, p4, etc) to be set as an option.

    - Transfer and translate branch specs and other metadata on
      p4->p4 transfers.

    - Extend revml to have a <client> tag to allow p4->* conversions
      to carry the p4 client through.

    - option to lock label specs (integrate specs)?

    - Handle differently localized timestamps.  RevML should be
      in GMT or use ISO8601's tz indicators.

    - VCP::Dest::cvs Investigate running cvs as root to spoof user
      names properly
        - setting the called executable name (another todo) could
          be extended to allow multi-word invocations
        - looks like cvs calls getlogin(), might need to spoof
          wtmp to do this

    - VCP::Dest::cvs Write RCS files directly
        - only way to get accurate data in to RCS file format
            - usernames, times

    - VCP::Filter::usermap to allow usernames to be altered

    - VCP::Filter::commentedit to allow comments to be edited (post-sort
      but pre change aggregation); this is to allow cvs2p4-like comments.

    - warn when a *Map filter is used and the default <<keep>> action
      fires, because this indicates a possible missing or faulty rule.

    - improve error reporting for .vcp files. Either use recursive descent
      to delegate each value to the appropriate object or capture line and
      column with each value for error reporting.

    - in p4->foo suppress revs with empty deltas immediately after an
      integrate so as to not introduce changless revs in other repos,
      including ->p4 conversions when --change-branch-rev-1 not
      specified.

    - give better diagnostics when the state file appears to be out of
      date:
      - when --continue is specified we can't.
      - when --continus is not specified and a revision already exists in
        the destination state files (but not in the destination rep).  We
        could keep track of the last known change to the destination in
        the state file and probe to see if that file is at the indicated
        revision.  Or just watch for revisions coming along that should be
        in the dest (according to the state file) but aren't.

    - Add a --skip-unchanged-revs option or VCP::Filter that skips
      unchanged revisions; all children of such a revision become
      children of that revision's parent.

    - Enable an "--append-revs" flag to allow a bootstrap file to be
      added.  This is dangerous (there's no checking to be sure that the
      first new version is the first version after the existing version in
      the repository) but useful.  This might be done already with
      --bootstrap specified on the source, but is completely untested

    - reports and queries against the state files to show:
      - head rev of each filebranch
      - what source revs ended up where (path & rev_id)
      - how branches got mapped
      - what the main branch_id is for foo->cvs imports.
      - what fields got edited how by StringEdit, Map, LabelMap, etc.

    - Make the transfers more transactional, so we can recover from
      where we left off when something fails.  We're part way there with
      the --continue support, but VCP needs to log what it's about to
      submit and sniff out how far the submittal got before it blew up.
      This would allow recovery by updating the state files to the
      correct state and not trying to double commit files.

    - Allow the state files to be checked in to the destination.  Probably
      as text, in order to avoid sdbm byte ordering issues if they are
      checked out on a differently byte ordered system.

    - Perhaps allow keyword expansion, but convert the expanded texts
      so that they are no longer seens as RCS style keywords.  This would
      allow imported files to have a "stamp of origin" in them.  Would
      also need an option to leave the keywords in place in this case, since
      the user might presumably want expansion to work correctly in the
      new repository too.  Suggested by Thomas Quinot <[email protected]>.

    - Add a link checker to vcp html

    - An option to not bring over deleted files Steve James <[email protected]>

    - A report destination that offers a preview of what a transfer will
      do, with summary and long views.

    - Limit the number of NtLkLy queries per command to prevent server lockup.
      Steve James <[email protected]>  (possibly URGENT, need to test).

    - Set CVS_PASSFILE for all cvs invocations to prevent mucking with the
      users' current .cvspass

    - Use ptys to handle CVS login, if available.  Recomend installing
      IO::Pty if needed but not installed.

    - implement VCP::Source::cvs -d option when scanning RCS files

    - PERHAPS checksum all non-binary files line by line, removing all \r's
      in order to reduce sensitivity to varying platform settings between
      the source and the destination.

    - allow VCP::{Source,Dest}::* to "sniff" at unknown directories /
      files to see if they can detect what kind of repository is there.
      This will make schemes optional, so tab completion will work again.


- Efficiency

    - Batch cvs checkouts somehow (see clkao's patch)

    - extend VCP::Source::cvs to build revisions directly from the RCS
      files, this will probably mean memorizing the offsets of the delta
      or full text chunk for each version in the RCS file, then applying
      them all as needed to get the desired version.  They may need to be
      reversed as a speed hack since RCS files tend to store the most
      recent revision in full text and uses deltas from that to encode
      older revisions, and we'll probably want the oldest revision first.
      This means that we can build the more recent revisions from the
      older revisions by reversing the deltas as we apply them to build
      the older revisions, then apply those reversed deltas.  Or
      something; not sure what's best here.

    - VCP::Source::revml should only keep on hand the versions it needs at
      each moment in order to conserve disk space.  The problem with this
      is that the RevML may be coming from a non-seek()able byte stream,
      like STDIN, so we need to patch as we go.  One alternative is to
      cache the revml off to the side and rescan it if this happens.
      Another is to only patch- as-you-go if the input is non-seek()able.
      NOTE: P'haps VCP::Source::revml and VCP::Source::cvs can share the
      RCS file scanner.  NOTE: P'haps VCP::Source::revml and
      VCP::Source::cvs can share the an internal file revision format; RCS
      is bass ackwards for our needs.

    - in --continue mode, VCP::Source::p4 could do a p4 files to get all
      the source_filebranch_ids and then get the last_rev_in_filebranch
      for each, which would probably be a lot quicker than running a full
      filelog and throwing away most of the data (ie for a --continue on a
      large tree with lots of changes, but only one or two files have
      changed since the last export).

    - Consider offering a repeat attribute in RevML <char code="0x00"
      repeat="34234"> (david d zuhn <[email protected]>)

    - the RevMapDB should be purged of any revisions descended from a
      revision being transferred.  Right now, if you restore the set
      repository from an earlier backup and don't rewind the vcp_state
      directory, you will end up with a mix of RevMapDB entries from the
      prior transfer and the current.  For now, a warning is generated.


- Cleanup

    - remove all references to $r->action "placeholder" and
      replace is_placeholder_rev() calls with is_branch_rev()
      and/or is_clone_rev() calls where appropriate.

    - replace assert_eq (and it's *huge* reams of output on failure)
      with ok_or_diff.

    - Factor VCP::{Source,Dest}::new()s up in to VCP::Driver
        - Rename parse_foo_repo_spec() in VCP::Utils::foo to
          parse_repo_spec and make sure VCP::Utils is first in
          all use base;es.

    - use VCP::Utils::is_win32() whereever $^O is tested for win32.

    - Refactor File::Spec->rel2abs( ..., start_dir ) calls in to a new
      VCP::Utils::start_dir();

    - Refactor $self->work_path( "co" ... ) calls in to a
      VCP::Plugin::checkout_path( ... ) or generalize the concept of a
      repository workspace.

    - use parent/child nomenclature instead of previous/next

    - Centralize underscorification of tag names for VSS.

- Rejected

    - VCP::Source::p4 should be able to create and read a metadata dump as
      an option.  Watch out for different schemas in different p4d
      versions.  Q: Read the btree files directly? 'twould be faster and
      more space efficient.  (rejected by Perforce so as to not tie VCP to
      p4d's schema; this may still be contributed as OSS by others, but
      will not be done by the core team any time soon unless the plans
      change)

    - VCP::Dest::p4 should write a metadata file directly, and be able to
      merge new data in to a destination's exported metafile for reimport.
      Q: Write the btree files directly?  This would bypass any checking
      p4d does on recovering from a metadata file.  (rejected by Perforce
      so as to not tie VCP to p4d's schema; this may still be contributed
      as OSS by others, but will not be done by the core team any time
      soon unless the plans change)

    - An option to prefix all labels with some user-defined string Steve
      James <[email protected]> (this is no longer necessary, as vcp does
      not add its own labels by default).

# Change User Description Committed
#1 4040 timothee_besset guest branch
//guest/perforce_software/revml/TODO
#94 4038 Barrie Slaymaker - TODO cleaned up
#93 3993 Barrie Slaymaker - Fold in changes from clkao's SVN work
#92 3930 Barrie Slaymaker - VCP::Source::cvs and VCP::Dest::p4 handle cloning deletes
- "placeholder" actions and is_placeholder_rev() deprecated in
  favor of is_branch_rev() and is_clone_rev().
- Misc cleanups and minor bugfixes
#91 3858 Barrie Slaymaker - Final tweaks for 0.5 release
#90 3814 Barrie Slaymaker - TODO items cleaned up a bit
#89 3781 Barrie Slaymaker - Note some testing TODOs
#88 3752 Barrie Slaymaker - TODO: --cd testing, renaming
#87 3720 Barrie Slaymaker - Note some future feature adds in TODO
#86 3692 Barrie Slaymaker - assert_eq() is deprecated
#85 3681 Barrie Slaymaker - VCP now scans much more of real_vss_1 and converts it to revml
#84 3679 Barrie Slaymaker - VCP::Source::vss respects --case-sensitive in more places
#83 3673 Barrie Slaymaker - rev_root sanity check is now case insensitive on Win32
#82 3670 Barrie Slaymaker - Note new TODO
#81 3666 Barrie Slaymaker - vcp can now edit existing .vcp files, for VSS sources and revml dests
#80 3659 Barrie Slaymaker - Minor merges from crashed HD
#79 3554 John Fetkovich added p42cvs_default_filters sub.
 Corrected documentation.
#78 3533 John Fetkovich remove completed rel2abs items
#77 3520 John Fetkovich add item
#76 3484 John Fetkovich fix a prompt name
#75 3483 John Fetkovich removed done item
#74 3482 John Fetkovich done item removed
#73 3468 Barrie Slaymaker - get --output-config-file working with ui (if ui completes,
  which may not happen if the user enters invalid input).
- Note various UI issue in TODO.
#72 3460 Barrie Slaymaker - Revamp Plugin/Source/Dest hierarchy to allow for
  reguritating options in to .vcp files
#71 3434 Barrie Slaymaker Note important testing TODOs
#70 3432 Barrie Slaymaker - "p4 print" to "p4 sync"
#69 3413 Barrie Slaymaker - Delete some completed TODOs
- Note s/$^O.*Win32/is_win32()/g cleanup needed
#68 3405 John Fetkovich added item
#67 3401 Barrie Slaymaker - Note that VCP::Source::cvs should deal gracefully with
  "/module" (as opposed to "module")
#66 3395 John Fetkovich various ui refinements
#65 3392 Barrie Slaymaker Note TODO to document need for VSS cleanup prior to conversion
#64 3375 John Fetkovich more ui changes
#63 3372 Barrie Slaymaker Note gerry's concerns about UI as must-do items
#62 3336 Barrie Slaymaker Add API TODO to add RunWait() calls
#61 3222 Barrie Slaymaker HTML build process tweak TODO
#60 3180 Barrie Slaymaker Add to TODOs
#59 3098 Barrie Slaymaker Convert all length p4 command line calls to us p4 -x -.
       All hail p4 -x -.
#58 3088 Barrie Slaymaker Note p4->cvs problem
#57 3078 Barrie Slaymaker Note more CVS oddities we really should be testing for but
       aren't.
#56 3066 Barrie Slaymaker Set time correctly on integrate and pre-delete add
       changes.
#55 3060 Barrie Slaymaker Note arglist too long error in p4->p4 conversion
#54 3048 Barrie Slaymaker Note some tweaks
#53 3030 Barrie Slaymaker Note cleanup needed for placeholder revs on empty branches
       for CVS.
#52 3021 Barrie Slaymaker Force integrate to branch from proper revision and allow
       it to branch from the predecessor of a deleted file (CVS
       allows file deletes).
#51 3009 John Fetkovich removed completed TODO item
#50 3007 Barrie Slaymaker Read CVS vendor branche tags
#49 3006 Barrie Slaymaker Note the bugs Gerry found.
#48 2954 John Fetkovich removed completed item
#47 2917 Barrie Slaymaker Reorganize TODO
#46 2911 Barrie Slaymaker Document changes, add a few TODOs
#45 2898 Barrie Slaymaker More TODO
#44 2861 Barrie Slaymaker Note need to test -r and -d options.
#43 2838 John Fetkovich Use parse_options rather than using Getopt::Long directly.
#42 2823 John Fetkovich removed already done item
#41 2809 Barrie Slaymaker Implement --repo-id in Plugin.pm, refactor source & dest
       options parsing starting in VCP::Source::cvs (need to
       roll out to other sources and dests), get t/91cvs2revml.t
       passing again (first time in months! branching and
       --continue support works in cvs->foo!).
#40 2802 John Fetkovich Added a source_repo_id to each revision, and repo_id to each
Source and Dest.  The repo_ids include repository type
(cvs,p4,revml,vss,...) and the repo_server fields.  Changed the
$self->...->set() and $self->...->get() lines in VCP::Dest::* to
pass in a conglomerated key value, by passing in the key as an
ARRAY ref.  Also various restructuring in VCP::DB.pm,
VCP::DB_file.pm and VCP::DB_file::sdbm.pm related to this
change.
#39 2776 Barrie Slaymaker Note need to test -r in test suite.
#38 2775 Barrie Slaymaker Note possible future culling improvement
#37 2773 Barrie Slaymaker TODO
#36 2772 Barrie Slaymaker TODO
#35 2771 Barrie Slaymaker Note need for source_repo_id in each rev & what to do with it
#34 2768 John Fetkovich Allow revml to be output in non-indented form with --no-indent
       option on dest::revml
#33 2764 John Fetkovich add --compress switch to dest::revml
       add --uncompress switch to source::revml

       use gzip to compress/uncompress revml files
#32 2763 John Fetkovich test a rerooted revml->p4 to someplace deep in the p4 tree, to make sure
       that VCP is configuring p4 to map the files to the right place.
#31 2758 John Fetkovich Check contents of t/cvsroot_2 in t/91revml2cvs.t
#30 2755 John Fetkovich Completed coversion of tests to IPC::Run3
#29 2746 Barrie Slaymaker Make current and flag things that are now feasible
#28 2721 Barrie Slaymaker VCP::Dest::p4 now uses RevMapDB
#27 2707 Barrie Slaymaker todo items
#26 2706 Barrie Slaymaker Interim checkin
#25 2682 Barrie Slaymaker more underscorification documentation.
#24 2307 Barrie Slaymaker get VCP::Filter::map working, update docs
#23 2293 Barrie Slaymaker Update CHANGES, TODO, improve .vcp files, add --init-cvs
#22 2281 Barrie Slaymaker foo
#21 2267 Barrie Slaymaker factor out cvs2revml, test both --use-cvs and direct modes, with times
#20 2242 Barrie Slaymaker add forthcoming p4 filelog -t
#19 2233 Barrie Slaymaker debug
#18 2229 Barrie Slaymaker reminders to self
#17 2042 Barrie Slaymaker Basic source::p4 branching support
#16 2017 Barrie Slaymaker Interim checkin of id=/base_version_id for revml: and
       branch_diagram:
#15 2009 Barrie Slaymaker lots of fixes, improve core support for branches and VCP::Source::cvs
       now supports branches.
#14 2006 Barrie Slaymaker more preparations for branching support,
       handling of cvs :foo:... CVSROOT specs,
       misc fixes, improvements
#13 2005 Barrie Slaymaker Add code cleanup note
#12 1855 Barrie Slaymaker Major VSS checkin.
 Works on Win32
#11 1809 Barrie Slaymaker VCP::Patch should ignore lineends
#10 1448 Barrie Slaymaker suggestions for NtLkly hack cleanups, reporting.
#9 1367 Barrie Slaymaker lots of docco updates
#8 1358 Barrie Slaymaker Win32 changes
#7 1330 Barrie Slaymaker Ignore cvs lock mgmt warnings in VCP::Source::cvs.
#6 824 Barrie Slaymaker Fix p4 backfilling path math to not mangle paths.
#5 719 Barrie Slaymaker vcp 0.221
#4 695 Barrie Slaymaker Cleaned up support for binary files in VCP::Dest::revml and
altered test suite to deal with it better.  Added some thoughts
to the TODO file.
#3 628 Barrie Slaymaker Cleaned up POD in bin/vcp, added BSD-style license.
#2 620 Barrie Slaymaker Underscorify CVS tags, only warn about undeleted files if
debugging.
#1 478 Barrie Slaymaker 0.05 Mon Dec 18 07:27:53 EST 2000
   - Use `p4 labels //...@label` command as per Rober Cowham's suggestion, with
     the '-s' flag recommended by Christopher Siewald and
     [email protected].  Though it's actually something like

       vcp: running /usr/bin/p4 -u safari -c safari -p localhost:5666 -s files
       //.../NtLkly //...@compiler_a3 //.../NtLkly //...@compiler_may3

     and so //on //for 50 parameters to get the speed up.  I use the
     //.../NtLkly "file" as //a separator between the lists of files in various
     //revisions.  Hope nobody has any files named that :-).  What I should do
     is choose a random label that doesn't occur in the labels list, I guess.
   - VCP::Source::revml and VCP::Dest::revml are now binary, control code, and
     "hibit ASCII" (I know, that's an oxymoron) clean.  The <comment>, <delta>,
     and <content> elements now escape anything other than tab, line feed,
     space, or printable chars (32 <= c <= ASCII 126) using a tag like '<char
     code="0x09">'.  The test suite tests all this.  Filenames should also
     be escaped this way, but I didn't get to that.
   - The decision whether to do deltas or encode the content in base64 is now
     based on how many characters would need to be escaped.
   - We now depend on the users' diff program to have a "-a" option to force it
     to diff even if the files look binary to it.  I need to use Diff.pm and
     adapt it for use on binary data.
   - VCP::Dest::cvs now makes sure that no two consecutive revisions of the
     same file have the same mod_time.  VCP::Source::p4 got so fast at pulling
     revisions from the repositories the test suite sets up that CVS was not
     noticing that files had changed.
   - VCP::Plugin now allows you to set a list of acceptable result codes, since
     we now use p4 in ways that make it return non-zero result codes.
   - VCP::Revs now croaks if you try to add two entries of the same VCP::Rev
     (ie matching filename and rev_id).
   - The <type> tag is now limited to "text" or "binary", and is meant to
     pass that level of info between foreign repositories.
   - The <p4_info> on each file now carries the one line p4 description of
     the file so that p4->p4 transferes can pick out the more detailed
     info.  VCP::Source::p4, VCP::Dest::p4 do this.
   - VCP::{Source,Dest}::{p4,cvs} now set binaryness on added files properly,
     I think.  For p4->p4, the native p4 type is preserved.  For CVS sources,
     seeing the keyword substitution flag 'o' or 'b' implies binaryness, for
     p4, seeing a filetype like qr/u?x?binary/ or qr/x?tempobj/ or "resource"
     implies binaryness (to non-p4 destinations).  NOTE: Seeing a 'o' or 'b'
     in a CVS source only ends up setting the 'b' option on the destination.
     That should be ok for most uses, but we can make it smarter for cvs->cvs
     transfers if need be.