CVS2SVN

Intro

Here I'll describe how we transformed our ScummVM CVS to SVN. It may be helpful for other big projects facing the same challenge.

Of course, SF.net has this nice "import CVS repository", but that one runs an automatic cvs2svn script and we wanted more:

• Restore connections between moved files
• That includes the merge of scummvm-old and scummvm modules. scummvm module was created when we performed a major project directory tree restructuring
• Keep subtree with tags and branches for each of our subprojects

Overall idea

Generally, the connection between moved files in SVN could be restored as shown in the following chunk of a real ScummVM repository dump:

 Revision-number: 89
 ...
 Node-path: trunk/scummvm-old/wince/missing/dirent.h
 Node-kind: file
 Node-action: add
 +Node-copyfrom-rev: 53
 +Node-copyfrom-path: trunk/scummvm-old/PocketSCUMM/missing/dirent.h
 ...
 Revision-number: 93
 Node-path: trunk/scummvm-old/PocketSCUMM/PocketSCUMM.vco
 Node-action: delete
 Node-path: trunk/scummvm-old/PocketSCUMM/missing/dirent.h
 Node-action: delete

Note lines marked by '+'. Those were added. This successfully restores the SVN history link. 53 is the last revision number where dirent.h file was altered.

The process

To successfully fullfill the task I wrote several simple scripts. I had to do it in scripts, not manually, because restoration of connections is pretty long process, and I wanted to minimize the repository freeze time. So first I modified the dump file locally, and later reapplied my changes to the fresh dump. Those scripts are intentionally kept simple and no error checking is performed.

The whole dump takes over 1.4GB, so there is no way to edit the file directly. Hence I came up with a simple idea of extracting just those Revision numbers and Node-paths, so they could be later replaced in the original dump.

automation-pass1.sh

 wget http://.../scummvm-cvsroot.tar.bz2
 rm -rf scummvm scummvm.cvs
 tar xjf scummvm-cvsroot.tar.bz2
 mv scummvm scummvm.cvs
 mkdir scummvm
 
 # We need to combine scummvm-old and scummvm modules
 mv scummvm.cvs/scummvm scummvm.cvs/scummvm-old scummvm
 
 # Due to bug in our CVS repository branch-0-5-0 is both tag and branch, so
 # we have to force it here
 cvs2svn --dump-only --force-branch=branch-0-5-0 --dumpfile scummvm.dump scummvm
 grep -E "^Node-|^Revision-number" scummvm.dump > scummvm.dump.nodes.in
 
 cvs2svn --dump-only --dumpfile tools.dump scummvm.cvs/tools
 grep -E "^Node-|^Revision-number" tools.dump > tools.dump.nodes.in
 
 # No efforts were performed with restoring links in following modules
 cvs2svn --dump-only --dumpfile web.dump scummvm.cvs/web
 cvs2svn --dump-only --dumpfile scummex.dump scummvm.cvs/scummex
 cvs2svn --dump-only --dumpfile residual.dump scummvm.cvs/residual
 cvs2svn --dump-only --dumpfile docs.dump scummvm.cvs/docs
 cvs2svn --dump-only --dumpfile engine-data.dump scummvm.cvs/engine-data
 
 # I performed all work offsite, so I had to transfer these dumps over slow dial-up line
 bzip2 -9 scummvm.dump.nodes.in tools.dump.nodes.in

This stage took about 1.5 hours. Bottleneck is the disk subsystem. The overall size of produced data is over 1.4GB.

Manual editing

Now I opened the dump.nodes.in files in XEmacs and started to add those links. First, I searched it for the word 'delete' and studied each case. I had to consult files layout and CVS log messages to see if those files were simply killed or really moved or renamed. Due to fact that some files were really renamed, and there are name clashes between files in directories, it was not possible to fully automate the task, although big chunk of it could be scripted.

So what I did is specified that Node-copyfrom-path: manually and left Node-copyfrom-rev blank. Then I recorded a simple macro in XEmacs, as that was quicker to do than writing yet another script. The macro was something like this:

• It starts on Node-copyfrom-path line.
• Put path to yank buffer
• Kill other windows
• Split window (thus we have 2 views of buffer at the same place)
• Switch to another window
• Search backwards contents of the yank buffer

With this I saw the revision number in another window. I doublechecked that this is the correct place and put that revision number to Node-copyfrom-rev field. However I didn't see any inconsistencies here, so I guess it could insert those numbers fully automatic.

Resulting .nodes file had inserted lines marked with leading + like on example at the top of this document.

automation-pass2.sh

This one is simple. It merges back those inserted lines and modifies all internal paths, so it will put all modules into separate directories of the SVN repository:

 perl merge-dump.pl scummvm.dump.nodes scummvm <scummvm.dump >scummvm.dump.new
 perl merge-dump.pl tools.dump.nodes tools <tools.dump >tools.dump.new
 
 for i in web scummex residual docs engine-data
 do
   perl prepare-dump.pl $i <$i.dump >$i.dump.new
 done

merge-dump.pl

 $logfile = shift;
 $module = shift;
 
 open(LOG, $logfile) or die "Can't open file $logfile";
 
 $logline = <LOG>;
 
 while(<>) {
   $line = $lineorig = $_;
 
   $line =~ s/^Node-path: /Node-path: $module\//;
   $line =~ s#^Node-copyfrom-path: (/?)#Node-copyfrom-path: $1$module/#;
 
   print $line;
 
   if ($lineorig eq $logline) {
     $logline = <LOG>;
     while ($logline =~ /^\+(.*)/) {
       print "$1\n";
       $logline = <LOG>;
     }
   }
 }
 
 close LOG;

prepare-dump.pl

 $module = shift;
 
 while(<>) {
   $line = $_;
 
   $line =~ s/^Node-path: /Node-path: $module\//;
   $line =~ s#^Node-copyfrom-path: (/?)#Node-copyfrom-path: $1$module/#;
 
   print $line;
 }

The second regexp here is tricky, since Node-copyfrom-path could contain either /trunk/scummvm/blah or trunk/scummvm/blah and we have to keep that leading slash if it is present.

So after this stage the amount of data on the disk doubles since we have both merged and non-merged dumps. I kept non-merged dumps intact, so pass2 could be redone without performing lengthy pass1 over again.

automation-pass3.sh

At this straightforward stage I create local svn repository

 rm -rf svn
 svnadmin create svn
 svnadmin load svn < init.dump
 
 for i in scummvm tools web scummex residual docs engine-data
 do
   svnadmin load svn < $i.dump.new
 done

It takes another hour, and then I can dump resulting repository with

 svnadmin dump svn >scummvmrepo.dump

init.dump files contains skeleton of our new repository layout

init.dump

 SVN-fs-dump-format-version: 2
 
 Revision-number: 1
 Prop-content-length: 112
 Content-length: 112
 
 K 8
 svn:date
 V 27
 2001-10-09T14:30:12.000000Z
 K 7
 svn:log
 V 38
 New repository initialized by cvs2svn.
 PROPS-END
 
 Node-path: scummvm
 Node-kind: dir
 Node-action: add
 
 
 Node-path: tools
 Node-kind: dir
 Node-action: add
 
 
 Node-path: web
 Node-kind: dir
 Node-action: add
 
 
 Node-path: docs
 Node-kind: dir
 Node-action: add
 
 
 Node-path: scummex
 Node-kind: dir
 Node-action: add
 
 
 Node-path: engine-data
 Node-kind: dir
 Node-action: add
 
 
 Node-path: residual
 Node-kind: dir
 Node-action: add
 
 

Final step

Last step is dump the repository, bzip it and upload to sf.net. Later SF.net added possibility to import an existing dump automagically, but at the time we performed the move, we had to submit a PR.