How we moved ScummVM from CVS to SVN
Intro
Here I'll describe how we transformed our ScummVM CVS to SVN. It may be helpful for other big projects facing same challenge.
Of course, SF.net has this nice "import CVS repository", but that one runs automatic cvs2svn script and wanted more:
- Restore connections between moved files
- That includes merge of scummvm-old and scummvm modules. scummvm module was created when we performed major project directory tree restructuring
- Keep subtree with tags and branches for each of our subprojects
Overall idea
Generally connection between moved files in SVN could be restored as shown in following chunk of real ScummVM repository dump:
Revision-number: 89 ... Node-path: trunk/scummvm-old/wince/missing/dirent.h Node-kind: file Node-action: add +Node-copyfrom-rev: 53 +Node-copyfrom-path: trunk/scummvm-old/PocketSCUMM/missing/dirent.h ...
Note lines marked by '+'. Those were added. This successfully restores SVN history link.
The process
To successfully fullfill the task I wrote several simple scripts. I had to do it in scripts, not manually because restoration of connections is pretty lengthy process, and I wanted to minimise repository freeze time. So I first modified dump file locally, and later reapplied my changes to fresh dump. Those scripts are intentionally kept simple and no error checking is performed.
Whole dump takes over 1.4G, so there is no way to edit the file directly. Hence I came with simple idea of extracting just those Revision numbers and Node-paths, so they could be later replaced in original dump.
automation-pass1.pl
wget http://.../scummvm-cvsroot.tar.bz2 rm -rf scummvm scummvm.cvs tar xjf scummvm-cvsroot.tar.bz2 mv scummvm scummvm.cvs mkdir scummvm # We need to combine scummvm-old and scummvm modules mv scummvm.cvs/scummvm scummvm.cvs/scummvm-old scummvm # Due to bug in our CVS repository branch-0-5-0 is both tag and branch, so # we have to force it here cvs2svn --dump-only --force-branch=branch-0-5-0 --dumpfile scummvm.dump scummvm grep -E "^Node-|^Revision-number" scummvm.dump > scummvm.dump.nodes.in cvs2svn --dump-only --dumpfile tools.dump scummvm.cvs/tools grep -E "^Node-|^Revision-number" tools.dump > tools.dump.nodes.in # No efforts were performed with restoring links in following modules cvs2svn --dump-only --dumpfile web.dump scummvm.cvs/web cvs2svn --dump-only --dumpfile scummex.dump scummvm.cvs/scummex cvs2svn --dump-only --dumpfile residual.dump scummvm.cvs/residual cvs2svn --dump-only --dumpfile docs.dump scummvm.cvs/docs cvs2svn --dump-only --dumpfile engine-data.dump scummvm.cvs/engine-data # I performed all work offsite, so I had to transfer these dumps over slow dial-up line bzip2 -9 scummvm.dump.nodes.in tools.dump.nodes.in
This stage took about 1.5 hours. Bottleneck is the disk subsytem. Overall size of produced data is over 1.4G.
Manual editing
Now I opened dump.nodes.in files in XEmacs and started to add those links. First, I searched all words 'delete' and studied each case. I had to consult files layout and CVS log messages to see either those files simply killed or really moved or renamed. Due to fact that some files were really renamed, and there are name clashes between files in directories this was not possible to fully automate the task, however big chunk of it could be scripted.
So what I did is to specify that Node-copyfrom-path: manually and left Node-copyfrom-rev blank. Then I recorded some simple macro in XEmacs, as that was quicker to do than writing yet another script. The macro was something like this:
- It starts on Node-copyfrom-path line.
- Put path in yank buffer
- Kill other windows
- Split window (thus we have 2 views of buffer at the same place)
- Switch to another window
- Search backwards contents of yank buffer
With this I saw revision number in another window. Doublecheck that this is correct place and put that revision number to Node-copyfrom-rev field. However I didn't see any inconsistencies here, so I guess it could insert those numbers fully automatic.
Resulting .nodes file has inserted lines marked with leading + like on example at the top of this page.
automation-pass2.pl
This one is simple. What it does is to merge back those inserted lines and modify all internal path, so it will put all modules into separate directories on SVN repository:
perl merge-dump.pl scummvm.dump.nodes scummvm <scummvm.dump >scummvm.dump.new perl merge-dump.pl tools.dump.nodes tools <tools.dump >tools.dump.new for i in web scummex residual docs engine-data do perl prepare-dump.pl $i <$i.dump >$i.dump.new done
merge-dump.pl
$logfile = shift;
$module = shift;
open(LOG, $logfile) or die "Can't open file $logfile";
$logline = <LOG>;
while(<>) {
$line = $lineorig = $_;
$line =~ s/^Node-path: /Node-path: $module\//;
$line =~ s#^Node-copyfrom-path: (/?)#Node-copyfrom-path: $1$module/#;
print $line;
if ($lineorig eq $logline) {
$logline = <LOG>;
while ($logline =~ /^\+(.*)/) {
print "$1\n";
$logline = <LOG>;
}
}
}
close LOG;
prepare-dump.pl
$module = shift;
while(<>) {
$line = $_;
$line =~ s/^Node-path: /Node-path: $module\//;
$line =~ s#^Node-copyfrom-path: (/?)#Node-copyfrom-path: $1$module/#;
print $line;
}
Second regexp here is tricky, since Node-copyfrom-path could contain either /trunk/scummvm/blah or trunk/scummvm/blah and we have to keep that leading slash of it is present.
So after this stage amount of data on disk doubles since we have both merged and non-merged dumps. I kept non-merget dumps, so pass2 could be redone without performing lengthy pass1 over again.
automation-pass3.sh
At this straightforward stage I create local svn repository
rm -rf svn svnadmin create svn svnadmin load svn < init.dump for i in scummvm tools web scummex residual docs engine-data do svnadmin load svn < $i.dump.new done
It takes another hour, and then I can dump resulting repository with
svnadmin dump svn >scummvmrepo.dump
init.dump files contains skeleton of our new repository layout
init.dump
SVN-fs-dump-format-version: 2 Revision-number: 1 Prop-content-length: 112 Content-length: 112 K 8 svn:date V 27 2001-10-09T14:30:12.000000Z K 7 svn:log V 38 New repository initialized by cvs2svn. PROPS-END Node-path: scummvm Node-kind: dir Node-action: add Node-path: tools Node-kind: dir Node-action: add Node-path: web Node-kind: dir Node-action: add Node-path: docs Node-kind: dir Node-action: add Node-path: scummex Node-kind: dir Node-action: add Node-path: engine-data Node-kind: dir Node-action: add Node-path: residual Node-kind: dir Node-action: add
Final step
Then repository was dumped, bzipped and uploaded to sf.net. Now it is possible to import existing dump, but at time when we do it, we had to submit a PR.