Difference between revisions of "Cine/Specifications"
(Found out that Cinematique's compression algorithm is probably Bytekiller 1.3.) |
(Added link to ScummVM's cine decompression routine for ease of comparison with the Colditz Escape's code.) |
||
Line 311: | Line 311: | ||
combined with a fixed non-adaptive [http://en.wikipedia.org/wiki/Entropy_coding entropy coding] | combined with a fixed non-adaptive [http://en.wikipedia.org/wiki/Entropy_coding entropy coding] | ||
scheme. It seems that the algorithm is [http://www.pouet.net/prod.php?which=47994 Bytekiller 1.3]. | scheme. It seems that the algorithm is [http://www.pouet.net/prod.php?which=47994 Bytekiller 1.3]. | ||
This is based on comparing the reverse engineered | This is based on comparing the reverse engineered | ||
[https://github.com/scummvm/scummvm/blob/branch-2-0-0/engines/cine/unpack.cpp#L92 decompression routine] | |||
in ScummVM with the | |||
[https://github.com/aperture-software/colditz-escape/blob/v1.2/low-level.c#L150 uncompress]-function | [https://github.com/aperture-software/colditz-escape/blob/v1.2/low-level.c#L150 uncompress]-function | ||
from [https://github.com/aperture-software/colditz-escape Colditz Escape]'s source code. | from [https://github.com/aperture-software/colditz-escape Colditz Escape]'s source code. |
Revision as of 13:34, 3 May 2018
Cine Specifications (version ε)
This is a place intended for information about the Cinématique engine's internals, file formats etc. Adding any additional information here is encouraged. Information probably is inaccurate in places (e.g. may be specific to Operation Stealth rather than apply to Future Wars and/or Cruise For A Corpse too) so if you have more accurate information, please add it here.
File formats
.ADL files
AdLib music.
.AMI files
NOTE: This file type seems to be specific to Amiga versions of Operation Stealth. Not sure if Future Wars uses this.
It looks like *.AMI files are samples (Loaded a couple of them into Audacity using raw 8-bit signed audio data import and played them and they sounded alright).
.ANI files
Animation.
.H32 files
Probably Roland MT-32 music.
.HP files
Probably PC speaker music files.
.IST files
NOTE: These seem to be specific to the PC versions of Operation Stealth. Don't know about Future Wars.
All filenames of this type start with "BOND" and the files are 15 bytes in size. Here's a list of all the 12 different IST-files I've seen so far:
- BOND1.IST, BOND4.IST, BOND10.IST, BOND11.IST, BOND14.IST, BOND16.IST, BOND20.IST, BOND22.IST, BOND23.IST, BOND24.IST, BOND31.IST, BOND32.IST
These probably have something to do with Roland MT-32 music as there are corresponding files without the IST extension as well (BOND1, BOND4, ..., BOND32) and the corresponding files without the IST extension contain names of .H32 files (e.g. DEEPSNAR.H32, ELECGTR1.H32, SLAPBAS1.H32, STRSECT1.H32) in addition to other data.
.MIG files
"INTRO.MIG" is the only file I've seen of this type (And that's with Operation Stealth, I'd guess Future Wars hasn't got any files with this extension but that's only a guess).
I'd guess "INTRO.MIG" is Operation Stealth's introduction sequence's music file.
Here are the MD5 checksums of the two versions of the INTRO.MIG file I've seen:
- 65613c9f127ecf8bd7988aef7e49b698 (1624 bytes, Amiga versions of Operation Stealth)
- efb187353d289d9b85e1e455734e38f9 (3424 bytes, PC and Atari ST versions of Operation Stealth)
The Amiga version's file starts with the characters "SONG" and then a bit later there reads "A_COEUR.AMI" (AMI files seem to be samples). In the PC and Atari ST versions' file there reads "COEUR___.H32" and values "02", "03", ..., "15". BTW coeur is French and means heart.
.MSG files
A formatted file containing a list of strings.
.MSK files
An image mask (1 bit per pixel).
.OBJ files
A formatted file containing descriptions of objects in a scene. Each object has info about X, Y, Mask, Frame, Costume, Name and Part.
.PI1 files
Possibly some kind of an image mask or image data?
.PRC files
Script files. PRC = Procedure?
.REL files
Script files. REL = Relation?
.SEQ files
Some type of animation data.
.SET files
An image set.
.SPL files
NOTE: This file type seems to be specific to Amiga and Atari ST versions of Operation Stealth. Not sure if Future Wars uses this.
Some type of animation data.
Files without any extension
Examples:
- BOND1, BOND4, BOND10, BOND11, BOND14, BOND16, BOND20, BOND22, BOND23, BOND24, BOND31, BOND32
- These files probably have something to do with music. For more info take look at .IST files.
Examples from Operation Stealth's Amiga demo:
- CELLO2, CELLOSEC, ETOMSIMO, FUZZGUIT, PRO1002, PRO11, PRO1120, PRO13, PRO3, PRO300, PRO69, PRO700, PRO800, PRO9874
- These seem to be sample files (Possibly identical in format to the .AMI files)
Savegames
NOTE: This is specific to Operation Stealth's PC version's savegame format at the moment.
- 0x0000: Current disk (Uint16BE)
- A chunk of current string variables:
- 0x0002: Current part name (String, 13 bytes)
- 0x000F: Current procedure name (String, 13 bytes)
- 0x001C: Current relation name (String, 13 bytes)
- 0x0029: Current message name (String, 13 bytes)
- 0x0036: Current background names (8 strings, 13 bytes each)
- 0x009E: Current Ct name (String, 13 bytes)
- Object structs:
- 0x00AB: Entry count (Uint16BE), value discarded on load and assumed to be 255 in the PC version of Operation Stealth
- 0x00AD: Entry size (Uint16BE), 32 in the PC version of Operation Stealth
- 0x00AF: Object structs (Entry count structs, entry size bytes each):
- struct objectStruct:
- x (Int16BE)
- y (Int16BE)
- mask (Uint16BE), seems to be a priority or a Z-buffer value this one
- frame (Int16BE), indexes the animDataTable
- costume (Int16BE), maybe this should be named room number instead?
- name (String, 20 bytes)
- part (Uint16BE)
- Palettes:
- 0x208F: Temporary color palette (256 colors, 3 bytes per color, 768 bytes total)
- TODO: Make sure whether this is the backup color palette or the active one
- 0x238F: Color palette (256 colors, 3 bytes per color, 768 bytes total)
- TODO: Make sure whether this is the active color palette or the backup one
- 0x208F: Temporary color palette (256 colors, 3 bytes per color, 768 bytes total)
- 0x268F: Global variables (255 Uint16BE values, 510 bytes total)
- 0x288D: Zone data table (16 Uint16BE values, 32 bytes total)
- 0x28AD: Command variables (4 Uint16BE values, 8 bytes total)
- 0x28B5: Command buffer (String, 80 bytes)
- 0x2905: Zone query table (16 Uint16BE values, 32 bytes total)
- 0x2925: Current music name (String, 13 bytes)
- A chunk of Uint16BE variables:
- 0x2932: Is music loaded? (Uint16BE, Boolean)
- 0x2934: Is music playing? (Uint16BE, Boolean)
- 0x2936: Player's command string's vertical position on-screen (Uint16BE)
- 0x2938: Some unknown zero value (Uint16BE)
- 0x293A: Allow player input? (Uint16BE, Boolean)
- 0x293C: Player command (Uint16BE)
- 0x293E: commandVar1 (Uint16BE)
- 0x2940: Is draw command enabled? (Uint16BE, Boolean)
- 0x2942: var5 (Uint16BE)
- 0x2944: var4 (Uint16BE)
- 0x2946: var3 (Uint16BE)
- 0x2948: var2 (Uint16BE)
- 0x294A: commandVar2 (Uint16BE)
- 0x294C: Default menu background color (Uint16BE)
- 0x295E: adBgVar1 (Uint16BE)
- 0x2950: currentAdditionalBgIdx (Uint16BE)
- 0x2952: currentAdditionalBgIdx2 (Uint16BE)
- 0x2954: additionalBgVScroll (Uint16BE)
- 0x2956: adBgVar0 (Uint16BE)
- 0x2958: Is system menu disabled? (Uint16BE, Boolean)
- Animation data structs:
- 0x295A: Entry count (Uint16BE), value discarded on load and assumed to be 255 in the PC version of Operation Stealth
- 0x295C: Entry size (Uint16BE), 36 in the PC version of Operation Stealth
- 0x295E: Animation data structs (Entry count structs, entry size bytes each):
- struct animData:
- Width (Uint16BE)
- Var1 (Uint16BE)
- Bits per pixel (Uint16BE), or maybe type?
- Height (Uint16BE)
- Data pointer (Uint32LE, Real mode far pointer, only test for equality or inequality with zero!)
- File index (Int16BE)
- Frame index (Int16BE)
- Name (String, 20 bytes)
- 0x4D3A: Unknown screen parameters (6 parameters, Uint16BE each, 12 bytes total)
- Global scripts:
- 0x4D46: Entry count (Uint16BE)
- 0x4D48: Global script structs (Entry count structs, entry size undetermined)
- TODO: Determine the entry size used by a global script
- Object scripts:
- Entry count (Uint16BE)
- Object script structs (Entry count structs, entry size undetermined)
- TODO: Determine the entry size used by an object script
- seqList elements:
- Entry count (Uint16BE)
- seqList element structs (Entry count structs, 32 bytes each)
- Overlays:
- Entry count (Uint16BE)
- Overlay structs (Entry count structs, 22 bytes each)
- Background incrusts
- Entry count (Uint16BE)
- Background incrust structs (Entry count structs, 22 bytes each)
And there the savegame file ends.
Part file format
NOTE: This applies to both Future Wars and Operation Stealth.
Part file's start: Byte Meaning ----- ----------------------------------------------------------- 0-1 Number of elements in this part file (Uint16BE) 2-3 Entry size (Uint16BE). Normally 0x1E i.e. 30 ----- -----------------------------------------------------------
Then comes info for each element (Entry size in length each):
Byte Meaning ------ ----------------------------------------------------------- 0-13 Name (ASCIIZ string) 14-17 Data's starting offset in this part file (Uint32BE) 18-21 Packed size (Uint32BE) 22-25 Unpacked size (Uint32BE) 26-29 ??? ------ -----------------------------------------------------------
After that it's the data for all the elements contained in this part file.
VOL.CNF
NOTE: This file is specific to Operation Stealth.
This file contains list of resource files (e.g. PROCS10, RSC04, SONS2, LABYBASE etc) and a list of files contained in each resource file (e.g. AUTO00.PRC and MASKG.REL in PROCS00).
If the file starts with "ABASECP" the file is compressed, otherwise it's uncompressed.
Compressed file's start: Byte Meaning ----- ----------------------------------------------------------- 0-7 Magic header ("ABASECP" string with the trailing zero) 8-11 Unpacked size of the data after this header (Uint32BE) 12-15 Packed size of the data after this header (Uint32BE) ----- -----------------------------------------------------------
For a compressed file just read the rest of the file after the header and uncompress it with Delphine's unpacking routine and you've got the same data as you'd have had if you'd have had an unpacked file to start with (Except the header, of course).
Uncompressed file's start: Byte Meaning ----- ----------------------------------------------------------- 0-1 Resource files count (Uint16BE) 2-3 Entry size (Uint16BE). 0x14 in all tested files so far ----- -----------------------------------------------------------
Then come resource files count times resource file info structs of entry size bytes length each:
Resource file info struct: x = entry size - 1 Byte Meaning ----- ----------------------------------------------------------- 0-7 Resource file name string (Possibly no trailing zero!) 8-x Unknown data ----- -----------------------------------------------------------
Then for each resource file comes a list of files that are in it.
Each list begins with an unsigned 32-bit big endian integer telling the size of the entry. After that come (size / 11) or (size / 13) filenames of length 11 or 13 respectively. Almost as a rule compressed files use filename length of 11 and uncompressed files use filename length of 13 but it's not always so (At least some Amiga version used a compressed 'vol.cnf' file but still used filenames of length 13).
Filenames of length 11 have no separation of the extension and the basename so that's why we have to convert them first. There's no trailing zero in them either and they're always of the full length 11 with padding spaces. Extension can be always found at offset 8 onwards. Filenames of length 13 are okay as they are, no need for converting them.
Examples of filename mappings: "AEROPORTMSG" -> "AEROPORT.MSG" "MITRAILLHP " -> "MITRAILL.HP" (Notice the trailing space after the extension) "BOND10 " -> "BOND10" "GIRL SET" -> "GIRL.SET"
Compression format
The compression algorithm used by all Delphine's adventure games uses sliding window compression (Quite like LZ77) combined with a fixed non-adaptive entropy coding scheme. It seems that the algorithm is Bytekiller 1.3. This is based on comparing the reverse engineered decompression routine in ScummVM with the uncompress-function from Colditz Escape's source code.
The compressed data is in big endian 32-bit chunks, working backwards from the buffer's end. So we start from the data's end and work backwards. Also outputting the unpacked data is done backwards, starting from the destination buffer's end and working backwards byte by byte.
Compression format: NOTE: As the whole data consists of unsigned big endian 32-bit integers, I use indexing in 32-bit addresses here. By -1 I mean the last 32-bits of the data (i.e. bytes src[srcLen-4], src[srcLen-3], src[srcLen-2] and src[srcLen-1]), by -2 the second to last 32-bits etc. Dword Meaning --------- ------------------------------------------------------------------------ -1 Unpacked length (Uint32BE). -2 Error code (Uint32BE). Xor of the whole packed data in Uint32BE chunks. 0 - (-3) The packed data (In Uint32BE chunks). -------- ------------------------------------------------------------------------
Bit sequences in the compressed stream
The whole packed data consists of commands and their parameters.
The source stream is read as a bit stream. Behind the scenes it is read in one unsigned big endian 32-bit integer value at a time and when such a chunk becomes depleted of bits then another chunk is read in etc. Each chunk's least significant bit is the first bit read from it, the most significant bit is the last bit read from it etc.
First 32-bit chunk
Because of the way the Delphine's decompressor routine handled the bit stream the first chunk acts in a sort of a peculiar way (Take a look at the functions rcr (Declaration, Definition) and nextBit (Declaration, Definition) in Cine's revision 32689). All other chunks in the source stream always contain full 32 bits but the first chunk contains only 0 to 31 bits. If the first chunk is zero, then we simply discard it and read another chunk. If the first chunk is not zero, then it contains as many source bits as its highest set bit's position is (e.g. 100b contains 2 bits because its highest set bit is at bit position 2, 110101b contains 5 bits, 1b contains 0 bits etc). So all the less significant bits than the first chunk's highest set bit are valid source stream bits (That means the highest set bit in the first chunk is always discarded).
Command length
First bit of a command always tells how many bits the whole command takes. If the first bit of a command is zero, then the whole command (Including the first bit) takes two bits total, otherwise it takes three bits (i.e. the first bit is one).
Command types
There are two possible types of commands:
- unpackRawBytes(N)
- Copies N bytes straight from the source stream and writes them to the destination
- copyRelocatedBytes(OFFSET, N)
- Copies N bytes from position OFFSET in the sliding window (i.e. from the unpacked buffer)
Commands may have predefined values or a restricted value range for some of their parameters. If a command's parameter isn't a predefined value then it is read from the source stream.
All possible commands
Here are all the possible commands (Including their first bit that tells the command's length):
Bits => Action: 0 0 => unpackRawBytes(3 bits + 1) i.e. unpackRawBytes(1..8) 1 1 1 => unpackRawBytes(8 bits + 9) i.e. unpackRawBytes(9..264) 0 1 => copyRelocatedBytes(8 bits, 2) i.e. copyRelocatedBytes(0..255, 2) 1 0 0 => copyRelocatedBytes(9 bits, 3) i.e. copyRelocatedBytes(0..511, 3) 1 0 1 => copyRelocatedBytes(10 bits, 4) i.e. copyRelocatedBytes(0..1023, 4) 1 1 0 => copyRelocatedBytes(12 bits, 8 bits + 1) i.e. copyRelocatedBytes(0..4095, 1..256)
Examples of parsing commands
Some examples of parsing the commands:
- We read one bit from the stream, it's 0. Okay, so the command has length of two bits. We read the second bit, it's 0, so the command is unpackRawBytes(3 bits + 1). That means we read 3 bits from the source stream, let's call that number X. Then we call unpackRawBytes(X + 1).
- We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 1 and 0 (i.e. 10b). So we've got command copyRelocatedBytes(12 bits, 8 bits + 1). Let's first read 12 bits from the source stream and call that OFFSET. Then let's read 8 bits from the source stream and call that X. Then we'll call copyRelocatedBytes(OFFSET, X + 1).
- We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 1 and 1 (i.e. 11b). So we've got command unpackedRawBytes(8 bits + 9). Let's then read 8 bits from the source stream and call that X. Then we'll call unpackRawBytes(X + 9).
- We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 0 and 0 (i.e. 00b). So we've got command copyRelocatedBytes(9 bits, 3). Let's read 9 bits from the source stream and call that OFFSET. Then we'll call copyRelocatedBytes(OFFSET, 3).
End of decompression
The unpacking ends when we've written unpacked length bytes into the destination buffer. If at that point the exclusive-or of all the read source stream chunks together equals the error code from the source stream's header (i.e. their exclusive-or is zero), that means the packed data is probably intact (Only probably because the exclusive-or of 32-bit chunks isn't a very good error detection method, it lets errors through relatively easily if compared to other more robust error detection methods like SHA-1 or even CRC).
Fonts
Fonts are loaded from file "TEXTE.DAT".
TEXTE.DAT file's format
Byte Meaning ------ ------------------------------------------------------------ 0-1 Entry size (16-bit big endian integer) 2-3 Entry count (16-bit big endian integer) 4-end Font data (4-bit bitplaned data in 16-bit big endian chunks) ------ ------------------------------------------------------------
- Characters are 16x8 in size
- Entry size has been 8 in all observed data
- Entry count divided by entry size gives the number of characters in the font
- Take a look at gfxConvertSpriteToRaw to see how the font data can be unpacked
Known different font versions
There are 4 known different font versions:
78 characters version
- Used by most PC, Amiga and Atari ST versions of Future Wars, but also by Operation Stealth's Amiga demo
85 characters version
- Used by all observed versions of German Future Wars (Amiga and PC), possibly by Spanish Future Wars too