Cine/Specifications

From ScummVM :: Wiki
< Cine
Revision as of 13:31, 3 May 2018 by Buddha^ (talk | contribs) (Found out that Cinematique's compression algorithm is probably Bytekiller 1.3.)
Jump to navigation Jump to search

Cine Specifications (version ε)

This is a place intended for information about the Cinématique engine's internals, file formats etc. Adding any additional information here is encouraged. Information probably is inaccurate in places (e.g. may be specific to Operation Stealth rather than apply to Future Wars and/or Cruise For A Corpse too) so if you have more accurate information, please add it here.

File formats

.ADL files

AdLib music.

.AMI files

NOTE: This file type seems to be specific to Amiga versions of Operation Stealth. Not sure if Future Wars uses this.

It looks like *.AMI files are samples (Loaded a couple of them into Audacity using raw 8-bit signed audio data import and played them and they sounded alright).

.ANI files

Animation.

.H32 files

Probably Roland MT-32 music.

.HP files

Probably PC speaker music files.

.IST files

NOTE: These seem to be specific to the PC versions of Operation Stealth. Don't know about Future Wars.

All filenames of this type start with "BOND" and the files are 15 bytes in size. Here's a list of all the 12 different IST-files I've seen so far:

  • BOND1.IST, BOND4.IST, BOND10.IST, BOND11.IST, BOND14.IST, BOND16.IST, BOND20.IST, BOND22.IST, BOND23.IST, BOND24.IST, BOND31.IST, BOND32.IST

These probably have something to do with Roland MT-32 music as there are corresponding files without the IST extension as well (BOND1, BOND4, ..., BOND32) and the corresponding files without the IST extension contain names of .H32 files (e.g. DEEPSNAR.H32, ELECGTR1.H32, SLAPBAS1.H32, STRSECT1.H32) in addition to other data.

.MIG files

"INTRO.MIG" is the only file I've seen of this type (And that's with Operation Stealth, I'd guess Future Wars hasn't got any files with this extension but that's only a guess).

I'd guess "INTRO.MIG" is Operation Stealth's introduction sequence's music file.

Here are the MD5 checksums of the two versions of the INTRO.MIG file I've seen:

  • 65613c9f127ecf8bd7988aef7e49b698 (1624 bytes, Amiga versions of Operation Stealth)
  • efb187353d289d9b85e1e455734e38f9 (3424 bytes, PC and Atari ST versions of Operation Stealth)

The Amiga version's file starts with the characters "SONG" and then a bit later there reads "A_COEUR.AMI" (AMI files seem to be samples). In the PC and Atari ST versions' file there reads "COEUR___.H32" and values "02", "03", ..., "15". BTW coeur is French and means heart.

.MSG files

A formatted file containing a list of strings.

.MSK files

An image mask (1 bit per pixel).

.OBJ files

A formatted file containing descriptions of objects in a scene. Each object has info about X, Y, Mask, Frame, Costume, Name and Part.

.PI1 files

Possibly some kind of an image mask or image data?

.PRC files

Script files. PRC = Procedure?

.REL files

Script files. REL = Relation?

.SEQ files

Some type of animation data.

.SET files

An image set.

.SPL files

NOTE: This file type seems to be specific to Amiga and Atari ST versions of Operation Stealth. Not sure if Future Wars uses this.

Some type of animation data.

Files without any extension

Examples:

  • BOND1, BOND4, BOND10, BOND11, BOND14, BOND16, BOND20, BOND22, BOND23, BOND24, BOND31, BOND32
    • These files probably have something to do with music. For more info take look at .IST files.

Examples from Operation Stealth's Amiga demo:

  • CELLO2, CELLOSEC, ETOMSIMO, FUZZGUIT, PRO1002, PRO11, PRO1120, PRO13, PRO3, PRO300, PRO69, PRO700, PRO800, PRO9874
    • These seem to be sample files (Possibly identical in format to the .AMI files)

Savegames

NOTE: This is specific to Operation Stealth's PC version's savegame format at the moment.

  • 0x0000: Current disk (Uint16BE)
  • A chunk of current string variables:
    • 0x0002: Current part name (String, 13 bytes)
    • 0x000F: Current procedure name (String, 13 bytes)
    • 0x001C: Current relation name (String, 13 bytes)
    • 0x0029: Current message name (String, 13 bytes)
    • 0x0036: Current background names (8 strings, 13 bytes each)
    • 0x009E: Current Ct name (String, 13 bytes)
  • Object structs:
    • 0x00AB: Entry count (Uint16BE), value discarded on load and assumed to be 255 in the PC version of Operation Stealth
    • 0x00AD: Entry size (Uint16BE), 32 in the PC version of Operation Stealth
    • 0x00AF: Object structs (Entry count structs, entry size bytes each):
    • struct objectStruct:
      • x (Int16BE)
      • y (Int16BE)
      • mask (Uint16BE), seems to be a priority or a Z-buffer value this one
      • frame (Int16BE), indexes the animDataTable
      • costume (Int16BE), maybe this should be named room number instead?
      • name (String, 20 bytes)
      • part (Uint16BE)
  • Palettes:
    • 0x208F: Temporary color palette (256 colors, 3 bytes per color, 768 bytes total)
      • TODO: Make sure whether this is the backup color palette or the active one
    • 0x238F: Color palette (256 colors, 3 bytes per color, 768 bytes total)
      • TODO: Make sure whether this is the active color palette or the backup one
  • 0x268F: Global variables (255 Uint16BE values, 510 bytes total)
  • 0x288D: Zone data table (16 Uint16BE values, 32 bytes total)
  • 0x28AD: Command variables (4 Uint16BE values, 8 bytes total)
  • 0x28B5: Command buffer (String, 80 bytes)
  • 0x2905: Zone query table (16 Uint16BE values, 32 bytes total)
  • 0x2925: Current music name (String, 13 bytes)
  • A chunk of Uint16BE variables:
    • 0x2932: Is music loaded? (Uint16BE, Boolean)
    • 0x2934: Is music playing? (Uint16BE, Boolean)
    • 0x2936: Player's command string's vertical position on-screen (Uint16BE)
    • 0x2938: Some unknown zero value (Uint16BE)
    • 0x293A: Allow player input? (Uint16BE, Boolean)
    • 0x293C: Player command (Uint16BE)
    • 0x293E: commandVar1 (Uint16BE)
    • 0x2940: Is draw command enabled? (Uint16BE, Boolean)
    • 0x2942: var5 (Uint16BE)
    • 0x2944: var4 (Uint16BE)
    • 0x2946: var3 (Uint16BE)
    • 0x2948: var2 (Uint16BE)
    • 0x294A: commandVar2 (Uint16BE)
    • 0x294C: Default menu background color (Uint16BE)
    • 0x295E: adBgVar1 (Uint16BE)
    • 0x2950: currentAdditionalBgIdx (Uint16BE)
    • 0x2952: currentAdditionalBgIdx2 (Uint16BE)
    • 0x2954: additionalBgVScroll (Uint16BE)
    • 0x2956: adBgVar0 (Uint16BE)
    • 0x2958: Is system menu disabled? (Uint16BE, Boolean)
  • Animation data structs:
    • 0x295A: Entry count (Uint16BE), value discarded on load and assumed to be 255 in the PC version of Operation Stealth
    • 0x295C: Entry size (Uint16BE), 36 in the PC version of Operation Stealth
    • 0x295E: Animation data structs (Entry count structs, entry size bytes each):
    • struct animData:
      • Width (Uint16BE)
      • Var1 (Uint16BE)
      • Bits per pixel (Uint16BE), or maybe type?
      • Height (Uint16BE)
      • Data pointer (Uint32LE, Real mode far pointer, only test for equality or inequality with zero!)
      • File index (Int16BE)
      • Frame index (Int16BE)
      • Name (String, 20 bytes)
  • 0x4D3A: Unknown screen parameters (6 parameters, Uint16BE each, 12 bytes total)
  • Global scripts:
    • 0x4D46: Entry count (Uint16BE)
    • 0x4D48: Global script structs (Entry count structs, entry size undetermined)
      • TODO: Determine the entry size used by a global script
  • Object scripts:
    • Entry count (Uint16BE)
    • Object script structs (Entry count structs, entry size undetermined)
      • TODO: Determine the entry size used by an object script
  • seqList elements:
    • Entry count (Uint16BE)
    • seqList element structs (Entry count structs, 32 bytes each)
  • Overlays:
    • Entry count (Uint16BE)
    • Overlay structs (Entry count structs, 22 bytes each)
  • Background incrusts
    • Entry count (Uint16BE)
    • Background incrust structs (Entry count structs, 22 bytes each)

And there the savegame file ends.

Part file format

NOTE: This applies to both Future Wars and Operation Stealth. 

Part file's start:

    Byte  Meaning
    ----- -----------------------------------------------------------
     0-1  Number of elements in this part file (Uint16BE)
     2-3  Entry size (Uint16BE). Normally 0x1E i.e. 30
    ----- -----------------------------------------------------------

Then comes info for each element (Entry size in length each): 

    Byte  Meaning
    ------ -----------------------------------------------------------
     0-13  Name (ASCIIZ string)
    14-17  Data's starting offset in this part file (Uint32BE)
    18-21  Packed size (Uint32BE)
    22-25  Unpacked size (Uint32BE)
    26-29  ???
    ------ -----------------------------------------------------------

After that it's the data for all the elements contained in this part file.

VOL.CNF

NOTE: This file is specific to Operation Stealth.

This file contains list of resource files (e.g. PROCS10, RSC04, SONS2, LABYBASE etc) and a list of files contained in each resource file (e.g. AUTO00.PRC and MASKG.REL in PROCS00).

If the file starts with "ABASECP" the file is compressed, otherwise it's uncompressed. 

Compressed file's start:

    Byte  Meaning
    ----- -----------------------------------------------------------
     0-7  Magic header ("ABASECP" string with the trailing zero)
     8-11 Unpacked size of the data after this header (Uint32BE)
    12-15 Packed size of the data after this header (Uint32BE)
    ----- -----------------------------------------------------------

For a compressed file just read the rest of the file after the header and uncompress it with Delphine's unpacking routine and you've got the same data as you'd have had if you'd have had an unpacked file to start with (Except the header, of course). 

Uncompressed file's start:

    Byte  Meaning
    ----- -----------------------------------------------------------
     0-1  Resource files count (Uint16BE)
     2-3  Entry size (Uint16BE). 0x14 in all tested files so far
    ----- -----------------------------------------------------------

Then come resource files count times resource file info structs of entry size bytes length each: 

Resource file info struct:
x = entry size - 1

    Byte  Meaning
    ----- -----------------------------------------------------------
     0-7  Resource file name string (Possibly no trailing zero!)
     8-x  Unknown data
    ----- -----------------------------------------------------------

Then for each resource file comes a list of files that are in it.

Each list begins with an unsigned 32-bit big endian integer telling the size of the entry. After that come (size / 11) or (size / 13) filenames of length 11 or 13 respectively. Almost as a rule compressed files use filename length of 11 and uncompressed files use filename length of 13 but it's not always so (At least some Amiga version used a compressed 'vol.cnf' file but still used filenames of length 13).

Filenames of length 11 have no separation of the extension and the basename so that's why we have to convert them first. There's no trailing zero in them either and they're always of the full length 11 with padding spaces. Extension can be always found at offset 8 onwards. Filenames of length 13 are okay as they are, no need for converting them. 

Examples of filename mappings:

    "AEROPORTMSG" -> "AEROPORT.MSG"
    "MITRAILLHP " -> "MITRAILL.HP" (Notice the trailing space after the extension)
    "BOND10     " -> "BOND10"
    "GIRL    SET" -> "GIRL.SET"

Compression format

The compression algorithm used by all Delphine's adventure games uses sliding window compression (Quite like LZ77) combined with a fixed non-adaptive entropy coding scheme. It seems that the algorithm is Bytekiller 1.3. This is based on comparing the reverse engineered code in ScummVM with the uncompress-function from Colditz Escape's source code.

The compressed data is in big endian 32-bit chunks, working backwards from the buffer's end. So we start from the data's end and work backwards. Also outputting the unpacked data is done backwards, starting from the destination buffer's end and working backwards byte by byte. 

Compression format:

NOTE: As the whole data consists of unsigned big endian 32-bit integers, I use indexing
      in 32-bit addresses here. By -1 I mean the last 32-bits of the data
      (i.e. bytes src[srcLen-4], src[srcLen-3], src[srcLen-2] and src[srcLen-1]),
      by -2 the second to last 32-bits etc.

    Dword     Meaning
    --------- ------------------------------------------------------------------------
    -1        Unpacked length (Uint32BE).
    -2        Error code (Uint32BE). Xor of the whole packed data in Uint32BE chunks.
    0 - (-3)  The packed data (In Uint32BE chunks).
    --------  ------------------------------------------------------------------------

Bit sequences in the compressed stream

The whole packed data consists of commands and their parameters.

The source stream is read as a bit stream. Behind the scenes it is read in one unsigned big endian 32-bit integer value at a time and when such a chunk becomes depleted of bits then another chunk is read in etc. Each chunk's least significant bit is the first bit read from it, the most significant bit is the last bit read from it etc.

First 32-bit chunk

Because of the way the Delphine's decompressor routine handled the bit stream the first chunk acts in a sort of a peculiar way (Take a look at the functions rcr (Declaration, Definition) and nextBit (Declaration, Definition) in Cine's revision 32689). All other chunks in the source stream always contain full 32 bits but the first chunk contains only 0 to 31 bits. If the first chunk is zero, then we simply discard it and read another chunk. If the first chunk is not zero, then it contains as many source bits as its highest set bit's position is (e.g. 100b contains 2 bits because its highest set bit is at bit position 2, 110101b contains 5 bits, 1b contains 0 bits etc). So all the less significant bits than the first chunk's highest set bit are valid source stream bits (That means the highest set bit in the first chunk is always discarded).

Command length

First bit of a command always tells how many bits the whole command takes. If the first bit of a command is zero, then the whole command (Including the first bit) takes two bits total, otherwise it takes three bits (i.e. the first bit is one).

Command types

There are two possible types of commands:

  • unpackRawBytes(N)
    • Copies N bytes straight from the source stream and writes them to the destination
  • copyRelocatedBytes(OFFSET, N)
    • Copies N bytes from position OFFSET in the sliding window (i.e. from the unpacked buffer)

Commands may have predefined values or a restricted value range for some of their parameters. If a command's parameter isn't a predefined value then it is read from the source stream.

All possible commands

Here are all the possible commands (Including their first bit that tells the command's length): 

Bits  => Action:
0 0   => unpackRawBytes(3 bits + 1)              i.e. unpackRawBytes(1..8)
1 1 1 => unpackRawBytes(8 bits + 9)              i.e. unpackRawBytes(9..264)
0 1   => copyRelocatedBytes(8 bits, 2)           i.e. copyRelocatedBytes(0..255, 2)
1 0 0 => copyRelocatedBytes(9 bits, 3)           i.e. copyRelocatedBytes(0..511, 3)
1 0 1 => copyRelocatedBytes(10 bits, 4)          i.e. copyRelocatedBytes(0..1023, 4)
1 1 0 => copyRelocatedBytes(12 bits, 8 bits + 1) i.e. copyRelocatedBytes(0..4095, 1..256)
Examples of parsing commands

Some examples of parsing the commands:

  • We read one bit from the stream, it's 0. Okay, so the command has length of two bits. We read the second bit, it's 0, so the command is unpackRawBytes(3 bits + 1). That means we read 3 bits from the source stream, let's call that number X. Then we call unpackRawBytes(X + 1).
  • We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 1 and 0 (i.e. 10b). So we've got command copyRelocatedBytes(12 bits, 8 bits + 1). Let's first read 12 bits from the source stream and call that OFFSET. Then let's read 8 bits from the source stream and call that X. Then we'll call copyRelocatedBytes(OFFSET, X + 1).
  • We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 1 and 1 (i.e. 11b). So we've got command unpackedRawBytes(8 bits + 9). Let's then read 8 bits from the source stream and call that X. Then we'll call unpackRawBytes(X + 9).
  • We read one bit from the stream, it's 1. So the command has length of three bits. We read two more bits, they're 0 and 0 (i.e. 00b). So we've got command copyRelocatedBytes(9 bits, 3). Let's read 9 bits from the source stream and call that OFFSET. Then we'll call copyRelocatedBytes(OFFSET, 3).

End of decompression

The unpacking ends when we've written unpacked length bytes into the destination buffer. If at that point the exclusive-or of all the read source stream chunks together equals the error code from the source stream's header (i.e. their exclusive-or is zero), that means the packed data is probably intact (Only probably because the exclusive-or of 32-bit chunks isn't a very good error detection method, it lets errors through relatively easily if compared to other more robust error detection methods like SHA-1 or even CRC).

Fonts

Fonts are loaded from file "TEXTE.DAT".

TEXTE.DAT file's format

    Byte  Meaning
    ------ ------------------------------------------------------------
     0-1   Entry size (16-bit big endian integer)
     2-3   Entry count (16-bit big endian integer)
     4-end Font data (4-bit bitplaned data in 16-bit big endian chunks)
    ------ ------------------------------------------------------------
  • Characters are 16x8 in size
  • Entry size has been 8 in all observed data
  • Entry count divided by entry size gives the number of characters in the font
  • Take a look at gfxConvertSpriteToRaw to see how the font data can be unpacked

Known different font versions

There are 4 known different font versions:

78 characters version
  • Used by most PC, Amiga and Atari ST versions of Future Wars, but also by Operation Stealth's Amiga demo
  • Cinematique-78 characters font.png
85 characters version
  • Used by all observed versions of German Future Wars (Amiga and PC), possibly by Spanish Future Wars too
  • Cinematique-85 characters font.png
90 characters version
  • Used by most PC, Amiga and Atari ST versions of Operation Stealth
  • Cinematique-90 characters font.png
93 characters version
  • Used by all observed versions of German Operation Stealth (Amiga and PC)
  • Cinematique-93 characters font.png
Retrieved from "https://wiki.scummvm.org/index.php?title=Cine/Specifications&oldid=24476"