Advanced Detector

Advanced Detector

If your engine supports a large number of games, or variants then detecting them can be tricky.

As some of the game variants will have files with the same name, but differing contents, detection by filename alone is not sufficient, and a fair number of variants may only differ by small sections of data within the entire file, so opening the file and looking for a "magic header" is not reliable either.
So instead you will take a checksum or even better a hash of the file to detect the exact version, then you will need to write this code to open the files and run this check into your custom MetaEngine...

Sounds like a lot of work?
Well to avoid every engine author having to do this themselves (and the codebase ending up with the maintenance headache of 20+ implementations of this which are almost, but not exactly the same!), the ScummVM Infrastructure Team have provided the Advanced Detector!
This provides a standard framework for filename and MD5 based game detection.

The code for this can be found in engines/advancedDetector.*

To use this, you will have to follow the instructions here, but you will subclass AdvancedMetaEngine instead within your engine's detection.h and detection.cpp.

All you will have to provide is a standard data table of ADGameDescription entries describing each game variant, which is usually placed in a separate detection_tables.h header, which is included in detection.cpp for use there.

This structure plus other parameters are passed to the AdvancedMetaEngine constructor, which can also contain overrides of the default parameters for detection e.g. _md5Bytes is the number of bytes used for the MD5 hash for each file, etc.

It is suggested you consult the code and header comments in engines/advancedDetector.* and look at the examples provided by current engines for a more complete example.

Game detection entry in ScummVM config file

When you look into your .scummvmrc or scummvm.ini (depending on the platform), you will find that generally it has following structure <syntax type="INI"> [scummvm] globalkey1=foo globalkey2=bar versioninfo=1.5.0git2516-g30a372d

[monkey2-vga] description=Monkey Island 2: LeChuck's Revenge (DOS/English) path=/Users/sev/games/scumm/monkey2 gameid=monkey2 language=en platform=pc </syntax>

What you see here is several sections designated by identifiers in square brackets and set of key/value pairs belonging to each such section.

The main section with predefined name 'scummvm' contains global options, which are mainly editable in Options dialog in GUI. Then there go sections for each separate game. Additionally some of ports define their own service sections.

Name of each game is what we are calling target. Target, which is in the sample above specified as monkey2-vga is user-editable identifier unique to the specific user, and could be used for launching the game from command line.

Then each entry has description which is also user-editable, path to the game, and gameid. gameid is a service name which identifies the game within whole ScummVM. There should be no clashes, and each engine knows which gameids it does support. First engine which finds a match for a given gameid will be used to run the game. This is why it is important to keep this ID unique, since there is no guarantee in sequence of engines which ScummVM probes when launching a game.

Keys platform and language are used for narrowing down the possible game candidate but are fully optional.

How Advanced Detector works

Advanced detector tries to match files in probed directory against specified lists of file characteristics provided in an array of ADGameDescription structures. It takes into account MD5 sum of the file or its first several hundred bytes, its size and name. It creates list of candidates which then it tries to narrow down to a single ADGameDescription instance unless it is told to do otherwise. In case of disambiguates it returns list of games.

It is important to know, that currently there are in fact two modes of Advanced Detector. First one is used during the game detection when user tries to add a game (detection mode), and second one when the user launches already detected game (running mode). Both uses call same method findGames() which potentially could return list of games. In detection mode the user is then represented with list of games to choose from, but in the running mode in case findGames() method returns more than one game, only first one in the list will be used. This may lead to situation when the game gets detected but doesn't run, thus it is important to test detection and avoid any disambiguates. This is also the main reason for existing of some features in Advanced Detector which are geared towards resolving such conflicts.

In the running mode Advanced Detector tries to match as many information stored in the config game entry as possible. Typical keys it matches against are gameid, platform and language, but it may also use extra when instructed to do so.

In case there are no matches in the ADGameDescription list, there are two additional fallback detection modes. One is file-based detection, which matches just the file names, and second one is a hook which gets called and could contain code of any complexity. Most prominent example of advanced fallback detection is SCI engine.

Generated targets

Targets generated by Advanced Detector have the following structure:

 GAMEID-DEMO-CD-PLATFORM-LANG

The target generation is highly affected by AD flags. The flags which have influence are: ADGF_CD, ADGF_DEMO, ADGF_DROPLANGUAGE.

PlainGameDescriptor table

<syntax type="C++"> struct PlainGameDescriptor { const char *gameid; const char *description; }; </syntax>

This table contains all gameids which are known by the engine. Also each gameid contains full human-readable description, which goes to description field in ScummVM configuration file.

Only gameid which are present in this table could be used in ADGameDescription table.

Typical PlainGameDescriptor table:

<syntax type="C++"> static const PlainGameDescriptor cineGames[] = { {"cine", "Cinematique evo.1 engine game"}, {"fw", "Future Wars"}, {"os", "Operation Stealth"}, {0, 0} }; </syntax>

Please note that it is NULL-terminated, and also contains generic gameid cine which is used by fallback detection.

ADGameDescription table

ADGameDescription table has the following structure:

<syntax type="C++"> struct ADGameDescription { const char *gameid; const char *extra; ADGameFileDescription filesDescriptions[14]; Common::Language language; Common::Platform platform; uint32 flags; const char *guioptions; }; </syntax>

gameid -- This is gameid. Mainly it is used for taking the game description from PlainGameDescriptor table.

extra -- This is used to distinguish between different variants of a game. The content of this field is inserted in the generated description for the config file game entry. In case kADFlagUseExtraAsHint ADFlag is set, contents of this field is stored in config file, and is used to additionally distinguish between game variants. In case ADGF_USEEXTRAASTITLE game flag is set, contents of this field will be put into description rather than one extracted from 'PlainGameDescriptor table.

filesDescriptions -- list of individual file entries used for detection. 13 files (last is zero terminated) is the max number of files used in ScummVM currently. We are forced to specify hardcoded number due to C++ limitation for defining const arrays.

language -- language of the game variant.

platform -- platofrm of the game variant

flags -- game feature flags. Contains both engine-specific ones as well as global ones (see ADGameFlags)

guioptions -- game features which are user controllable. Basically this list reflects which features of GUI should be turned on or off in order to minimize user confusion. For instance, there is no point in changing game language in single language games or have MIDI controls with game which supports only digital music. (See GUI Options)

Typical ADGameDescription table will look as follows:

<syntax type="C++"> static const ADGameDescription gameDescriptions[] = { { "fw", "", AD_ENTRY1("part01", "61d003202d301c29dd399acfb1354310"), Common::EN_ANY, Common::kPlatformPC, ADGF_NO_FLAGS, GUIO0() }, { AD_TABLE_END_MARKER, 0, 0 } }; </syntax>

ADGameFileDescription structure

<syntax type="C++"> struct ADGameFileDescription { const char *fileName; ///< Name of described file. uint16 fileType; ///< Optional. Not used during detection, only by engines. const char *md5; ///< MD5 of (the beginning of) the described file. Optional. Set to NULL to ignore. int32 fileSize; ///< Size of the described file. Set to -1 to ignore. }; </syntax>

fileName -- name of the file. It is case insensitive, but historically we use lowercase names.

fileType -- rarely used field where ADGameFileDescription structure is used by the engine. May specify music file, script file, etc.

md5 -- MD5 of the file. Most often it is MD5 of the beginning of the file for performance reasons. See _md5Bytes setting of AdvancedMetaEngine. If set to NULL, the md5 is not used in detection and the entry matches against any content.

fileSize -- file size in bytes. Optional too, set to -1 in order to match against any file size.

Game Entry flags ADGameFlags

Game flags are used to tell the engine which features this particular game has. There are both engine-specific and Advanced Detector-specific game flags. The latter besides being more or less universal also affect the detection behaviour.

ADGF_ADDENGLISH -- Used for dual language games. In this case user will be present with selection between localised and English version of the game. Affects GUIOs.

ADGF_CD -- Specifies a CD version. Generated target will get '-cd' suffix.

ADGF_DEMO -- Specifies a game demo. Generated target will get '-demo' suffix.

ADGF_DROPLANGUAGE -- generated target will not have language specified. Used mainly for multilanguage games which have language selector internally. Thus the language will be selected within the game and the setting stored in the config file, but the game entry will stay intact.

ADGF_MACRESFORK

ADGF_NO_FLAGS -- No flags are set.

ADGF_PIRATED -- Specifies a blacklisted game. The game will be detected but refuse to run.

There are known hacked variants for some of the games exist in the wild. We used to ignore user reports on them, but with the number of engines growing it became tough to remember that this particular game is really a hack. If it is widespread enough, we were getting recurrent reports that the game is not detected. To avoid this situation we now accept md5s of such games but mark them accordingly.

ADGF_TESTING -- Specifies game which was announced for public testing. The user will get relevant warning when launching the game.

ADGF_UNSTABLE -- Specifies game which is not publicly supported and is in a heavy development. The user will get relevant warning when launching the game.

ADGF_USEEXTRAASTITLE -- Instead of description specified in PlainGameDescriptor table, extra field will be used as game description. Good example is AGI fan games where the game title is known but it is not feasible to add it to PlainGameDescriptor table, or minor composer engine demos with games combined for the same reason.

Advanced Detector flags ADFlags

kADFlagUseExtraAsHint -- Specify this flag in situation when there are more than single game stored in the same directory. E.g. there is no way to know which game the user wants to run without asking him. The typical example is VGA version of Lure of the Temptress which contained both EGA and VGA datafiles in game directory.

Upgrading obsolete gameids

<syntax type="C++"> static const Engines::ObsoleteGameID obsoleteGameIDsTable[] = {

       {"simon1acorn", "simon1", Common::kPlatformAcorn},
       {"simon1amiga", "simon1", Common::kPlatformAmiga},
       {"simon2talkie", "simon2", Common::kPlatformPC},
       {"simon2mac", "simon2", Common::kPlatformMacintosh},
       {"simon2win", "simon2", Common::kPlatformWindows},
       {0, 0, Common::kPlatformUnknown}

}; </syntax>

AdvancedMetaEngine

Is a generic MetaEngine wrapper aware of Advanced Detector. It should be used whenever AD is used.

Engine constructor

<syntax type="C++"> AdvancedMetaEngine(const void *descs, uint descItemSize, const PlainGameDescriptor *gameids); </syntax>

descs is pointed to list ADGameDescription structures, or their supersets.

descItemSize is sizeof of the descs element used for iterating over it.

gameids is pointed to list of PlainGameDescriptor structures defining supported gameids.

Additional Advanced MetaEngine parameters

_md5bytes -- number of bytes used to compute md5. If set to 0 then whole file will be used. Beware of long files though since that can dramatically slow down the detection. Typically sane value is 5000 bytes, but often experiment is required for many game variants with subtle differences.

_singleid -- Used to override gameid. A recommended setting to prevent global gameid pollution. With this option set, the gameid effectively turns into engineid.

In the past we started to have clashes in game names, thus the option was introduced. Also it was mentioned that in the ideal world it should be enough to point just the game directory and ScummVM correctly detects and runs the game. This is a step towards this direction, however there are several cases when it is not possible to identify the game to run, particularly in those cases when there are more than single game stored in a directory.

_flags -- same as individual game flags but user for engine-wide settings. For instance, we know for sure that all games in the engine are unstable, so instead of modifying every game entry, we specify it here.

_guioptions -- same as individual GUI options, but applied engine-wide. For example, when none of the games have speech, we may specify it in this spot.

_maxScanDepth -- Maximum traversal depth for directories. Default is 1, that is do not go inside of subdirectories for detection.

_directoryGlobs -- Case-insesitive list of nested directoriy globs AD will search games in. Null-terminated. Must be set if detection should go into subdirectories.

AdvancedMetaEngine usage

TODO