HOWTO-Convert-Plugins

As part of GSoC'20, there is new structure for plugins: detection code is now included in the main executable or present as a stand-alone plugin.

The existing engines' MetaEngine, thus, needs to be split in two.

New Structure

AdvancedMetaEngineDetection is split out of AdvancedMetaEngine.
- MetaEngineDetection: Houses all methods that are always linked statically. As before, each engine subclasses AdvancedMetaEngineDetection, but only provides detection and related methods.
- MetaEngine: Houses all things that "can" be linked dynamically. These are bridge functions, that connect an Engine with MetaEngine, thus the name. It houses methods like createInstance, which instantiates an actual game engine.
- AdvancedMetaEngineDetection: Inherits from MetaEngineDetection and provides a default detection method.
- AdvancedMetaEngine: Inherits MetaEngine to derive a structure like before.
Most changes are centered around plugins. Two main new helpers are getEngineFromMetaEngine and getMetaEngineFromEngine which do the same thing as they sound like. These help out in various scenarios where you would need the matching engine-metaengine to do something.

The format of each individual engine is largely similar to before, but I will explain in a bit how some things will change.

Changes to existing engines

Most changes are pretty simple to follow. The new structure will need to be kept in mind when working on new engines, To keep things short, I'll use "ME" for MetaEngine, "MED" for MetaEngineDetection, AME & AMED for their advanced counterparts. I'll also use the AGI engine as an example, so it's clear what everything means.

If an engine inherits MED, it should also inherit ME.
The class MED for AGI lives in agi/detection.cpp, and inherits MED. The detection-related code goes here, detection tables, etc.

Example:

class AgiMetaEngineDetection : public MetaEngineDetection {...}

The same engine should use the equivalent meta-counterpart in agi/metaengine.cpp, a file specifically for ME/AME.

Example:

class AgiMetaEngine : public MetaEngine {...}

If an engine uses AMED, it should also use AME.

That's the basic engine split. As for the engines themselves,

Many engines have a custom description for the game. For example, in addition to the ADGameDescription struct, they would have their own, something like:

struct AGIGameDescription {
	ADGameDescription desc;

	int gameID;
	int gameType;
	uint32 features;
	uint16 version;
};

These go into agi/detection.h.
Sometimes the detection-tables will need some enumeration values present in the main engine header file, for example, agi.. To avoid including the whole header which would contain irrelevant information for the executable, a copy of these enums is provided to both - the engine & metaengine.

Simply shift these enumeration values to agi/detection.h. Include it in the main engine header, so everything returns to normal. However, now we can include it in the detection.cpp translation unit, while avoiding the engine header file.

Engines inheriting from MetaEngineDetection will provide their own detection method. An example of this is Scumm. However, detection is not only used during detecting of games, but also during creating an instance. Scumm provides some internal static methods, called detectionImpl which is used both - by MetaEngineDetection::detectGames(...) & by MetaEngine::createInstance(...). The solution for this is to shift the common code inside another header file, detection_internal.h. Then, this is included by both - detection.cpp & metaengine.cpp
- Remember to mark the methods as static inside the header, so the visibility is limited to the TU.
MetaEngineDetect now go into the executable. They provide some naming functions - getName, getEngineId & getOrignalCopyright. Because MetaEngineDetect will need to find a relevant MetaEngine, the implemented MetaEngine should override the getName method, and provide it a name similar to the one found in getEngineId of MetaEngineDetect.
Lastly, after the creation of your files, you need to register them as plugins.

For detection.cpp, use REGISTER_PLUGIN_STATIC(...) macro, as they always go into the executable.
- Keep in mind that the ID should be suffixed with "DETECTION". This is because the ID cannot conflict with the one in metaengine. The type should be a metaengine. Example:

REGISTER_PLUGIN_STATIC(AGI_DETECTION, PLUGIN_TYPE_ENGINE_DETECTION, AgiMetaEngine);

For metaengine.cpp the story is as-is, if the plugin is dynamic, use the dynamic macro - otherwise the static version.

Module objects / Building detection features

Each engine's module.mk file will be adding object files to DETECTION_OBJS. It looks something like this:

...
...
# Include common rules
include $(srcdir)/rules.mk

# Detection objects
DETECT_OBJS += $(MODULE)/detection.o

Remember to add metaengine.o and remove detection.o from the enginename module.
Additional dependencies for detection are also ok. However, remember that engines could also use those files, so it wouldn't always be the case to separate the module-object completely. I will give some examples:
- AGI uses something called wagparser to help in its fallback detection. The engine itself doesn't need it, so we can separate it completely i.e remove from MODULE_OBJS in engines/agi module and add it to DETECT_OBJS inside the same module list.
- Scumm uses 2 files - file.cpp, file_nes.cpp to help with detection. However, the engines themselves need those files too, so we can't separate it completely. Hence, the updated module will look like

MODULE := engines/scumm

...
# Rest of the file
...

# Detection objects
DETECT_OBJS += $(MODULE)/detection.o

# Skip building the following objects if a static
# module is enabled, because it already has the contents.
ifneq ($(ENABLE_SCUMM), STATIC_PLUGIN)
DETECT_OBJS += $(MODULE)/file.o
DETECT_OBJS += $(MODULE)/file_nes.o
endif

The makefiles then take care of adding all the DETECT_OBJS from each of engine module files, regardless of which engine is being enabled or not.

Some engines that would require review

While most of the engines adapted pretty nicely to this structure, some had a somewhat different scenario because of which the base structure was changed a bit. I will list all of them below.

Sci & Wintermute (fallback-detection)

These engines have a struct similar to ADGameDescription, whose contents are overwritten by the fallback detection method. The problem is that these fallback detection methods are heavily dependent on engine resources.
- I tried to make a "lite" version for these classes just to be helpful in detection, but that was a fast fail because it's too dependent on engine resources.
Thus, a workaround would need to be added.
A solution for this is that you would still override fallbackDetect in MetaEngine (which lives in detection.cpp, but instead of detecting games here, you provide a hook to the relevant engine plugin and call a relevant method from there.
- Override a function called fallbackDetectExtern in the MetaEngineConnect class which lives in metaengine.cpp and provide the actual detection here, which is engine-resources dependant. See https://github.com/aryanrawlani28/scummvm/commit/eb960d0814a3dd1d6cd2e7f91d53518f88176476 for more info, and an in-code example.

Mohawk (dialogs)

A while back https://github.com/scummvm/scummvm/pull/2164 added support for in-game custom dialogs.
If you look at some of the engines currently scummvm master tree, which inherit from AMED, you'll see that they provide some extra gui options via an array, ADExtraGuiOptionsMap gameGuiOptions[] and then pass this along to the AMED constructor.
- To not disturb the engine-structure too much, these are still as-is, meaning that without an engine plugin loaded, you'll be able to see these options. (Other things like Keymaps, achievements are still coming from the engine.)
Mohawk however, has the new dialogs in which they could query information from the game, then setup dialog accordingly. As such, it's probably best that they remain in the engine plugin itself.
So, MED's & ME's both have the ability to create custom dialogs. MED's only provide the "Engine" tab inside "Edit Game" option, while ME's provide in-game dialogs, etc.
For now, to clearly distinguish between them, the name buildEngineOptionsWidgetDynamic is given, but this is not really proper. extern & intern like the ones I did with fallback detection were ok, but it could be confusing in this scenario. Hence, this name can probably be changed into something better.

Detection features as a plugin themselves

On some low-hardware end platforms, the executable size bloat would prove difficult.
Making detection features available as a plugin itself would be beneficial in that scenario.

How is it done?

We already had the detection features separated from Engines. They are normally supposed to link directly with the executable, but since we have to have a plugin, we can group all the detection files themselves into a DETECTION type plugin.
In the ScummVM engines/ directory, look for detection.cpp. This is where we include the detection tables and mark the plugin as a PLUGIN_TYPE_DETECTION.
When using configure, --enable-detection-dynamic will be available, which enables the detection objects to be packed into a library.
- When using this, helper methods in PluginManagerUncached will help in loading the detection library, linking the MetaEngines, and eventually clearing those out of memory once we don't need them.
- If not using that, they will be linked statically, but still have an explicit option - --enable-detection-static available.

Modules objects / Building detection features for dynamic detection plugins

As said above, each engine is already populating the detect_objs variable.
In the configuration script, when --enable-detection-dynamic is passed, the Makefiles.common includes the below module:

MODULE := detection // Only added if dynamic-detection enabled

DETECT_OBJS_DYNAMIC=$(addprefix ../,$(DETECT_OBJS)) // Since it lives outside the engines subdirectory, add "../" prefix.

MODULE_OBJS := \
	../engines/detection.o \
	$(DETECT_OBJS_DYNAMIC) // Reuse the already existing list of detection objects

# Reset detect objects, so none of them build into the executable.
DETECT_OBJS :=

PLUGIN := 1

# Include common rules
include $(srcdir)/rules.mk