View source for HOWTO-Convert-Plugins

As part of GSoC'20, there is new structure for plugins: detection code is now included in the main executable or present as a stand-alone plugin. [https://github.com/scummvm/scummvm/pull/2403 Pull request].

The existing engines' MetaEngine, thus, needs to be split in two.

<font color=red>'''If you started your engine before October 3, 2020, this guide is for you.'''</font>

== New Structure ==

* AdvancedMetaEngineDetection is split out of AdvancedMetaEngine.
** MetaEngineDetection: Houses all methods that are always linked statically. As before, each engine subclasses AdvancedMetaEngineDetection, but only provides detection and related methods.
** MetaEngine: Houses all things that "can" be linked dynamically. These are bridge functions, that connect an Engine with MetaEngine, thus the name. It houses methods like <tt>createInstance</tt>, which instantiates an actual game engine.
** AdvancedMetaEngineDetection: Inherits from MetaEngineDetection and provides a default detection method.
** AdvancedMetaEngine: Inherits MetaEngine to derive a structure like before.
* Most changes are centered around plugins. Two main new helpers are <tt>getEngineFromMetaEngine</tt> and <tt>getMetaEngineFromEngine</tt> which do the same thing as they sound like. These help out in various scenarios where you would need the matching engine-metaengine to do something.

The format of each individual engine is largely similar to before, but I will explain in a bit how some things will change.

== Changes to existing engines ==

Most changes are pretty simple to follow. The new structure will need to be kept in mind when working on new engines,
To keep things short, I'll use "ME" for MetaEngine, "MED" for MetaEngineDetection, AME & AMED for their advanced counterparts.
I'll also use the '''AGI''' engine as an example, so it's clear what everything means.

* If an engine inherits MED, it should also inherit ME. 
* The class MED for AGI lives in <tt>agi/detection.cpp</tt>, and inherits MED. The detection-related code goes here, detection tables, etc.
Example:
<syntaxhighlight lang="cpp">
class AgiMetaEngineDetection : public MetaEngineDetection {...}
</syntaxhighlight>

* The same engine should use the equivalent meta-counterpart in <tt>agi/metaengine.cpp</tt>, a file specifically for ME/AME.

Example:

<syntaxhighlight lang="cpp">
class AgiMetaEngine : public MetaEngine {...}
</syntaxhighlight>

* If an engine uses AMED, it should also use AME.

-----------------------------

That's the basic engine split. As for the engines themselves,
* Many engines have a custom description for the game. For example, in addition to the <tt>ADGameDescription</tt> struct, they would have their own, something like:

<syntaxhighlight lang="cpp">
struct AGIGameDescription {
	ADGameDescription desc;

	int gameID;
	int gameType;
	uint32 features;
	uint16 version;
};
</syntaxhighlight>

* These go into <tt>agi/detection.h</tt>.
* Sometimes the detection-tables will need some enumeration values present in the main engine header file, for example, <tt>agi.</tt>. To avoid including the whole header which would contain irrelevant information for the executable, a copy of these enums is provided to both - the engine & metaengine.
Simply shift these enumeration values to <tt>agi/detection.h</tt>. Include it in the main engine header, so everything returns to normal. However, now we can include it in the <tt>detection.cpp</tt> translation unit, while avoiding the engine header file.

* Engines inheriting from <tt>MetaEngineDetection</tt> will provide their own detection method. An example of this is Scumm. However, detection is not only used during detecting of games, but also during creating an instance. Scumm provides some internal static methods, called <tt>detectionImpl</tt> which is used both - by <tt>MetaEngineDetection::detectGames(...)</tt> & by <tt>MetaEngine::createInstance(...)</tt>. The solution for this is to shift the common code inside another header file, <tt>detection_internal.h</tt>. Then, this is included by both - <tt>detection.cpp</tt> & <tt>metaengine.cpp</tt>
** Remember to mark the methods as static inside the header, so the visibility is limited to the TU.
* MetaEngineDetect now go into the executable. They provide some naming functions - <tt>getName</tt>, <tt>getEngineId</tt> & <tt>getOrignalCopyright</tt>. Because MetaEngineDetect will need to find a relevant MetaEngine, the implemented MetaEngine should override the <tt>getName</tt> method, and provide it a name similar to the one found in <tt>getEngineId</tt> of MetaEngineDetect.
* Lastly, after the creation of your files, you need to register them as plugins.

* For <tt>detection.cpp</tt>, use <tt>REGISTER_PLUGIN_STATIC(...)</tt> macro, as they always go into the executable.
** Keep in mind that the ID should be suffixed with "DETECTION". This is because the ID cannot conflict with the one in metaengine. The type should be a metaengine. Example:

<syntaxhighlight lang="cpp">
REGISTER_PLUGIN_STATIC(AGI_DETECTION, PLUGIN_TYPE_ENGINE_DETECTION, AgiMetaEngine);
</syntaxhighlight>

* For <tt>metaengine.cpp</tt> the story is as-is, if the plugin is dynamic, use the dynamic macro - otherwise the static version.
-------------------------------

=== Module objects / Building detection features ===

* Each engine's module.mk file will be adding object files to <tt>DETECTION_OBJS</tt>. It looks something like this:

<syntaxhighlight lang="cpp">
...
...
# Include common rules
include $(srcdir)/rules.mk

# Detection objects
DETECT_OBJS += $(MODULE)/detection.o
</syntaxhighlight>

* Remember to ''add'' <tt>metaengine.o</tt> and ''remove'' <tt>detection.o</tt> from the <tt>enginename</tt> module.
* Additional dependencies for detection are also ''ok''. However, remember that engines could also use those files, so it wouldn't always be the case to separate the module-object completely. I will give some examples:
** AGI uses something called <tt>wagparser</tt> to help in its fallback detection. The engine itself doesn't need it, so we can separate it completely i.e remove from <tt>MODULE_OBJS</tt> in <tt>engines/agi</tt> module and add it to <tt>DETECT_OBJS</tt> inside the same module list.
** Scumm uses 2 files - <tt>file.cpp</tt>, <tt>file_nes.cpp</tt> to help with detection. However, the engines themselves need those files too, so we can't separate it completely. Hence, the updated module will look like

<syntaxhighlight lang="make">
MODULE := engines/scumm

...
# Rest of the file
...

# Detection objects
DETECT_OBJS += $(MODULE)/detection.o

# Skip building the following objects if a static
# module is enabled, because it already has the contents.
ifneq ($(ENABLE_SCUMM), STATIC_PLUGIN)
DETECT_OBJS += $(MODULE)/file.o
DETECT_OBJS += $(MODULE)/file_nes.o
endif
</syntaxhighlight>

* The makefiles then take care of adding all the <tt>DETECT_OBJS</tt> from each of engine module files, regardless of which engine is being enabled or not.

=== Some engines that would require review ===
While most of the engines adapted pretty nicely to this structure, some had a somewhat different scenario because of which the base structure was changed a bit. I will list all of them below.

==== Sci & Wintermute (fallback-detection) ====
* These engines have a struct similar to <tt>ADGameDescription</tt>, whose contents are overwritten by the fallback detection method. The problem is that these fallback detection methods are heavily dependent on engine resources.
** I tried to make a "lite" version for these classes just to be helpful in detection, but that was a fast fail because it's too dependent on engine resources.
* Thus, a workaround would need to be added.
* A solution for this is that you would still override <tt>fallbackDetect</tt> in MetaEngine (which lives in <tt>detection.cpp</tt>, but instead of detecting games here, you provide a hook to the relevant engine plugin and call a relevant method from there.
** Override a function called <tt>fallbackDetectExtern</tt> in the <tt>MetaEngineConnect</tt> class which lives in <tt>metaengine.cpp</tt> and provide the actual detection here, which is engine-resources dependant. See https://github.com/aryanrawlani28/scummvm/commit/eb960d0814a3dd1d6cd2e7f91d53518f88176476 for more info, and an in-code example.

==== Mohawk (dialogs) ====
* A while back https://github.com/scummvm/scummvm/pull/2164 added support for in-game custom dialogs.
* If you look at some of the engines currently scummvm master tree, which inherit from AMED, you'll see that they provide some extra gui options via an array, <tt>ADExtraGuiOptionsMap gameGuiOptions[]</tt>  and then pass this along to the AMED constructor.
** To not disturb the engine-structure too much, these are still as-is, meaning that without an engine plugin loaded, you'll be able to see these options. (Other things like Keymaps, achievements are still coming from the engine.)
* Mohawk however, has the new dialogs in which they could query information from the game, then setup dialog accordingly. As such, it's probably best that they remain in the engine plugin itself.
* So, MED's & ME's both have the ability to create custom dialogs. MED's only provide the "Engine" tab inside "Edit Game" option, while ME's provide in-game dialogs, etc.
* For now, to clearly distinguish between them, the name <tt>buildEngineOptionsWidgetDynamic</tt> is given, but this is not really proper. <tt>extern & intern</tt> like the ones I did with fallback detection were ok, but it could be confusing in this scenario. Hence, this name can probably be changed into something better.

== Detection features as a plugin themselves ==
* On some low-hardware end platforms, the executable size bloat would prove difficult.
* Making detection features available as a plugin itself would be beneficial in that scenario.

=== How is it done? ===
* We already had the detection features separated from Engines. They are normally supposed to link directly with the executable, but since we have to have a plugin, we can group all the detection files themselves into a <tt>DETECTION</tt> type plugin.
* In the ScummVM <tt>engines/</tt> directory, look for <tt>detection.cpp</tt>. This is where we include the detection tables and mark the plugin as a <tt>PLUGIN_TYPE_DETECTION</tt>.
* When using configure, <tt>--enable-detection-dynamic</tt> will be available, which enables the detection objects to be packed into a library.
** When using this, helper methods in <tt>PluginManagerUncached</tt> will help in loading the detection library, linking the MetaEngines, and eventually clearing those out of memory once we don't need them.
** If not using that, they will be linked statically, but still have an explicit option - <tt>--enable-detection-static</tt> available.

=== Modules objects / Building detection features for dynamic detection plugins ===
* As said above, each engine is already populating the detect_objs variable.
* In the configuration script, when <tt>--enable-detection-dynamic</tt> is passed, the <tt>Makefiles.common</tt> includes the below module:

<syntaxhighlight lang="make">
MODULE := detection // Only added if dynamic-detection enabled

DETECT_OBJS_DYNAMIC=$(addprefix ../,$(DETECT_OBJS)) // Since it lives outside the engines subdirectory, add "../" prefix.

MODULE_OBJS := \
	../engines/detection.o \
	$(DETECT_OBJS_DYNAMIC) // Reuse the already existing list of detection objects

# Reset detect objects, so none of them build into the executable.
DETECT_OBJS :=

PLUGIN := 1

# Include common rules
include $(srcdir)/rules.mk
</syntaxhighlight>