HOWTO-Dynamic Modules

NOTE: many parts of this HOWTO assume the "gsoc2010-plugins" branch has been integrated into trunk.

Introduction

This page is meant as a HOWTO which roughly outlines the steps needed to implement dynamic engine plugins for a previously unsupported backend (port) of ScummVM. This HOWTO only applies to targets that can use the ELF file-format and will generally only be useful for smaller targets that cannot support POSIX plugins.

I will assume that you are at least roughly familiar with ScummVM, and have a fresh checkout of our Subversion repository. Note that it's strongly advised to base your work on the current development version of ScummVM, and not on a release version. This will ease integration of your work.

Overview

ScummVM has a PluginManager class that potentially keeps track of multiple plugin providers. As of this writing, a StaticPluginProvider is always added to this PluginManager that keeps track of all statically-linked plugins. Your job will be to add a dynamic plugin provider specific to the backend you're working on that will provide for any dynamic plugins that are enabled on that backend and to tell the backend to use this provider in the case that dynamic modules are enabled.

Step by step

For the rest of this document, I will assume you are attempting to implement loadable modules for the "foobar" port of ScummVM :-).

1. Create a new backends/plugins subdirectory matching the name of the port you're providing for, i.e. backends/plugins/foobar/.
2. Create a provider file in this subdirectory, such as foobar-provider.h that contains the class declaration of the provider for your port. This class should subtype ELFPluginProvider. This class should nest a class declaration that subtypes ELFPlugin but is specific to your port (i.e. FOOBARPlugin) and override ELFPluginProvider's "createPlugin" method with its own that returns a new FOOBARPlugin object (See example below).
3. The FOOBARPlugin class needs to override ELFPlugin's "makeDLObject" method and have it return a new object of an appropriate subclass of DLObject. Currently, the subclasses of DLObject are processor-specific, but depending on the port, you may find you have to subclass one of these processor-specific subclasses with a platform-specific one, such as FOOBARDLObject(). (See "Subclassing DLObject" below).
4. Find where scummvm_main(argc, argv) is invoked in your backend's main and somewhere shortly before it add the FOOBARPluginProvider to the PluginManager, i.e. PluginManager::instance().addPluginProvider(new FOOBARPluginProvider(); (Be sure to guard this call with #ifdef DYNAMIC_MODULES). This, of course, will require that you include the FOOBARPluginProvider header at the top of the file.
5. Now we need to get the backend generating custom engine plugin files that can be loaded/unloaded to/from memory at will during runtime. To make the runtime linking (loading) of plugin files easier to handle, we let ld do some of the work for us at compile-time. First, we build the non-relocatable main executable scummvm.elf (Manually implementing Dynamic Modules on a smaller target that doesn't allow the main executable to be non-relocatable has not yet been successfully done and, as such, is not covered by this HOWTO). Then we link the plugins files together with a custom ld linker script; ld uses the absolute addresses from the main executable to resolve messy jumps from the plugin back to the main code and thus does much of the more complicated dynamic work for us. I would suggest adding the necessary custom linker script into the backends/plugins/foobar/ directory you made earlier (See "Making the plugin linker script for your backend" below).
6. Modify the build system for your backend (See "Necessary Build Modifications" below).
7. Cross your fingers and be prepared to work at things for a while if it doesn't immediately function ;-)

Example of foobar-provider.h

<syntax type="C++">#if defined(DYNAMIC_MODULES) && defined(FOOBAR)

1. include "backends/plugins/elf-provider.h"
2. include "backends/plugins/foobarprocessor-loader.h"

class FOOBARPluginProvider : public ELFPluginProvider { class FOOBARPlugin : public ELFPlugin { public: FOOBARPlugin(const Common::String &filename) : ELFPlugin(filename) {}

DLObject *makeDLObject() { return new FOOBARProcessorDLObject(); } };

public: Plugin* createPlugin(const Common::FSNode &node) const { return new FOOBARPlugin(node.getPath()); } };

1. endif // defined(DYNAMIC_MODULES) && defined(FOOBAR)

</syntax>

Making the plugin linker script for your backend

To generate this script, first find the linker script used to link a static build of the target platform (usually this is the default script for the linker, which you can dump via the command foobar-compiler-ld --verbose). You'll want to modify this linker in a few key ways.

• First, you can remove the ENTRY(symbol) command. This command normally tells the linker which instruction should be the first to execute in the program, but since you're generating a plugin that will be linked in and jumped to from the main executable, it is not needed.
• Next, define an ELF program header for a "plugin" segment to be loaded from the file. Do this by adding the following before the SECTIONS command:

<syntax type="C++">PHDRS {

  plugin PT_LOAD ;

}</syntax>

• Set the start address of the file to be 0 (since this will be linked in at a different address later), usually you can do this by replacing the first line in SECTIONS with . = 0;, though different ld scripts may require different modifications.
• Place the first section (as listed under the SECTIONS command) into the "plugin" segment you defined in PHDRS by appending :plugin to the first sections-command, i.e .text . : { *(.text) } :plugin. As long as there aren't any other PHDRS, this should ensure the entire linked file is put in the "plugin" segment, since future sections assume they are to be put in the same segment as the previous section unless specified otherwise (via :phdr or :NONE).
• Find the .ctors and .dtors sections (constructors and destructors) and delete any output-section-commands within them that reference crtbegin or crtend, i.e KEEP (*crtbegin*.o(.dtors)) or KEEP (*(EXCLUDE_FILE (*crtend*.o ) .dtors)). Put ___plugin_ctors = .; as the first output-section-command in the .ctors section and ___plugin_ctors_end = .; as the last output-section-command. Do the same for the .dtors section but with ___plugin_dtors, etc. This gives these symbols values that our run-time loader can use. In the end, .ctors and .dtors should look something like this:

<syntax type="C++">.ctors  : {

 ___plugin_ctors = .;
 KEEP (*(SORT(.ctors.*)))
 KEEP (*(.ctors))
 ___plugin_ctors_end = .;

} .dtors  : {

 ___plugin_dtors = .;
 KEEP (*(SORT(.dtors.*)))
 KEEP (*(.dtors))
 ___plugin_dtors_end = .;

}</syntax>

• That's it! If you have trouble with any of these instructions or need to further modify the linker script for something specific to your platform, see [1] for linker script documentation and peruse the various plugin.ld files in the subdirectories of backends/plugins/. TODO: Add stuff about MIPS-specific linker script modifications, namely the "shorts" segment.

Necessary Build Modifications

There will undoubtedly be a number of things specific to your platform to deal with when it comes to modifying the building, but you should definitely add the following, bordered by #ifdef guards to make sure DYNAMIC_MODULES is set to 1 (along with telling the desired engines that you want them to be separate plugin files, i.e. "ENABLE_SCUMM = DYNAMIC_PLUGIN"):

<syntax type="C++">DEFINES += ELF_LOADER_TARGET //also add any other necessary defines PLUGIN_SUFFIX := .plg //or whatever your suffix is PLUGIN_EXTRA_DEPS = $(EXECUTABLE) PLUGIN_LDFLAGS += -nostartfiles -Wl,-q,--just-symbols,$(EXECUTABLE),-T$(srcdir)/backends/plugins/foobar/plugin.ld,--retain-symbols-file,$(srcdir)/backends/plugins/plugin.syms PRE_OBJS_FLAGS := -Wl,--whole-archive POST_OBJS_FLAGS := -Wl,--no-whole-archive </syntax>

As you can see, these modifications mainly deal with making sure the plugins are dependent on the main executable and use the custom linker script.

You will also need to make sure your main executable is not being overly stripped as the thing doing the stripping may think the executable doesn't use certain symbols simply because they reference portions of code that will be linked in later in a plugin.

TODO: Add stuff about "ONE_PLUGIN_AT_A_TIME" once it is working perfectly.

Subclassing DLObject

At some point, the PluginManager will call getPlugins() on the FOOBARProvider, which will trigger a search for plugin files in your backend's file system and return a list of FOOBARPlugins based on the results of this search. The PluginManager will then be able to call loadplugin() and unloadPlugin, and on any of these FOOBARPlugins, which will involve calls to open(const char *path) on a new object that's a subclass of DLObject (which subclass is determined by how you coded makeDLObject() in the FOOBARPlugin class) with the path to the plugin file the FOOBARPlugin represents as its argument (The multiple levels of abstraction here can get confusing :-)). A subclass of DLObject, then, is ultimately responsible for opening the plugin files and loading them/unloading them from memory.

TODO: Finish explaining what the DLObject class does and how to subclass it effectively.