Difference between revisions of "HOWTO-Dynamic Modules"

Jump to navigation Jump to search
2,705 bytes added ,  15:15, 25 October 2018
m
Text replacement - "</source>" to "</syntaxhighlight>"
(Filled out the "making the plugin linker script for your backend")
m (Text replacement - "</source>" to "</syntaxhighlight>")
 
(7 intermediate revisions by 3 users not shown)
Line 5: Line 5:
This page is meant as a HOWTO which roughly outlines the steps needed to implement dynamic engine plugins for a previously unsupported backend (port) of ScummVM. This HOWTO only applies to targets that can use the ELF file-format and will generally only be useful for smaller targets that cannot support POSIX plugins.  
This page is meant as a HOWTO which roughly outlines the steps needed to implement dynamic engine plugins for a previously unsupported backend (port) of ScummVM. This HOWTO only applies to targets that can use the ELF file-format and will generally only be useful for smaller targets that cannot support POSIX plugins.  


I will assume that you are at least roughly familiar with ScummVM, and have a fresh checkout of our Subversion repository. Note that it's strongly advised to base your work on the current development version of ScummVM, and not on a release version. This will ease integration of your work.
I will assume that you are at least roughly familiar with ScummVM, and have a fresh checkout of our source code repository. Note that it's strongly advised to base your work on the current development version of ScummVM, and not on a release version. This will ease integration of your work.


=== Overview ===
=== Overview ===
Line 21: Line 21:
# Now we need to get the backend generating custom engine plugin files that can be loaded/unloaded to/from memory at will during runtime. To make the runtime linking (loading) of plugin files easier to handle, we let ld do some of the work for us at compile-time. First, we build the non-relocatable main executable <tt>scummvm.elf</tt> (Manually implementing Dynamic Modules on a smaller target that doesn't allow the main executable to be non-relocatable has not yet been successfully done and, as such, is not covered by this HOWTO). Then we link the plugins files together with a custom ld linker script; ld uses the absolute addresses from the main executable to resolve messy jumps from the plugin back to the main code and thus does much of the more complicated dynamic work for us. I would suggest adding the necessary custom linker script into the <tt>backends/plugins/foobar/</tt> directory you made earlier (See "Making the plugin linker script for your backend" below).
# Now we need to get the backend generating custom engine plugin files that can be loaded/unloaded to/from memory at will during runtime. To make the runtime linking (loading) of plugin files easier to handle, we let ld do some of the work for us at compile-time. First, we build the non-relocatable main executable <tt>scummvm.elf</tt> (Manually implementing Dynamic Modules on a smaller target that doesn't allow the main executable to be non-relocatable has not yet been successfully done and, as such, is not covered by this HOWTO). Then we link the plugins files together with a custom ld linker script; ld uses the absolute addresses from the main executable to resolve messy jumps from the plugin back to the main code and thus does much of the more complicated dynamic work for us. I would suggest adding the necessary custom linker script into the <tt>backends/plugins/foobar/</tt> directory you made earlier (See "Making the plugin linker script for your backend" below).
# Modify the build system for your backend (See "Necessary Build Modifications" below).
# Modify the build system for your backend (See "Necessary Build Modifications" below).
# Cross your fingers and be prepared to work at things for a while if it doesn't immediately function ;-)
# Some miscellaneous things: You should add a platform-specific call to a function that flushes the data cache into <code>static void flushDataCache()</code> in <tt>backends/plugins/elf-loader.cpp</tt>. You will also need to add a check for the proper machine (processor) type into <code>bool DLObject::readElfHeader(Common::SeekableReadStream* DLFile, Elf32_Ehdr *ehdr)</code> in that same file, adding the machine type to elf32.h if it's not already there.
# That's the gist of it; Cross your fingers and be prepared to work at things for a while if it doesn't immediately function ;-).


==== Example of foobar-provider.h====
==== Example of foobar-provider.h====
<syntax type="C++">#if defined(DYNAMIC_MODULES) && defined(FOOBAR)
<syntaxhighlight lang="cpp">#if defined(DYNAMIC_MODULES) && defined(FOOBAR)


#include "backends/plugins/elf-provider.h"
#include "backends/plugins/elf-provider.h"
Line 44: Line 45:


#endif // defined(DYNAMIC_MODULES) && defined(FOOBAR)
#endif // defined(DYNAMIC_MODULES) && defined(FOOBAR)
</syntax>
</syntaxhighlight>


==== Making the plugin linker script for your backend ====
==== Making the plugin linker script for your backend ====
Line 51: Line 52:
*First, you can remove the <tt>ENTRY(symbol)</tt> command. This command normally tells the linker which instruction should be the first to execute in the program, but since you're generating a plugin that will be linked in and jumped to from the main executable, it is not needed.  
*First, you can remove the <tt>ENTRY(symbol)</tt> command. This command normally tells the linker which instruction should be the first to execute in the program, but since you're generating a plugin that will be linked in and jumped to from the main executable, it is not needed.  
*Next, define an ELF program header for a "plugin" segment to be loaded from the file. Do this by adding the following before the SECTIONS command:
*Next, define an ELF program header for a "plugin" segment to be loaded from the file. Do this by adding the following before the SECTIONS command:
<syntax type="C++">PHDRS
<syntaxhighlight lang="cpp">PHDRS
{
{
   plugin PT_LOAD ;
   plugin PT_LOAD ;
}</syntax>
}</syntaxhighlight>
*Set the start address of the file to be 0 (since this will be linked in at a different address later), usually you can do this by replacing the first line in SECTIONS with <code>. = 0;</code>, though different ld scripts may require different modifications.
*Set the start address of the file to be 0 (since this will be linked in at a different address later), usually you can do this by replacing the first line in SECTIONS with <code>. = 0;</code>, though different ld scripts may require different modifications.
*Place the first section (as listed under the SECTIONS command) into the "plugin" segment you defined in PHDRS by appending <tt>:plugin</tt> to the first sections-command, i.e <code>.text . : { *(.text) } :plugin</code>. As long as there aren't any other PHDRS, this should ensure the entire linked file is put in the "plugin" segment, since future sections assume they are to be put in the same segment as the previous section unless specified otherwise (via <tt>:phdr</tt> or <tt>:NONE</tt>).
*Place the first section (as listed under the SECTIONS command) into the "plugin" segment you defined in PHDRS by appending <tt>:plugin</tt> to the first sections-command, i.e <code>.text . : { *(.text) } :plugin</code>. As long as there aren't any other PHDRS, this should ensure the entire linked file is put in the "plugin" segment, since future sections assume they are to be put in the same segment as the previous section unless specified otherwise (via <tt>:phdr</tt> or <tt>:NONE</tt>).
*Find the <tt>.ctors</tt> and <tt>.dtors</tt> sections (constructors and destructors) and delete any output-section-commands within them that reference crtbegin or crtend, i.e <code>KEEP (*crtbegin*.o(.dtors))</code> or <code>KEEP (*(EXCLUDE_FILE (*crtend*.o ) .dtors))</code>. Put <code>___plugin_ctors = .;</code> as the first output-section-command in the <tt>.ctors</tt> section and <code>___plugin_ctors_end = .;</code> as the last output-section-command. Do the same for the <tt>.dtors</tt> section but with <tt>___plugin_dtors</tt>, etc. This gives these symbols values that our run-time loader can use. In the end, <tt>.ctors</tt> and <tt>.dtors</tt> should look something like this:
*Find the <tt>.ctors</tt> and <tt>.dtors</tt> sections (constructors and destructors) and delete any output-section-commands within them that reference crtbegin or crtend, i.e <code>KEEP (*crtbegin*.o(.dtors))</code> or <code>KEEP (*(EXCLUDE_FILE (*crtend*.o ) .dtors))</code>. Put <code>___plugin_ctors = .;</code> as the first output-section-command in the <tt>.ctors</tt> section and <code>___plugin_ctors_end = .;</code> as the last output-section-command. Do the same for the <tt>.dtors</tt> section but with <tt>___plugin_dtors</tt>, etc. This gives these symbols values that our run-time loader can use. In the end, <tt>.ctors</tt> and <tt>.dtors</tt> should look something like this:
<syntax type="C++">.ctors          :
<syntaxhighlight lang="cpp">.ctors          :
  {
{
    ___plugin_ctors = .;
  ___plugin_ctors = .;
    KEEP (*(SORT(.ctors.*)))
  KEEP (*(SORT(.ctors.*)))
    KEEP (*(.ctors))
  KEEP (*(.ctors))
    ___plugin_ctors_end = .;
  ___plugin_ctors_end = .;
  }
}
  .dtors          :
.dtors          :
  {
{
    ___plugin_dtors = .;
  ___plugin_dtors = .;
    KEEP (*(SORT(.dtors.*)))
  KEEP (*(SORT(.dtors.*)))
    KEEP (*(.dtors))
  KEEP (*(.dtors))
    ___plugin_dtors_end = .;
  ___plugin_dtors_end = .;
  }</syntax>
}</syntaxhighlight>
*That's it! If you have trouble with any of these instructions or need to further modify the linker script for something specific to your platform, see [http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gnu-linker/scripts.html] for linker script documentation and peruse the various <tt>plugin.ld</tt> files in the subdirectories of <tt>backends/plugins/</tt>. TODO: Add stuff about MIPS-specific linker script modifications, namely the "shorts" segment.
*That's it! If you have trouble with any of these instructions or need to further modify the linker script for something specific to your platform, see [http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gnu-linker/scripts.html] for linker script documentation and peruse the various <tt>plugin.ld</tt> files in the subdirectories of <tt>backends/plugins/</tt>. TODO: Add stuff about MIPS-specific linker script modifications, namely the "shorts" segment.


Line 77: Line 78:
There will undoubtedly be a number of things specific to your platform to deal with when it comes to modifying the building, but you should definitely add the following, bordered by #ifdef guards to make sure DYNAMIC_MODULES is set to 1 (along with telling the desired engines that you want them to be separate plugin files, i.e. "ENABLE_SCUMM = DYNAMIC_PLUGIN"):
There will undoubtedly be a number of things specific to your platform to deal with when it comes to modifying the building, but you should definitely add the following, bordered by #ifdef guards to make sure DYNAMIC_MODULES is set to 1 (along with telling the desired engines that you want them to be separate plugin files, i.e. "ENABLE_SCUMM = DYNAMIC_PLUGIN"):


<syntax type="C++">DEFINES += ELF_LOADER_TARGET //also add any other necessary defines
<syntaxhighlight lang="Makefile">DEFINES += ELF_LOADER_TARGET //also add any other necessary defines
PLUGIN_SUFFIX := .plg //or whatever your suffix is
PLUGIN_SUFFIX := .plg //or whatever your suffix is
PLUGIN_EXTRA_DEPS = $(EXECUTABLE)
PLUGIN_EXTRA_DEPS = $(EXECUTABLE)
Line 83: Line 84:
PRE_OBJS_FLAGS := -Wl,--whole-archive
PRE_OBJS_FLAGS := -Wl,--whole-archive
POST_OBJS_FLAGS := -Wl,--no-whole-archive
POST_OBJS_FLAGS := -Wl,--no-whole-archive
</syntax>
</syntaxhighlight>


As you can see, these modifications mainly deal with making sure the plugins are dependent on the main executable and use the custom linker script.
As you can see, these modifications mainly deal with making sure the plugins are dependent on the main executable and use the custom linker script.
Line 92: Line 93:


==== Subclassing DLObject ====
==== Subclassing DLObject ====
At some point, the PluginManager will call <code>getPlugins()</code> on the FOOBARProvider, which will trigger a search for plugin files in your backend's file system and return a list of FOOBARPlugins based on the results of this search. The PluginManager will then be able to call <code>loadplugin()</code> and <code>unloadPlugin</code>, and  on any of these FOOBARPlugins, which will involve calls to <code>open(const char *path)</code> on a new object that's a subclass of DLObject (which subclass is determined by how you coded <code>makeDLObject()</code> in the FOOBARPlugin class) with the path to the plugin file the FOOBARPlugin represents as its argument (The multiple levels of abstraction here can get confusing :-)). A subclass of DLObject, then, is ultimately responsible for opening the plugin files and loading them/unloading them from memory.
At some point, the PluginManager will call <code>getPlugins()</code> on the FOOBARProvider, which will trigger a search for plugin files in your backend's file system and return a list of FOOBARPlugins based on the results of this search. The PluginManager will then be able to call <code>loadplugin()</code> and <code>unloadPlugin</code> on any of these FOOBARPlugins, which will involve calls to <code>open(const char *path)</code> on a new object that's a subclass of DLObject (which subclass this will be is determined by how you coded <code>makeDLObject()</code> in your FOOBARPlugin class) with the path to the plugin file the FOOBARPlugin represents as its argument (The multiple levels of abstraction here can get confusing, I know :-)). A subclass of DLObject, then, is ultimately responsible for opening the plugin files and loading them/unloading them from memory.
 
If there is already a DLObject subclass that matches the processor for your smaller target (i.e <tt>MIPSDLObject</tt>, <tt>ARMDLObject</tt>), use it! Assuming that class is in use for another working backend with dynamic modules implemented, most everything should work correctly. The one function where one should expect problems is <code>bool DLObject::relocate(Common::SeekableReadStream* DLFile, unsigned long offset, unsigned long size, void *relSegment)</code> in that your platform may generate plugins that use relocation types as of yet unsupported by that subclass of DLObject. If this is the case, you will hopefully get a helpful message at runtime, such as "Unknown relocation type 12" or something like that. You should then look up which relocation type this number refers to in the ABI for your processor (relocation types are processor-specific), add a define for the type into elf32.h next to the other relocation types for your processor (i.e. <code>#define R_ARM_ABS32 2</code>), and then add code to handle it into the relocate function (again, the ABI for your processor and pre-existing relocation code will be your friend when trying to figure this out).


TODO: Finish explaining what the DLObject class does and how to subclass it effectively.
If there isn't already a DLObject subclass that matches the processor for your smaller target, you will have to make one that overrides <tt>relocate</tt> and <tt>relocateRels</tt>. You can look to these functions in <tt>backends/plugins/arm-loader.cpp</tt> and <tt>backends/plugins/mips-loader.cpp</tt> for a template of how to code for them. Again, the hardest part will be the code for the actual relocations. You'll want to first detect what these relocations are (you can do this using objdump on generated plugin files), then look up how they are supposed to be handled in the ABI for your processor. Since we pre-link to let ld do some of this relocation work for us, you should make builds that don't do this pre-linking with the plugins and ones that do and compare to see exactly what this work is. That way, you won't write relocation code for difficult things the linker already does for you (a lot of the time for absolute jumps, all you have to do is add the main offset of the plugin). Careful reading and persistence will pay off at this point :-)
TrustedUser
2,147

edits

Navigation menu