Summer of Code/GSoC2010

This pages lists students and projects for the Google Summer of Code 2010. See also Google's ScummVM organization info page.

Adding a testing framework for ScummVM's subsystems

Student

Neeraj Kumar

Mentors

Eugene Sandulenko, Jordi Vialta Prat

Code

http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/branches/gsoc2010-testbed/

Outcome

Success. The resulting "testbed" engine has been merged into trunk.

Original task description

Technical Contact: Johannes Schickel, Max Horn

Background:

ScummVM is a highly portable application. To achieve that, there are different interfaces to access platform specific functionality. Our APIs are usually well documented, thus giving porters and engine authors a clear vision of how different functions / methods are supposed to work. Apart from the lower level platform abstraction APIs we usually offer more high level APIs for engine authors.

For example we have a low-level filesystem abstraction API: FSNode. This can be used to browse directories, open files, create new files for writing etc. Since the API is pretty much low level, engine authors usually do not use it directly, but use some wrapper code to access it. To prevent code duplication and take work from engine authors we introduced the Archive and SearchMan classes. SearchMan can be easily used to access game data files without fiddling with FSNode.

However, experience has shown that the documentation of all these classes is not always as clear and complete as it should be, and sometimes might be misunderstood. In addition, the implementations for the various systems we run on may not adhere to the documented specification and/or regressions were introduced when doing changes to the code. Currently we have only a very minimalistic test suite in "tests/", testing only a tiny subset of our code: container classes, string handling and some stream code. Currently it is hard to do proper regression/implementation testing for other APIs, for example FSNode API.

The Task:

As visible from the description above, we need an automated test system to help ensure our implementations are bug free. We need both a framework for the tests, as well as an extensive test suite. There are several different approaches to testing which all are of independent interest and complement each other:

1. Unit tests: Add many more tests to our test suite in "tests/". This would also require some enhancements to our build system, since currently we do not link against all of our code for the test suite. Also new test mechanisms are needed to make it possible to e.g. test audio & graphics output (e.g. code that can load an existing reference image and compare it to the output of a graphics primitive). It might also be helpful to enhance the "null" backend so that all unit tests can be run against that, headless (good for automated tests on a server with actual graphics/audio output).
2. A "test" engine: Unit tests would probably only be runnable on "desktop" ports like Windows and Linux, and not on ports like PSP or NDS, due to run environment restrictions. To allow extended coverage on *all* ports, implementing a "test" engine would be helpful. In addition to running on all ports, the "test" engine also provides an environment to test things in an integrated, non-isolated way, unlike unit tests. It should test as many features as possible: Game detection, file loading/saving, navigating the file system, testing graphics and audio output, etc. -- and it must be possible to verify everything. I.e. compare results to reference data, output progress data on the screen, create dump logs, and so on. Lots can be done here.

The test suite extension would be easiest to use on Linux etc. and there is already some existing code to build on. A test engine would work on all ports, including game consoles and phones, and is nice for testing in a more "holistic" and complete way. Of course best would be to have both, and also make it possible to run all unit tests from inside the test engine.

Overall the test engine has the highest priority, since it would give us the possibility to test all ports without additional work. Thus the primary task would be to implement and document at least a test engine, which tests (preferably) all of ScummVM's subsystems extensively.

Required Skills:

• Reasonable C++ skills.
• Experience with unit tests.
• Knowledge about various ScummVM subsystems would be useful, but not required.
• Knowledge about how to create a new engine for ScummVM would also be useful. See also our Engines HOWTO.

Also required:

• Access to a non-desktop testing device (PSP, NDS, PS2, WinCE or Symbian mobile, etc.) and the ability to compile and run ScummVM on it (the ScummVM team would help with compiling and running).

Implementing support for Loadable Modules on MIPS and ARM-based platforms

Student

Anthony Puccinelli

Mentors

Yotam Barnoy, John Willis

Code

http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/branches/gsoc2010-plugins/

Outcome

Success, see also the Dynamic Modules HOWTO. To be merged.

Original task description

Technical Contact: Max Horn, Yotam Barnoy

Background:

ScummVM by now contains about two dozen engines for different adventure games. Each of these takes up lots of storage space in the ScummVM binary, and hence also RAM, even if not used. Since both are scarce resources on some of our smaller targets (e.g. the Nintendo DS, which only has 4 MB RAM), this leads to problems. Some of the ports work around this by shipping multiple ScummVM binaries, each with a different subset of engines. This is not ideal, as it means more work for the developers and is confusing to users.

Putting all engines into loadable modules reduces memory usage, as this way during game play only a single module needs to be in RAM. ScummVM itself is prepared for loadable module support. In fact several of our ports, notably the Windows, Linux and Mac OS X ports, already support loadable modules. This is easy for them as the operating system already provides loadable modules. But this is not the case on systems like the Dreamcast, Nintendo Gamecube & DS & Wii, the PlayStation Portable, Symbian and others.

The Task:

Implement support for loadable modules for at least two "small" targets where this is currently not possible. For the Dreamcast, Marcus Comstedt solved this by implementing a module loader on his own. Yotam Barnoy built on Marcus' work and made a similar loader for the PSP. You should probably use this version as a starting point.

The module loader consists of a linker script which is used to generate custom loadable modules "manually". Furthermore, we use a custom loader for ELF binaries which takes care of the run-time linking.

The original Dreamcast code is for SuperH processors, and the PSP code runs on a MIPS processor. You would have to adapt these to generate and load ELF binaries for ARM processors (for Nintendo DS, WinCE, Symbian, GP2x), PowerPC (Nintendo Wii & Gamecube), or the PS2's MIPS version.

Ideally, your code should be generic enough to be usable for all platforms. With two examples of the loader already in existence, this should be quite doable.

Another factor to consider is that should you wish to build the loader for very memory-starved platforms such as the Nintendo DS, you will have to deal with an issue within the ScummVM architecture. Currently, the loadable modules are all initially loaded into memory. When a game is chosen by the user, the irrelevant modules are erased from memory. In order for the modules to fit into the limited Nintendo DS RAM, this method will have to change -- only one module should be loaded into memory at a time. We have some ideas on how to achieve that, talk to us.

Once loadable modules work, there are potential tasks beyond this, which could be pursued if enough time is left. For example, on the NDS, even when only one engine is loaded, memory can still be too tight unless the core binary itself is reduced. Therefore, the NDS port disables MP3 support in certain engines, as well as certain other core features. But we would prefer to have a single binary which only loads modules as required. So, maybe one can turn the MP3 decoder (as well as other audio and video decoders and maybe further core components) into plugins, too, and only load them when required. Of course this opens up issues of its own, like now modules would have to be able to declare dependencies on each other.

Required Skills:

• Good C++ skills.
• Some knowledge of the ELF binary format (great documentation is available on the net, start e.g. at Wikipedia).
• Any of the listed "small" devices (Nintendo DS, Wii, Gamecube, PS2, GP2x, any WinCE or Symbian device supporting ScummVM, ...) -- ideally multiple. Or figure out how to setup compilers for them.
• Basic knowledge regarding the target architecture, be it ARM, PowerPC, or MIPS.

Refactoring of the SDL backend and OpenGL support

Student

Alejandro Marzini

Mentors

Joost Peters, Jordi Vialta Prat, John Willis

Code

http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/branches/gsoc2010-opengl/

Outcome

Success. To be merged.

Original task description

Technical Contact: John Willis, Joost Peters

Background:

Since its inception, ScummVM has offered a default cross platform backend that makes use of the extensive and prolific SDL API. We have no intention to change this but we would like to extend the default backend to take advantage of things like OpenGL, and its little brother OpenGL ES, that is becoming commonplace on the latest generations of devices.

We would also like to refactor the common aspects of the SDL backend into a common base class to better support backends that derive from it. Architecturally, this could be very elegant and allow platforms to derive from the common SDL base (inc. the OpenGL and default existing one) and future platforms could also derive from the OpenGL base etc.

There are a number of required features that the extended backend must have.

The Task:

• Allow for arbitrary rate scaling using OpenGL/OpenGL ES.
• Be well written and maintainable!
  • This could become part of the default choice for ScummVM’s user base so really does have to be well architected.
• Offer a fallback to the existing rendering code if OpenGL support is not available OR not desired by the user. This should be configurable at run time and maybe via a config setting?
• Support removing the new OpenGL code at compile time by the use of structured defines so that NO OpenGL code need be built if not required.
  • We have a number of ports that derive from the SDL backend and really could not cope with the code bloat.

Also required:

• This task will require testing under both OpenGL and OpenGL ES APIs.
  • There are sample OpenGL ES and OpenGL implementations for development available from several sources. (For OpenGL ES, the PowerVR SGX SDK should be of interest.)

Required Skills:

• Reasonable C++ skills.
• Refactoring skills.
• Knowledge of the OpenGL and SDL APIs.
• Knowledge of the default SDL ScummVM platform backend is desirable.

Game script (bytecode) decompiler

Student

Michael Madsen

Mentors

Johannes Schickel, Max Horn

Code

http://scummvm.svn.sourceforge.net/viewvc/scummvm/tools/branches/gsoc2010-decompiler/

Outcome

Success. To be merged.

Original task description

Technical Contact: Max Horn, Johannes Schickel

Background:

Most adventure engines ScummVM supports are driven by bytecode scripts, which control the game behavior. For various reasons (in particular, debugging), it's very useful to be able to take such a script and decompile it into something resembling regular code (with variables, loops, if statements etc.). We have a tool for doing this for the bytecode used by SCUMM games. This tool is called descumm. We also have similar (but less advanced) tools for various other engines, in our tools module in SVN.

Currently, descumm uses a relatively crude heuristic to detect "if" statements, loops and so on. In particular, it only supports loops with the condition at the start (think "while(...) { ... }" ), not such with the condition at the end (as in "do { ... } while(...)" and also suffers from various other limitations (like not being able to correctly recognize "continue" and "break" statements.

The whole thing is somewhat complicated by the fact that there are two main versions of this bytecode: V5 and older (register based bytecode), and V6 and newer (stack based). We handle the registed based code quite well, but not so the stack based one. And in fact most non-SCUMM engines use a stack based virtual machine.

The Task:

Write a generic bytecode decompiler with the goal of ultimately being able to decompile bytecode of all games which use a stack based bytecode virtual machine. You can look at existing work from GSoC 2007, our descumm compiler (only SCUMM v6 and newer is stack based) and some other de*.cpp files in our tools SVN module. Your goal should be to support at least two different bytecode systems (i.e., from two different ScummVM engines). But everything should be designed and implemented with the ultimate goal of being able to support all such bytecode systems (e.g. by providing a suitable API or subclassing facilities that make it possible to hook in parsers for arbitrary bytecode variants). Engines which use a stack based virtual machine include SCUMM v6+, KYRA, SWORD1, SWORD2, ... For many of these, free game demos are available, too. We would help you by providing dumps of scripts etc.

A good starting point for this might be the Jode Java bytecode decompiler which does a rather good job at decompiling. Another useful site might be [1], and this thesis. The technical contact has several ideas on how to approach this project, too -- as usual, you are expected to talk to us and ask for help and ideas!

The tool should run on at least Linux, Mac OS X and Windows. Acceptable languages include C/C++, Python, Perl -- other languages might be OK, but please consult with us first -- after all, other developers also will have to use and maintain your code. For the same reason it is obviously mandatory that your code be well documented.

Actually, we probably would prefer if it was written in C++ and was written as kind of a library, with the command line tool just a frontend to the library. This "library" approach would make it possible to include the decompiler into ScummVM itself, making it possible to decompile scripts on the fly from ScummVM's built-in debugger console.

Note:

This task was being worked on as part of the Google Summer of Code 2007 and again 2009, but unfortunately in both cases the result sadly is incomplete and never reached a satisfactory status. It is probably best to start from scratch, but you can find the code from 2007 here and the code from 2009 here.