1,079
edits
Difference between revisions of "OpenTasks"
→Add a testing framework for ScummVM's subsystems
(→Generic infrastructure tasks: first version of a subsystem test suite task.) |
|||
| Line 171: | Line 171: | ||
''Background:'' | ''Background:'' | ||
ScummVM is a highly portable application | ScummVM is a highly portable application. To achieve that there are different interfaces to access platform specific functionality. Our APIs are usually well documented, thus giving porters and engine authors a clear vision of how different functions / methods are supposed to work. Apart from the lower level platform abstraction APIs we usually offer more high level APIs for engine authors. | ||
For example we have | For example we have a low-level filesystem abstraction API: FSNode. This can be used to browse directories, open files, create new file for writing etc. Since the API is pretty much low level, engine authors usually do not use it directly, but use some wrapper code to access it. To prevent code duplication and take work from engine authors we introduced the Archive and SearchMan classes. SearchMan can be easily used to access game data files without fiddling with FSNode. | ||
However, experience has shown that the documentation of all these classes is not always as clear and complete as it should be, and sometimes might be misunderstood. In addition, the implementations for the various systems we run on may not adhere to the documented specification and/or regressions were introduced when doing changes to the code. Currently we have only a very minimalistic test suite in "tests/", testing only a tiny subset of our code: container classes, string handling and some stream code. Currently it is hard to do proper regression / implementation testing for other APIs, for example FSNode API. | |||
''The Task:'' | ''The Task:'' | ||
As visible from the description above, we need a | As visible from the description above, we need a automated test system to help ensure our implementations are bug free. We need both a framework for the tests, as well as an extensive test suite. There are several different approaches to testing which all are of independent interest and complement each other: | ||
# Unit tests: Add many more tests to our test suite in "tests/". This would also require some enhancements to our build system, since currently we do not link against all of our code for the test suite. Also new test mechanisms are needed to make it possible to e.g. test audio & graphics output (e.g. code that can load an existing reference image and compare it to the output of a graphics primitive). It might also be helpful to enhance the "null" backend so that all unit tests can be run against that, headless (good for automated tests on a server with actual graphics/audio output. | |||
# A "test" engine: Unit tests probably would likely only be runnable on "desktop" ports like Windows and Linux, and not on ports like PSP or NDS, due to run environment restrictions. To allow extended coverage on *all* ports, [[HOWTO-Engines|implementing a "test" engine]] would be helpful. In addition to running on all ports, the "test" engine also provides an environment to test things in an integrate, non-isolated way, unlike unit tests.It should test as many features as possible: game detection, file loading/saving, navigating the file system, testing graphics and audio output, etc. -- and it must be possible to verify everything. I.e. compare results to reference data, output progress data on the screen, create dump logs, and so on. Lots can be done here. | |||
# A framework for testing actual games: At some point it would also be useful (but quite likely beyond this task) to be able to perform testing on actual games, to ensure certain bugs do not occur anymore (regression tests). This might include loading a savegame, feeeding a set of virtual user events to the game, then comparing screen snapshots, internal variables etc. to reference data. | |||
The test suite extension would be easiest to use on Linux etc. and there is already some existing code to build on. An test engine would work on all ports, including game consoles and phones, and is nice for testing in a more "holistic" and complete way. Of course best would be to have both, and also make it possible to run all unit test from inside the test engine. | |||
Overall the test engine has the highest priority, since it would give us the possibility to test ''all'' ports without additional work. Thus the primary task would be to implement and document at least a test engine, which tests (preferably) all of ScummVM's subsystems extensively. | |||
Implementing a testing framework for games / game engines would also be of high interest (esp. if it worked on all ports), as many bugs we have to deal with occur on only certain tests systems. Having automated tests for these would help our porters a lot, but are also difficult to implement, and require the tester to own copies of the games to be tested. This could be considered as an alternative primary task, but would require substantially more knowledge about ScummVM. | |||
''Required Skills:'' | ''Required Skills:'' | ||
| Line 194: | Line 196: | ||
* Experience with unit tests. | * Experience with unit tests. | ||
* Knowledge about various ScummVM subsystems would be useful, but not required. | * Knowledge about various ScummVM subsystems would be useful, but not required. | ||
* Knowledge about how to create a new engine for ScummVM would also be useful. | * Knowledge about how to create a new engine for ScummVM would also be useful. See also our [[HOWTO-Engines|Engines HOWTO]]. | ||
''Also required:'' | |||
* Access to a non-desktop testing device (PSP, NDS, PS2, WinCE or Symbian mobile, etc.) and the ability to compile and run ScummVM on it (the ScummVM team would help with compiling and running). | |||
== Audio related tasks == | == Audio related tasks == | ||