Difference between revisions of "HOWTO-Reverse Engineering"

It's up to the individual whether they want to use a debugger when reverse engineering a program. Some prefer a more cerebral challenge of only figuring out code execution using a debugger, whereas others may find using a debugger useful for figuring out what values are passed to functions. I would recommend using a debugger particularly when reversing a game for the purpose of adding ScummVM support.  When you start implementing code to implement game functionality once you've got portions of the game disassembled, it can be immensely useful for tracking down bugs, particularly if you initial implement your methods to closely match the methods in the disassembly.

For debugging purposes, if the game is a DOS game the DosBox Debugger is the best tool I've found for executing and debugging DOS programs. The default distribution of DosBox doesn't have it enabled, but you can either compile DosBox with it enabled, or download a previously compiled executable. See the [http://vogons.zetafleet.com/viewtopic.php?t=7323 DosBox Debugger Thread] for more information.

- Get the value from the beginning of the segment. This is just to make the calculations easier, since the start of the segment will have an instruction offset between 0h and 0Fh, which means it won't be messing with  

Methods can be renamed using the general 'N' hotkey (as well as via the menus), and the 'Y' can be used to specify a C-like prototype for a method. This is particularly useful when some of the parameters for a method are passed using registers. By explicitly documenting what the method expects, it makes it easier to remember later on when you're reversing methods that call it. If a method does have parameters passed in registers, prototypes like the below can be used:

In this case, the method takes an 8-bit parameter in al, and another 16-bit value in bx, then returns a result in the ax register.

In this case, an initial index in the ax register is multiplied by 30h (30 hexadecimal = 48 decimal). So from this we can determine that the given structure is 48 bytes in size. For smaller sized structures, you may want to create as many 2 byte word fields as needed to make up the correct size for the structure. For larger sizes, the easiest way is to simply declare an array of the needed structure size - 1, and follow it with a single byte field. You can then delete/undefine the array. The remaining byte will keep the structure at the correct size, and you can then later fill in the fields as you find references to them.

In such cases, the only way to tell for sure is to start searching for immediate values in the program of values one byte backwards at a time, until you can't find any values. For example, if you find references in the code of values '2CFFh', and '2CFEh', each with previous multiplications by 30h/48, but none for '2CFDh', then you can probably be confident that the array starts at offset 2CFDh.  

One of the easiest places to start a disassembly is generally by identifying file accesses. Using IDA, you can, for example, do a text search for 'open file' to find occurrences of file opening. Likewise for file reading, writing, and closing. Normally, a program will encapsulate these calls into a method of it's own, so your first disassembly step can be in identifying the methods and naming them appropriately with names like 'File_open', 'File_read', and so on. Likewise, giving the passed parameters an appropriate name. In IDA, the 'Y' command can be used to set up an appropriate method signature for methods. By properly naming the method and it's parameters, this will help you in all the methods that call those methods.  

For example, if a read method has a 'size' parameter and a 'buffer' parameter, then if a method that calls passes '200' for the size, and a reference into the stack, you can be confident that the stack entry can be called something like 'readBuffer', and use the '*' (array size) key when looking at the Stack View to set the size of the array to 200 bytes.

The DosBox debugger may prove useful when dealing with games using large resources - try using 'BPINT 21 42' to put a breakpoint on any calls to seek within a file. By clearly identifying the 'Seek method', you can then step out of that routine to find what called it. Hopefully, you can then examine the logic in the disassembly used to generate the file offset to help you figure out how file offsets are generated for specific resources, and from that figure out how the resource's index works.

Remember that game resource managers not only typically merge multiple individual resources into one single bigger file, they frequently also compress them as well, to save space and prevent people from seeing textual resources when viewing the contents of the file. If the resource manager tends to If you can figure out the strategy used for extracting single resources, it may be worthwhile taking the time to code a standalone program to extract and, if necessary, decompress single resources into separate output files. That way, you can more easily look at individual resources that are used by the game without having to worry about manually locating them in the archive/resource file.

Another place to get started on the disassembly is the graphic draw routines, those responsible for copying raw pixels to the screen surface. Graphic display is complicated in the early PC days by different modes writing to memory in different ways. In the Monochrome/Hercules mode, for example, 8 pixels are stored per 8-bit byte. In EGA, the addressing can be complicated by how the display is configured - the same areas of memory may  

For most of the graphics mode, you can look at them in a similar manner - as a block of data in memory starting at offset A000h:0, and a number of bytes per line that will vary per line. Assembly routines that deal with the graphics sreen will typically have code to figure out screen offsets based on a passed and y parameter, so it will frequently be easy to identify the parameters and figure out how the screen offsets work. For example, in 320x200x256 MCGA mode, an offset will be calculated using the formula (y * 320) + x.

For finding the graphic routines you have two options. The first is to entirely use IDA, and simply search for immediate values of 'A000h'. Since this is the area of memory graphics and displayed in, it can be a quick way to locate graphic routines.

The other alternative is to use the DosBox Debugger. It has a use command called 'bpm' that allow you to set a memory breakpoint, which then gets triggered if the given memory address changes. So you could do 'bpm A000:0' to set a breakpoint on the first byte of the screen memory (i.e. the top left hand corner of the screen). Then whichever routine modifies it first will trigger the breakpoint. Using the previous discussed techniques, you can find the same place in your IDA disassembly, and look into reversing that method first. Generally speaking, it will likely be a clear screen or fill rectangle routine.

It will be likely that realted functions will be next to each other, so once you've looked into the given identified function, you may also be able to review previous or following functions to see if they have identifiable graphic routines.

The strings in the data segment can be an excellent source for identifying the purposes. If you're very lucky, there may be error messages that contain the name of the function as part of the error. In which case, you can then out what code references it, and name the method appropriately.

Even if any error messages don't contain method names, an error message can prove invaluable. For example, an error message like "Unable to initialise mouse" tells you that whatever method uses it is seting up the mouse for access, so could be given a name like 'initialise_mouse', or 'initialise_events'.  

Another possibility for data segment strings is to pick strings that may be descriptive enough to guess their puprose. For example, if a string has a name like 'FNT20', or 'UI.FNT', it's likely that it's a font file, containing the images for each character for use in displaying on the screen. In that case, any code that uses it is likely to be passing it to a 'font_open' function, which loads up the font. If you can disassemble that method, you may be able to determine how the loaded font is stored in memory. From there, you can use the cross references function of IDA to find other methods that use the same memory address as where the font is stored, and which will likely give you the methods for actually displaying text on the screen.

Another method for identifying methods may be simply executing the program itself. When running the program in DosBox,you may find it useful to start stepping through the main procedure to see what happens from stepping over every method. IDA may be helpful in identifying the main procedure, but even if not, most programming languages have a series of method calls for setting up the initial application state,and then a single final call to the 'main' method.

With the main method identified, stepping over each method may produce interesting results. For example, a single method call may contain all the code for showing the game's introduction sequence. If this can be identified, you could name the method appropriately. You may then be able to gleam information from how the method is called and for the method itself:

Particularly for cases like that, identifying and naming the graphic/screen methods may be helpful, since you could work the disassembly from both the front end reading the animation, and from the low level drawing of the graphics.