User:Buddha^/GSoC2007-ProjectDiary

2007-05-28 (Monday) - GSoC starts!

So the GSoC 2007's coding period started. I'm just getting started, I installed Ubuntu 7.04 on my Dell CPX laptop last saturday and got ScummVM to compile after some trying (Just playing with packages, knowing what to install etc. After all the needed packages were there, ./configure and make did the trick).

Right now I'm getting acquainted with ScummVM's internals & workings. And after knowing enough to do so I'll be working on AGI version fallback detection (sev did work on that previously).

Just installed Code::Blocks 2007-05-23 nightly build on my laptop.

So, I took some screenshots of most official AGI games. And submitted my first patch, woot! It was screenshots of Larry I (640x480, hq2x + aspect ratio correction turned on, opting + advpng -optimized).

2007-05-29 (Tuesday) - Taking screenshots of AGI games etc

Today I took and submitted screenshots for the following games (In addition to the Larry I screenshots I submitted yesterday):

Official Sierra AGI games:

• King's Quest I, II, III & IV
• Space Quest I & II
• Police Quest I
• Manhunter I & II
• Gold Rush!
• The Black Cauldron
• Mixed-Up Mother Goose

Fanmade AGI games:

• Voodoo Girl: Queen of the Darned
• Space Quest 0: Replicated
• Space Quest X: The Lost Chapter
• Serguei's Destiny 1

And I also made small thumbs for most of the games like "Larry's head thumb" for the Larry series and a "Sarien head thumb" for the Space Quest series etc.

2007-05-30 (Wednesday) - Starting work on WinAGI's wag-file loading

Installed MinGW and MSYS on my desktop machine (It's got Windows 2000 on it) and compiled ScummVM on it.

Worked on *.wag file loading.

2007-05-31 (Thursday) - Made a quick & ugly rudimentary parser for WinAGI's wag-file loading

Worked some more on *.wag file loading. I've already got a *.wag file parser but it's not pretty :). I'll have to convert it into something better suited for ScummVM.

2007-06-03 (Sunday) - Made class for WinAGI's properties

Made a Doxygen documented WagProperty class that represents a single property from a *.wag file (That can be the game's ID, the game's author's name, an about message about the game, the last edit date of the game or something else). Made it so that you can just read a property's header and choose whether to read the property's data also into memory or just skip it (Useful when you know beforehand what properties you are interested in and what you aren't).

Made WagFileParser class use WagProperty when parsing the *.wag file.

2007-06-04 (Monday) - More work on the WinAGI's wag-file parser

Modified WagProperty class and implemented more of WagFileParser's needed functionality. Documented WagFileParser's implemented parts.

2007-06-05 (Tuesday) - Fallback detection and WinAGI's wag-file parsing continues

Worked on the WagFileParser and Agi::fallbackDetector related code.

2007-06-06 (Wednesday) - Ran into trouble with fallback detection

Installed Visual Studio 2005 Express edition because MinGW+MSYS-combo jammed my Windows 2000 machine some of the time when using make. Hopefully this'll work better...

Ran into trouble with the AGI engine's fallbackDetector-routine needing dynamic ADGameDescription-structs. ADGameDescription has constant strings in it and we'd need to use dynamic strings so it looks like something will have to change. Sev said he'll talk with someone about this. Otherwise it looks like I might be getting to submit my first code patch real soon now. We'll see how soon that'll be...

And about Visual Studio 2005 Express... it does work quite nicely at the moment, thank you :) (Knocks wood).

2007-06-07 (Thursday) - Finished fallback detection for the AGI engine

After some talk with a friend of mine and much talk with my mentor Sev about the problems with the current fallback detection system I set out to implement Fingolfin's suggestion.

Got it done! It took some hours though ;) And it compiles too... let's see about testing and hopefully my first code patch tomorrow.

2007-06-08 (Friday) - Communication lessons and first code patch

Talked with my mentor Sev and got a little wiser about portability. Now I know that seeking in files can be very slow on some platforms, like e.g. GP32. I had ignored Sev's advice on some code choices because I didn't think seeking in files would be an issue at all. There was some discussion and I'll try to be more frank and communicative about my own decisions and ask about the reasons behind others' choices in the future.

I think it's good to know the real reasons behind choices... in this particular case the reason for a certain coding choice was "We shouldn't do much file seeking because file seeking is very slow on some platform (e.g. GP32) that we support".

So I changed my WAG-file parsing code to read in the whole file and handle parsing it in memory rather than go seeking in it on the disk. I separated the changes needed for the dynamic string content generation in the fallback detection from my WAG-file parsing code and submitted the former's changes as a patch. Woot! My first code patch to ScummVM :)

2007-06-11 (Monday) - Changes to the first patch and cleaning WAG-parsing

During the weekend Fingolfin had given feedback to my first code patch in the tracker. And his second alternative approach was clearly better so I took it, made some fixes to it and submitted that to the tracker.

I also took my WAG-file parsing code and simplified and cleaned it. I took some code out to make the code more maintainable (Things like allowing to read only the header of a property and afterwards also its data or skipping that data altogether if we know we don't need that property's data). At least in this case I think it was better to not have them there, rather have more maintainable code than some little added performance boost or less memory usage in this particular non-critical area.

Oh, and I also came to notice that I had somehow managed to get non-normalized line endings in some of the source code files in my local ScummVM copy. Probably something to do with using diff and patch, applying Fingolfin's version of the fallback detection patch to my own local copy etc (It had 0x0A (LF) as the line ending, my local source code files had 0x0D & 0x0A (CR & LF)). Oh, the joys of choice between CR, LF, CR & LF etc :P

2007-06-12 (Tuesday) - Got first patch accepted to the trunk and submitted second patch (WAG file parsing)

Okay, so today I got my first patch with additional changes by Fingolfin & Sev accepted to the trunk. Yay!

I also submitted my second code patch that deals with WAG file parsing. Here's its description:

"Here's a patch that makes use of WinAGI's (http://www.winagi.com/) *.wag file format in AGI games' fallback detection. WAG-files may include information like game ID, game description, used AGI interpreter version, game's last edit date etc. This patch makes the AGI fallback detector use those information, if present."

After this I started looking into AGI256 & AGI256-2 related stuff because that's what's next!

2007-06-13 (Wednesday) - Started looking into AGI256

Read some AGI specifications on the ScummVM wiki. Read some of the ScummVM's AGI engine code. Now I know that in the ScummVM's AGI engine the priorities ("Sort of a Z-buffer") are saved in the same buffer as the picture data (Each pixel's lower 4 bits give the picture data and upper 4 bits give the priority data). So that'll make it a bit more of an effort to support 256 color images in the AGI engine.

Also found that Ctrl-D brings up the debug console and that first 16 colors are for the EGA/Amigaish-palette and the next 16 colors are for the console's transparency (So you can still see the underlying game with the console window on top). We'll see what's to be done with those console transparency colors when we need to use the whole 256 colors for the images' palette. Maybe map them through a table lookup to the static AGI256 256 color palette?

Looked at the AGI 256 pictures' static palette with GIMP. There's a pattern in it. Here's my current theory on the subject:

• AGI256 pictures' palette's structure:
  • First 16 colors are the default EGA palette.
  • Next 16 colors are greyscale colors from black to white.
  • Next 72 colors are probably a HSL/HSI (Or something like that) color sweep with full saturation and brightness.
  • Then the same color sweep as before but with less saturation and/or brightness.
  • And again the same color sweep as before but with even less saturation and/or brightness.
  • Last 8 colors are greyscale colors from low intensity grey to almost black.

Also looked at differences in AGI.EXE between the original Sierra On-Line's AGI interpreter version 2.936 and the hacked AGI256 AGI.EXE. Used HT editor for that. Didn't learn much with that approach though. But then I took Interactive Disassembler Freeware v4.3 and started looking at AGI256's AGIGRAF.OVL file.

Using IDA I got some info out of the AGIGRAF.OVL. Commented some of the functions in it. But without understanding what the functions are that are called from outside I can't understand it fully (AGIGRAF.OVL is like a DOS age DLL so it references functions that aren't in AGIGRAF.OVL but should be in memory when it's loaded into memory).

• Some guesses for the function names from AGI256's AGIGRAF.OVL (*HIGHLY* preliminary and subject to change):
  • setVideoMode (320x200x256c)
  • setTextMode (40x25x16c)
  • showAgi256Pic?
  • calcHeightLUT
    • Calculates table of y*320 values in range 0 <= y < 200
  • setPalette?
  • readNextAgi256Pic?
  • readAgi256Pic?
  • setClrCurrPixel
  • setWhiteMenuBar

2007-06-14 (Thursday) - Taking a look at the AGI256 hacked interpreter

Started to convert ScummVM's AGI engine to a 256 color version so that the color screen and priority screen would be separated into totally different buffers (At the moment they are married/merged together so that each pixel in the 160x168 AGI screen buffer contains 4 bits of color data and 4 bits of priority data). Doing the conversion in this way would require changes in many places in the AGI engine... so Sev hinted that it would be good first to take a look at the hacked AGI256 AGI interpreter with IDA because whoever did that hack highly probably didn't just rewrite half of the engine :-).

So started to look into AGI256 hacked interpreter files. Also found Nick Sonneveld's commented disassemblies for various AGI versions at http://www.agidev.com/projects/nagi/dev.php and they've been helpful.

2007-06-15 (Friday) - Disassembling and reading specifications

Looked more into AGI256's hacked files and read AGI specifications etc.

Commented almost all the differences between the original AGI 2.936's AGI.EXE and the hacked AGI256's AGI.EXE.

2007-06-18 (Monday) - Reverse engineering AGI256

Got more into working with IDA. Using info from Sonneveld's IDBs for various AGI.EXE versions I could make out where in the memory the "DLL"-files are loaded (AGIDATA.OVL, AGIGRAF.OVL, AGIOBJS.OVL). So I made a flat AGI.EXE that has them included in the correct positions. It helps with IDA debugging because IDA can now see all the functions and give cross references between them etc.

So after making the flattened AGI.EXE for the AGI256 hack inspection I read more of the Sonneveld's IDBs, commented and tried to understand parts of the flattened AGI.EXE, made some structs in IDA, fiddled with IDA's segments etc.

Figured out some functions like the agi256ReadPic -function and also a weird looking function that did something with the palette... as a matter of fact it turned out to be a synchronized palette blinking effect that's called from the screen shaking routine (Command number 110: cmd_shake_screen).

Learned a few things today... like that call pushes the location of the next instruction after it (In retrospect that's actually very rational) rather than the location of the call instruction itself. There was some interesting "call $+3 followed by a pop to get the IP" -usage in the agi256ReadPic routine and that's how I came to ponder the call instruction's true workings. HelpPC helped.

It looks like some call is jumping in the middle of a "shr al, 1" instruction. I'll see about that tomorrow...

2007-06-19 (Tuesday) - Got prototypes for AGI256 and AGI256-2 working

The jump instruction that was pointing in the middle of a "shr al, 1" instruction (I talked about that in yesterday's post) was part of some obsolete Hercules graphics mode code. So it was never used in AGI256's or AGI256-2's code.

Reverse engineered some more of the AGI256. Tried a prototype in ScummVM based on a theory about how AGI256 works. Didn't work straight away... some things were off. So took another look at the blitting function in AGIOBJS.OVL as the blitting function was one of the points were I wanted/needed more information. Finally at some point it worked! AGI256 prototype support for ScummVM is here :-).

Encouraged or just on a plain roll I looked at the AGIOBJS.OVL differences in the blitting function between AGI256 and AGI256-2. There wasn't much difference in them. Just that the AGI256-2 didn't use mirroring or run length encoding for its views (That info I had already got from AGI256-2's readme, though) and little else (How it wrote the pixels into memory for one). Tried another prototype for supporting AGI256-2 and other some fiddling and coding, it now works too.

There's some discrepancy in AGI256-2's intro (The background area surrounding the AGI256-logo, the color bar and the authors text isn't all black, it's partially grey). But I get something very similar when I run the AGI256-2's hacked AGI.EXE from command line in full screen mode under Windows 2000 (Although as I tried it just now, it doesn't always do that). Running the AGI256-2 demo just by clicking on the AGI256-2's hacked AGI.EXE in Explorer works correctly. So I'm not sure what's causing it exactly. Gotta look into it...