Difference between revisions of "HOWTO-Debug-Endian-Issues"

HOWTO-Debug-Endian-Issues (view source)

Revision as of 18:46, 22 October 2022

4,085 bytes added , 18:46, 22 October 2022

Update this page for 2022 and the tweaked VM

Dwatteau

186

edits

@@ Line 1: / Line 1: @@
-As most desktop machines are now Little-Endian i.e. x86, debugging endian bugs is becoming difficult.<br>
+Most desktop development machines being [https://en.wikipedia.org/wiki/Endianness little-endian] nowadays, proper endianness testing and debugging is becoming difficult.
-Since (as of Q4 2012) all common Big-Endian machines are not desktop systems e.g. Wii, getting run-time debug information from them is problematic and slow. The only solution available generally was to set up a legacy big-endian machine, which would generally also be very problematic and slow.
-Apart from auditing the codebase for known non-portable code constructs as per [[Coding_Conventions#Endianess_issues]], without a working test machine to replicate the issue and debug with, fixing these bugs can prove impossible.
+The [[PlayStation_3|PS3]], [[Nintendo_Wii|Wii]] and [[AmigaOS]] ports are some examples of systems running in big-endian mode and where endianness issues show up from time to time. Unfortunately, these ports are not very well suited for efficient iterative development and debugging. Apart from auditing the codebase for [[Coding_Conventions#Endianness_issues|known non-portable code constructs]], without a working test machine to replicate the issue and debug with, fixing these bugs can prove impossible.
-This page details a solution to this.
+Various solutions for working from a big-endian development environment exist, though:
-The basic idea is to use an open source Virtual Machine (VM) software package to emulate a machine with a different endian CPU to the host system, install a Linux distribution for the emulated platform onto this, and then install the standard GCC toolchain and required libraries necessary to compile ScummVM from source, run and debug with gdb or valgrind as per normal Unix debugging.
+# Running a native, modern and powerful big-endian development system:
+#* Examples include the [https://www.raptorcs.com/content/BK1B01/intro.html Raptor Blackbird™] workstation. It is a powerful system which can run up-to-date Linux/BSD distributions, both in little-endian and big-endian modes.
+#* Unfortunately, it's a pricey niche system, and there's no guarantee that big-endian OS options will be maintained for long.
+# Running a native, older big-endian development system:
+#* Examples include buying an older G4 or G5 Apple PowerPC system, or an older SPARC64 Sun station. They can often be bought second hand at very reasonable prices.
+#* However, running a modern development environment on them in getting harder (but not impossible), because the big-endian desktop ecosystem receives less and less maintenance, so things often tend to break. G5 systems (in particular) also require careful maintenance and can be very power-hungry.
+#* (Some MIPS and ARM boards also exist, but their quality can vary a lot, and although the MIPS and ARM architectures are theoretically bi-endian, in practice these development boards often only run in little-endian mode, nowadays. They can be useful for strict-alignment testing, though.)
+# '''Emulating a big-endian development system from your regular development machine''':
+#* Any reasonably powerful desktop system should be able to emulate a big-endian architecture, thanks to [https://www.qemu.org QEMU].
+#* The main drawback is that this currently requires running some older/unmaintained Linux distributions, and, as of late 2022, audio and graphics acceleration support is missing. Since this is emulation (and not virtualization), there is also a noticeable (but usually tolerable) performance impact.
-Of the open source VM packages available, [https://www.virtualbox.org/ Oracle Virtualbox] is probably the easiest to setup and configure. Unfortunately, this is only capable of emulating x86 and x86_64 machines, which are both little endian.
+This howto focuses on the last option, since it is the most accessible one, and it's still a way of fixing the majority of endianness issues we encounter.
-However, it is derived from [http://www.qemu.org QEMU], which is capable of emulating a far larger number of architectures, including PPC, which is probably the most common big-endian architecture.
+== Linux big-endian PowerPC emulation with QEMU ==
-== Setting up a PPC (Big-Endian) VM using QEMU on a x86 (Little-Endian) host ==
+The current reference VM is a pre-configured Debian 8.11 PowerPC system. It has been modified to feature an updated C++11 toolchain (GCC 5.5.0).
-* Install QEMU for your host operating system i.e. Win32, Linux, OSX
+The VM image is quite large, so ask the rest of the team for its URL.
-** Linux: QEMU should be available via your distribution's package management libraries. Examples for standard distributions are as follows, but consult your distribution documentation for the exact method:
-*** Debian: <pre>apt-get install qemu</pre>
-** Win32: Precompiled builds for Win32 are available from [http://lassauge.free.fr/qemu/ here].
-** OSX: This is packaged with an OSX GUI as the Q Emulator available from [http://www.kju-app.org/ here].
-** For all other systems, source code be downloaded from [http://www.qemu.org here]. Refer to your system and QEMU's documentation for help on compiling and installing this.
-* Configure QEMU to emulate a PPC machine.
-** This should just be using the right binary i.e. qemu-system-ppc(.exe, .app), with some extra options to select an emulated sound device. Consult the QEMU documentation on your machine for full instructions.
-** QEMU may exit with an application crash/DLL error on Win32. This generally indicates bad parameter settings, rather than an executable/dependency problem.
-** For reference, the following Win32 batch file may be useful as a basis:<br>
- @ECHO OFF
- SET SDL_VIDEODRIVER=directx
- SET QEMU_AUDIO_DRV=dsound
- SET SDL_AUDIODRIVER=dsound
- SET QEMU_AUDIO_LOG_TO_MONITOR=1
- START qemu-system-ppcw.exe ^
- -name debianPPC ^
- -L Bios -vga std -soundhw es1370 ^
- -m 512M ^
- -boot menu=on,splash=./bootsplash.bmp,splash-time=3000 ^
- -rtc base=localtime,clock=host ^
- -hda debian_squeeze_powerpc_desktop.qcow2 ^
- -cdrom GAME.iso ^
- -net nic,model=ne2k_pci -net user ^
- -no-reboot
- pause
-<br>
+Some important notes:
-Note that for the '''soundhw''' parameter, some valid options are es1370 and ac97. Choose the one that works for you.
-<br>
-* Install a PPC Linux distribution onto the QEMU PPC VM.
-** The full method here is to create a blank QCOW2 disk image file with the qemu-tools and then use this as the main emulated hard disk via the "-hda" option, then add a Linux PPC CDROM ISO image using the "-cdrom" option, start QEMU and do the distribution installation procedure as normal.
-** However, '''Debian Linux Squeeze for PPC''' is suggested as prebuilt disk images are available [http://people.debian.org/~aurel32/qemu/powerpc/ here]. Squeeze is preferred over Wheezy for this.
-* Enable VM network access proxied via the host system.
-** By default, QEMU should be providing a virtual ethernet network adapter to the VM, which will provide access to the internet via the host's connection. This can be checked by the normal methods such as ICMP ping within the VM OS. If this is not working, check on your host machine, then refer to the QEMU documentation for any missing configuration setting and to see if the hosted OS requires any device driver to allow this virtual device to work.
-* Install any OS updates to the VM.
-** Especially when using the prebuilt images, OS system updates should be done via the normal method to ensure up to date system libraries are installed.
-*** Debian: <pre>apt-get update</pre>
-* Install the ScummVM source code, required libraries and compile a debug build.
-* Within the VM, this can now be done as per a normal Linux build. For further information, check our [[Compiling ScummVM/GCC|GCC compilation instructions]] page
-** Getting ScummVM source tree: <pre>git clone git://github.com/scummvm/scummvm.git</pre>
-*** Refer to [[Developer_Central#Getting_started |here]] for more help.
-** Install any required libraries: <pre>apt-get install libsdl12dev</pre>
-** Compiling ScummVM: <pre>cd scummvm && ./configure && make clean && make</pre>
-*** Refer to [[Compiling_ScummVM/GCC |here]] for more help.
-* Provide game data to the VM.
-** Though QEMU can provide access to host directories as emulated FAT formatted drives by this option: <pre> -drive file=fat:ro:some-directory</pre>, this only supports FAT-16 and is thus limited to 2GB maximum. It is suggested to use the <pre> -cdrom </pre> option with a ISO image instead. CD swapping is not possible, but it is fairly easy to create a ISO image from a set of files in the same way as prior to writing a CD.
-** Transferring data to the host VM by normal network file transfer is also possible i.e. SSH from the VM to host or another machine, HTTP download, etc.
-* Install debugging tools.
-** <pre>apt-get install gdb</pre>
-** <pre>apt-get install valgrind</pre>
-*** Note: Valgrind is currently broken on Debian Squeeze PPC. If you want to install this, you will need to add the apt repositories for "Wheezy" and install the updated version from there.
-* Debugging can then be done as per a normal native Linux machine, though patience may be required as the VM will be much slower than a native machine.
-A preinstalled standard desktop Debian Squeeze PPC QEMU hard disk image in qcow2 format can be provided by the ScummVM development team, but since this is over 5GB, this is currently only provided on request, rather than on the download site.
+* Security support updates have been discontinued for Debian 8 in late 2018. Older cryptographic ciphers and certificates (such as in TLS or SSH) in the base system may also cause various issues. For this reason, this VM should only be run ''on a local, trusted environment''.
+* Modern versions of Debian are actually still built for big-endian PowerPC, but it's not a ''release'' architecture anymore, which means that it's only available through Debian ''unstable''. Debian unstable is harder to maintain than a stable release, and bugs/reliability issues often appear, especially  on non-mainstream architectures. This is why we're sticking with a Debian 8 VM for now.
+* '''3D games and audio content will be hard to debug''' on this environment, since QEMU only provides a limited, unaccelerated framebuffer, and no sound card support yet. Make sure that the ScummVM component you want to debug/test won't be impacted by this.
+* The bigger your CPU clock rate, the better: a 4 GHz CPU will bring noticeable improvement over a 3 GHz CPU, which is itself much better than a 2 GHz CPU, and so on. Note that QEMU emulation is mostly single-threaded, so having many CPU cores isn't really useful for this.
+=== Starting the VM ===
+Once you've downloaded and extracted the VM archive, you'll need to install QEMU for your system with your usual package manager (Windows builds are available [https://qemu.weilnetz.de/w64/ here]).
+Then, create one of the following scripts in the same directory as the VM image.
+<code>run.bat</code> for Windows:
+<syntaxhighlight lang="batch">
+"%ProgramFiles%\qemu\qemu-system-ppcw.exe" ^
+-L pc-bios ^
+-M mac99,via=pmu ^
+-m 2048 ^
+-g 1024x750x32 ^
+-no-reboot ^
+-boot c ^
+-prom-env "boot-device=hd:,\yaboot" ^
+-prom-env "boot-args=conf=hd:,\yaboot.conf" ^
+-hda hda-debian8-scummvm-ppc.qcow2
+</syntaxhighlight>
+<code>run.sh</code> for macOS and other Unix-like systems:
+<syntaxhighlight lang="shell">
+#!/bin/sh
+qemu-system-ppc \
+-L pc-bios \
+-M mac99,via=pmu \
+-m 2048 \
+-g 1024x750x32 \
+-no-reboot \
+-boot c \
+-prom-env 'boot-device=hd:,\yaboot' \
+-prom-env 'boot-args=conf=hd:,\yaboot.conf' \
+-hda hda-debian8-scummvm-ppc.qcow2
+</syntaxhighlight>
+Most options should be kept as-is, but you may want to tweak the following ones:
+* <code>-m 2048</code>: the amount of memory given to the VM, in megabytes. Using more than 2 GiB is not possible in QEMU at the moment. Moreover, this is a 32-bit system and it wouldn't change the VM performance much.
+* <code>-g 1024x750x32</code>: VM screen resolution and bit depth. You can try suiting it to your needs, but strange results may happen with some resolutions, and, since there's no graphics acceleration, making the window too big may worsen performance. Reducing the bit-depth from <code>32</code> to <code>24</code> bits could help in some cases.
+* <code>-cdrom /path/to/host/game.iso</code>: this is a quick way of sharing some game or development files from your host to the VM (it will then appear in its file manager). Other file-sharing options between the two systems are possible ([[#Various options for a more convenient setup|see below]]).
+Then, run that script. A QEMU window should appear, and a Linux system should boot. Wait until a full [https://docs.xfce.org/4.10/start XFCE desktop] appears (this may take a couple of minutes, depending on your host system performance).
+Keyboard layout can be changed in Applications Menu > Settings > Keyboard > Layout, or with <code>setxkbmap</code> (see also https://wiki.debian.org/Keyboard), if necessary. Default credentials are <code>scummvm</code> / <code>scummvm</code>, since this is just a local development environment.
+=== Building ScummVM in the Debian PPC VM ===
+Open a terminal by clicking on Applications Menu > Terminal Emulator. The password for any <code>sudo</code> command is also <code>scummvm</code>.
+If you need to shut down the VM at any point, click on Applications Menu > Log out > Shut Down, and make sure that the VM is completely halted before closing the QEMU window.
+Don't expect the included web browser to be remotely useful for anything.
+For development purposes, you may want to install the following set of tools, if you use them:
+<syntaxhighlight lang="shell">
+sudo apt-get install vim tmux ccache ddd valgrind
+</syntaxhighlight>
+Compilers, GNU Make, GDB, Git and SDL development files are already installed in this image. The APT package manager is also pre-configured to use the older [https://www.debian.org/distrib/archive.en.html Debian 8 archive](] files. Some Debian 8 GPG keys have expired since then, though, so APT will print some security warnings.
+Cloning the repository and compiling ScummVM is done through the [[Compiling_ScummVM/GCC|usual means]], with some important points:
+* Everything is going to be slower than your usual environment; building ScummVM through <code>qemu-system-ppc</code> with only the SCUMM engine and no optimizations takes around 20 minutes with a 4.2 GHz Intel i7, for example.
+* Since QEMU emulation is single-threaded, there is no point is running <code>make</code> with any <code>-j</code> flag for parallel compilation.
+* It is highly suggested to only enable the engines and features that you need for your test, e.g. <code>./configure --disable-detection-full --disable-all-engines --enable-engine=tinsel --disable-lua --disable-tinygl --disable-cloud --disable-hq-scalers --disable-optimizations --enable-debug</code> will save you a lot of time if you're only interested in testing the Tinsel engine.
+* Some tools such as Valgrind become unbearably slow when emulated. <code>--enable-asan</code> is available, but its implementation is from 2015.
+* The older 1.2 branch of SDL is used, since rendering is done through an unaccelerated framebuffer at the moment, and SDL1.2 is a better option than SDL2 for this case.
+You can then run the resulting <code>scummvm</code> binary, while making sure that some ScummVM options such as Global Options > Paths > Extra Path are properly configured for development. Then, add your game, and see how it behaves on big-endian! Run <code>gdb</code> on it if necessary, make the appropriate code changes, and iterate with <code>make</code> until it works as expected.
+=== Various options for a more convenient setup ===
+If you need a powerful and user-friendly IDE inside the VM, you won't have many options, because this is an old and limited Linux system. If you don't like console text editors, you're probably out of luck (unless you want to try <code>sudo apt-get install codeblocks</code>). One option could be to work from your usual IDE on your regular desktop, and share its files with the VM.
+The VM can access its host system via the <code>10.0.2.2</code> IP address (useful if you want to share files from the host to the VM with an NFS, SMB, HTTP, or FTP server).
+If you just need to access the VM through SSH:
+* add the <code>-nic user,hostfwd=tcp::60022-:22</code> option to your <code>qemu-system-ppc</code> script
+* inside the VM, install the OpenSSH server: <code>sudo apt-get install openssh-server</code>
+* from your host, run: <code>ssh -p 60022 scummvm@127.0.0.1</code>. SSH access means that you can also <code>rsync</code> and so on.
+[https://wiki.archlinux.org/title/QEMU#VNC Connecting through VNC] is also possible, but it's probably not going to be a great experience.
+If you need a bit more storage space inside the VM, some big and unnecessary tools can be removed:
+<syntaxhighlight lang="shell">
+sudo apt-get remove --purge 'vlc.*' 'libreoffice.*' 'gimp.*' 'firefox.*' 'iceweasel.*'
+sudo apt-get autoremove --purge
+</syntaxhighlight>
 == Debugging for Other Architectures ==