Open main menu

Difference between revisions of "Buildbot"

30 bytes removed ,  17:55, 9 May 2021
Update to new buildbot
(Update to new buildbot)
 
Line 2: Line 2:
url=http://buildbot.scummvm.org|
url=http://buildbot.scummvm.org|
purpose=Provides automated build services for an increasing number of our supported platforms.|
purpose=Provides automated build services for an increasing number of our supported platforms.|
maintainer=Andre Heider ([[User:Dhewg|Dhewg]])<br>John Willis ([[User:DJWillis|DJWillis]])
maintainer=LePhilousophe
}}
}}


Line 15: Line 15:
You can find the ScummVM buildbot page [http://buildbot.scummvm.org/ here].
You can find the ScummVM buildbot page [http://buildbot.scummvm.org/ here].


== Goals ==
All the code need to setup the platforms and the Buildbot server is located in [https://github.com/scummvm/dockerized-bb dockerized-bb repository].


* Compile all ports when a commit is done (both trunk and the active branch) to check if it broke one or more ports.
==Goals==
 
*Compile all ports when a commit is done (both trunk and the active branch) to check if it broke one or more ports.
* Notify developers about breakage (and if a following commit fixes it again).
* Notify developers about breakage (and if a following commit fixes it again).
* Provide useful statistics.
* Provide useful statistics.
* Provide nightly builds of all ports.
* Provide nightly builds of all ports.


== Status ==
==Status==


What we have so far:
What we have so far:
* The bot already polls the [[Git|Git Server]] for changes on trunk and the current branch.
*The bot polls the [[Git|Git Server]] for changes on trunk and the current branch.
* When a change occurs, all ports are being built incrementally.
*When a change occurs, all ports are being built incrementally.
* Once every 24h (currently 4am CET), a full and clean rebuild is compiled for all ports.
*Once every 24h (currently 4am UTC), a full and clean rebuild is compiled for all ports.
* Provide the [http://buildbot.scummvm.org/builds.html nightly builds via HTTP].
* Provide the [https://buildbot.scummvm.org/#/snapshots nightly builds via HTTP].
* Notify developers upon problems through an IRC bot in [[IRC Channel]].
*Notify developers upon problems through an IRC bot in [[IRC Channel]] and a Web hook in [[Discord Server|Discord server]].


What's missing:
What's missing:
* Add more toolchains for all missing ports - when possible (see below).
*Add more toolchains for all missing ports - when possible (see below).
* Notify developers upon problems via mail.


== Toolchains ==
==Toolchains ==


A few requirements must be met to add a toolchain to the buildbot server:
A few requirements must be met to add a toolchain to the buildbot server:
* It must be possible to cross-compile the port from Linux (that might change in the future).
* The toolchain must be created using Docker build command.
* No custom Makefiles from the ''backend/'' folder, the port has to use the ''./configure'' script.
* No custom Makefiles from the ''backend/'' folder, the port has to use the ''./configure'' script.
* The compile process must neither modify nor add anything in the source tree. All builds are performed in external build directories, by invoking the ''./configure'' script in these external directories.
*The compile process must neither modify nor add anything in the source tree. All builds are performed in external build directories, by invoking the ''./configure'' script in these external directories.
* The port must be fully buildable from scratch with only ''./configure'' arguments and environment variables for it.
*The port must be fully buildable from scratch with only ''./configure'' arguments, environment variables for it and specific Make targets.


The toolchain should be primarily maintained by the port maintainer, but since this requires a little Linux experience, its not a must ;). We will provide assistance with this where possible.
The toolchain should be primarily maintained by the port maintainer, but since this requires a little Linux experience, its not a must ;). We will provide assistance with this where possible.


If your toolchain/port is ready to be added, ask ''sev'', ''joostp'' or ''dhewg'' for an account.
If your toolchain/port is ready to be added, ask ''lephilousophe'' for guidance on adding it to the repository. A good idea would be to start off existing toolchain and adapt it to your platform.


== Administration ==
==Administration on server==


When the buildbot config files are changed, a user with shell access and sudo privileges needs to run the following commands. It's good practice to do this once buildbot is idle. In order to prevent the buildbot from accidentally starting building processes, it's recommended to stop the buildbot-slave processes by invoking
===Managing Buildbot master configuration===
When the Buildbot configuration files are changed, a user with shell access and sudo privileges needs to run the following commands:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo systemctl stop buildbot-slave
sudo -u buildbot -s
cd ~/dockerized-bb
git pull
</syntaxhighlight>
</syntaxhighlight>


before you continue with the following steps:
You can check that configuration is good by running the following command in the working tree:<syntaxhighlight lang="bash">
 
make master-check
</syntaxhighlight>When changing Buildbot version (by editing the Makefile variable), it's necessary to install it. This can be done with the following command:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo su -s /bin/bash buildbot
make master
cd /var/lib/buildbot/scummvm-sites
git pull --ff-only
</syntaxhighlight>
</syntaxhighlight>


Exit out from buildbot's shell by running "exit" or pressing CTRL+D.
Exit out from Buildbot's shell by running "exit" or pressing CTRL+D.
Then, you can restart the buildbot master and slave process(es) through the systemd services:
Then, you can reload or restart the Buildbot process with systemd.
 
Once Buildbot is upgraded or code not related to configuration has changed, Buildbot must be restarted. Reloading configuration is not enough.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo systemctl restart buildbot-master
sudo systemctl reload buildbot
sudo systemctl restart buildbot-slave
sudo systemctl restart buildbot
</syntaxhighlight>
</syntaxhighlight>
===Upgrading platforms===
Platforms are never built on the Buildbot server. They are pulled from [https://hub.docker.com/r/scummvm/dockerized-workers ScummVM docker registry] or transmitted directly from maintainer computer using ssh when the needed pieces are not freely redistributable (Apple SDK for example).


If problems arise, check if all files in the master (~buildbot/scummvm-master) and slave (~buildbot/scummvm-slave) directories are owned by the correct user (buildbot:buildbot). If the daemons were incorrectly started, new files might have been created under another user account. In that case:
On the server, the command to download platforms from the registry must be run from working tree:<syntaxhighlight lang="bash">
 
make pull-workers
* stop the daemons (see above)
</syntaxhighlight>Once the newly downloaded platforms are fully tested, don't forget to cleanup old images with the following command:<syntaxhighlight lang="bash">
* fix the permissions with
docker image prune
<syntaxhighlight lang="bash">
chown -R buildbot:buildbot ~buildbot/scummvm-master/*
chown -R buildbot:buildbot ~buildbot/scummvm-slave/*
</syntaxhighlight>
</syntaxhighlight>
* start the daemons


=== Rebuilding Toolchain Libraries ===
==Managing Docker images==
All images are built on the maintainer computer using Make from the code pushed to dockerized-bb.


It is periodically necessary to update the toolchain libraries for
The build is done in two phases: toolchain and worker. Toolchain is where all the tools are built and worker is built using the toolchain and Buildbot software.
newer version releases or to add new library dependencies.
This is a basic outline procedure for how to build these on the
buildbot.


* Toolchains are located at /opt/toolchains/<host target> in general, with the following exceptions:
To build necessary images, the following tools are needed :
** Though some of the MinGW 32 toolchain is present in
/opt/toolchains/i586-mingw32msvc
which should be used as the host/prefix in the following
instructions, the binaries i.e. i586-mingw32msvc-gcc are located
at /usr/bin. This is probably as they are precompiled to be
installed at this location, so this is necessary.


However, if the PATH step to the bin is omitted in the following,
*Git,
this will still work for this target, which is confusing and could
*Docker,
result in odd builds as some of toolchain bin contents may be needed.
* GNU M4,
*GNU Make


* All source tarballs used and any patches should be left in /opt/toolchains for reference.
Once everything is installed, you need to checkout the dockerized-bb repository. You can create a Makefile.user file out of the example one to customize the platforms you are interested in building and downloading.
 
* /opt/toolchains is actually a symlink to the real location of the external storage drive, but /opt/toolchains should be used in all configuration to ensure that this can be freely moved in future if necessary.
 
* Check for any platform specific library build instructions at the relevant page [[Compiling_ScummVM|here]].
 
libSDL:
 
* Decompress the latest libSDL tarball to a temporary location i.e.
 
<syntaxhighlight lang="bash">
tar -xvzf SDL-1.2.15.tar.gz -d /home/digitall
</syntaxhighlight>


* Apply any required patches i.e. Currently libSDL-1.2.15 snapshot has a error in the Win32 version strings.. This may occur again on future releases :/
When it's done, running the following command will build the toolchain and the worker for the desired platform and push it to the Docker registry:<syntaxhighlight lang="bash">
make toolchains/<platform> workers/<platform> workers/<platform>/push
</syntaxhighlight>Dependencies are handled so running only the last command will create the toolchain, the worker and upload it.


<syntaxhighlight lang="bash">
You need an account with the proper rights to push on ScummVM Docker registry. Registry can be customized in Makefile.user if it's needed.
patch -p0 < SDL-1.2.15-Fix-Windows-DLL-Version.patch
</syntaxhighlight>


* Add toolchain binaries directory to path.
To optimize the disk space when pulled on the server, it's better to build all the platform images on the same machine. They will share all the layers they have in common.


<syntaxhighlight lang="bash">
===Rebuilding Toolchain Libraries===
PATH=/opt/toolchains/x86_64-w64-mingw32/bin:$PATH
export PATH
</syntaxhighlight>


* Add out of tree build directory and configure the SDL codebase here with the host target and prefix to use the correct cross toolchain and target for make install.
It is periodically necessary to update the toolchain libraries for newer version releases or to add new library dependencies.


<syntaxhighlight lang="bash">
This is a basic outline procedure for how to do this.
mkdir SDL-1.2.15-x86_64-w64-mingw32
cd SDL-1.2.15-x86_64-w64-mingw32
../SDL-1.2.15/configure --host=x86_64-w64-mingw32 --prefix=/opt/toolchains/x86_64-w64-mingw32
</syntaxhighlight>


* Compile the codebase.
On your local copy of dockerized-bb repository, you can run the following command (needs Python):<syntaxhighlight lang="bash">
make check-versions
</syntaxhighlight>This command will list all outdated packages versions. It makes easier to track all updates.


<syntaxhighlight lang="bash">
Do the needed edits to the various files (for example update the date of the Debian image to have the latest one available).
make
</syntaxhighlight>


* Install the compiled files. As you become root, you need to make the same modification to the path to get the required cross toolchain tools such as ranlib.
When possible tags with specific versions are used (dates, git commits identifiers, version number, ...). It ensures images can be replicated everywhere and helps to debug problems.


<syntaxhighlight lang="bash">
When all versions have been updated rebuild needed images with:<syntaxhighlight lang="bash">
sudo bash
make toolchains workers
PATH=/opt/toolchains/x86_64-w64-mingw32/bin:$PATH
</syntaxhighlight>Again, specifying workers alone will rebuild toolchains before when needed.
export PATH
make install
</syntaxhighlight>


* IMPORTANT:
Eventually, you can push all the images with the following command:<syntaxhighlight lang="bash">
** If installing libSDL for Win32 / Win64 MinGW toolchains, please check the note [[Compiling_ScummVM/MinGW#SDL|here]] and make the relevant fixes to bin/sdl-config and lib/pkg-config/sdl2.pc etc.
make push-toolchains push-workers
** If installing libSDL2 for macOS toolchains, please check the note [[Compiling_ScummVM/macOS#Important_note_about_SDL2|here]] and make the relevant fixes to bin/sdl2-config.
</syntaxhighlight>As the images are independent, both must be specified when you want to push all types of images.