Difference between revisions of "Buildbot"

From ScummVM :: Wiki
Jump to navigation Jump to search
m (minor cleanup)
(Update to new buildbot)
 
(26 intermediate revisions by 10 users not shown)
Line 1: Line 1:
{{Infobox Project Service Information|
url=http://buildbot.scummvm.org|
purpose=Provides automated build services for an increasing number of our supported platforms.|
maintainer=LePhilousophe
}}
== Introduction ==
== Introduction ==


Line 9: Line 15:
You can find the ScummVM buildbot page [http://buildbot.scummvm.org/ here].
You can find the ScummVM buildbot page [http://buildbot.scummvm.org/ here].


== Goals ==
All the code need to setup the platforms and the Buildbot server is located in [https://github.com/scummvm/dockerized-bb dockerized-bb repository].
 
==Goals==


* Compile all ports when a commit is done (both trunk and the active branch) to check if it broke one or more ports.
*Compile all ports when a commit is done (both trunk and the active branch) to check if it broke one or more ports.
* Notify developers about breakage (and if a following commit fixes it again).
* Notify developers about breakage (and if a following commit fixes it again).
* Provide useful statistics.
* Provide useful statistics.
* Provide nightly builds of all ports.
* Provide nightly builds of all ports.


== Status ==
==Status==


What we have so far:
What we have so far:
* The bot already polls the SVN repository for changes on trunk and the current branch.
*The bot polls the [[Git|Git Server]] for changes on trunk and the current branch.
* When a change occurs, all ports are being built incrementally.
*When a change occurs, all ports are being built incrementally.
* Once every 24h (currently 4am CET), a full and clean rebuild is compiled for all ports.
*Once every 24h (currently 4am UTC), a full and clean rebuild is compiled for all ports.
* Provide the [https://buildbot.scummvm.org/#/snapshots nightly builds via HTTP].
*Notify developers upon problems through an IRC bot in [[IRC Channel]] and a Web hook in [[Discord Server|Discord server]].


What's missing:
What's missing:
* Add more toolchains for all missing ports - when possible (see below).
*Add more toolchains for all missing ports - when possible (see below).
* Notify developers upon problems (IRC and/or mail is possible).
* Provide the nightly builds via HTTP.


== Toolchains ==
==Toolchains ==


A few requirements must be met to add a toolchain to the buildbot server:
A few requirements must be met to add a toolchain to the buildbot server:
* It's possible to cross-compile the port from Linux (that might change in the future).
* The toolchain must be created using Docker build command.
* No custom Makefiles from the ''backend/'' folder, the port has to use the ''./configure'' script.
* No custom Makefiles from the ''backend/'' folder, the port has to use the ''./configure'' script.
* The compile process doesn't modify nor add anything in the source tree (The ''./configure'' script is called from another directory).
*The compile process must neither modify nor add anything in the source tree. All builds are performed in external build directories, by invoking the ''./configure'' script in these external directories.
* The port is fully buildable from scratch with only ''./configure'' arguments and environment variables for it.
*The port must be fully buildable from scratch with only ''./configure'' arguments, environment variables for it and specific Make targets.
 
The toolchain should be primarily maintained by the port maintainer, but since this requires a little Linux experience, its not a must ;). We will provide assistance with this where possible.
 
If your toolchain/port is ready to be added, ask ''lephilousophe'' for guidance on adding it to the repository. A good idea would be to start off existing toolchain and adapt it to your platform.
 
==Administration on server==
 
===Managing Buildbot master configuration===
When the Buildbot configuration files are changed, a user with shell access and sudo privileges needs to run the following commands:
 
<syntaxhighlight lang="bash">
sudo -u buildbot -s
cd ~/dockerized-bb
git pull
</syntaxhighlight>
 
You can check that configuration is good by running the following command in the working tree:<syntaxhighlight lang="bash">
make master-check
</syntaxhighlight>When changing Buildbot version (by editing the Makefile variable), it's necessary to install it. This can be done with the following command:
<syntaxhighlight lang="bash">
make master
</syntaxhighlight>
 
Exit out from Buildbot's shell by running "exit" or pressing CTRL+D.
Then, you can reload or restart the Buildbot process with systemd.
 
Once Buildbot is upgraded or code not related to configuration has changed, Buildbot must be restarted. Reloading configuration is not enough.
 
<syntaxhighlight lang="bash">
sudo systemctl reload buildbot
sudo systemctl restart buildbot
</syntaxhighlight>
===Upgrading platforms===
Platforms are never built on the Buildbot server. They are pulled from [https://hub.docker.com/r/scummvm/dockerized-workers ScummVM docker registry] or transmitted directly from maintainer computer using ssh when the needed pieces are not freely redistributable (Apple SDK for example).
 
On the server, the command to download platforms from the registry must be run from working tree:<syntaxhighlight lang="bash">
make pull-workers
</syntaxhighlight>Once the newly downloaded platforms are fully tested, don't forget to cleanup old images with the following command:<syntaxhighlight lang="bash">
docker image prune
</syntaxhighlight>
 
==Managing Docker images==
All images are built on the maintainer computer using Make from the code pushed to dockerized-bb.
 
The build is done in two phases: toolchain and worker. Toolchain is where all the tools are built and worker is built using the toolchain and Buildbot software.
 
To build necessary images, the following tools are needed :
 
*Git,
*Docker,
* GNU M4,
*GNU Make
 
Once everything is installed, you need to checkout the dockerized-bb repository. You can create a Makefile.user file out of the example one to customize the platforms you are interested in building and downloading.
 
When it's done, running the following command will build the toolchain and the worker for the desired platform and push it to the Docker registry:<syntaxhighlight lang="bash">
make toolchains/<platform> workers/<platform> workers/<platform>/push
</syntaxhighlight>Dependencies are handled so running only the last command will create the toolchain, the worker and upload it.
 
You need an account with the proper rights to push on ScummVM Docker registry. Registry can be customized in Makefile.user if it's needed.
 
To optimize the disk space when pulled on the server, it's better to build all the platform images on the same machine. They will share all the layers they have in common.
 
===Rebuilding Toolchain Libraries===
 
It is periodically necessary to update the toolchain libraries for newer version releases or to add new library dependencies.
 
This is a basic outline procedure for how to do this.
 
On your local copy of dockerized-bb repository, you can run the following command (needs Python):<syntaxhighlight lang="bash">
make check-versions
</syntaxhighlight>This command will list all outdated packages versions. It makes easier to track all updates.
 
Do the needed edits to the various files (for example update the date of the Debian image to have the latest one available).
 
When possible tags with specific versions are used (dates, git commits identifiers, version number, ...). It ensures images can be replicated everywhere and helps to debug problems.


The toolchain should be maintained by the port maintainer, but since this requires a little Linux experience, its not a must ;)
When all versions have been updated rebuild needed images with:<syntaxhighlight lang="bash">
make toolchains workers
</syntaxhighlight>Again, specifying workers alone will rebuild toolchains before when needed.


If your toolchain/port is ready to be added, ask ''sev'', ''joostp'' or ''dhewg'' for an account.
Eventually, you can push all the images with the following command:<syntaxhighlight lang="bash">
make push-toolchains push-workers
</syntaxhighlight>As the images are independent, both must be specified when you want to push all types of images.

Latest revision as of 17:55, 9 May 2021

Buildbot Project Service
URL(s) http://buildbot.scummvm.org
Purpose Provides automated build services for an increasing number of our supported platforms.
Maintainer(s) LePhilousophe

Introduction

From the buildbot homepage:

The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.
The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin.

You can find the ScummVM buildbot page here.

All the code need to setup the platforms and the Buildbot server is located in dockerized-bb repository.

Goals

  • Compile all ports when a commit is done (both trunk and the active branch) to check if it broke one or more ports.
  • Notify developers about breakage (and if a following commit fixes it again).
  • Provide useful statistics.
  • Provide nightly builds of all ports.

Status

What we have so far:

  • The bot polls the Git Server for changes on trunk and the current branch.
  • When a change occurs, all ports are being built incrementally.
  • Once every 24h (currently 4am UTC), a full and clean rebuild is compiled for all ports.
  • Provide the nightly builds via HTTP.
  • Notify developers upon problems through an IRC bot in IRC Channel and a Web hook in Discord server.

What's missing:

  • Add more toolchains for all missing ports - when possible (see below).

Toolchains

A few requirements must be met to add a toolchain to the buildbot server:

  • The toolchain must be created using Docker build command.
  • No custom Makefiles from the backend/ folder, the port has to use the ./configure script.
  • The compile process must neither modify nor add anything in the source tree. All builds are performed in external build directories, by invoking the ./configure script in these external directories.
  • The port must be fully buildable from scratch with only ./configure arguments, environment variables for it and specific Make targets.

The toolchain should be primarily maintained by the port maintainer, but since this requires a little Linux experience, its not a must ;). We will provide assistance with this where possible.

If your toolchain/port is ready to be added, ask lephilousophe for guidance on adding it to the repository. A good idea would be to start off existing toolchain and adapt it to your platform.

Administration on server

Managing Buildbot master configuration

When the Buildbot configuration files are changed, a user with shell access and sudo privileges needs to run the following commands:

sudo -u buildbot -s
cd ~/dockerized-bb
git pull

You can check that configuration is good by running the following command in the working tree:

make master-check

When changing Buildbot version (by editing the Makefile variable), it's necessary to install it. This can be done with the following command:

make master

Exit out from Buildbot's shell by running "exit" or pressing CTRL+D. Then, you can reload or restart the Buildbot process with systemd.

Once Buildbot is upgraded or code not related to configuration has changed, Buildbot must be restarted. Reloading configuration is not enough.

sudo systemctl reload buildbot
sudo systemctl restart buildbot

Upgrading platforms

Platforms are never built on the Buildbot server. They are pulled from ScummVM docker registry or transmitted directly from maintainer computer using ssh when the needed pieces are not freely redistributable (Apple SDK for example).

On the server, the command to download platforms from the registry must be run from working tree:

make pull-workers

Once the newly downloaded platforms are fully tested, don't forget to cleanup old images with the following command:

docker image prune

Managing Docker images

All images are built on the maintainer computer using Make from the code pushed to dockerized-bb.

The build is done in two phases: toolchain and worker. Toolchain is where all the tools are built and worker is built using the toolchain and Buildbot software.

To build necessary images, the following tools are needed :

  • Git,
  • Docker,
  • GNU M4,
  • GNU Make

Once everything is installed, you need to checkout the dockerized-bb repository. You can create a Makefile.user file out of the example one to customize the platforms you are interested in building and downloading.

When it's done, running the following command will build the toolchain and the worker for the desired platform and push it to the Docker registry:

make toolchains/<platform> workers/<platform> workers/<platform>/push

Dependencies are handled so running only the last command will create the toolchain, the worker and upload it.

You need an account with the proper rights to push on ScummVM Docker registry. Registry can be customized in Makefile.user if it's needed.

To optimize the disk space when pulled on the server, it's better to build all the platform images on the same machine. They will share all the layers they have in common.

Rebuilding Toolchain Libraries

It is periodically necessary to update the toolchain libraries for newer version releases or to add new library dependencies.

This is a basic outline procedure for how to do this.

On your local copy of dockerized-bb repository, you can run the following command (needs Python):

make check-versions

This command will list all outdated packages versions. It makes easier to track all updates.

Do the needed edits to the various files (for example update the date of the Debian image to have the latest one available).

When possible tags with specific versions are used (dates, git commits identifiers, version number, ...). It ensures images can be replicated everywhere and helps to debug problems.

When all versions have been updated rebuild needed images with:

make toolchains workers

Again, specifying workers alone will rebuild toolchains before when needed. Eventually, you can push all the images with the following command:

make push-toolchains push-workers

As the images are independent, both must be specified when you want to push all types of images.