Buildbot server configuration

This page contains notes on the setup of the MIT krb5 buildbot master and workers. The buildbot master is hosted on krbdev-buildbot.mit.edu. Workers are listed in machines.txt in the krbdev-services repository.

Operational notes

To force a new build on all workers, log into krbdev-buildbot.mit.edu, run "su -s /bin/bash - buildbot", then run "buildbot sendchange -b master -m localhost:9989 -W yourusername@mit.edu" (or similarly for a different branch).

Packages

The buildbot package is required for the buildbot master. The git package is required for the krb5 repository mirror.

buildbot master setup

buildbot 0.9 changes its terminology to refer to "workers" rather than "slaves". At the current time the buildbot server runs on Ubuntu 16.04, where the buildbot package is version 0.8.12. These notes will need to be updated when we move to 0.9 or later.

Run:

 su -s /bin/bash - buildbot
 touch .k5login
 rmdir masters slaves
 buildbot create-master -r master

Install buildbot/master.cfg from krbdev-services into /var/lib/buildbot/master. Copy /var/lib/buildbot/master/slaves.py from the old server. Make sure both files are owned by and readable by buildbot. If it is necessary to reconstruct slaves.py, has the form:

 from buildbot.buildslave import BuildSlave
 slaves = [
     BuildSlave('v09', '<password>', properties={'platform': 'amd64-u1204'}),
     ...
 ]

The name and password must match the values used on the worker. The platform is mostly arbitrary and will be used to construct builder entries. Multiple workers can have the same platform. The third argument may be omitted for workers which run special tasks (such as the documentation build) which aren't part of the regular platform builds.

Copy over .ssh/authorized_keys from the old server's /var/lib/buildbot. If it is necessary to reconstruct it, it must contain the ssh key for each worker entry (.ssh/id_rsa.pub from the buildbot account on the worker).

As root, edit /etc/default/buildmaster and change the values so they read:

 MASTER_ENABLED[1]=1
 MASTER_NAME[1]="master"
 MASTER_USER[1]="buildbot"
 MASTER_BASEDIR[1]="/var/lib/buildbot/master"
 MASTER_OPTIONS[1]=""
 MASTER_PREFIXCMD[1]=""

Run "service buildmaster restart".

Install a krbsnap keytab into /var/lib/buildbot, readable only by root. Install buildbot/doc-update.sh from krbdev-services into /var/lib/buildbot. Add the following cron job:

 0 4 * * * /var/lib/buildbot/doc-update.sh

git mirror setup

The buildbot master host runs a mirror of the drugstore krb5 git repository, for access by workers and to send change notifications to the buildbot master.

Create a krbsnap account using the uid of the Athena krbsnap user:

 useradd -m -u 38160 -s /bin/bash krbsnap

As krbsnap, create ~/.ssh and add the krbsnap key from /git/krb5.git/hooks/krbsnap_rsa_key.pub on drugstore.mit.edu to ~/.ssh/authorized_keys. (This file may be managed by ops along with the .k5login file, and ops may have created the krbsnap account during provisioning.)

As krbsnap, run:

 mkdir ~/krb5.git
 cd ~/krb5.git
 git init --bare

To populate the git repository, log into drugstore as root, "su -s /bin/bash - yourusername", and run:

 cd /git/krb5.git
 GIT_SSH=/git/krb5.git/hooks/ssh-as-krbsnap git push krbsnap

Make sure that /git/krb5.git/config contains a remote named "krbsnap" for krbsnap@krbdev-buildbot.mit.edu, and that the [hooks] section contains an entry "push-to = krbsnap".

As krbsnap, run:

 cd ~/krb5.git
 touch git-daemon-export-ok
 cd hooks
 cp /usr/share/buildbot/contrib/git_buildbot.py .
 ln -s git_buildbot.py post-receive

As krbsnap, run "crontab -e" and add this entry:

 @reboot git daemon --detach --base-path=$HOME

Run the command by hand to start the daemon initially.

buildbot worker setup

For the Solaris buildbot worker, see Solaris_Build_Environment for platform-specific instructions.

Install the buildbot worker software, using the python3-buildbot-worker package (buildbot-slave prior to Ubuntu 18.04) or the platform equivalent.

Install the following Ubuntu packages: autoconf bison build-essential dejagnu git keyutils ldap-utils libcmocka-dev libkeyutils-dev libldap2-dev liblmdb-dev libsasl2-dev libssl-dev libtool pkg-config python3-kdcproxy python3-pip tcl-dev

Install the slapd package. apt may ask for a master password twice; the value is unimportant and can be left blank.

On Ubuntu 18.04 or later, install libresolv-wrapper.

Run "pip3 install pyrad".

(TBD: enumerate packages needed for documentation build worker.)

If the platform package does not create a buildbot account, create one with a home directory. Create an empty .k5login file in the buildbot home directory. These instructions will assume that the buildbot home directory is /var/lib/buildbot.

As the buildbot account ("su -s /bin/bash - buildbot"), generate a key using:

 ssh-keygen -q -N  -f .ssh/id_rsa -t rsa

Add the contents of .ssh/id_rsa.pub to ~buildbot/.ssh/authorized_keys on krbdev-buildbot.mit.edu.

Run "ssh -l buildbot krbdev-buildbot.mit.edu" to get the master host key into .ssh/known_hosts. The correct host key fingerprint of the master can be obtained by running "ssh-keygen -l -E sha256 -f /etc/ssh/ssh_host_ecdsa_key.pub" on krbdev-buildbot.mit.edu (or perhaps ssh_host_rsa_key.pub or ssh_host_dsa_key.pub if the worker has an old ssh client).

Create a "workers" directory in buildbot's home directory and run "buildbot-worker create-worker /var/lib/buildbot/workers/NAME 127.0.0.1:9989 NAME PASSWORD", using the name and password from the slaves.py entry for the worker. Prior to Ubuntu 18.04, the command is "buildslave create-slave" and the subdirectory should be named "slaves".

Arrange for the buildslave process to be started on boot. On Ubuntu 18.04, this is accomplished by editing /etc/default/buildbot-worker as root and setting:

 WORKER_ENABLED[1]=1
 WORKER_NAME[1]="NAME"
 WORKER_USER[1]="buildbot"
 WORKER_BASEDIR[1]="/var/lib/buildbot/workers/NAME"
 WORKER_OPTIONS[1]=""
 WORKER_PREFIXCMD[1]=""

On earlier versions of Ubuntu, the file is /etc/default/buildslave and each occurrence of "worker" is replaced with "slave".

Add a cron job for the buildbot account (run "crontab -e" as buildbot) to maintain the ssh tunnel to the master:

 */5 *  *   *   *     exec ssh -oExitOnForwardFailure=yes -l buildbot -N -L9989:127.0.0.1:9989 krbdev-buildbot.mit.edu

Run the command manually (backgrounded, without the "exec") to start it for the current session.

The worker which runs the documentation build needs the python3-lxml package, and either the python3-cheetah package (requires Ubuntu 18.10 or higher) or the python3-pip package and "pip3 install cheetah3" to be run.

snapshot service

This service is probably no longer needed, but the setup details are covered here in case it becomes necessary to resurrect it.

In the krbsnap home directory, create a subdirectory "snap" and copy the krbdev/gensnap script from krbdev-services into it. Also create a keytab for the krbsnap principal in ~/snap/krbsnap.keytab. Add the cron job to run gensnap from krbdev/krbsnap-crontab to the crontab for the krbsnap account. (Do not install sync_gitsvn or its cron job; it is defunct.)

The gensnap script updates a working copy for each branch, runs mkrel, and installs the results in krbsnap@aeneas.mit.edu:/var/ftp/pub/kerberos/dist/vaporware-r-us . These snapshots are used by the old nightly build infrastructure (scripts in /mit/krbdev/testing), which has been supplanted by other CI systems.