Buildbot server configuration

This page contains notes on the setup of the MIT krb5 buildbot master and workers. The buildbot master is hosted on krbdev-buildbot.mit.edu. Workers are listed in machines.txt in the krbdev-services repository.

Operational notes

To force a new build on all workers, log into krbdev-buildbot.mit.edu, run "su -s /bin/bash - buildbot", then run "buildbot sendchange -b master -m localhost:9989 -W yourusername@mit.edu" (or similarly for a different branch).

Packages

The buildbot package is required for the buildbot master. The git package is required for the krb5 repository mirror.

buildbot master setup

buildbot 0.9 changes its terminology to refer to "workers" rather than "slaves". At the current time the buildbot server runs on Ubuntu 16.04, where the buildbot package is version 0.8.12. These notes will need to be updated when we move to 0.9 or later.

Run:

 su -s /bin/bash - buildbot
 touch .k5login
 rmdir masters slaves
 buildbot create-master -r master

Install buildbot/master.cfg from krbdev-services into /var/lib/buildbot/master. Copy /var/lib/buildbot/master/slaves.py from the old server. Make sure both files are owned by and readable by buildbot. If it is necessary to reconstruct slaves.py, has the form:

 from buildbot.buildslave import BuildSlave
 slaves = [
     BuildSlave('v09', '<password>', properties={'platform': 'amd64-u1204'}),
     ...
 ]

The name and password must match the values used on the worker. The platform is mostly arbitrary and will be used to construct builder entries. Multiple workers can have the same platform. The third argument may be omitted for workers which run special tasks (such as the documentation build) which aren't part of the regular platform builds.

Copy over .ssh/authorized_keys from the old server's /var/lib/buildbot. If it is necessary to reconstruct it, it must contain the ssh key for each worker entry (.ssh/id_rsa.pub from the buildbot account on the worker).

As root, edit /etc/default/buildmaster and change the values so they read:

 MASTER_ENABLED[1]=1
 MASTER_NAME[1]="master"
 MASTER_USER[1]="buildbot"
 MASTER_BASEDIR[1]="/var/lib/buildbot/master"
 MASTER_OPTIONS[1]=""
 MASTER_PREFIXCMD[1]=""

Run "service buildmaster restart".

Install a krbsnap keytab into /var/lib/buildbot, readable only by root. Install buildbot/doc-update.sh from krbdev-services into /var/lib/buildbot. Add the following cron job:

 0 4 * * * /var/lib/buildbot/doc-update.sh

git mirror setup

The buildbot master host runs a mirror of the drugstore krb5 git repository, for access by workers and to send change notifications to the buildbot master.

Create a krbsnap account using the uid of the Athena krbsnap user:

 useradd -m -u 38160 -s /bin/bash krbsnap

As krbsnap, create ~/.ssh and add the krbsnap key from /git/krb5.git/hooks/krbsnap_rsa_key.pub on drugstore.mit.edu to ~/.ssh/authorized_keys. (This file may be managed by ops along with the .k5login file, and ops may have created the krbsnap account during provisioning.)

As krbsnap, run:

 mkdir ~/krb5.git
 cd ~/krb5.git
 git init --bare

To populate the git repository, log into drugstore as root, "su -s /bin/bash - yourusername", and run:

 cd /git/krb5.git
 GIT_SSH=/git/krb5.git/hooks/ssh-as-krbsnap git push krbsnap

Make sure that /git/krb5.git/config contains a remote named "krbsnap" for krbsnap@krbdev-buildbot.mit.edu, and that the [hooks] section contains an entry "push-to = krbsnap".

As krbsnap, run:

 cd ~/krb5.git
 touch git-daemon-export-ok
 cd hooks
 cp /usr/share/buildbot/contrib/git_buildbot.py .
 ln -s git_buildbot.py post-receive

As krbsnap, run "crontab -e" and add this entry:

 @reboot git daemon --detach --base-path=$HOME

Run the command by hand to start the daemon initially.

buildbot worker setup

For the Solaris buildbot worker, see Solaris_Build_Environment for platform-specific instructions.

Install the buildbot slave software, using the buildbot-slave package or the platform equivalent.

If the platform package does not create a buildbot account, create one with a home directory. Make sure it has an empty .k5login file, or one containing developer principals. These instructions will assume that the buildbot home directory is /var/lib/buildbot.

As the buildbot account ("su -s /bin/bash - buildbot"), generate a key using:

 ssh-keygen -q -N  -f .ssh/id_rsa -t rsa

Add the contents of .ssh/id_rsa.pub to .ssh/authorized_keys on krbdev-buildbot.mit.edu.

Run "ssh -l buildbot krbdev-buildbot.mit.edu" to get the master host key into .ssh/known_hosts. The correct host key fingerprint of the master can be obtained by running "ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ecdsa_key.pub" on krbdev-buildbot.mit.edu (or perhaps ssh_host_rsa_key.pub or ssh_host_dsa_key.pub if the worker has an old ssh client).

Create a "slaves" directory in buildbot's home directory and run "buildslave create-slave /var/lib/buildbot/slaves/NAME 127.0.0.1:9989 NAME PASSWORD", using the name and password from the slaves.py entry for the worker.

Arrange for the buildslave process to be started on boot. On Ubuntu, this is accomplished by editing /etc/default/buildslave and setting:

 SLAVE_ENABLED[1]=1
 SLAVE_NAME[1]="NAME"
 SLAVE_USER[1]="buildbot"
 SLAVE_BASEDIR[1]="/var/lib/buildbot/slaves/NAME"
 SLAVE_OPTIONS[1]=""
 SLAVE_PREFIXCMD[1]=""

Add a cron job for the buildbot account (run "crontab -e" as buildbot) to maintain the ssh tunnel to the master:

 */5 *  *   *   *     exec ssh -oExitOnForwardFailure=yes -l buildbot -N -L9989:127.0.0.1:9989 krbdev-buildbot.mit.edu

Run the command manually (backgrounded) to start it for the current session.

snapshot service

This service is probably no longer needed, but the setup details are covered here in case it becomes necessary to resurrect it.

In the krbsnap home directory, create a subdirectory "snap" and copy the krbdev/gensnap script from krbdev-services into it. Also create a keytab for the krbsnap principal in ~/snap/krbsnap.keytab. Add the cron job to run gensnap from krbdev/krbsnap-crontab to the crontab for the krbsnap account. (Do not install sync_gitsvn or its cron job; it is defunct.)

The gensnap script updates a working copy for each branch, runs mkrel, and installs the results in krbsnap@aeneas.mit.edu:/var/ftp/pub/kerberos/dist/vaporware-r-us . These snapshots are used by the old nightly build infrastructure (scripts in /mit/krbdev/testing), which has been supplanted by other CI systems.