Difference between revisions of "Solaris Build Environment"
(→Problem log) |
(→Install required OpenCSW packages) |
||
(41 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
− | This page contains notes on the setup of a Solaris buildbot worker for MIT krb5. |
+ | This page contains notes on the setup of a Solaris buildbot worker for MIT krb5. The current buildbot worker is krbdev-sparc-build.mit.edu. |
Our current Solaris build hardware is a Sun Fire V240 running Solaris 10 U10. |
Our current Solaris build hardware is a Sun Fire V240 running Solaris 10 U10. |
||
− | Solaris 10 includes a variety of free software tools in /usr/sfw, but not the full set of dependencies needed to checkout, build, and test the krb5 tree. We have chosen to use |
+ | Solaris 10 includes a variety of free software tools in /usr/sfw, but not the full set of dependencies needed to checkout, build, and test the krb5 tree. We have chosen to use [https://opencsw.org/ OpenCSW] to add most of the dependencies, and local builds in /usr/local for the remaining few. |
− | == |
+ | ==Set up a useful shell== |
− | 1. Set up a shell with a path containing the needed system utilities: |
||
+ | A root login by default uses a very minimal shell (/bin/sh) with a very minimal path (/usr/sbin:/usr/bin). For any operations performed as root, we begin by starting a functional shell as follows: |
||
− | + | bash |
|
− | export PATH=/usr/sbin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin |
+ | export PATH=/opt/csw/bin:/usr/sbin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin |
+ | export MANPATH=/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man |
||
− | 2. Create /usr/local/src: |
||
+ | For now, we are not modifying /etc/passwd or root's dotfiles to make this happen automatically. |
||
− | mkdir /usr/local/src |
||
+ | ==Set up OpenCSW== |
||
− | cd /usr/local/src |
||
− | 3. From https://ftp.gnu.org/, fetch the latest versions of gmp, mpfr, and mpc, and fetch gcc 5.4.0. Untar them in /usr/local/src. |
||
+ | Installing the pkgutil command (it will be placed in /opt/csw/bin) and update its catalog as follows: |
||
− | 4. Build gmp, mpfr, and mcc as follows. |
||
+ | pkgadd -d http://get.opencsw.org/now |
||
+ | pkgutil -U |
||
− | cd /usr/local/src/gmp-''version'' |
||
+ | "pkgutil -l" lists installed packages. "pkgutil -a string" looks up string in the catalog. "pkgutil -i packagename" installs a package; the -y flag can be used to skip prompts. "pkgutil -U" followed by "pkgutil -yu" updates all installed packages. |
||
− | ./configure --disable-shared ABI=32 |
||
− | gmake && gmake install |
||
− | cd /usr/local/src/mpfr-''version'' |
||
− | ./configure --disable-shared LDFLAGS=-L/usr/local/lib |
||
− | gmake && gmake install |
||
− | cd /usr/local/src/mpc-''version'' |
||
− | ./configure --disable-shared LDFLAGS=-L/usr/local/lib |
||
− | gmake && gmake install |
||
− | 5. Build gcc as follows. This build takes many hours. |
||
+ | ==Install required OpenCSW packages== |
||
− | cd /usr/local/src/gcc-5.4.0 |
||
+ | pkgutil -y -i git |
||
− | edit gcc/config/sol2.h; where it says "#ifndef USE_GLD" and subsequently talks about using a mapfile, change it to: |
||
+ | pkgutil -y -i libssl_dev |
||
− | #if !defined(USE_GLD) && defined(ENABLE_SHARED_LIBGCC) |
||
+ | mkdir /opt/csw/sbin/sparcv9 # to work around an apparent bug in openldap package |
||
− | ./configure --with-gmp=/usr/local --with-mpfr=/usr/local --with-mpc=/usr/local --disable-shared --enable-languages=c,c++ |
||
+ | pkgutil -y -i openldap |
||
− | gmake && gmake install |
||
+ | pkgutil -y -i openldap_back_bdb |
||
+ | pkgutil -y -i openldap_client |
||
+ | pkgutil -y -i openldap_dev |
||
+ | pkgutil -y -i autoconf |
||
+ | pkgutil -y -i tcl |
||
+ | pkgutil -y -i tcl_dev |
||
+ | pkgutil -y -i expect |
||
+ | pkgutil -y -i bind_utils |
||
+ | pkgutil -y -i bison |
||
+ | pkgutil -y -i python33 |
||
+ | pkgutil -y -i buildbot_slave |
||
+ | pkgutil -y -i emacs |
||
+ | pkgutil -y -i gdb |
||
+ | edit /etc/passwd and change the home directory of "games" to / |
||
− | ==Setting up pkgsrc== |
||
+ | We have decided not to support the Solaris 10 native LDAP library and tools (Solaris 11 ships with OpenLDAP and Solaris 12 will make it the default), so we install OpenLDAP from OpenCSW. OpenSSL 0.9 is present in /usr/sfw/lib, but we need 1.0 or later for PKINIT so we install the OpenCSW version. The bison in /usr/sfw/bin is also too old for our x-deltat.y file. buildbot_slave requires Python 2.7 but does not list it as a formal dependency. emacs and gdb are not needed to build krb5, but are handy to have around for manual testing and debugging work. |
||
− | If it becomes necessary to restart this process, "rm -rf /usr/pkgsrc /usr/pkg /var/db/pkg" will clean the slate. |
||
+ | buildbot 0.9 changes its terminology to refer to "workers" rather than "slaves". At the time of this writing, OpenCSW only includes buildbot 0.8.4; when it switches to buildbot 0.9, the package we need will likely change to buildbot_worker. |
||
− | 1. Set up the shell and fetch the pkgsrc tar file: |
||
+ | One of the above packages appears to create a "games" account with the home directory set to /opt/csw. This would ordinarily allow ssh access by the Kerberos principal games@ATHENA.MIT.EDU. We change the home directory of this account to / so that /.k5login governs access. |
||
− | /bin/bash |
||
+ | ==Local builds of remaining dependencies== |
||
− | export PATH=/usr/pkg/sbin:/usr/pkg/bin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin |
||
− | cd /tmp |
||
− | wget https://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz |
||
− | cd /usr |
||
− | gtar xzf /tmp/pkgsrc.tar.gz |
||
− | 2. Bootstrap pkgsrc: |
||
+ | dejagnu is not present in OpenCSW, so we need to build it ourselves: |
||
− | + | mkdir /usr/local /usr/local/src |
|
− | + | From https://ftp.gnu.org/gnu/dejagnu/ fetch the latest dejagnu; untar it in /usr/local/src |
|
− | + | chown -R root:root /usr/local/src/dejagnu-''version'' |
|
− | + | cd /usr/local/src/dejagnu-''version'' |
|
− | + | ./configure && gmake install |
|
− | exit out of the shell, run /usr/pkg/bin/bash, and set the path as in step 1 |
||
− | (cd /usr/pkgsrc/devel/scmcvs && bmake install) |
||
− | (cd /usr/pkgsrc && cvs update) |
||
− | 3. Build packages useful for krb5 development: |
||
+ | ==Set up buildbot== |
||
− | (cd /usr/pkgsrc/devel/git && bmake install) |
||
+ | Create and switch to the buildbot account: |
||
− | ==Creating user accounts== |
||
+ | useradd -d /var/lib/buildbot -u 101 -s /bin/bash -m buildbot |
||
+ | su - buildbot |
||
+ | touch .k5login |
||
+ | edit .profile and add: |
||
+ | PATH=/usr/local/bin:/opt/csw/bin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin |
||
+ | MANPATH=/usr/local/share/man:/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man |
||
+ | USER=buildbot |
||
+ | export PATH MANPATH USER |
||
− | By default, /home on Solaris is controlled by the automounter. To avoid needing to change the automounter configuration, we create user accounts with home directories in /export/home: |
||
+ | Principals may be added to .k5login, but make sure it exists so that the Kerberos principal "buildbot" does not have access to the account. |
||
− | useradd -d /export/home/ghudson -m ghudson |
||
+ | Set up ssh for the ssh tunnel to krbdev-buildbot.mit.edu: |
||
− | Creating an account allows the Kerberos principal of the same name in the ATHENA.MIT.EDU realm to log in on that account. |
||
+ | mkdir .ssh |
||
+ | ssh-keygen -q -N <nowiki>''</nowiki> -f .ssh/id_rsa -t rsa |
||
+ | cat .ssh/id_rsa.pub |
||
+ | In a separate shell, log into krbdev-buildbot.mit.edu, "su -s /bin/bash - buildbot" and add the contents of id_rsa.pub to .ssh/authorized_keys |
||
+ | Run "/usr/bin/ssh -l buildbot krbdev-buildbot.mit.edu" to add the host key to .ssh/known-hosts. Check the key fingerprint against "ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_rsa_key.pub" |
||
− | ==Problem log== |
||
+ | Create the slave directory: |
||
− | A number of problems were encountered when trying simpler or mistaken variants of the above procedure. For the sake of easier debugging, here is a record of them: |
||
+ | mkdir slaves |
||
+ | buildslave create-slave /var/lib/buildbot/slaves/s01 127.0.0.1:9989 s01 ''password'' |
||
− | * It initially looked like cvs wasn't present in pkgsrc. The directory is named devel/scmcvs, because devel/cvs would conflict with the CVS metadata directory on case-insensitive filesystems. |
||
+ | where ''password'' should match the entry for s01 in slaves.py on krbdev-buildbot.mit.edu. |
||
− | * When attempting to build pkgsrc with /usr/sfw/bin/gcc, a circular dependency between pkgtools/digest and lang/gcc2 was encountered. This resulted from running bmake without /usr/sfw/bin in $PATH. After correcting the path, it was necessary to run "bmake clean" in pkgtools/digest to clear the issue. |
||
+ | As root, create /etc/init.d/buildslave with the contents: |
||
− | * When attempting to build pgksrc with /usr/sfw/bin/gcc, devel/p5-gettext and other packages would fail when they included <stdbool.h>, because that Solaris header is incompatible with the default standards environment of gcc 3.4.3. This can be worked around by bootstrapping pkgsrc with "gcc -std=gnu99 -Wno-implicit declaration" and setting CC to that value in /usr/pkg/etc/mk.conf after bootstrap. (Without "-Wno-implicit-declaration", the bootstrap process fails when building bsdinstall.) Building with gcc 5.4.x works around the <stdbool.h> problem, as it defaults to the gnu11 standards environment, but it also requires -Wno-implicit-function-declaration for bsdinstall to build correctly during bootstrap. |
||
+ | #!/sbin/sh |
||
+ | case "$1" in |
||
+ | start) |
||
+ | su buildbot -c 'ssh -l buildbot -N -L9989:127.0.0.1:9989 krbdev-buildbot.mit.edu &' |
||
+ | su - buildbot -c 'buildslave start --quiet /var/lib/buildbot/slaves/s01' |
||
+ | ;; |
||
+ | stop) |
||
+ | su buildbot -c '/opt/csw/bin/buildslave stop --quiet /var/lib/buildbot/slaves/s01' |
||
+ | ;; |
||
+ | esac |
||
+ | exit 0 |
||
− | * After removing /usr/pkgsrc and /usr/pkg, bootstrapping pkgsrc again wouldn't install anything in /usr/pkg except /usr/pkg/etc/mk.conf. To start over, it is necessary to also remove /var/db/pkg. |
||
+ | Make it executable with "chmod u+x /etc/init.d/buildslave". Create the following links: |
||
− | * When attempting to build pkgsrc with "/usr/sfw/bin/gcc -std=gnu99 -Wno-implicit-declaration", /bin/bash segmentation faults were encountered in several packages (starting with p5-Error) when executing long shell lines. To work around this issue, the above procedure builds the pkgsrc bash and uses it for the build after the bootstrap. |
||
+ | ln -s /etc/init.d/buildslave /etc/rc2.d/S99buildslave |
||
+ | ln -s /etc/init.d/buildslave /etc/rc2.d/K00buildslave |
||
− | * When attempting to build pkgsrc with "/usr/sfw/bin/gcc -std=gnu99 -Wno-implicit-declaration", pkg_create for some packages failed with stat failures, where the offending pathnames contained gcc error messages such as "gcc: unrecognized option `-print-multiarch': No such file or directory". To work around this problem, we build a newer version of gcc in /usr/local/bin before building pkgsrc. |
||
+ | We do not currently stop the ssh tunnel automatically, because it isn't easy to do. The ssh tunnel must be manually restarted if it breaks; on other workers, we use a cron job which runs "ssh -oExitOnForwardFailure=yes ...", but the Solaris 10 ssh does not support that option. The start rule will display an unwanted copy of /etc/motd; an alternative would be to explicitly set the path when running buildbot, instead of relying on buildbot's .profile. |
||
− | * When building gcc's dependencies, mpc complains that libgmp uses an incompatible ABI. This is because gmp overrides the default 32-bit ABI and chooses the 64-bit ABI for performance reasons. Configuring gmp with "ABI=32" works around the issue. |
||
+ | ==Create user accounts== |
||
− | * gcc does not use the configured LDFLAGS when linking some of its programs; therefore, configuring with LDFLAGS=-L/usr/local/lib is not sufficient to find its dependencies there, and configuring with LDFLAGS=-R/usr/local/lib does not allow built programs to find the shared library. To work around this problem, we configure the dependencies with --disable-shared and configure gcc with --with-gmp=/usr/local --with-mpfr=/usr/local --with-mpc=/usr/local. |
||
+ | By default, /home on Solaris is controlled by the automounter. To avoid needing to change the automounter configuration, we create user accounts with home directories in /export/home. For example: |
||
− | * If gcc is configured without --disable-shared, programs built with /usr/local/bin/gcc will not run wth the default runtime linker path. We could perhaps modify the runtime linker path to include /usr/local/lib, but we prefer solutions that require fewer global system configuration changes. |
||
+ | useradd -u 3622 -d /export/home/ghudson -s /bin/bash -m ghudson |
||
− | * If gcc is configured with --disable-shared, the build fails with "ld: fatal: file libgcc-unwind.map: stat failed: No such file or directory". (see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65725). To work around this problem, we apply a small change to fix the bug, taken from the aforementioned bug report. |
||
+ | Creating an account allows the Kerberos principal of the same name in the ATHENA.MIT.EDU realm to log in on that account. Matching the local UID to the Moira UID may be unnecessary as long as we do not make use of remote filesystems on this machine. |
||
− | * If gcc is configured with --disable-shared, the build fails with "libtool: compile: not configured to build any kind of library" while building Java classpath materials. To work around this problem, we configure gcc with support for just C and C++. |
||
+ | To set a reasonable path for development work, the user can edit .profile to add: |
||
− | * When building the bash pkgsrc package with gcc 5.4.0 from /usr/local/bin, the subsidiary build of devel/gettext-tools fails with "pkg_create: lstat failed for file # ### BEGIN LIBTOOL TAG CONFIG: disable-static: No such file or directory". The offending string appears in work/.PLIST. A workaround has not yet been found. |
||
+ | PATH=/usr/local/bin:/opt/csw/bin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin |
||
+ | MANPATH=/usr/local/share/man:/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man |
||
+ | export PATH MANPATH |
||
+ | |||
+ | The following shell function, or a variant of it, may be useful for configuring a build with the correct paths. This variant is designed to work in a separate build directory placed next to the src directory within a checkout. |
||
+ | |||
+ | k5configure() { ../src/configure --enable-maintainer-mode --prefix=$HOME/inst --with-ldap CFLAGS=-g CPPFLAGS="-I/opt/csw/include" LDFLAGS="-L/opt/csw/lib -R/opt/csw/lib" "$@"; } |
||
+ | |||
+ | ==To do== |
||
+ | |||
+ | * We should install the SunPro compiler and do automated builds with that compiler as well as gcc. |
||
+ | |||
+ | ==Hardware notes== |
||
+ | |||
+ | The 8-pin modular serial connector for console/ALOM takes a shielded UTP cable. (The shielding probably isn't too important except for EMI reasons.) The modular to female DE-9 adapter is wired as a RS-232 DCE so it will plug directly into most USB to DE-9 RS-232 adapters. (The modular to male DB-25 adapter seems to be wired as a DTE and might need a null modem.) |
Latest revision as of 11:54, 12 July 2018
This page contains notes on the setup of a Solaris buildbot worker for MIT krb5. The current buildbot worker is krbdev-sparc-build.mit.edu.
Our current Solaris build hardware is a Sun Fire V240 running Solaris 10 U10.
Solaris 10 includes a variety of free software tools in /usr/sfw, but not the full set of dependencies needed to checkout, build, and test the krb5 tree. We have chosen to use OpenCSW to add most of the dependencies, and local builds in /usr/local for the remaining few.
Contents
Set up a useful shell
A root login by default uses a very minimal shell (/bin/sh) with a very minimal path (/usr/sbin:/usr/bin). For any operations performed as root, we begin by starting a functional shell as follows:
bash export PATH=/opt/csw/bin:/usr/sbin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin export MANPATH=/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man
For now, we are not modifying /etc/passwd or root's dotfiles to make this happen automatically.
Set up OpenCSW
Installing the pkgutil command (it will be placed in /opt/csw/bin) and update its catalog as follows:
pkgadd -d http://get.opencsw.org/now pkgutil -U
"pkgutil -l" lists installed packages. "pkgutil -a string" looks up string in the catalog. "pkgutil -i packagename" installs a package; the -y flag can be used to skip prompts. "pkgutil -U" followed by "pkgutil -yu" updates all installed packages.
Install required OpenCSW packages
pkgutil -y -i git pkgutil -y -i libssl_dev mkdir /opt/csw/sbin/sparcv9 # to work around an apparent bug in openldap package pkgutil -y -i openldap pkgutil -y -i openldap_back_bdb pkgutil -y -i openldap_client pkgutil -y -i openldap_dev pkgutil -y -i autoconf pkgutil -y -i tcl pkgutil -y -i tcl_dev pkgutil -y -i expect pkgutil -y -i bind_utils pkgutil -y -i bison pkgutil -y -i python33 pkgutil -y -i buildbot_slave pkgutil -y -i emacs pkgutil -y -i gdb edit /etc/passwd and change the home directory of "games" to /
We have decided not to support the Solaris 10 native LDAP library and tools (Solaris 11 ships with OpenLDAP and Solaris 12 will make it the default), so we install OpenLDAP from OpenCSW. OpenSSL 0.9 is present in /usr/sfw/lib, but we need 1.0 or later for PKINIT so we install the OpenCSW version. The bison in /usr/sfw/bin is also too old for our x-deltat.y file. buildbot_slave requires Python 2.7 but does not list it as a formal dependency. emacs and gdb are not needed to build krb5, but are handy to have around for manual testing and debugging work.
buildbot 0.9 changes its terminology to refer to "workers" rather than "slaves". At the time of this writing, OpenCSW only includes buildbot 0.8.4; when it switches to buildbot 0.9, the package we need will likely change to buildbot_worker.
One of the above packages appears to create a "games" account with the home directory set to /opt/csw. This would ordinarily allow ssh access by the Kerberos principal games@ATHENA.MIT.EDU. We change the home directory of this account to / so that /.k5login governs access.
Local builds of remaining dependencies
dejagnu is not present in OpenCSW, so we need to build it ourselves:
mkdir /usr/local /usr/local/src From https://ftp.gnu.org/gnu/dejagnu/ fetch the latest dejagnu; untar it in /usr/local/src chown -R root:root /usr/local/src/dejagnu-version cd /usr/local/src/dejagnu-version ./configure && gmake install
Set up buildbot
Create and switch to the buildbot account:
useradd -d /var/lib/buildbot -u 101 -s /bin/bash -m buildbot su - buildbot touch .k5login edit .profile and add: PATH=/usr/local/bin:/opt/csw/bin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin MANPATH=/usr/local/share/man:/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man USER=buildbot export PATH MANPATH USER
Principals may be added to .k5login, but make sure it exists so that the Kerberos principal "buildbot" does not have access to the account.
Set up ssh for the ssh tunnel to krbdev-buildbot.mit.edu:
mkdir .ssh ssh-keygen -q -N '' -f .ssh/id_rsa -t rsa cat .ssh/id_rsa.pub In a separate shell, log into krbdev-buildbot.mit.edu, "su -s /bin/bash - buildbot" and add the contents of id_rsa.pub to .ssh/authorized_keys Run "/usr/bin/ssh -l buildbot krbdev-buildbot.mit.edu" to add the host key to .ssh/known-hosts. Check the key fingerprint against "ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_rsa_key.pub"
Create the slave directory:
mkdir slaves buildslave create-slave /var/lib/buildbot/slaves/s01 127.0.0.1:9989 s01 password
where password should match the entry for s01 in slaves.py on krbdev-buildbot.mit.edu.
As root, create /etc/init.d/buildslave with the contents:
#!/sbin/sh case "$1" in start) su buildbot -c 'ssh -l buildbot -N -L9989:127.0.0.1:9989 krbdev-buildbot.mit.edu &' su - buildbot -c 'buildslave start --quiet /var/lib/buildbot/slaves/s01' ;; stop) su buildbot -c '/opt/csw/bin/buildslave stop --quiet /var/lib/buildbot/slaves/s01' ;; esac exit 0
Make it executable with "chmod u+x /etc/init.d/buildslave". Create the following links:
ln -s /etc/init.d/buildslave /etc/rc2.d/S99buildslave ln -s /etc/init.d/buildslave /etc/rc2.d/K00buildslave
We do not currently stop the ssh tunnel automatically, because it isn't easy to do. The ssh tunnel must be manually restarted if it breaks; on other workers, we use a cron job which runs "ssh -oExitOnForwardFailure=yes ...", but the Solaris 10 ssh does not support that option. The start rule will display an unwanted copy of /etc/motd; an alternative would be to explicitly set the path when running buildbot, instead of relying on buildbot's .profile.
Create user accounts
By default, /home on Solaris is controlled by the automounter. To avoid needing to change the automounter configuration, we create user accounts with home directories in /export/home. For example:
useradd -u 3622 -d /export/home/ghudson -s /bin/bash -m ghudson
Creating an account allows the Kerberos principal of the same name in the ATHENA.MIT.EDU realm to log in on that account. Matching the local UID to the Moira UID may be unnecessary as long as we do not make use of remote filesystems on this machine.
To set a reasonable path for development work, the user can edit .profile to add:
PATH=/usr/local/bin:/opt/csw/bin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin MANPATH=/usr/local/share/man:/opt/csw/share/man:/usr/share/man:/usr/sfw/share/man export PATH MANPATH
The following shell function, or a variant of it, may be useful for configuring a build with the correct paths. This variant is designed to work in a separate build directory placed next to the src directory within a checkout.
k5configure() { ../src/configure --enable-maintainer-mode --prefix=$HOME/inst --with-ldap CFLAGS=-g CPPFLAGS="-I/opt/csw/include" LDFLAGS="-L/opt/csw/lib -R/opt/csw/lib" "$@"; }
To do
- We should install the SunPro compiler and do automated builds with that compiler as well as gcc.
Hardware notes
The 8-pin modular serial connector for console/ALOM takes a shielded UTP cable. (The shielding probably isn't too important except for EMI reasons.) The modular to female DE-9 adapter is wired as a RS-232 DCE so it will plug directly into most USB to DE-9 RS-232 adapters. (The modular to male DB-25 adapter seems to be wired as a DTE and might need a null modem.)