Projects/Hierarchical iprop
Contents
Description
This project adds the ability for incremental propagation slaves to act as masters for other slaves, forming a hierarchy of masters and slaves. This feature can be useful for controlling propagation load in environments with many slaves, or to address incomplete network connectivity between masters and slaves. The project is based on an existing implementation by Richard Basch.
Existing Design
Since release 1.7, MIT krb5 has had support for incremental propagation for changes to principal information. The slave runs a kpropd daemon in standalone mode, which periodically polls for updates from kadmind on the master, using a GSSRPC-based protocol. Updates are normally transmitted to the slave as marshalled operations in the response to the polling RPC. If the slave drifts too far out of date or if there have been changes to policies on the master, then a full dump is performed from the master to the slave using kprop.
The master and slave store the state needed for incremental propagation in a memory-mapped file called the ulog, which contains a header and a circular array of fixed-size update entries. Each update entry has a timestamp and a serial number; serial numbers begin at 1 and increment for each update. The ulog header contains:
- kdb_hmagic: A magic number (to detect corruption)
- db_version_num: A version number (currently set to 1 and unused by readers)
- kdb_num: The number of updates in the ulog
- kdb_first_time: The timestamp of the ulog's first entry
- kdb_last_time: The timestamp of the ulog's last entry
- kdb_first_sno: The timestamp of the ulog's first entry
- kdb_last_sno: The timestamp of the ulog's last entry
- kdb_state: The stability state of the ulog
- kdb_block: The entry size (normally 2048)
The size of the circular entry array is determined by the iprop_master_ulogsize krb5.conf variable, which is not recorded in the header. The kdb_num, kdb_first_sno, and kdb_last_sno fields are redundant; this redundancy can be used to detect if the ulogsize parameter changed since the last time the ulog was updated.
After the ulog is initialized or reinitialized (with kproplog -R), the kdb_num, kdb_first_time, kdb_first_sno, and kdb_last_sno fields are set to 0, and the kdb_last_time field is set to the time of initialization. The first database update after ulog initialization increments the kdb_num, kdb_first_sno, and kdb_last_sno fields to 1, and sets the kdb_first_time and kdb_last_time fields to the time of the update.
XXX enumerate uses of each header field on masters
Slaves do not use the update entry array and only make use of two header fields, kdb_last_sno and kdb_last_time, to track the most recently received update from the master. XXX enumerate which pieces of code use and set these fields on slaves
Proposed changes
Three changes are needed to allow hierarchical iprop:
1. On slaves, maintain a complete ulog. When update entries are received from the upstream master and processed by ulog_replay(), add them to the ulog in addition to modifying the KDB.
2. Add a -proponly option to kadmind so that it can be run on slaves as an iprop server.
3. Add a "-A server" option to kpropd to make it talk to a specified upstream master rather than the krb5.conf value of admin_server.
The ulog on a slave can be in states that are impossible on the master:
- After receiving a full dump from upstream, the kdb_num, kdb_first_time, and kdb_first_sno fields are 0, but the kdb_last_sno and kdb_last_time fields reflect the most recent update from the upstream master when the dump file was created.
- After a ulog reset or full dump, subsequent updates reflect the serial numbers of the updates received from upstream, instead of starting at 1.
XXX what pieces of code need to change as a result? Partial list:
- Detection of ulogsize change in ulog_map
XXX examine corner cases associated with full resyncs, upstream and downstream
