logo_kerberos.gif

Difference between revisions of "Projects/Reporting-friendly KDB dump format"

From K5Wiki
Jump to: navigation, search
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{project-early}}
+
{{project-rel|1.14}}
{{project-target|1.14}}
 
   
This is split from [[Projects/KDB reporting and bulk operations]].
 
  +
This is split from [[Projects/KDB reporting and bulk operations]]. Release 1.14 contains the initial implementation described below. Further improvements, mostly consisting of additional output tables, are at [[Projects/Reporting-friendly KDB dump format improvements]].
   
 
==Background==
 
==Background==
Line 9: Line 9:
 
==Format choice==
 
==Format choice==
   
Tab-separated value formats are probably acceptable to the largest variety of tools. These tools range from simple awk scripts to SQL databases.
+
Tab-separated value formats are probably acceptable to the largest variety of tools. These tools range from simple awk scripts to SQL databases. Comma-separated values are easy to add. Various quoting and escaping options exist in various dialects of CSV format. Opinions vary on whether tab-separated formats can use quoting, but in any case, most of the fields will not require quoting if tab-separated.
   
There are several conceptual database tables, each of which will have a different set of columns. To allow combining these in a single dump file, each dump line will have a first column that indicates the conceptual table to which it belongs. Another option is to provide command line options to select an individual conceptual table to dump.
+
There are several conceptual database tables, each of which will have a different set of columns. To allow a single combined dump format, each dump line will have a first column that indicates the conceptual table to which it belongs. Another option is to provide command line options to select an individual conceptual table to dump, in which case the table name prefix would be omitted from each line. Headers should be optional, because some tools work better without them.
  +
  +
==Examples==
  +
  +
These are some examples of how commonly available tools could be used to manipulate output from one of these dumps.
  +
  +
<pre>
  +
$ cat keyinfo.txt
  +
name keyindex kvno enctype salttype salt
  +
foo@EXAMPLE.COM 0 1 aes128-cts-hmac-sha1-96 normal -1
  +
bar@EXAMPLE.COM 0 1 aes128-cts-hmac-sha1-96 normal -1
  +
bar@EXAMPLE.COM 1 1 des-cbc-crc normal -1
  +
$ sqlite3
  +
sqlite> .mode tabs
  +
sqlite> .import keyinfo.txt keyinfo
  +
sqlite> select * from keyinfo where enctype like 'des-cbc-%';
  +
bar@EXAMPLE.COM 1 1 des-cbc-crc normal -1
  +
sqlite> .quit
  +
$ awk -F'\t' '$4 ~ /des-cbc-/ { print }' keyinfo.txt
  +
bar@EXAMPLE.COM 1 1 des-cbc-crc normal -1
  +
</pre>
  +
  +
==Formatting options==
  +
  +
===Symbolic names===
  +
  +
Some values such as enctypes and salttypes can be printed numerically or using symbolic names. The default will be for symbolic names.
  +
  +
===Date/time===
  +
  +
Time stamps print as ISO 8601 UTC date and time. Numeric output is decimal POSIX time_t values.
  +
  +
Durations can print as decimal seconds. Later options could include ISO 8601 durations (although technically ISO 8601 duration representations are only supposed to designate intervals by context, and not pure durations).
   
 
==Conceptual tables==
 
==Conceptual tables==
   
This is an attempt to split KDB data fields into a somewhat more normalized schema that is somewhat easier to manipulate for reporting purposes.
+
This is an attempt to split KDB data fields into a somewhat more normalized schema that is somewhat easier to manipulate for reporting purposes. The rectype is the string used to request a specific conceptual table. Initially, only one table can print at a time. Eventually, there will be a combined output format where each line is prefixed with the rectype. Additional possibilities include outputting a directory with one file per table.
   
 
===Principal metadata===
 
===Principal metadata===
   
* Principal name
 
  +
;rectype: princ_meta
* Last modified by (principal name)
 
  +
;name: Principal name (krb5_kdb_entry)
* Last modification date
 
  +
;modby: Last modified by (principal name) (KRB5_TL_MOD_PRINC)
  +
;modtime: Last modification date (KRB5_TL_MOD_PRINC)
  +
;lastpwd: Last password change (KRB5_TL_LAST_PWD_CHANGE)
  +
;policy: Policy object name (osa_princ_ent_rec)
  +
;mkvno: Master key version used for this principal's key data (KRB5_TL_MKVNO)
  +
;hist_kvno: kadmin history principal kvno for this principal's key history (osa_princ_ent_rec)
   
 
===Principal keys===
 
===Principal keys===
Line 27: Line 32:
 
There will generally be multiple dump key data dump lines per principal. The order is significant (though typically it's only important within the active kvno), so there will need to be a key index number in case the user imports the dump into a data store that doesn't preserve the ordering of input lines (such as most relational databases).
 
There will generally be multiple dump key data dump lines per principal. The order is significant (though typically it's only important within the active kvno), so there will need to be a key index number in case the user imports the dump into a data store that doesn't preserve the ordering of input lines (such as most relational databases).
   
Some open questions include whether to use numeric values or string representations for enctype or salt type. Maybe this can be a runtime option?
 
  +
;rectype: keyinfo (or keydata)
 
  +
;name: Principal name (krb5_kdb_entry)
* Principal name
 
  +
;keyindex: Key index
* Key index
 
  +
;kvno: Key version number
* Key version number (kvno)
 
  +
;enctype: Enctype
* Enctype
 
  +
;key: hex string (only for keydata rectype)
* Salt type
+
;salttype: Salt type
* Salt data as hex string (might be "-1" to denote no salt or normal/default salt)
+
;salt: Salt data as hex string (might be "-1" to denote no salt or normal/default salt)
   
 
A sample implementation of a "keyinfo" dump format is at https://github.com/tlyu/krb5/tree/keyinfo
 
A sample implementation of a "keyinfo" dump format is at https://github.com/tlyu/krb5/tree/keyinfo
Line 40: Line 45:
 
===Per-principal policy data===
 
===Per-principal policy data===
   
* Principal name
 
  +
;rectype: princ_tktpolicy
* Principal expiration date
+
;name: Principal name
* Password expiration date
+
;expiration: Principal expiration date
* Max ticket lifetime
+
;pw_expiration: Password expiration date
* Max renewable ticket lifetime
+
;max_life: Max ticket lifetime
* Password policy name
+
;max_renew_life: Max renewable ticket lifetime
   
 
===Per-principal lockout data===
 
===Per-principal lockout data===
Line 51: Line 56:
 
These are the per-KDC (non-replicated) data that track failed logins due to incorrect passwords.
 
These are the per-KDC (non-replicated) data that track failed logins due to incorrect passwords.
   
* Principal name
 
  +
;rectype: princ_lockout
* Last success
 
  +
;name: Principal name
* Last failure
 
  +
;last_success: Last success
* Count of failed attempts
 
  +
;last_failed: Last failure
  +
;fail_count: Count of failed attempts
   
 
===Principal boolean attributes===
 
===Principal boolean attributes===
   
This is currently an boolean flag word; it's probably best to make it a set of strings. This is a bit tricky because there are some flags that are of the form disallow_*.
 
  +
;rectype: princ_flags
  +
;name: Principal name
  +
;flag: Attribute name (string form, or hexadecimal if numeric is requested)
  +
;value: Boolean value (1 or 0)
   
* Principal name
 
  +
Feedback from operators indicates that having a row for each attribute, regardless of whether or not it's set, can be useful to satisfy auditors.
* Attribute name if set (string form)
 
  +
  +
Unknown attributes print as hexadecimal numbers, if set. If the user requests numeric output, output all flag settings, including unset flags, as hexadecimal.
   
 
===Principal string attributes===
 
===Principal string attributes===
   
* Principal name
 
  +
;rectype: princ_stringattrs
* Attribute name
+
;name: Principal name
* Attribute value
+
;key: Attribute name
+
;value: Attribute value
===Password policy===
 
 
* Policy name
 
* Min password life
 
* Max password life
 
* Min password length
 
* Min password character classes
 
* Password history length
 
 
===Lockout policy===
 
 
* Policy name
 
* Max failures
 
* Failure count reset interval
 
* Lockout duration
 
 
===Ticket policy===
 
 
* Policy name
 
* Max ticket lifetime
 
* Max renewable ticket lifetime
 
 
===Policy boolean attributes===
 
 
As for principal boolean attributes
 
 
===Policy allowed keysalts===
 
 
(Is this an ordered list?)
 
 
* Policy name
 
* Enctype
 
* Salt type
 

Latest revision as of 16:11, 14 September 2015

This project was completed in release 1.14.


This is split from Projects/KDB reporting and bulk operations. Release 1.14 contains the initial implementation described below. Further improvements, mostly consisting of additional output tables, are at Projects/Reporting-friendly KDB dump format improvements.

Background

Operators often want to perform reporting operations on KDB data, but the dump format is optimized for loading by kdb5_util, not human reading or reporting using simple scripts. For example, the number of columns on each principal dump line depends on the number of keydata entries associated with that principal. Also, some useful metadata such as modification date are only present in a human-unfriendly hexadecimal format, as an artifact of being stored in the tl_data of the principal.

Format choice

Tab-separated value formats are probably acceptable to the largest variety of tools. These tools range from simple awk scripts to SQL databases. Comma-separated values are easy to add. Various quoting and escaping options exist in various dialects of CSV format. Opinions vary on whether tab-separated formats can use quoting, but in any case, most of the fields will not require quoting if tab-separated.

There are several conceptual database tables, each of which will have a different set of columns. To allow a single combined dump format, each dump line will have a first column that indicates the conceptual table to which it belongs. Another option is to provide command line options to select an individual conceptual table to dump, in which case the table name prefix would be omitted from each line. Headers should be optional, because some tools work better without them.

Examples

These are some examples of how commonly available tools could be used to manipulate output from one of these dumps.

$ cat keyinfo.txt
name	keyindex	kvno	enctype	salttype	salt
foo@EXAMPLE.COM	0	1	aes128-cts-hmac-sha1-96	normal	-1
bar@EXAMPLE.COM	0	1	aes128-cts-hmac-sha1-96	normal	-1
bar@EXAMPLE.COM	1	1	des-cbc-crc	normal	-1
$ sqlite3
sqlite> .mode tabs
sqlite> .import keyinfo.txt keyinfo
sqlite> select * from keyinfo where enctype like 'des-cbc-%';
bar@EXAMPLE.COM	1	1	des-cbc-crc	normal	-1
sqlite> .quit
$ awk -F'\t' '$4 ~ /des-cbc-/ { print }' keyinfo.txt
bar@EXAMPLE.COM	1	1	des-cbc-crc	normal	-1

Formatting options

Symbolic names

Some values such as enctypes and salttypes can be printed numerically or using symbolic names. The default will be for symbolic names.

Date/time

Time stamps print as ISO 8601 UTC date and time. Numeric output is decimal POSIX time_t values.

Durations can print as decimal seconds. Later options could include ISO 8601 durations (although technically ISO 8601 duration representations are only supposed to designate intervals by context, and not pure durations).

Conceptual tables

This is an attempt to split KDB data fields into a somewhat more normalized schema that is somewhat easier to manipulate for reporting purposes. The rectype is the string used to request a specific conceptual table. Initially, only one table can print at a time. Eventually, there will be a combined output format where each line is prefixed with the rectype. Additional possibilities include outputting a directory with one file per table.

Principal metadata

rectype
princ_meta
name
Principal name (krb5_kdb_entry)
modby
Last modified by (principal name) (KRB5_TL_MOD_PRINC)
modtime
Last modification date (KRB5_TL_MOD_PRINC)
lastpwd
Last password change (KRB5_TL_LAST_PWD_CHANGE)
policy
Policy object name (osa_princ_ent_rec)
mkvno
Master key version used for this principal's key data (KRB5_TL_MKVNO)
hist_kvno
kadmin history principal kvno for this principal's key history (osa_princ_ent_rec)

Principal keys

There will generally be multiple dump key data dump lines per principal. The order is significant (though typically it's only important within the active kvno), so there will need to be a key index number in case the user imports the dump into a data store that doesn't preserve the ordering of input lines (such as most relational databases).

rectype
keyinfo (or keydata)
name
Principal name (krb5_kdb_entry)
keyindex
Key index
kvno
Key version number
enctype
Enctype
key
hex string (only for keydata rectype)
salttype
Salt type
salt
Salt data as hex string (might be "-1" to denote no salt or normal/default salt)

A sample implementation of a "keyinfo" dump format is at https://github.com/tlyu/krb5/tree/keyinfo

Per-principal policy data

rectype
princ_tktpolicy
name
Principal name
expiration
Principal expiration date
pw_expiration
Password expiration date
max_life
Max ticket lifetime
max_renew_life
Max renewable ticket lifetime

Per-principal lockout data

These are the per-KDC (non-replicated) data that track failed logins due to incorrect passwords.

rectype
princ_lockout
name
Principal name
last_success
Last success
last_failed
Last failure
fail_count
Count of failed attempts

Principal boolean attributes

rectype
princ_flags
name
Principal name
flag
Attribute name (string form, or hexadecimal if numeric is requested)
value
Boolean value (1 or 0)

Feedback from operators indicates that having a row for each attribute, regardless of whether or not it's set, can be useful to satisfy auditors.

Unknown attributes print as hexadecimal numbers, if set. If the user requests numeric output, output all flag settings, including unset flags, as hexadecimal.

Principal string attributes

rectype
princ_stringattrs
name
Principal name
key
Attribute name
value
Attribute value