FreeIPA is pretty cool but it is a complex beast with a lot of moving parts. Its documentation is alright but there are many things that were (as least to me) not obvious about it.
Contents
Directory suffix
The examples below assume a FreeIPA domain of ipa.example.com. When naming directory entries, replace SUFFIX, with dc=ipa,dc=example,dc=com.
If you use the integrated CA feature, then Dogtag's state will be stored in the directory at another suffix: o=ipaca. You mostly don't have to worry about that detail except when configuring and monitoring replication status.
Connecting to the directory via UNIX sockets
Rather than having to keep the cn=Directory Manager password handy, you can connect directly to the LDAP server via a UNIX socket by using the -H ldapi://%2frun%2fslapd-IPA-EXAMPLE-COM.socket -Y EXTERNAL options to ldapsearch et. al.
Root DSE attributes
As with all LDAP directories, some interesting details can be queried anonymously by performing a search for the root entry:
$ ldapsearch -LLL -x -H ldaps://ipa0.ipa.exampe.com -s base -b '' vendorname vendorversion netscapemdsuffix namingcontexts defaultnamingcontext dn: vendorname: 389 Project vendorversion: 389-Directory/1.4.0.2010 B2019.175.2029 netscapemdsuffix: cn=ldap://dc=ipa0,dc=ipa,dc=example,dc=com namingcontexts: cn=changelog namingcontexts: dc=ipa,dc=example,dc=com namingcontexts: o=ipaca defaultnamingcontext: dc=ipa,dc=example,dc=com
See Root DSE Attributes for an explanation of what they mean.
Replication status monitoring
dsconf URL replication monitor: a port of repl-monitor.pl to Python. Added in as part of #50545 which made it into 389-ds 1.4.2.2.
Prior to that, we can run dsconf -j URL replication list and then for each suffix run dsconf -j URL replication status --suffix=SUFFIX. This outputs one line of JSON per replication agreement (to stderr, WTF?)
389-ds 1.4.1.4 adds replicaLastUpdateStatusJSON and replicaLastInitStatusJSON to the nsds5replicationagreemet object: #49602. The JSON includes helpful fields such as state which is green, amber, red and is ideal for alerting without having to interpret nsds5replicaLastUpdateStatus ourselves.
.. although it's not that hard: match ^Error (\d+) and alert if non-zero.
The replication-status and replication-lag-time won't be meaningful unless dsconf can connect to the remote consumer server for each agreement.
If using ldap/ldaps and a simple bind to connect to the local server, these credentials will be re-used. But FreeIPA configures its agreements with nsDS5ReplicaTransportInfo set to LDAP, so dsconf won't secure its connections to the consumer with TLS, so the credentials will be exposed to an insecure network!
If using ldapi, dsconf will attempt an anonymous connection to the consumer unless --bind-dn and --bind-pass are also specified. So instead we can check if the replica is up-to-date by comparing the Changelog Sequence Numbers (CSN) from the Replica Update Vectors (RUV)s as described below.
dsconf URL repl-agmt status --suffix=SUFFIX AGREEMENT-NAME is similar but runs for a particular replication agreement instead of each one.
dsconf URL replication get --suffix=SUFFIX just dumps the nds5replica object at cn=replica,SUFFIX,cn=mapping tree,cn=config object, in LDIF format (or JSON with -j).
dsconf URL repl-agmt list --suffix=SUFFIX just dumps the nsds5replicationagreement objects underneath the nds5replica object for SUFFIX, in LDIF format (or JSON with -j).
dsconf URL healthcheck presumably exits non-zero if there's a problem with the server.
Checking RUVs manually
For each replicated suffix, on the supplier get the replica id and the maximum CSN of each replication agreement:
ldapsearch -b SUFFIX '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' nsDS5ReplicaId nsds5agmtmaxcsn
There will be one attribute per agreement in the format:
SUFFIX;NAME;SERVER;PORT;ID;MAXCSN
For each of these agreements, *on the consumer*:
ldapsearch -b SUFFIX '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' nsds50ruv
There will be one attribute per agreement in the format:
{replica ID URL} MINCSN MAXCSN
Match the suppler and consumer replica agreement IDs up and check that the max CSNs are the same.
If they are different, calculate the lag time by extracting the timestamps from the max CSNs (first 8 digits), parsing the hexadecimal number, taking the difference as the number of seconds.
The Red Hat Directory Server manual has all the info about how replication works.
Monitoring the Replication Status explains how to use dsconf to do the same as the above.
Forcing Replication Updates explains what to do to ensure a server is up-to-date after it goes down and is brougt back up, having missed updates while it was down. Interestingly this does not mention dsconf URL repl-agmt poke --suffix=SUFFIX REPL-AGMT...
Exposing to the Internet
- Create a CA certificate offline, and use it to sign FreeIPA's own CA certificate
- I'm not sure how to handle revocation of FreeIPA's certificate without creating another set of infrastructure for OCSP and CRL publishing, and even then, are all TLS clients actaully able to make use of them? Certainly not...
Use a firewall and only expose the correct ports (the freeipa-ldap{,s} services in firewalld are outdated, newer firewalld has a freeipa-4 service which is better to use; and don't bother with freeipa-replication as it hasn't been used since FreeIPA 4).
- Delegate a zone to your IPA servers properly, so that clients find them via the public DNS.
- Configure the IPA servers to use your operator's DNS servers rather than the BIND running locally.
Has the side effect of keeping DNS working when named-pkcs11.service is stopped.
The jury's out on whether the local server and/or other servers should have entries hardcoded in /etc/hosts. It might make things easier if named-pkcs11.service on all servers went down; on the other hand it might also mask problems when things are running smoothly.
Set recursion: no in BIND's options section.
Change FreeIPA's default for allow-recursion to none
Configure TLS version min/max for httpd and dirsrv to TLSv1.2/TLSv1.3
RHEL8 uses system-wide crypto-policies settings that might make this un-necessary for httpd, but I don't think dirsrv (which still uses NSS rather than OpenSSL) obeys them yet
Configure sensible TLS ciphers for httpd and nss.
- Originally I also did this for Dogtag's tomcat, but this was before I realised that this is only accessed from the local host so isn't necessary.
Disable access to dirsrv's root user by enabling the rootdn_access plugin and configuring a whitelist of IP addresses able to use it (only ::1/127.0.0.1 is a good start).
repl-monitor.pl won't like this. It can't use TLS or GSSAPI to protect the transport so we definitely don't want it running over the internet. SSH tunnel?
ds-replcheck can use TLS, so the whitelist approach will work.
But let's just use SSH tunnelling which will work for both (as long as the source address on the server for forwarded connections is ::1/127.0.0.1.
Disable the default admin user.
Create a new admin user with a different name and put it in the admins group.
Make sure your HBAC and Sudo rules refer to the admins group rather than the admin user.
Log in as the new admin user, and then disable the original admin user.
- It would be neat to dynamically adjust firewall rules to include the current address of each host in a particular group. But that relies on Kerberos being up and DNS updates working correctly (i.e., not being intercepted and blocked by Sky).
- Then again there's always the operator console for troubleshooting.
- Just disable password authentication entirely and rely on GSSAPI or public key authentication... the basic SSH stuff really!
Additional options for dirsrv:
nsslapd-minssf: requires use of a protected transport.
Also known to break realmd but who uses that to join FreeIPA?
nsslapd-allow-anonymous-access to prevent reading the directory without authentication.
nsslapd-require-secure-binds: require a protected transport for authentication requests
- Can't stop a misconfigured client from blurting out a password via a simple bind on the non-TLS port however!
Also has the effect of breaking repl-monitor.pl, which is denied from doing a simple bind to localhost - the server isn't able to waive nsslapd-require-secure-binds (and presumably nsslapd-minssf for connections from ::1/127.0.0.0/8.
Certificate storage locations
Trust Store
The list of known CA certificates that the FreeIPA installation trusts is kept in the directory under cn=certificates,cn=ipa,cn=etc,SUFFIX. For a stand-alone CA installation, the store will normally only contain the CA's certificate (IPA.EXAMPLE.COM IPA CA). But for an externally signed CA installation, it will contain the external CA certificate as well. This bug shows examples of the schema.
The ipa-cacert-manage list command will perform an LDAP search, and print an entry for each certificate it finds.
Each entry can have more than one certificate, since the cACertificate and ipaCertIssuerSerial fields are multi-valued. (Not sure whether the order is guaranteed to be preserved, so what happens if the multi-values of these two attributes don't match?)
New certificates can be installed with ipa-cacert-manage install. There's no command to remove them.
The ipa-certupdate can be run on a client to make sure any new or renewed certificates added to the trust store are applied to the local machine. On a server it will additionally ensure that new or renewed certificates added to the trust store are installed into the the KDC, web and directory server certificate databases (or is this done instead by ipa-server-certinstall?)
![/!\](/moin_static199/modernized/img/alert.png "/!\"){kind=small}
If you have more than one certificate in the trust store, this bug means that OpenSSL on Debian systems won't trust *any* of them.
Legacy CAcert container
The entry cn=CAcert,cn=ipa,cn=etc,SUFFIX contains the IPA CA certificate. It's used by legacy clients (RHEL6) that can only trust a single CA.
When certmonger starts up, it uses this entry to refresh its cached copy of the IPA CA certificate (ref: fetch_roots function). This in turn is written out when using ipa-cercert ... -F chain.crt.
IPA CAs
cn=cas,cn=ca,SUFFIX contains the IPA CA certificates themselves (plural because IPA supports multiple 'lightweight sub-CAs'.
$ ipa ca-find --raw --pkey-only --all ------------ 1 CA matched ------------ dn: cn=ipa,cn=cas,cn=ca,dc=ipa,dc=example,dc=com cn: ipa ---------------------------- Number of entries returned 1 ----------------------------
Host Aliases
Say you have a host, foo.ipa.example.com, with an IP address of 192.0.2.0.
The host is behind a NAT gateway with an address on the Internet of 203.0.113.0.
If you enable dyndns_update, a foo.ipa.example.com will point at 192.0.2.0. Although this leaks information about foo's local network to the public, at least users on foo's network will be able to reach it using that FQDN.
But what if the NAT gateway is provided by Sky Broadband? Sky's routers intercept DNS traffic, preventing their customers from using third-party DNS resolvers, doubtless so that Sky can sell information about the web sites their customers visit to advertisers. As a side effect of this, some more exotic forms of DNS traffic are blocked. This includes the GSS-TSIG messages from nsupdate, which is the mechanism that sssd uses to perform DNS updates. (Debugging this was not fun.)
One alternative is for users on foo's network to make use of mDNS and reach it at foo.local. In order to make sure that they can use Kerberos to log in, we must make use of FreeIPA's Kerberos Principal Alias feature:
$ ipa host-add-principal foo.ipa.example.com host/foo.local ------------------------------------------------------ Added new aliases to host "foo.ipa.example.com" ------------------------------------------------------ Host name: foo.ipa.example.com Principal alias: host/foo.ipa.example.com@IPA.EXAMPLE.COM, host/foo.local@IPA.EXAMPLE.COM
Users can now obtain a ticket for host/foo.local and use it to authenticate to the SSH server running on foo.
However, users will still be prompted to confirm foo's SSH host key; this is because sss_ssh_knownhostsproxy requests the host keys for foo.local, and SSSD's SSH responder isn't able to search the directory for hosts by alias.
CRL Generation Master promotion on CentOS 8
The Starting CRL generation on RHEL 8 chapter of the "Migrating IdM from RHEL 7 to RHEL 8 and keeping it up-to-date" section of the RHEL 8 Installing IdM manual doesn't make much sense. The new replica is running RHEL 8, so why do the prerequisites talk about RHEL 7.6/7.7?
I guess the text was sloppily copied from the previous chapter: https://bugzilla.redhat.com/show_bug.cgi?id=1785595
After following these steps on a CentOS 8.0.1905 machine, the CRL is not available:
$ curl -s -I http://ipa2.ipa.robots.org.uk/ipa/crl/MasterCRL.bin | head -n1 HTTP/1.1 404 Not Found
The manual mentions a ipa-crlgen-manage command, which does not exist in CentOS 8, but it does exist in CentOS 7:
centos8$ ipa-crlgen-manage --version -bash: ipa-crlgen-manage: command not found $ rpm -q centos-release ipa-server centos-release-8.0-0.1905.0.9.el8.x86_64 ipa-server-4.7.1-11.module_el8.0.0+79+bbd20d7b.x86_64
$ ipa-crlgen-manage --version 4.6.5 $ rpm -q centos-release ipa-server centos-release-7-7.1908.0.el7.centos.x86_64 ipa-server-4.6.5-11.el7.centos.3.x86_64
Examining the FreeIPA source reveals that this script was added in FreeIPA 4.8:
$ git checkout master $ git log --oneline ./install/tools/ipa-crlgen-manage.in 6d02eddd3 Replace PYTHONSHEBANG with valid shebang 0d23fa927 CRL generation master: new utility to enable|disable $ git describe --contains 0d23fa927 rc_4-8-0-1~126
And backported to FreeIPA 4.7.3 and 4.6.5:
$ git checkout ipa-4-7 $ git log --oneline ./install/tools/ipa-crlgen-manage.in 452fe2fdc Replace PYTHONSHEBANG with valid shebang 52770aa5f CRL generation master: new utility to enable|disable $ git describe --contains 52770aa5f release-4-7-3~202
$ git checkout ipa-4-6 $ git log --oneline ./install/tools/man/ipa-crlgen-manage.1 af5abe0d7 CRL generation master: new utility to enable|disable $ git describe --contains af5abe0d7 release-4-6-5~4
So I guess it will show up in CentOS 8.1. But until then, examining the source code reveals that, after manually enabling CRL generation, there is an additional undocumented step:
1 # make sure a CRL is generated if setup_crl is True 2 if setup_crlgen: 3 logger.info("Forcing CRL update") 4 api.Backend.ra.override_port = 8443 5 result = api.Backend.ra.updateCRL(wait='true') 6 if result.get('crlUpdate', 'Failure') == 'Success': 7 logger.debug("Successfully updated CRL") 8 api.Backend.ra.override_port = None
Unfortunately the updateCRL method was likewise only added in FreeIPA 4.7.3. Fortunately it's not too complex to prevent me from bodging together the following:
1 from pprint import pprint 2 3 from ipalib import api 4 from ipaplatform.paths import paths 5 6 api.bootstrap(in_server=True, confdir=paths.ETC_IPA) 7 api.finalize() 8 9 api.Backend.ldap2.connect() 10 11 pprint(api.Backend.ra._sslget('/ca/agent/ca/updateCRL', 12 8443, 13 crlIssuingPoint='MasterCRL', 14 waitForUpdate=True, 15 xml='true'))
# python3 force.py (200, <http.client.HTTPMessage object at 0x7f8ddd4e9a58>, b'<?xml version="1.0" encoding="UTF-8" standalone="no"?><xml><header><crlIssui' b'ngPoint>MasterCRL</crlIssuingPoint><crlUpdate>Scheduled</crlUpdate></header>' b'<fixed/><records/></xml>')
And behold, we now have a CRL on the new master:
$ curl -s -I http://ipa2.ipa.robots.org.uk/ipa/crl/MasterCRL.bin | head -n1 HTTP/1.1 200 OK
Remaining questions
There are still seemingly significant differences in Tomcat's configuration between the old and new masters:
property
ipa0
ipa2
ca.certStatusUpdateInterval
unset
0
ca.listenToCloneModifications
true
false
master.ca.agent.host
unset
ipa0.ipa.robots.org.uk
master.ca.agent.port
unset
443
These settings are mentioned in older documentation and on a post to the mailing list:
https://docs.fedoraproject.org/en-US/Fedora/15/html/FreeIPA_Guide/promoting-replica.html
https://www.redhat.com/archives/freeipa-users/2012-May/msg00389.html
So I'm still not sure whether my new master CRL generator will actually listen to updates from replicas and/or periodically check the status of certificates for updates. I also wonder whether master.ca.agent.host has to be updated on ipa1...
Extending FreeIPA
Extending FreeIPA (also found in PDF form attached to https://www.redhat.com/archives/freeipa-users/2012-February/msg00230.html)
https://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf
Custom subtrees should go in an nsContainer directly under SUFFIX.
DNS: long TXT records
DMARC requires rather long TXT records. Attempting to create one of these in the web UI or on the command line works, but then named-pkcs11 logs:
failed to parse RR entry: resource record DN 'idnsname=0._domainkey,idnsname=example.com.,cn=dns,dc=ipa,dc=example,dc=com': data 'v=DKIM1; t=s; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApIoCiX7nfTcACGVVNxUbxIrhtYrBCDeRFVxGwAbkraMI+6qtVicNu1XB6i26TcVz+azCGl6oZALS32oJJGrcdJlqmHb9egzwUMs9cgksiaLI5D54fT7rWi9JEgsu59PJE9uFPQ2PBnWeDpblQc5q1D0G15/udTKp06QV17qkCemdvLoNMYh+BtPT4cn1AqDvXKHPSt/H5Lrr9ZpNNbw5hhIHAho7zJRUsL4wTImL5WFahX4HUc3XDmTrdw9CJbSU4gZRODfh0G7g28MxybfiltxNWDYPelgxsPFJFifo6vrXyUZQbQwyqvCvpyJbJZ66u2EOtYQ6VCUH6l9CJdkADQIDAQAB': syntax error
The reason for this is not obvious.
So, RFC 1035 describes the data of a TXT RR as simply "One or more <character-string>s". And describes <character-string> as "a single length octet followed by that number of characters. <character-string> is treated as binary information, and can be up to 256 characters in length (including the length octet)."
So our everyday experience of a TXT record is wrong! We naïvely think of it as "just a string", when really it has structure; in this case, one or more arrays of bytes each of which can be up to 255 characters. BIND is choking because it's trying to compile a TXT record consisting of a single <character-string> of well over 300 bytes!
The RFC also defines how to represent a <character-string> in a Zone file, as parsed by BIND:
- Entries are predominantly line-oriented, though parentheses can be used to continue a list of items across a line boundary
and
<character-string> is expressed in one or two ways: as a contiguous set of characters without interior spaces, or as a string beginning with a " and ending with a ". Inside a " delimited string any character can occur, except for a " itself, which must be quoted using \ (back slash).
So it turns out that zone file format is more of a literal transcription of DNS data into a binary format than I had expected. So where you express the following in a zone file:
0._domainkey IN TXT ( "v=DKIM1;" "t=s;" "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq1cxaCim2Tx+HfuB4oGG" "IpQaynGyJV0vi0uQZPBE7/uCVL7/+yJKll+Ec07RHkUg4j3tMltpeuSJd6Hddkj8" "OPDtFYI3EyqexUQe04NsTB9gNAmA1ag4vYb3YCCA2KsbAipIDchkUL31I1XYW8cL" "U/4R2aWYxEJNnPkt/Y0ljy5oKx5HLUs46Miqvz4BvQPKy6MvTEci7yJ5pLWcYGzd" "PjGRZuuBcQGPXB26dVZ7LTEHMrhiJ2uQ7ukoFw//nDU8weypvymSX9AVIWmq9ZXr" "CORBeCspOCupO5jAyGNJjBvs8IoVzis/iICqCgFv7XNgGTEs8nC5L9QXvX/lrSI6" "HwIDAQAB" )
... the compiled RRDATA for the record consists of *nine* length-prefixed byte strings. It's only RFC 7489, sec. 6.1 that determines that they are stitched together into a single string by the entity making use of the records:
- a TXT record can comprise several "character-string" objects. Where this is the case, the module performing DMARC evaluation MUST concatenate these strings by joining together the objects in order and parsing the result as a single string.
Stitching this knowledge together we can now create the record in the web UI or the ipa command, simply by adding spaces to the record data so that no one run of characters lasts for more than 255 bytes. For example:
$ ipa dnsrecord-mod example.com 0._domainkey --txt-rec='"v=DKIM1; t=s; p=" "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApIoCiX7nfTcACGVVNxUb" "xIrhtYrBCDeRFVxGwAbkraMI+6qtVicNu1XB6i26TcVz+azCGl6oZALS32oJJGrc" "dJlqmHb9egzwUMs9cgksiaLI5D54fT7rWi9JEgsu59PJE9uFPQ2PBnWeDpblQc5q" "1D0G15/udTKp06QV17qkCemdvLoNMYh+BtPT4cn1AqDvXKHPSt/H5Lrr9ZpNNbw5" "hhIHAho7zJRUsL4wTImL5WFahX4HUc3XDmTrdw9CJbSU4gZRODfh0G7g28Mxybfi" "ltxNWDYPelgxsPFJFifo6vrXyUZQbQwyqvCvpyJbJZ66u2EOtYQ6VCUH6l9CJdkA" "DQIDAQAB"' Record name: 0._domainkey TXT record: "v=DKIM1; t=s; p=" "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApIoCiX7nfTcACGVVNxUb" "xIrhtYrBCDeRFVxGwAbkraMI+6qtVicNu1XB6i26TcVz+azCGl6oZALS32oJJGrc" "dJlqmHb9egzwUMs9cgksiaLI5D54fT7rWi9JEgsu59PJE9uFPQ2PBnWeDpblQc5q" "1D0G15/udTKp06QV17qkCemdvLoNMYh+BtPT4cn1AqDvXKHPSt/H5Lrr9ZpNNbw5" "hhIHAho7zJRUsL4wTImL5WFahX4HUc3XDmTrdw9CJbSU4gZRODfh0G7g28Mxybfi" "ltxNWDYPelgxsPFJFifo6vrXyUZQbQwyqvCvpyJbJZ66u2EOtYQ6VCUH6l9CJdkA" "DQIDAQAB"
While figuring out the above I ran into RFC 1464, which proposes a way to store structured data within TXT records, by having each <character-string> within the RDATA for a TXT record represent an attribute and value pair, separated by =. e.g.:
host.widgets.com IN TXT "printer=lpr5" sam.widgets.com IN TXT "favorite drink=orange juice"
It does not state, although I presume, that you can have multiple attributes by using multiple <character-string>s within the RDATA:
record.example.com IN TXT "foo=bar" "baz=qux"
... up to the maximum length of the RRDATA for a record which is 65536 bytes. I guess that one of the reasons this never took off is that the length of each value would be limited to 254 - (length of attribute) bytes.