FreeIPA is pretty cool but it is a complex beast with a lot of moving parts. Its documentation is alright but there are many things that were (as least to me) not obvious about it.

Contents

  1. Official Documentation
    1. Documentation for components
  2. Directory suffix
  3. Connecting to the directory via UNIX sockets
  4. Root DSE attributes
  5. Password storage
  6. Log files
    1. Directory server
    2. Kerberos
    3. Web server
    4. PKI server
      1. Tomcat logs
      2. Tomcat access logs
      3. Subsystem logs
      4. Subsystem debug logs
    5. Name server logs
    6. Custodia
    7. OTP
  7. Replication status monitoring
    1. Adding a dedicated system account for replication status monitoring
  8. Exposing to the Internet
  9. Certificate storage locations
    1. Trust Store
    2. Legacy CAcert container
    3. IPA CAs
  10. Host Aliases
  11. CA renewal server/CRL publisher promotion
    1. Remaining questions
  12. PKI topology goes out of sync with LDAP server topology
  13. Extending FreeIPA
  14. DNS: long TXT records

Official Documentation

The best place to start when looking for reference documentation is Red Hat's Red Hat Identity Management Documenatation index; then the FreeIPA workshop.

The FreeIPA wiki has a documentation index, although many pages are out of date. Google will often take you here, or to various other outdated sources of information, so always check the above documentation first!

Documentation for components

FreeIPA configures its underlying components in an opinionated way. Keep their documentation to hand as well:

Directory suffix

The examples below assume a FreeIPA domain of ipa.example.com. When naming directory entries, replace SUFFIX, with dc=ipa,dc=example,dc=com.

If you use the integrated CA feature, then Dogtag's state will be stored in the directory at another suffix: o=ipaca. You mostly don't have to worry about that detail except when configuring and monitoring replication status.

Connecting to the directory via UNIX sockets

To administer the directory server, you have to use Simple Authentication, specifying cn=Directory Manager as the password. Having to keep the password handy is a bit annoying.

There is an alternative: a process connecting via a UNIX socket can use SASL EXTERNAL authentication in order to be identified by their UID/GID. On the command line: 

# ldapwhoami -H ldapi://%2frun%2fslapd-IPA-EXAMPLE-COM.socket -Y EXTERNAL
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
dn: cn=Directory Manager

Note that we weren't prompted for a password--the root user is mapped to cn=Directory Manager by default.

This can be made the default for the root user by putting the following in /root/.ldaprc: 

URI ldapi://%2frun%2fslapd-IPA-EXAMPLE-COM.socket
SASL_MECH EXTERNAL

Root DSE attributes

As with all LDAP directories, some interesting details can be queried anonymously by performing a search for the root entry:

See Root DSE Attributes for an explanation of what they mean.

Password storage

The cn=Directory Manager password is stored on the nsslapd-rootpw attribute of cn=config.

Password hashes are stored in the userPassword attribute.

The default hash format is (now) {PBKDF2_SHA256}; it used to be the (weak) {SSHA512}.

The AllowNThash password plugin policy is enabled by default. If ipa-adtrust-install has been run, then the NT hash (unsalted MD4, gotta love that security) will be written to the sambaNTPassword and/or ipaNTHash attributes during password change.

Kerberos keys are stored in the krbPrincipalKey attribute.

Log files

Directory server

Under /var/log/dirsrv/slapd-IPA-EXAMPLE-COM you'll find access, audit and errors.

Audit logging is not enabled by default. Enable it with:

Kerberos

KDC logs are found at /var/log/krb5kdc.log

kadmin (password change) are at /var/log/kadmind.log.

Web server

The FreeIPA web console and also API logs are found in /var/log/httpd.

PKI server

Has a lot of log files some of which are not purged by default, so they can grow very large if not removed manually.

Tomcat logs

These are:

They're documented in pki-server-logging(5).

They are rotated daily but old log files are not purged by default.

To enable purging, replace the /etc/pki/pki-tomcat/logging.properties symlink with a copy of the target file (see the man page for details) and then set:

Tomcat access logs

This is found at /var/log/pki/pki-tomcat/localhost_access_log.*.txt. It's rotated daily old log files are not purged by default.

To enable purging, edit /etc/pki/pki-tomcat/server.xml and find the Valve element for org.apache.catalina.valves.AccessLogValve. Add an attribute maxDays="7".

Subsystem logs

These are /var/log/pki/pki-tomcat/*/{selftests.log,system,transactions,signedAudit/*_audit}.

They are rotated after 30 days or after they grow to 2000 KiB. This can be figured in the subsystem's corresponding CS.cfg file; however this file only exists for the ca and kra subsystems; there's no corresponding file for acme and pki. Old log files are not purged automtaically.

There is an expirationDate property for these log files in CS.cfg but a comment in RollingLogFile.java says that it is not supported.

Subsystem debug logs

These are /var/log/pki/pki-tomcat/*/debug.*.log.

They're documented at Configuring Subsystem Debug Log.

They're rotated daily, but old log files are not purged by default (even through the documentation says otherwise).

Name server logs

These are found in /var/named/data/*.log. They are rotated automatically and old log files are purged, however not very often.

Custodia

I think this is a component that handles requests from the API server for vault-related operations. It in turn talks to the PKI KRA subsystem.

Audit log is found at /var/log/ipa-custodia.audit.log. Doesn't seem to produce operational logs to a file nor to the journal.

OTP

Logs to the journal. Because this is a per-connection socket-activated service, the easiest way to filter for its messages is journalctl -t ipa-otpd.

Replication status monitoring

{i} There's a general purpose dsctl IPA-EXAMPLE-COM healthcheck command. This outputs a list of problems detected with the directory server. It doesn't seem to be called by ipa-healthcheck, maybe it's a relatively new command. I don't know if it checks for problems with replication status.

Monitoring the Replication Topology in the RHDS documentation leads us to:

Comparing two Directory Server instances describes the ds-replcheck command, which can be use to check if two servers are in-sync. It can also compare replicated suffixes and output any differences found.

Here's an alternative that you can run on each server to check its view of each of its replication agreements:

On RHEL 7, there's no dsconf command, nor replicaLastUpdateStatusJSON attribute; so here's a pure-ldapsearch equivalent instead:

All replication agreements can be 'poked' to force them to send data with:

You can view details for a single replication agreement with:

Some low-level details of replication agreements can be viewed with:

Those are pretty technical. Probably if you need to interpret this information you're best off asking for help on the mailing lists.

Two other parts of the RHDS documentation are worth pointing out:

But generally the entire Managing Replication chapter is worth a read.

In any case, check the directory server error log to investigate the cause of replication errors.

Adding a dedicated system account for replication status monitoring

This is an account that exists only in the directory. It's not a FreeIPA user, it can't log in to any systems, etc; it can only bind to the directory and query.

I'm using ldapvi instead of ldapadd/ldapmodify; this assumes that /root/.ldaprc has been created as described above.

Test binding as the user:

Grant the user permission to view replication agreements:

Fetch replication info from a server with the system account:

Now use ds-replcheck with the system account:

Exposing to the Internet

Certificate storage locations

Trust Store

The list of known CA certificates that the FreeIPA installation trusts is kept in the directory under cn=certificates,cn=ipa,cn=etc,SUFFIX. For a stand-alone CA installation, the store will normally only contain the CA's certificate (IPA.EXAMPLE.COM IPA CA). But for an externally signed CA installation, it will contain the external CA certificate as well. This bug shows examples of the schema.

The ipa-cacert-manage list command will perform an LDAP search, and print an entry for each certificate it finds.

New certificates can be installed with ipa-cacert-manage install. There's no command to remove them.

The ipa-certupdate can be run on a client to make sure any new or renewed certificates added to the trust store are applied to the local machine. On a server it will additionally ensure that new or renewed certificates added to the trust store are installed into the the KDC, web and directory server certificate databases (or is this done instead by ipa-server-certinstall?)

/!\ If you have more than one certificate in the trust store, this bug means that OpenSSL on Debian-based systems won't trust *any* of them. I'm working on a fix for this.

When certmonger starts up, it uses these entries to refresh its cached copies of the CA certificates for the IPA CA (ref: fetch_roots function).

Legacy CAcert container

The entry cn=CAcert,cn=ipa,cn=etc,SUFFIX contains the IPA CA certificate. It's used by:

IPA CAs

cn=cas,cn=ca,SUFFIX contains the IPA CA certificates themselves (plural because IPA supports multiple 'lightweight sub-CAs'.

Host Aliases

Say you have a host, foo.ipa.example.com, with an IP address of 192.0.2.0.

The host is behind a NAT gateway with an address on the Internet of 203.0.113.0.

If you enable dyndns_update, a foo.ipa.example.com will point at 192.0.2.0. Although this leaks information about foo's local network to the public, at least users on foo's network will be able to reach it using that FQDN.

But what if the NAT gateway is provided by Sky Broadband? Sky's routers intercept DNS traffic, preventing their customers from using third-party DNS resolvers, doubtless so that Sky can sell information about the web sites their customers visit to advertisers. As a side effect of this, some more exotic forms of DNS traffic are blocked. This includes the GSS-TSIG messages from nsupdate, which is the mechanism that sssd uses to perform DNS updates. (Debugging this was not fun.)

One alternative is for users on foo's network to make use of mDNS and reach it at foo.local. In order to make sure that they can use Kerberos to log in, we must make use of FreeIPA's Kerberos Principal Alias feature:

Users can now obtain a ticket for host/foo.local and use it to authenticate to the SSH server running on foo.

However, users will still be prompted to confirm foo's SSH host key; this is because sss_ssh_knownhostsproxy requests the host keys for foo.local, and SSSD's SSH responder isn't able to search the directory for hosts by alias.

CA renewal server/CRL publisher promotion

The documentation for removing a server from the topology doesn't mention that you need to move the CA renewal server and CRL publisher roles to another server with the CA role.

If you don't do this, it's not fatal, but your CRL file won't be updated.

Both steps are explained in a separate chapter, Decommissioning a server that performs the CA renewal server and CRL publisher role. There is overlap with content from the Managing Replication Topology chapter.

If you want to check the status of your CRL file: 

$ curl -sS -L http://ipa-ca.ipa.ipa.example.com/ipa/crl/MasterCRL.bin | openssl crl -inform der -noout -lastupdate
lastUpdate=Nov 16 15:13:34 2021 GMT

... you can view the whole CRL with -text and [other options are available|https://www.openssl.org/docs/manmaster/man1/openssl-req.html].

The rest of the notes under this heading are obsolete now that I've straightened things up, linked to the docs and filed the above bugs to try to get them cross linked... just skip past them. I'll remove them eventually...

The Starting CRL generation on RHEL 8 chapter of the "Migrating IdM from RHEL 7 to RHEL 8 and keeping it up-to-date" section of the RHEL 8 Installing IdM manual doesn't make much sense. The new replica is running RHEL 8, so why do the prerequisites talk about RHEL 7.6/7.7?

After following these steps on a CentOS 8.0.1905 machine, the CRL is not available:

The manual mentions a ipa-crlgen-manage command, which does not exist in CentOS 8, but it does exist in CentOS 7:

Examining the FreeIPA source reveals that this script was added in FreeIPA 4.8:

And backported to FreeIPA 4.7.3 and 4.6.5:

So I guess it will show up in CentOS 8.1. But until then, examining the source code reveals that, after manually enabling CRL generation, there is an additional undocumented step:

Unfortunately the updateCRL method was likewise only added in FreeIPA 4.7.3. Fortunately it's not too complex to prevent me from bodging together the following:

And behold, we now have a CRL on the new master:

Remaining questions

There are still seemingly significant differences in Tomcat's configuration between the old and new masters:

These settings are mentioned in older documentation and on a post to the mailing list:

So I'm still not sure whether my new master CRL generator will actually listen to updates from replicas and/or periodically check the status of certificates for updates. I also wonder whether master.ca.agent.host has to be updated on ipa1...

PKI topology goes out of sync with LDAP server topology

After removing some servers with the CA/KRA roles, the list of servers in Dogtag will get out of sync with the replication topology in the directory.

This will cause ipa-healthcheck to complain about non-contactable CA/KRA servers.

The upstream bug shows how to fix this:

... run that for each removed CA/KRA server.

Extending FreeIPA

Custom subtrees should go in an nsContainer directly under SUFFIX.

DNS: long TXT records

DMARC requires rather long TXT records. Attempting to create one of these in the web UI or on the command line works, but then named-pkcs11 logs:

The reason for this is not obvious.

So, RFC 1035 describes the data of a TXT RR as simply "One or more <character-string>s". And describes <character-string> as "a single length octet followed by that number of characters. <character-string> is treated as binary information, and can be up to 256 characters in length (including the length octet)."

So our everyday experience of a TXT record is wrong! We naïvely think of it as "just a string", when really it has structure; in this case, one or more arrays of bytes each of which can be up to 255 characters. BIND is choking because it's trying to compile a TXT record consisting of a single <character-string> of well over 300 bytes!

The RFC also defines how to represent a <character-string> in a Zone file, as parsed by BIND:

and

So it turns out that zone file format is more of a literal transcription of DNS data into a binary format than I had expected. So where you express the following in a zone file:

... the compiled RRDATA for the record consists of *nine* length-prefixed byte strings. It's only RFC 7489, sec. 6.1 that determines that they are stitched together into a single string by the entity making use of the records:

Stitching this knowledge together we can now create the record in the web UI or the ipa command, simply by adding spaces to the record data so that no one run of characters lasts for more than 255 bytes. For example:

While figuring out the above I ran into RFC 1464, which proposes a way to store structured data within TXT records, by having each <character-string> within the RDATA for a TXT record represent an attribute and value pair, separated by =. e.g.:

It does not state, although I presume, that you can have multiple attributes by using multiple <character-string>s within the RDATA:

... up to the maximum length of the RRDATA for a record which is 65536 bytes. I guess that one of the reasons this never took off is that the length of each value would be limited to 254 - (length of attribute) bytes.

CategoryTechnote

robots.org.uk: FreeIPA (last edited 2023-05-12 10:04:21 by sam)

© Sam Morris <sam@robots.org.uk>.
Content may be distributed and modified providing this notice is preserved.