FreeIPA is pretty cool but it is a complex beast with a lot of moving parts. Its documentation is alright but there are many things that were (as least to me) not obvious about it.

Contents

  1. Official Documentation
    1. Documentation for components
  2. Directory suffix
  3. Connecting to the directory via UNIX sockets
  4. Root DSE attributes
  5. Password storage
  6. Logging
    1. Audit logs
    2. CLI
    3. Directory server
    4. Kerberos
    5. Web server
    6. PKI server
      1. Tomcat logs
      2. Tomcat access logs
      3. Subsystem logs
      4. Subsystem debug logs
    7. Name server logs
    8. Custodia
    9. OTP
  7. Replication status monitoring
    1. Adding a dedicated system account for replication status monitoring
  8. Automating host enrollment with PKINIT
  9. Exposing to the Internet
  10. Certificate storage locations
    1. Trust Store
    2. Legacy CAcert container
    3. IPA CAs
  11. Host Aliases
  12. CA renewal server/CRL publisher promotion
    1. Remaining questions
  13. PKI topology goes out of sync with LDAP server topology
  14. Extending FreeIPA
  15. DNS: long TXT records

Official Documentation

The best place to start when looking for reference documentation is Red Hat's Red Hat Identity Management Documentation index; then the FreeIPA workshop.

The FreeIPA wiki has a documentation index, although many pages are out of date. Google will often take you here, or to various other outdated sources of information, so always check the above documentation first!

Documentation for components

FreeIPA configures its underlying components in an opinionated way. Keep their documentation to hand as well:

Directory suffix

The examples below assume a FreeIPA domain of ipa.example.com. When naming directory entries, replace SUFFIX, with dc=ipa,dc=example,dc=com.

If you use the integrated CA feature, then Dogtag's state will be stored in the directory at another suffix: o=ipaca. You mostly don't have to worry about that detail except when configuring and monitoring replication status.

Connecting to the directory via UNIX sockets

To administer the directory server, you have to use Simple Authentication, specifying cn=Directory Manager as the password. Having to keep the password handy is a bit annoying.

There is an alternative: a process connecting via a UNIX socket can use SASL EXTERNAL authentication in order to be identified by their UID/GID. On the command line: 

# ldapwhoami -H ldapi://%2frun%2fslapd-IPA-EXAMPLE-COM.socket -Y EXTERNAL
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
dn: cn=Directory Manager

Note that we weren't prompted for a password--the root user is mapped to cn=Directory Manager by default.

This can be made the default for the root user by putting the following in /root/.ldaprc: 

URI ldapi://%2frun%2fslapd-IPA-EXAMPLE-COM.socket
SASL_MECH EXTERNAL

Root DSE attributes

As with all LDAP directories, some interesting details can be queried anonymously by performing a search for the root entry:

See Root DSE Attributes for an explanation of what they mean.

Password storage

The cn=Directory Manager password is stored on the nsslapd-rootpw attribute of cn=config.

Password hashes are stored in the userPassword attribute.

The default hash format is (now) {PBKDF2_SHA256}; it used to be the (weak) {SSHA512}.

The AllowNThash password plugin policy is enabled by default. If ipa-adtrust-install has been run, then the NT hash (unsalted MD4, gotta love that security) will be written to the sambaNTPassword and/or ipaNTHash attributes during password change.

Kerberos keys are stored in the krbPrincipalKey attribute.

Logging

Red Hat have a knowledge base article detailing how to add debug logging on an IPA server. I'll not repeat _all_ of that article in the information below, so read the article if you want more logging for a particular FreeIPA component.

Audit logs

Newer version of FreeIPA have an audit logging feature.

Messages are logged to the journal. They can be viewed with journalctl -g '^\[IPA\.API\]'.

CLI

The ipa command logs to ~/.ipa/log/cli.conf.

Verbosity can be increased by placing debug = true in the [global] section of /etc/ipa/cli.conf (or ~/.ipa/default.conf). This can also be done for a single ipa invocation with the -e debug=true command line option.

Directory server

Under /var/log/dirsrv/slapd-IPA-EXAMPLE-COM you'll find access, security and errors.

Additional audit and audit-failure logs can be enabled:

Kerberos

KDC logs are found at /var/log/krb5kdc.log

kadmin (password change) are at /var/log/kadmind.log.

Web server

The FreeIPA web console and also API logs are found in /var/log/httpd.

Verbosity can be increased by placing debug = true in the [global] section of /etc/ipa/server.conf and restarting httpd.service.

PKI server

Has a lot of log files some of which are not purged by default, so they can grow very large if not removed manually.

By the way, the PKI server is hosted by a Tomcat web server on ports 8080 and 8443. https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/R7AXDSRRHI2ZCYPM3SPFDEHHL2EQYJWX/, so it will be accessible from the network unless firewalled off. (Once I have created a separate section that descibes all the components of FreeIPA, this information will move there).

Tomcat logs

These are:

They're documented in pki-server-logging(5).

They are rotated daily but old log files are not purged by default.

To enable purging, replace the /etc/pki/pki-tomcat/logging.properties symlink with a copy of the target file (see the man page for details) and then set:

Tomcat access logs

This is found at /var/log/pki/pki-tomcat/localhost_access_log.*.txt. It's rotated daily old log files are not purged by default.

To enable purging, edit /etc/pki/pki-tomcat/server.xml and find the Valve element for org.apache.catalina.valves.AccessLogValve. Add an attribute maxDays="7".

Subsystem logs

These are /var/log/pki/pki-tomcat/*/{selftests.log,system,transactions,signedAudit/*_audit}.

They are rotated after 30 days or after they grow to 2000 KiB. This can be figured in the subsystem's corresponding CS.cfg file; however this file only exists for the ca and kra subsystems; there's no corresponding file for acme and pki. Old log files are not purged automtaically.

There is an expirationDate property for these log files in CS.cfg but a comment in RollingLogFile.java says that it is not supported.

Subsystem debug logs

These are /var/log/pki/pki-tomcat/*/debug.*.log.

They're documented at Configuring Subsystem Debug Log.

They're rotated daily and old log files should be removed after 7 days, but this doesn't seem to work.

Name server logs

These are found in /var/named/data/*.log. They are rotated automatically and old log files are purged, however not very often.

Custodia

This is the component that transfers the directory manager password, the CA and system certificates & private keys, the Kerberos master key, and so on between FreeIPA servers when installing a new replica.

Audit log is found at /var/log/ipa-custodia.audit.log.

Doesn't seem to produce operational logs to a file, nor to the journal.

OTP

Logs to the journal. Because this is a per-connection socket-activated service, the easiest way to filter for its messages is journalctl -t ipa-otpd.

Replication status monitoring

{i} There's a general purpose dsctl IPA-EXAMPLE-COM healthcheck command. This outputs a list of problems detected with the directory server. It doesn't seem to be called by ipa-healthcheck, maybe it's a relatively new command. I don't know if it checks for problems with replication status.

Monitoring the Replication Topology in the RHDS documentation leads us to:

Comparing two Directory Server instances describes the ds-replcheck command, which can be use to check if two servers are in-sync. It can also compare replicated suffixes and output any differences found.

Here's an alternative that you can run on each server to check its view of each of its replication agreements:

On RHEL 7, there's no dsconf command, nor replicaLastUpdateStatusJSON attribute; so here's a pure-ldapsearch equivalent instead:

All replication agreements can be 'poked' to force them to send data with:

You can view details for a single replication agreement with:

Some low-level details of replication agreements can be viewed with:

Those are pretty technical. Probably if you need to interpret this information you're best off asking for help on the mailing lists.

Two other parts of the RHDS documentation are worth pointing out:

But generally the entire Managing Replication chapter is worth a read.

In any case, check the directory server error log to investigate the cause of replication errors.

Adding a dedicated system account for replication status monitoring

This is an account that exists only in the directory. It's not a FreeIPA user, it can't log in to any systems, etc; it can only bind to the directory and query.

I'm using ldapvi instead of ldapadd/ldapmodify; this assumes that /root/.ldaprc has been created as described above.

Test binding as the user:

Grant the user permission to view replication agreements:

Fetch replication info from a server with the system account:

Now use ds-replcheck with the system account:

Automating host enrollment with PKINIT

Public Key Cryptography for Initial Authentication (PKINIT) can be used for unattended enrollment of hosts. The advantage over using an admin account (or a dedicated account for enrollment) is that you don't need to manage the credentials of such an account, or enter them on the system being enrolled.

To begin, we need to create a certificate mapping rule.

When a KDC recieves a certauth request from a client, it will go through all these rules (in priority order, ascending, with no priority sorting last), until it finds one where the matching rule accepts the client's certificate.

In this case, the matching rule has two component rules; both must match in order for the matching rule to accept the certificate.

  1. <ISSUER> matches the client certificate's issuer. It's a regular expression search; note the start/end of string symbols at the beginning and end of the expression, and that . is escaped. Here, we want to match any certificate issued by our IPA domain's own certificate authority.

  2. <SAN> matches the 'subject alternative name' extension, by looking through each name for a Kerberos principal name. It too is a regular expression match. Here we want to match any certificate issued to a host principal.

The syntax of the matching rules is quite obscure; it is documented in krb5.conf(5) and in sss-certmap(5).

Once a certificate mapping rule's matching rule accepts the client's certificate, the mapping rule is used as a template, parametized with values taken from the client certificate. The template produces an LDAP filter which will be used to search the directory. If a single entry is found, then it identifies the DN of the identity that will be authenticated.

The available template parameters are listed in sss-certmap(5).Here we're looking for an entry that has both an fqdn matching the DNS-ID from the certificate, and a krbPrincipalName matching the Kerberos principal name from the certificate.

Now we create the host object to be enrolled:

Next we create a private key and a certificate signing request.

We must consider how we want the Kerberos principal name to be included in the certificate. There are two supported methods. Both involve the addition of an otherName entry to the subject alternative name extension (for details see RFC 52809).

  1. The szOID_NT_PRINCIPAL_NAME OID is followed by a UTF8String encoding the principal name (including realm).

  2. The id-pkinit-san (also known as id-kerberos-san) OID is followed by a KRB5PrincipalName sequence, comprises a realm (a GeneralName string) followed by a 2-element sequence of a name-type (1) and a name-string, which is itself a sequence of GeneralNames, in this case being the primary (host) followed by the instance (the hostname).

Fortunately, the <SAN> component-rule in the matching rule of the certificate mapping rule we created earlier can match OIDs (although I've only tested it with certificates that contain szOID_NT_PRINCIPAL_NAME alone and both szOID_NT_PRINCIPAL_NAME and id-pkinit-san). So the choice comes down to how easy they are to work with:

Enough talk, let's create a certificate signing request including the Kerberos principal name:

OpenSSL maps msUPN and Microsoft User Principal Name to szOID_NT_PRINCIPAL_NAME, so fortunately we can use them in our command, rather than the full OID.

Now let's issue a certificate:

Note that the response includes the 'subject UPN'. Let's double-check that the Kerberos principal name is really present in the issued certificate:

Now we have a certificate that can be used to authenticate as the host.

/!\ Before we try to use the certificate to authenticate, we need to wait 5 minutes. This is because KDCs cache certificate mapping rules for 300 seconds when they first recieve a certauth request, and we don't want a stale ruleset to cause an authentication failure.

While we wait, we can use a handy tool to evaluate our certificate mapping rule against the certificate, and check that it produces the LDAP filter expression that we expect.

And we can even search the directory with the produced filter to check that it finds the correct entry:

Ok, enough stalling for time. The KDC certificate mapping rule cache has expired and we're able to authenticate with the certificate:

In our example, it worked! Now it's time to copy the client certificate, private key and CA certificate to the host and run ipa-client-install!

This will look something like:

The same certificate and private key can be used again and again, as many times as needed, before the certificate expires.

Aside from the official IdM documentation and the man pages linked above, additional information can be found at:

Exposing to the Internet

Certificate storage locations

Trust Store

The list of known CA certificates that the FreeIPA installation trusts is kept in the directory under cn=certificates,cn=ipa,cn=etc,SUFFIX. For a stand-alone CA installation, the store will normally only contain the CA's certificate (IPA.EXAMPLE.COM IPA CA). But for an externally signed CA installation, it will contain the external CA certificate as well. This bug shows examples of the schema.

The ipa-cacert-manage list command will perform an LDAP search, and print an entry for each certificate it finds.

New certificates can be installed with ipa-cacert-manage install. There's no command to remove them.

The ipa-certupdate can be run on a client to make sure any new or renewed certificates added to the trust store are applied to the local machine. On a server it will additionally ensure that new or renewed certificates added to the trust store are installed into the the KDC, web and directory server certificate databases (or is this done instead by ipa-server-certinstall?)

/!\ If you have more than one certificate in the trust store, this bug means that OpenSSL on Debian-based systems won't trust *any* of them. I'm working on a fix for this.

When certmonger starts up, it uses these entries to refresh its cached copies of the CA certificates for the IPA CA (ref: fetch_roots function).

Legacy CAcert container

The entry cn=CAcert,cn=ipa,cn=etc,SUFFIX contains the IPA CA certificate. It's used by:

IPA CAs

cn=cas,cn=ca,SUFFIX contains the IPA CA certificates themselves (plural because IPA supports multiple 'lightweight sub-CAs'.

Host Aliases

Say you have a host, foo.ipa.example.com, with an IP address of 192.0.2.0.

The host is behind a NAT gateway with an address on the Internet of 203.0.113.0.

If you enable dyndns_update, a foo.ipa.example.com will point at 192.0.2.0. Although this leaks information about foo's local network to the public, at least users on foo's network will be able to reach it using that FQDN.

But what if the NAT gateway is provided by Sky Broadband? Sky's routers intercept DNS traffic, preventing their customers from using third-party DNS resolvers, doubtless so that Sky can sell information about the web sites their customers visit to advertisers. As a side effect of this, some more exotic forms of DNS traffic are blocked. This includes the GSS-TSIG messages from nsupdate, which is the mechanism that sssd uses to perform DNS updates. (Debugging this was not fun.)

One alternative is for users on foo's network to make use of mDNS and reach it at foo.local. In order to make sure that they can use Kerberos to log in, we must make use of FreeIPA's Kerberos Principal Alias feature:

Users can now obtain a ticket for host/foo.local and use it to authenticate to the SSH server running on foo.

However, users will still be prompted to confirm foo's SSH host key; this is because sss_ssh_knownhostsproxy requests the host keys for foo.local, and SSSD's SSH responder isn't able to search the directory for hosts by alias.

CA renewal server/CRL publisher promotion

The documentation for removing a server from the topology doesn't mention that you need to move the CA renewal server and CRL publisher roles to another server with the CA role.

If you don't do this, it's not fatal, but your CRL file won't be updated.

Both steps are explained in a separate chapter, Decommissioning a server that performs the CA renewal server and CRL publisher role. There is overlap with content from the Managing Replication Topology chapter.

If you want to check the status of your CRL file: 

$ curl -sS -L http://ipa-ca.ipa.example.com/ipa/crl/MasterCRL.bin | openssl crl -inform der -noout -lastupdate
lastUpdate=Nov 16 15:13:34 2021 GMT

... you can view the whole CRL with -text and other options are available.

The rest of the notes under this heading are obsolete now that I've straightened things up, linked to the docs and filed the above bugs to try to get them cross linked... just skip past them. I'll remove them eventually...

The Starting CRL generation on RHEL 8 chapter of the "Migrating IdM from RHEL 7 to RHEL 8 and keeping it up-to-date" section of the RHEL 8 Installing IdM manual doesn't make much sense. The new replica is running RHEL 8, so why do the prerequisites talk about RHEL 7.6/7.7?

After following these steps on a CentOS 8.0.1905 machine, the CRL is not available:

The manual mentions a ipa-crlgen-manage command, which does not exist in CentOS 8, but it does exist in CentOS 7:

Examining the FreeIPA source reveals that this script was added in FreeIPA 4.8:

And backported to FreeIPA 4.7.3 and 4.6.5:

So I guess it will show up in CentOS 8.1. But until then, examining the source code reveals that, after manually enabling CRL generation, there is an additional undocumented step:

Unfortunately the updateCRL method was likewise only added in FreeIPA 4.7.3. Fortunately it's not too complex to prevent me from bodging together the following:

And behold, we now have a CRL on the new master:

Remaining questions

There are still seemingly significant differences in Tomcat's configuration between the old and new masters:

These settings are mentioned in older documentation and on a post to the mailing list:

So the operation performed by ipa-crlgen-manage is incomplete, it does not manage ca.certStatusUpdateInterval and this setting is absolutely required for functioning CRL generation!

I also wonder whether master.ca.agent.host has to be updated on ipa1...

PKI topology goes out of sync with LDAP server topology

After removing some servers with the CA/KRA roles, the list of servers in Dogtag will get out of sync with the replication topology in the directory.

This will cause ipa-healthcheck to complain about non-contactable CA/KRA servers.

The upstream bug shows how to fix this:

... run that for each removed CA/KRA server.

Extending FreeIPA

Custom subtrees should go in an nsContainer directly under SUFFIX.

DNS: long TXT records

DMARC requires rather long TXT records. Attempting to create one of these in the web UI or on the command line works, but then named-pkcs11 logs:

The reason for this is not obvious.

So, RFC 1035 describes the data of a TXT RR as simply "One or more <character-string>s". And describes <character-string> as "a single length octet followed by that number of characters. <character-string> is treated as binary information, and can be up to 256 characters in length (including the length octet)."

So our everyday experience of a TXT record is wrong! We naïvely think of it as "just a string", when really it has structure; in this case, one or more arrays of bytes each of which can be up to 255 characters. BIND is choking because it's trying to compile a TXT record consisting of a single <character-string> of well over 300 bytes!

The RFC also defines how to represent a <character-string> in a Zone file, as parsed by BIND:

and

So it turns out that zone file format is more of a literal transcription of DNS data into a binary format than I had expected. So where you express the following in a zone file:

... the compiled RRDATA for the record consists of *nine* length-prefixed byte strings. It's only RFC 7489, sec. 6.1 that determines that they are stitched together into a single string by the entity making use of the records:

Stitching this knowledge together we can now create the record in the web UI or the ipa command, simply by adding spaces to the record data so that no one run of characters lasts for more than 255 bytes. For example:

While figuring out the above I ran into RFC 1464, which proposes a way to store structured data within TXT records, by having each <character-string> within the RDATA for a TXT record represent an attribute and value pair, separated by =. e.g.:

It does not state, although I presume, that you can have multiple attributes by using multiple <character-string>s within the RDATA:

... up to the maximum length of the RRDATA for a record which is 65536 bytes. I guess that one of the reasons this never took off is that the length of each value would be limited to 254 - (length of attribute) bytes.

CategoryTechnote

robots.org.uk: FreeIPA (last edited 2024-10-31 17:04:26 by sam)

© Sam Morris <sam@robots.org.uk>.
Content may be distributed and modified providing this notice is preserved.