Little Known Features of Illumio ASP – Policy Compute Engine Certificate Checks

In this quick series, the Illumio product management team will highlight the lesser known (but no less powerful) features of Illumio ASP.

Transport Layer Security (TLS) protocol is a cryptographic protocol for securing communication over computer networks, utilizing public key certificates. The most common format for public key certificates is X.509, described in RFC 5280. X.509 certificates are complex cryptographic documents with options for diverse uses such as authentication, encryption, trust, etc. Following industry best practices, Illumio ASP communication is secured using TLS.

For securing communication in Illumio ASP, certain options must be present in the certificate with specific deployment criteria, and the certificates must be installed on the Policy Compute Engine (PCE), the “brain” of Illumio ASP that centralizes visibility and policy management. While these are common criteria for certificates, they can be onerous and require attention to detail on the user’s part.

In this blog post, I will describe the deployment criteria for Illumio PCE certificates and a little known feature for certificate checks.

TLS Protocol and Certificates

TLS is used for securing the following communication sessions:

• User access to the PCE web console and REST API over the HTTPS (HTTP over TLS) protocol.
• Communication between the PCE and VENs, with REST API over HTTPS and Event Service, a custom protocol using TLS over TCP.
• Communication between PCE services on different PCE nodes in a multi-node cluster, such as cluster management and service discovery.

For TLS, an X.509 server certificate must be installed on each PCE node during installation. When any client (e.g., the VEN) opens a TLS session to the PCE, the PCE presents the X.509 server certificate to secure the communication with the client. During installation, the server certificate is uploaded to the PCE as part of a certificate bundle, which contains the server certificate and a set of CA certificates (Intermediate or Root) to establish the chain of trust back to a Root CA.

X.509 certificates are generally issued by a public Certificate Authority (CA) such as Digicert, Verisign, GoDaddy, and LetsEncrypt. Additionally, customers may use certificates issued by their own Private CAs (internal CA) or use self-signed certificates. Regardless of how a certificate is issued, the client must be able to validate the chain of trust back to the Root CA for this certificate; otherwise, the TLS handshake fails and a secure communication channel cannot be established.

PCE Certificate Check

During the initial deployment of the PCE, the user generally has to validate the certificates to ensure the varied and complicated requirements are met. Illumio PCE ships with a Certificate Check tool to assist users by automating the checking of certificates against the specific deployment criteria.

The certificate package for the Illumio PCE must meet the following criteria:

• Must contain PEM-encoded certificates.
• Certificates are signed using an acceptable signature algorithm.
• The package must include all of the CA certificates (Intermediate and/or Root) needed to establish the chain of trust back to the Root CA. If the certificate is generated by a Private CA, the package must include all certificates in the chain of trust back to the Root CA and any Intermediate CA certificates.
• The server certification must be the first certificate in the bundle.
• All certificates in the bundle must be valid for the current date.
• The certificate must match the PCE FQDN, in both Subject and Subject Alternative Name (SAN) fields.
• The certificate must support both Server and Client authentication.

Similarly, the private key that matches the X.509 certificate must be installed on each PCE node with the following criteria:

• The private key must be PEM-encoded.
• The file must not be encoded in other ways, e.g. DER, PKCS7/P7B, PKCS8 or PKCS12/PFX. 
• The file must not be password protected.

Over time, Illumio has discovered that PCE deployment at enterprises face multiple challenges when it comes to certificate creation and deployment.

For example, the certificates may not be issued for the PCE FQDN; or the multiple nodes’ hostnames are not correctly included in the SAN field; or the correct set of options are not provided in the certificates’ Extended Key Usage field. Similarly, during the installation process, the certificates may be installed without the correct permissions for the files or directories, or some pieces of the trust chain may be missed during installation.

The challenges are reserved for the initial installation process, and can extend beyond. If the PCE is scaled from a 4 node cluster to a 6 node cluster, the certificates will need to be updated to include the new node's FQDN in the SAN field. To reduce the manual effort with certificates, Illumio provides an administrative command-line tool called "illumio-pce-env” to check certificates automatically.

This tool provides several options to validate certificates for a variety of use cases, some of which are:

1. To validate the TLS certificate with basic tests, including the chain of trust and other aspects, run the following command:
2. illumio-pce-env setup --list
4. The specifying “--list” option checks your configuration and certificates and indicates possible problems by setting the return code. This is a quick way to check for issues and can be invoked as part of the Chef, Ansible, or other installation scripts.
6. To validate the TLS certificate with basic tests, including the chain of trust and other aspects, run the following command:
7. illumio-pce-env setup –list –test 5
9. If the previous command fails, then an administrator may want to find exactly what went wrong. To help with this, it is possible to specify a verbosity level argument—1 (least) to 5 (most)—with the --test option. Using a verbosity level of 5, the command display details results with each step of its certificate validation, including information on tests that were run and each test’s results. This helps diagnose the exact issue with the certificates or the trust chain.
11. To validate your certificates with alternative domain names, run the following command:
12. illumio-pce-env setup --list –test 5:some.alternative.hostAndDomainName
14. In the case where the FQDN plan to use for the PCE in production is different from the test deployment or if a PCE is being expanded from a 4 node cluster to a 6 node cluster, the certificate will need to be checked if the SAN field includes the correct hostnames. The SAN field permits the use of wildcards, which makes it a bit tricky to manually confirm valid hostnames. This syntax checks the certificate and chain against the specified some.alternative.hostAndDomainName.
16. To validate your certificates with an end-to-end test using ‘+’ syntax, run the following command:
17. illumio-pce-env setup --list --test 5+
19. Here, the argument to test is “5+”. Sometimes the static tests of a certificate may not suffice, and an administrator may want to validate the system with a full end-to-end test of the certificate with the operating system TLS libraries. This emulates the actual runtime use of certificate. The ‘+’ syntax creates a loopback OpenSSL server running on port 4433 and attempts to use the curl command to establish a TLS connection.
21. To validate your certificates before copying them to your planned production location, run the following command:
22. illumio-pce-env setup --batch --list \email=required@emailaddress node=value \cert=/path/to/cert \pkey=/path/to/private_key \trust=/path/to/certificate_chain \--test 5
24. If an administrator wants to check the certificate without actually installing it, there is a syntax to allow that. This would be used if a new certificate is deployed and the administrator wants to check if it meets the PCE requirements.

Please see the Illumio PCE documentation for a complete list of messages displayed by the illumio-pce-env tool as well as a comprehensive list of potential error messages.

In summary, the X.509 certificates used for securing communication with TLS have varied and complicated requirements and can be burdensome for the administrator to validate. Illumio ASP provides a command-line tool with a rich set of options to validate certificates for different needs – from basic testing to full end-to-end testing to more accurately and precisely test before deployment.

I hope this blog post helped provide insight into this little known, useful feature of the Illumio PCE. For further questions, please contact us at [email protected] and don't forget to check out the other posts from this series, which cover:

• Color Vision Deficiency Filter
• Log Export to Amazon S3 Buckets
• Broadcast and Multicast Filters