PKI & Certificates

Key Generation

openssl genrsa -out private.key 4096

openssl ecparam -genkey -name prime256v1 \
  -out ec.key

CSR & Signing

openssl req -new -key private.key \
  -out request.csr

openssl req -x509 -newkey rsa:4096 \
  -nodes -keyout key.pem \
  -out cert.pem -days 365

Certificate Inspection

openssl x509 -in cert.pem -text -noout

openssl x509 -enddate -noout -in cert.pem

openssl verify -CAfile ca.crt \
  -untrusted intermediate.crt server.crt

openssl s_client -connect example.com:443 \
  -showcerts

Format Conversion

openssl x509 -outform der \
  -in cert.pem -out cert.der

openssl pkcs12 -export -out cert.pfx \
  -inkey key.pem -in cert.pem

openssl x509 -noout -modulus -in cert.pem \
  | openssl md5
openssl rsa -noout -modulus -in key.pem \
  | openssl md5

openssl pkcs12 -in cert.pfx \
  -out cert.pem -nodes

Certificate Fields

An X.509 certificate is an ASN.1 structure defined by RFC 5280. Every certificate contains these core fields, encoded using DER (Distinguished Encoding Rules) and typically wrapped in PEM (Base64) for transport.

Version 1 vs. Version 3

Critical v3 Extensions

Extensions marked as critical must be understood by the verifying application; if an extension is critical and unrecognized, the certificate must be rejected. Non-critical extensions can be safely ignored.

Key Usage Bits

The Key Usage extension is a 9-bit field that restricts what cryptographic operations a key may perform. Setting the wrong bits is a common source of interoperability failures.

Extended Key Usage OIDs

While Key Usage restricts cryptographic operations, Extended Key Usage (EKU) restricts the application-level purpose. OIDs are dot-delimited numbers from the ITU/ISO object identifier tree.

Decoded Certificate Example

Output from openssl x509 -in server.crt -text -noout showing a typical TLS server certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:e3:a5:b8:c7:2d:9f:01:aa:bb:cc:dd:ee:ff:00:11
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Jan 15 00:00:00 2025 GMT
            Not After : Apr 15 23:59:59 2025 GMT
        Subject: CN=www.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                A1:B2:C3:D4:E5:F6:01:02:03:04:05:06:07:08:09:10
            X509v3 Authority Key Identifier:
                14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D
            X509v3 Subject Alternative Name:
                DNS:www.example.com, DNS:example.com
            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/
            X509v3 CRL Distribution Points:
                Full Name:
                    URI:http://r3.c.lencr.org/
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1

CA Hierarchy

Public Key Infrastructure forms a tree of trust. The root CA sits at the apex, delegating signing authority through one or more intermediate CAs down to end-entity (leaf) certificates.

CA Roles Explained

Cross-Signing

Cross-signing allows a new CA to be trusted by existing clients before its own root is widely distributed. A well-established CA signs the new CA's public key, creating an alternative trust path.

Example: Let's Encrypt's ISRG Root X1 was cross-signed by IdenTrust's DST Root CA X3, allowing Android devices and older browsers to trust Let's Encrypt certificates before ISRG Root X1 was added to their trust stores.

Trust Store Overview

Every operating system and application maintains a curated list of trusted root CA certificates. This is the foundation of all public PKI trust decisions.

Chain Building & Validation (RFC 5280)

When a client receives a server certificate, it must build a chain from the leaf up to a trusted root and validate every link. Here are the validation steps in order:

Self-Signed vs. CA-Signed

Adding a Custom CA to Trust Stores

For internal/enterprise CAs, you must distribute the root certificate to all client machines. Here are the commands for each platform:

# Copy CA cert to the trusted directory
sudo cp corp-ca.crt \
  /usr/local/share/ca-certificates/corp-ca.crt

# Rebuild the trust store
sudo update-ca-certificates

# Copy to anchors directory
sudo cp corp-ca.crt \
  /etc/pki/ca-trust/source/anchors/corp-ca.crt

# Update the consolidated bundle
sudo update-ca-trust

# Add to System keychain as trusted root
sudo security add-trusted-cert \
  -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  corp-ca.crt

# Import to Trusted Root store (machine-wide)
Import-Certificate `
  -FilePath .\corp-ca.crt `
  -CertStoreLocation Cert:\LocalMachine\Root

# Import to Java trust store (cacerts)
keytool -import -trustcacerts \
  -alias corp-ca \
  -file corp-ca.crt \
  -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit

# Firefox ignores OS trust stores by default.
# Option 1: Import via Preferences > Certificates
# Option 2: Deploy policies.json:
{
  "policies": {
    "Certificates": {
      "ImportEnterpriseRoots": true
    }
  }
}

CSR Generation Process

A Certificate Signing Request (CSR) is a signed data structure sent to a CA containing your public key, identifying information (subject DN, SANs), and a proof-of-possession signature made with the corresponding private key. The CA verifies the signature to confirm you hold the private key, then issues a certificate binding that key to your identity.

# Generates CSR, prompts for subject fields
openssl req -new \
  -key private.key \
  -out request.csr

# Non-interactive with Subject Alternative Names
openssl req -new \
  -key private.key \
  -out request.csr \
  -subj "/CN=example.com/O=Example Inc./C=US" \
  -addext "subjectAltName=DNS:example.com,DNS:www.example.com,DNS:api.example.com"

# Create new key and CSR in one command
openssl req -new -newkey rsa:4096 \
  -nodes \
  -keyout private.key \
  -out request.csr \
  -subj "/CN=example.com"

# Inspect CSR before submitting to CA
openssl req -in request.csr \
  -text -noout -verify

# Output includes:
#   verify OK
#   Subject: CN = example.com
#   Public Key Algorithm: rsaEncryption
#   Requested Extensions:
#     X509v3 Subject Alternative Name:
#       DNS:example.com, DNS:www.example.com

Certificate Issuance — Validation Levels

Certificate Authorities offer three levels of identity validation before issuing a certificate. Each level provides increasing assurance about who controls the domain and organization behind it.

Renewal Strategies

# Test renewal without making changes
certbot renew --dry-run

# Add to crontab (runs twice daily)
0 0,12 * * * certbot renew --quiet \
  --post-hook "systemctl reload nginx"

# Issue with auto-renewal configured
acme.sh --issue \
  -d example.com \
  -d www.example.com \
  --nginx \
  --reloadcmd "systemctl reload nginx"

# acme.sh installs its own cron job
# automatically on first use

Revocation Methods

When a certificate's private key is compromised, the certificate must be revoked before its natural expiration. Four mechanisms exist for communicating revocation status, each with significant trade-offs.

CRL — Certificate Revocation Lists

A CRL is a signed list of revoked certificate serial numbers published by the CA at the URL specified in the certificate's CRL Distribution Points extension. Clients download the full list and cache it until the nextUpdate timestamp.

OCSP — Online Certificate Status Protocol

OCSP (RFC 6960) provides real-time revocation checking. The client sends the certificate's serial number to the CA's OCSP responder URL and receives a signed response: good, revoked, or unknown.

OCSP Stapling

With OCSP Stapling (RFC 6066, TLS Certificate Status Request), the server periodically fetches and caches the OCSP response, then includes ("staples") it in the TLS handshake. The client receives fresh revocation status without contacting the CA directly.

server {
    ssl_stapling on;
    ssl_stapling_verify on;

    # CA chain for verifying OCSP response
    ssl_trusted_certificate /etc/ssl/chain.pem;

    # Resolver for OCSP responder lookup
    resolver 1.1.1.1 8.8.8.8 valid=300s;
    resolver_timeout 5s;
}

SSLUseStapling On
SSLStaplingCache \
  "shmcb:logs/ssl_stapling(32768)"
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors Off

# Optional: force staple even if
# responder is slow
SSLStaplingFakeTryLater Off

# Check if server is stapling OCSP response
openssl s_client -connect example.com:443 \
  -status -tlsextdebug 2>&1 \
  | grep "OCSP Response Status"

# Expected: OCSP Response Status: successful (0x0)
# If missing: OCSP response: no response sent

OCSP Must-Staple (Deprecated)

The OCSP Must-Staple extension (RFC 7633, OID 1.3.6.1.5.5.7.1.24) was designed to solve the soft-fail problem by requiring the server to include a valid OCSP staple. If the staple is missing, the client must hard-fail and reject the connection.

TLS 1.2 Full Handshake

The TLS 1.2 handshake requires two round trips (2-RTT) before application data can flow. Each message serves a specific purpose in negotiating algorithms, authenticating the server, and establishing shared secrets.

TLS 1.3 Simplified Handshake

TLS 1.3 (RFC 8446) dramatically simplifies the handshake to a single round trip (1-RTT) and supports zero round-trip resumption (0-RTT) for repeat connections. It removes legacy features, mandates PFS, and encrypts most handshake messages.

Cipher Suites

A cipher suite defines the algorithms used for each phase of the TLS connection: key exchange, authentication, bulk encryption, and message integrity. Understanding the naming format helps you choose secure configurations.

TLS 1.2 Cipher Suite Format

TLS 1.3 Simplified Format

TLS 1.3 separates key exchange from cipher suite naming. Cipher suites only specify the AEAD algorithm and hash, because key exchange is always (EC)DHE and authentication is determined by the certificate type.

Recommended Modern Cipher Suites

Deprecated & Weak Ciphers

Server Configuration

# Modern configuration (TLS 1.3 + strong 1.2)
ssl_protocols TLSv1.2 TLSv1.3;

ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';

# Server chooses cipher order
ssl_prefer_server_ciphers on;

# TLS 1.3 suites (configured separately)
ssl_conf_command Ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256;

# Modern configuration
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on

SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

# TLS 1.3 cipher suites
SSLCipherSuite TLSv1.3 TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

Perfect Forward Secrecy

Perfect Forward Secrecy (PFS) ensures that if a server's long-term private key is compromised in the future, past recorded TLS sessions cannot be decrypted. Each session generates a unique ephemeral key pair that is discarded after use.

# Check cipher suite of a live connection
openssl s_client -connect example.com:443 \
  2>/dev/null | grep "Cipher    :"

# PFS active if result contains ECDHE or DHE:
#   Cipher: TLS_AES_256_GCM_SHA384 (TLS 1.3)
#   Cipher: ECDHE-RSA-AES256-GCM-SHA384 (TLS 1.2)

# Check server's supported key exchange
openssl s_client -connect example.com:443 \
  2>/dev/null | grep "Server Temp Key"

# Expected: Server Temp Key: X25519, 253 bits
# or:       Server Temp Key: ECDH, P-256, 256 bits

How mTLS Works

Standard TLS authenticates only the server: the client verifies the server's certificate, but the server has no cryptographic proof of the client's identity. Mutual TLS (mTLS) adds a second direction — the server also requests and verifies a client certificate, creating two-way authentication.

Common Use Cases

Configuration Examples

server {
    listen 443 ssl;
    server_name api.example.com;

    # Server certificate
    ssl_certificate     /etc/ssl/server.crt;
    ssl_certificate_key /etc/ssl/server.key;

    # Client certificate verification
    ssl_client_certificate /etc/ssl/client-ca.crt;
    ssl_verify_client on;
    ssl_verify_depth  2;

    # Optional: CRL checking
    ssl_crl /etc/ssl/client-ca.crl;

    # Pass client cert info to upstream
    location / {
        proxy_set_header X-Client-DN
            $ssl_client_s_dn;
        proxy_set_header X-Client-Verify
            $ssl_client_verify;
        proxy_pass http://backend;
    }
}

<VirtualHost *:443>
    ServerName api.example.com

    # Server certificate
    SSLEngine on
    SSLCertificateFile    /etc/ssl/server.crt
    SSLCertificateKeyFile /etc/ssl/server.key

    # Client certificate verification
    SSLCACertificateFile /etc/ssl/client-ca.crt
    SSLVerifyClient require
    SSLVerifyDepth  2

    # Optional: CRL checking
    SSLCARevocationFile /etc/ssl/client-ca.crl
    SSLCARevocationCheck chain

    # Pass client cert info to app
    SSLOptions +StdEnvVars +ExportCertData
</VirtualHost>

Client Certificate Generation

Generating client certificates requires a private CA (or a designated sub-CA) that the server trusts. Here is the complete workflow from CA creation to PKCS#12 bundle:

# Generate CA private key
openssl genrsa -out client-ca.key 4096

# Create self-signed CA certificate
openssl req -new -x509 \
  -key client-ca.key \
  -out client-ca.crt \
  -days 3650 \
  -subj "/CN=Client CA/O=Example Inc."

# Generate client private key
openssl genrsa -out client.key 2048

# Or use EC key for better performance
openssl ecparam -genkey \
  -name prime256v1 \
  -out client.key

# Generate CSR with client identity
openssl req -new \
  -key client.key \
  -out client.csr \
  -subj "/CN=service-a/O=Example Inc./OU=Engineering"

# Sign with Client CA (valid 1 year)
openssl x509 -req \
  -in client.csr \
  -CA client-ca.crt \
  -CAkey client-ca.key \
  -CAcreateserial \
  -out client.crt \
  -days 365 \
  -extfile <(printf "extendedKeyUsage=clientAuth")

# Bundle key + cert for browser/app import
openssl pkcs12 -export \
  -out client.p12 \
  -inkey client.key \
  -in client.crt \
  -certfile client-ca.crt \
  -name "Service A Client Cert"

# Test mTLS connection with curl
curl --cert client.crt \
     --key client.key \
     --cacert server-ca.crt \
     https://api.example.com/health

# Or using PKCS#12 bundle
curl --cert-type P12 \
     --cert client.p12:password \
     https://api.example.com/health

Distribution Challenges

Solutions

Format Overview

Certificates and keys can be stored in several formats. The choice depends on the platform, the tool consuming the certificate, and whether you need to bundle the private key alongside the certificate chain.

PEM Format

PEM (Privacy-Enhanced Mail) is the most common certificate format on Linux/Unix systems. It is Base64-encoded DER data wrapped with plain-text header and footer lines. A single .pem file can contain multiple objects concatenated together (e.g., a certificate followed by its chain).

-----BEGIN CERTIFICATE-----
MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQEL
BQAwTzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5
IFJlc2VhcmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTYx
MDA2MTU0MzU1WhcNMjExMDA2MTU0MzU1WjBKMQswCQYDVQQGEwJVUzEWMBQG
... (Base64-encoded DER data) ...
-----END CERTIFICATE-----

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA0Z3VS5JJcds3xfn/ygWyF8PbnGy5AHB8VBjP...
-----END RSA PRIVATE KEY-----

# Common PEM header types:
# -----BEGIN CERTIFICATE-----          (X.509 certificate)
# -----BEGIN RSA PRIVATE KEY-----      (PKCS#1 RSA key)
# -----BEGIN PRIVATE KEY-----          (PKCS#8 private key)
# -----BEGIN ENCRYPTED PRIVATE KEY---- (encrypted PKCS#8)
# -----BEGIN CERTIFICATE REQUEST-----  (CSR)

DER Format

DER (Distinguished Encoding Rules) is the binary form of a certificate. PEM is simply Base64-encoded DER with header/footer lines added. DER files contain exactly one object (certificate or key) and cannot be concatenated like PEM.

PKCS#12 / PFX

PKCS#12 (also known as PFX) is a password-protected binary container that bundles a private key, its certificate, and the full certificate chain into a single file. It is the standard format for importing and exporting identity credentials across platforms.

Java KeyStore (JKS)

JKS is a proprietary Java keystore format that stores certificates and keys, protected by a keystore password and individual key passwords. Since Java 9, PKCS#12 is the default keystore format, and JKS is considered legacy.

Conversion Commands

A comprehensive reference for converting between certificate formats using openssl and keytool.

openssl x509 -outform der \
  -in cert.pem \
  -out cert.der

openssl x509 -inform der \
  -in cert.der \
  -out cert.pem

openssl pkcs12 -export \
  -out cert.pfx \
  -inkey private.key \
  -in cert.pem \
  -certfile chain.pem

openssl pkcs12 -in cert.pfx \
  -out all.pem \
  -nodes

# Extract private key only
openssl pkcs12 -in cert.pfx \
  -nocerts -nodes \
  -out private.key

# Extract certificate only
openssl pkcs12 -in cert.pfx \
  -clcerts -nokeys \
  -out cert.pem

# Extract CA chain only
openssl pkcs12 -in cert.pfx \
  -cacerts -nokeys \
  -out chain.pem

openssl crl2pkcs7 -nocrl \
  -certfile cert.pem \
  -certfile chain.pem \
  -out cert.p7b

openssl pkcs7 \
  -print_certs \
  -in cert.p7b \
  -out cert.pem

# Step 1: PEM → PKCS#12
openssl pkcs12 -export \
  -in cert.pem \
  -inkey private.key \
  -out cert.p12 \
  -name myalias

# Step 2: PKCS#12 → JKS
keytool -importkeystore \
  -srckeystore cert.p12 \
  -srcstoretype PKCS12 \
  -destkeystore keystore.jks \
  -deststoretype JKS

keytool -importkeystore \
  -srckeystore keystore.jks \
  -srcstoretype JKS \
  -destkeystore keystore.p12 \
  -deststoretype PKCS12

keytool -list -v \
  -keystore keystore.jks

# Show specific alias
keytool -list -v \
  -keystore keystore.jks \
  -alias myalias

ACME Protocol Flow

ACME (Automatic Certificate Management Environment, RFC 8555) automates the entire certificate lifecycle. The client proves domain control to the CA, which then issues a signed certificate — all without human intervention.

Challenge Types

ACME Clients

Certbot

certbot certonly --standalone \
  -d example.com \
  -d www.example.com

certbot certonly --webroot \
  -w /var/www/html \
  -d example.com

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials \
    ~/.secrets/cloudflare.ini \
  -d '*.example.com' \
  -d example.com

certbot renew

# Dry run (test without renewing)
certbot renew --dry-run

certbot certificates

certbot revoke \
  --cert-path /etc/letsencrypt/live/example.com/cert.pem \
  --reason keycompromise

certbot certonly --standalone \
  --staging \
  -d test.example.com

# Staging certs are not trusted
# but don't count against rate limits

certbot --nginx \
  -d example.com \
  -d www.example.com

# Auto-configures Nginx SSL blocks

acme.sh

acme.sh --issue \
  -d example.com \
  -w /var/www/html

acme.sh --install-cert \
  -d example.com \
  --key-file /etc/ssl/key.pem \
  --fullchain-file /etc/ssl/fullchain.pem \
  --reloadcmd "systemctl reload nginx"

acme.sh --renew \
  -d example.com --force

export CF_Key="your-api-key"
export CF_Email="you@example.com"

acme.sh --issue \
  --dns dns_cf \
  -d '*.example.com'

Caddy

example.com {
    reverse_proxy localhost:3000
}

Rate Limits

Auto-Renewal Setup

Let's Encrypt certificates are valid for 90 days. Automated renewal should run at least twice daily to ensure certificates are refreshed well before expiry (Certbot only renews within 30 days of expiration).

systemd Timer (Recommended)

[Unit]
Description=Certbot Renewal

[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet \
  --deploy-hook "systemctl reload nginx"

[Unit]
Description=Run Certbot Twice Daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target

Cron Job Alternative

# Run twice daily at random minutes past the hour
0 0,12 * * * root /usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"

ACME Endpoints

Microsoft AD CS

Active Directory Certificate Services (AD CS) is Microsoft's enterprise PKI platform. It integrates deeply with Active Directory for automated certificate enrollment, Group Policy distribution, and template-based issuance.

Two-Tier PKI Deployment

Auto-Enrollment via Group Policy

EJBCA & Alternatives

Federal PKI (FPKI)

The Federal Public Key Infrastructure (FPKI) is the U.S. government's trust framework for digital identity and certificate management. It enables secure, interoperable digital signatures and authentication across federal agencies and with external partners.

Federal Bridge CA (FBCA)

The Federal Bridge Certification Authority (FBCA) acts as a trust anchor that enables cross-certification between federal agencies, state governments, and external partners. Rather than each agency trusting every other agency's root CA, they each cross-certify with the FBCA, creating a hub-and-spoke trust model.

FPKI Assurance Levels

PIV & CAC Cards

Personal Identity Verification (PIV) and Common Access Card (CAC) are smart card-based identity credentials used across the U.S. federal government and Department of Defense.

PIV Certificate Types

NIST SP 800-57: Key Management

NIST Special Publication 800-57 provides comprehensive recommendations for cryptographic key management, including key lifecycle, algorithm selection, and cryptoperiods (how long a key should be used).

Key Lifecycle Stages

Hardware Security Modules

A Hardware Security Module (HSM) is a dedicated, tamper-resistant physical device that generates, stores, and manages cryptographic keys. HSMs perform cryptographic operations (signing, encryption, decryption) internally so that private keys never leave the protected hardware boundary.

HSMs matter because they provide a root of trust for PKI deployments. Software-based key storage is vulnerable to memory extraction, malware, and insider threats. HSMs address these risks through tamper-resistant enclosures, dedicated cryptographic processors, and strict access controls that meet regulatory requirements such as FIPS 140, Common Criteria, and PCI DSS.

HSM Types

FIPS 140 Validation Levels

PKCS#11 Interface

PKCS#11 (also called "Cryptoki") is the standard API for communicating with HSMs and other cryptographic tokens. Defined by RSA Laboratories (now OASIS), it provides a vendor-neutral C interface that applications use to perform key generation, signing, encryption, and other operations without direct access to key material.

Most HSM vendors ship a PKCS#11 shared library (.so on Linux, .dll on Windows) that OpenSSL, Java (via SunPKCS11 provider), and other applications load to interact with the hardware. This abstraction allows swapping HSM vendors without rewriting application code.

# Configure OpenSSL to use HSM via PKCS#11
openssl req -new -x509 \
  -engine pkcs11 \
  -keyform engine \
  -key "pkcs11:token=MyHSM;object=root-ca-key" \
  -out root-ca.crt -days 3650 \
  -subj "/CN=Root CA"

# List available tokens
pkcs11-tool --module /usr/lib/libCryptoki2.so \
  --list-tokens

# List objects on a specific token
pkcs11-tool --module /usr/lib/libCryptoki2.so \
  --list-objects --login

Key Ceremony Procedures

A key ceremony is a formally documented, witnessed process for generating, activating, or revoking a critical cryptographic key—typically a root CA key. Key ceremonies enforce dual control and split knowledge to prevent any single person from accessing or misusing key material.

Root CA Key Ceremony Steps

Key Lifecycle

Every cryptographic key follows a lifecycle from creation to destruction. Proper lifecycle management ensures keys are used securely, rotated before compromise, and destroyed when no longer needed.

Key Escrow vs Key Recovery

Code Signing

Code signing uses digital signatures to verify the integrity (code has not been tampered with) and attribution (code comes from a known publisher) of software. It is a critical component of supply chain security—operating systems, app stores, and enterprise policies use code signatures to decide whether software is trusted.

Platform Signing Tools

S/MIME for Email

S/MIME (Secure/Multipurpose Internet Mail Extensions) uses X.509 certificates to provide digital signatures and encryption for email messages. It is the standard for enterprise email security, supported natively by Outlook, Apple Mail, and Thunderbird.

S/MIME vs PGP

Certificate Pinning

Certificate pinning restricts which certificates or public keys a client will accept for a given host, providing protection beyond standard CA-based validation. If an attacker obtains a fraudulent certificate from a compromised CA, pinning prevents it from being accepted.

Modern Pinning Approaches

<!-- res/xml/network_security_config.xml -->
<network-security-config>
  <domain-config>
    <domain includeSubdomains="true">
      api.example.com
    </domain>
    <pin-set expiration="2025-06-01">
      <!-- Primary pin (leaf or intermediate) -->
      <pin digest="SHA-256">
        base64-encoded-sha256-of-spki
      </pin>
      <!-- Backup pin (REQUIRED) -->
      <pin digest="SHA-256">
        base64-encoded-backup-pin
      </pin>
    </pin-set>
  </domain-config>
</network-security-config>

Certificate Transparency (CT)

Certificate Transparency is an open framework (RFC 6962) that makes TLS certificate issuance publicly auditable. CAs must submit every certificate they issue to one or more append-only CT logs, creating a public record that domain owners and browsers can monitor for unauthorized issuance.

How CT Works

Monitoring with crt.sh

crt.sh is a free CT log search engine operated by Sectigo. It aggregates data from all major CT logs and allows searching for certificates by domain name, SHA-256 fingerprint, or issuer. Use it to monitor for unauthorized certificates issued for your domains.

# Search for all certificates issued for a domain
curl "https://crt.sh/?q=example.com&output=json" \
  | jq '.[] | {id, name_value, not_after, issuer_name}'

# Search for wildcard certificates
curl "https://crt.sh/?q=%25.example.com&output=json" \
  | jq '.[] | {id, name_value}'

# View SCTs embedded in a certificate
openssl s_client -connect example.com:443 \
  -servername example.com 2>/dev/null \
  | openssl x509 -noout -text \
  | grep -A 5 "CT Precertificate SCTs"

"certificate verify failed"

# Connect and inspect the certificate chain
openssl s_client -connect host:443 \
  -servername host 2>&1 | head -30

# Look for:
# - "Verify return code: 0 (ok)" = chain is valid
# - "Verify return code: 20" = unable to get local issuer
# - "Verify return code: 10" = certificate has expired

# Update system CA bundle
sudo update-ca-certificates     # Debian/Ubuntu
sudo update-ca-trust extract    # RHEL/CentOS

# Specify CA bundle explicitly
curl --cacert /path/to/ca-bundle.crt \
  https://host

# Python: set CA bundle
export REQUESTS_CA_BUNDLE=/path/to/ca.crt

"unable to get local issuer certificate"

# Show all certificates the server sends
openssl s_client -connect host:443 \
  -showcerts -servername host 2>&1 \
  | grep -E "^(Certificate chain| [0-9]+ s:)"

# If depth stops at 0, intermediate is missing
# Should show: 0 (leaf) -> 1 (intermediate) -> 2 (root)

# Correct order: leaf first, then intermediate(s)
cat server.crt intermediate.crt > fullchain.pem

# Nginx configuration
ssl_certificate     /etc/ssl/fullchain.pem;
ssl_certificate_key /etc/ssl/private/server.key;

# Apache configuration
SSLCertificateFile    /etc/ssl/server.crt
SSLCertificateChainFile /etc/ssl/intermediate.crt

"certificate has expired"

# Check certificate expiration from file
openssl x509 -enddate -noout -in cert.pem

# Check remote server certificate expiration
openssl s_client -connect host:443 \
  -servername host 2>/dev/null \
  | openssl x509 -noout -dates

# Check days until expiration
openssl x509 -checkend 86400 -noout -in cert.pem
# Exit 0 = valid for at least 24h; Exit 1 = expires within 24h

# Renew with Let's Encrypt
sudo certbot renew

# Check system clock
date -u
timedatectl status

# Sync NTP if clock is wrong
sudo systemctl restart chronyd   # RHEL/CentOS
sudo systemctl restart ntp       # Debian/Ubuntu

"hostname mismatch"

# View SAN from a certificate file
openssl x509 -in cert.pem -noout -text \
  | grep -A 1 "Subject Alternative Name"

# View SAN from a remote server
openssl s_client -connect host:443 \
  -servername host 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

# Generate CSR with multiple SANs
openssl req -new -key server.key \
  -out server.csr \
  -subj "/CN=example.com" \
  -addext "subjectAltName=\
    DNS:example.com,\
    DNS:www.example.com,\
    DNS:api.example.com"

SAN vs CN

# Show Subject (CN)
openssl x509 -in cert.pem -noout -subject

# Show SAN extension
openssl x509 -in cert.pem -noout \
  -ext subjectAltName

# Show both in one command
openssl x509 -in cert.pem -noout -text \
  | grep -E "(Subject:|DNS:)" | head -5

Clock Skew

Certificate validation depends on accurate system time. If the client's clock is significantly wrong, valid certificates may appear expired (notAfter in the past) or not yet valid (notBefore in the future). Clock skew is a common cause of certificate errors on embedded devices, VMs, and freshly provisioned servers.

# Check current time source (chrony)
chronyc tracking
chronyc sources -v

# Force immediate sync
sudo chronyc makestep

# Restart chrony service
sudo systemctl restart chronyd

# Alternative: ntpd
ntpq -p
sudo systemctl restart ntp

:: Check Windows time service status
w32tm /query /status

:: Force resync
w32tm /resync /force

:: Show time source
w32tm /query /source

:: Configure NTP server
w32tm /config /manualpeerlist:"pool.ntp.org" \
  /syncfromflags:manual /update

Intermediate Missing from Chain

The most common TLS misconfiguration is a server that sends only its leaf certificate without the intermediate CA certificate(s). Some clients (notably desktop browsers) work around this with AIA fetching, but most tools (curl, OpenSSL, Python, Java) will fail with "unable to get local issuer certificate."

# Check chain completeness
openssl s_client -connect host:443 \
  -servername host 2>&1 \
  | grep "verify error"

# Show full chain with certificate subjects
openssl s_client -connect host:443 \
  -showcerts -servername host 2>/dev/null \
  | grep "s:" | head -5

# Download missing intermediate from AIA
openssl x509 -in leaf.crt -noout -text \
  | grep "CA Issuers" \
  | sed 's/.*URI://'

# Order: server cert FIRST, then intermediate(s)
# Do NOT include the root CA
cat server.crt intermediate.crt > fullchain.pem

# Verify the chain
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt \
  -untrusted intermediate.crt server.crt

Web Server Configuration

server {
    listen 443 ssl;
    server_name example.com;

    # fullchain.pem = server.crt + intermediate.crt
    ssl_certificate     /etc/ssl/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/server.key;
}

<VirtualHost *:443>
    ServerName example.com

    SSLCertificateFile      /etc/ssl/server.crt
    SSLCertificateKeyFile   /etc/ssl/private/server.key
    SSLCertificateChainFile /etc/ssl/intermediate.crt
</VirtualHost>

Java Truststore Issues

Java applications use a dedicated trust store (cacerts) rather than the operating system's CA bundle. This means a certificate trusted by curl or a browser may still fail in Java if the CA is not in the JVM's trust store.

# Default location (Java 9+)
$JAVA_HOME/lib/security/cacerts

# Default location (Java 8 and earlier)
$JAVA_HOME/jre/lib/security/cacerts

# Default password: "changeit"

# List all trusted CAs
keytool -list -keystore \
  $JAVA_HOME/lib/security/cacerts \
  -storepass changeit | head -20

# Search for a specific CA
keytool -list -keystore \
  $JAVA_HOME/lib/security/cacerts \
  -storepass changeit \
  | grep -i "letsencrypt"

# Import CA cert into Java trust store
keytool -import -trustcacerts \
  -alias myca \
  -file ca.crt \
  -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit

# Verify import
keytool -list -alias myca \
  -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit

# Full SSL debug output
java -Djavax.net.debug=ssl:handshake \
  -jar myapp.jar

# More targeted debugging
java -Djavax.net.debug=ssl:handshake:verbose \
  -jar myapp.jar

# Specify custom trust store at runtime
java -Djavax.net.ssl.trustStore=/path/to/custom-cacerts \
  -Djavax.net.ssl.trustStorePassword=changeit \
  -jar myapp.jar

Debugging Tools Summary