inforge pki

Manage an environment's git-committed encrypted PKI store — the certificate hierarchies that secure the service mesh (and a daemon's trust tree). The store lives at resources/<env>/pki.enc.yaml, the structural twin of the secret store: certificates are committed as plaintext PEM (they are public), private keys as age ciphertext. See ADR-0024.

:::info Two key custodians The store records two age recipients. The offline root recipient owns cold two-tier root keys — its identity is kept offline by an operator and never reaches CI, so CI literally cannot sign with a root. The CI recipient (the same one the secret store uses, INFORGE_SECRETS_KEY) owns intermediate and root-only issuer keys, which deploy and inforge pki renew use. :::

Subcommands

Command	Purpose
inforge pki init <env>	Create the env's store. Reuses the secret store's CI recipient; generates the offline root recipient and prints its identity once.
inforge pki add <env> <name> --topology two-tier|root-only	Generate a PKI root and record it. A two-tier root is cold (encrypted to the offline recipient); a root-only root is delivered to an online issuer (encrypted to CI).
inforge pki intermediate <env> <name> <scope>	Offline, operator-run. Mint a per-scope intermediate CA for a two-tier PKI, signed by its cold root. Needs the offline root identity in INFORGE_PKI_ROOT_KEY.
inforge pki rotate <env> <name> --leaf|--intermediate <scope>|--root	Rotate a tier. --leaf documents leaf renewal; --intermediate re-mints one scope's intermediate from the cold root (offline); --root runs a dual-root overlap (--finalize ends it).
inforge pki recover-intermediate <env> <name> <scope>	Offline. Compromise recovery for one intermediate: fresh-key re-mint + forced, immediate host re-projection.
inforge pki renew <env>	Mint fresh mesh leaf certificates for every service and SSH-push them directly to each host's leaf.age. Decoupled from inforge deploy.
inforge pki ls <env>	List the PKIs in the store with their topology and the tiers present.

:::tip Operator runbooks Step-by-step procedures for adding a region and for rotating or recovering each tier live in the PKI runbooks. :::

Topologies

• two-tier — the service mesh: a cold (offline) root plus one intermediate per active scope (the global scope, and each region in regions.yaml). inforge mints short-TTL leaves from the scope's intermediate at renew time. A service joins a mesh via its pki: field.
• root-only — a single root with no intermediate, delivered to a designated online issuer (e.g. a daemon). inforge does not mint leaves for it.

Scopes and the regional boundary

A two-tier mesh keys its intermediates by scope: the literal global, or an abstract region name (e.g. us-east-1). A service's leaf is minted from its scope's intermediate — a global service from global, a regional service from each region it deploys to. Each service is also delivered the trust bundle of the scopes it may talk to (a regional service trusts {its region, global}; a global service trusts {all regions, global}), which enforces the mesh's regional boundary: intra-region ✅, region→global ✅, global→global ✅, cross-region ❌, global→region ❌.

Minting intermediates (offline)

Intermediates are signed by the cold root, so this step runs offline with the root identity that inforge pki init printed once:

export INFORGE_PKI_ROOT_KEY="AGE-SECRET-KEY-…"   # the offline root identity, kept offline
inforge pki intermediate prd wardnet-mesh global
inforge pki intermediate prd wardnet-mesh us-east-1

Commit the resulting pki.enc.yaml. inforge validate fails if a service's pki: references a mesh with no intermediate for one of the service's scopes.

Renewing leaves

export INFORGE_SECRETS_KEY="AGE-SECRET-KEY-…"   # the CI master identity
inforge pki renew prd

inforge pki renew mints a fresh leaf for every mesh member (one per scope) and SSH-pushes the material directly to each host's leaf.age, then reload-or-restarts the consumer:

• Per mesh host, every co-located member's leaf + key and the host's trust bundle land in one leaf.age at /etc/wardnet/mesh/ — this is the copy the per-host mesh proxy serves east-west traffic with (the proxy, not the service, holds mesh cert material). A host running the north-south gateway also gets the gateway's client leaf (CN=<scope>/gateway) in the same aggregate — so a scope whose only mesh member is a gateway still needs INFORGE_SECRETS_KEY and SSH reachability to renew.
• Per mtls_files: true service (the raw-mTLS-plane exception, e.g. a node↔node forward listener), additionally the service's own leaf.age under /etc/wardnet/services/<service>/mtls.

Leaves are valid for 90 days and carry a SPIFFE identity (spiffe://<base-domain>/<env>/<scope>/<member>) so peers can authorize on the encoded scope.

:::info Renew is not a deploy — schedule it inforge pki renew never runs the infra program (Pulumi), so it is safe to run while your working tree has un-shipped infra changes — but it does connect over SSH to push the renewed material to every mesh/mtls host, unlike a plain secret rotation. Every run re-mints a fresh 90-day leaf, so the effective rotation interval is how often you run it — schedule it on a cron (e.g. weekly, well inside the 90-day window) so leaves never approach expiry. The targets must already exist — run inforge deploy for the environment first; renew reuses them, never creates them. :::

How a renewed leaf reaches a running host

Renewal is push-based: inforge pki renew SSHes directly to each mesh/mtls host, writes the fresh material into its persistent leaf.age (age-encrypted to the host's own SSH key), and unconditionally reload-or-restarts the consumer — a fresh leaf always differs, so there is no hash-gating on this path.

• The mesh proxy (every host running ≥1 pki: service) receives one aggregate leaf.age at /etc/wardnet/mesh/ holding every co-located service's leaf + the trust bundle. inforge-agent decrypts it locally and reloads the mesh nginx — no downtime, and the services themselves are untouched (they hold no cert material). Because leaf.age is persistent, an ordinary reboot re-decrypts the same real material from disk with no network round-trip — a reboot's normal boot flow is the self-heal.
• An mtls_files: true service additionally receives its own leaf.age under /etc/wardnet/services/<service>/mtls, decrypted the same way, and then reloads the unit when it declares a reload: command, else restarts it.

So renewal needs SSH reachability to every target host at renewal time — inforge pki renew mints and pushes in one step, with no separate on-host timer or pull to converge afterward. Leaf private keys are decrypted only in memory at boot/reload — never written to disk in plaintext.

The deploy baseline

Right after pulumi up, inforge deploy (and inforge ephemeral up) runs a mesh baseline step: it mints the environment's mesh material with the same core as inforge pki renew and SSH-pushes each mesh host's leaf.age, then reload-or-restarts the proxy — so the freshly (re)configured proxies pick up real leaves immediately instead of serving their self-signed placeholder certificates. It needs the deploy SSH key (--ssh-key or INFORGE_DEPLOY_KEY); a failed push is reported per host — rerun inforge pki renew to retry.

The release-time mint (mtls_files services only)

An mtls_files: true service only starts running on its first inforge releases deploy. Because its boot path decrypts whatever leaf.age currently holds, that first start would crash-loop with no material yet pushed. To close the gap, inforge releases deploy mints the released service's leaf and pushes its leaf.age before it restarts the unit. Every other service — mesh member or not — skips this step: a plain mesh member's leaf is delivered to the mesh proxy by deploy/renew, independent of releases.

Rotation and recovery

Each tier rotates on its own cadence and with its own custody:

• Leaves are short-TTL — re-minting (inforge pki renew) is rotation; expiry is revocation. See rotate a leaf.
• Intermediates rotate from the cold root (offline). A planned roll (inforge pki rotate <env> <name> --intermediate <scope>) is invisible to other regions thanks to the regional boundary; a suspected key leak uses inforge pki recover-intermediate for an immediate, no-overlap replacement. See rotate an intermediate and recover a compromised intermediate.
• The root rotates with a dual-root overlap (inforge pki rotate <env> <name> --root, then --root --finalize): both roots verify during the window while root-anchoring consumers (e.g. the daemon fleet, cross-repo) come to trust the new root. Mesh services are undisturbed — re-signing preserves intermediate keys. See rotate the root.

The store file

resources/prd/pki.enc.yaml

# managed by `inforge pki` — do not edit by hand
rootRecipient: age1…        # offline operator key — cold two-tier root keys encrypt to it; CI never holds it
recipient: age1…            # the CI recipient (== the secret store's) — intermediate / root-only keys encrypt to it
pkis:
  wardnet-mesh:
    topology: two-tier
    root:
      cert: |               # plaintext PEM (public)
        -----BEGIN CERTIFICATE-----
      key: |                # age ciphertext → rootRecipient (cold)
        -----BEGIN AGE ENCRYPTED FILE-----
    intermediates:
      global:
        cert: |             # plaintext PEM
        key: |              # age ciphertext → recipient (CI)
      us-east-1:
        cert: |
        key: |
    # During a `--root` overlap only, the store also carries the retired tier:
    # previousRoots:                 # old root cert+key, dropped on --finalize
    #   - cert: |
    #     key: |
    # previousIntermediates:         # old intermediate certs (per scope), public
    #   global:
    #     - |

Because every certificate is committed in the clear, inforge validate checks a service's pki: references against the store credential-free — no decryption keys needed, the same model as vault: against the secret store.