Code Signing Infrastructure

Motivation

One of the reasons that I often see other systems recommended to new users is that the software required to support a large number of common devices is harder to set up and more troublesome to maintain on Fedora than it is on other systems.

That makes sense to me. I’m not a huge fan of DKMS/akmods. With an SRE background, I tend to believe that software should follow a build->test->deploy sequence, but installation from source (as in DKMS/akmods) is more of a deploy->build->test sequence. If the build process is disrupted (e.g. power loss, reboot during the silent background build, running out of free disk space, etc), or if tests fail, recovery options are limited because the software has already been deployed. That’s not a recipe for reliability.

Moreover, I advocate the use of Secure Boot. While there are systems in place to support local builds on a host that has Secure Boot enabled, they require the signing key to be located on the host, and usable by automated processes. If your package manager can build and sign kernel modules, then a rootkit can do the same thing. These systems defeat the purpose of the Secure Boot system.

I want:

  • a system that can build and sign code for use with Secure Boot
  • using an HSM so that the signing key cannot be exfiltrated
  • providing transparency logs so that the key cannot be quietly misused
  • on infrastructure that can be readily forked, deployed, discussed, and improved

The Challenge: Securing Code Signing in CI/CD

Code signing is fundamental to software security—it proves that binaries haven’t been tampered with and come from a trusted source. But some signing configurations have a critical vulnerability: the private key must exist somewhere, and wherever it exists, it can potentially be stolen.

This project tackles that problem head-on by building a code signing infrastructure where:

  • Keys cannot be exfiltrated
  • Keys cannot be used by unauthorized repositories
  • Every signing operation is transparent and auditable

The Solution: Hardware-Backed Signing with Public Transparency

The architecture uses AWS KMS (Key Management Service) asymmetric keys—RSA-4096 keys backed by FIPS 140-2 Level 2 validated hardware security modules (HSMs). The private key material never leaves the HSM boundary. Ever. Not in memory, not on disk, not over the network.

Core Security Properties

1. Key Exfiltration is Cryptographically Impossible

Unlike traditional signing where a private key file exists on disk (even if encrypted), AWS KMS keys exist only within HSM boundaries:

  • The private key is generated inside the HSM
  • All signing operations happen inside the HSM
  • The private key material is never exposed in plaintext
  • Even AWS administrators cannot extract the key
  • The key can be used only via authenticated AWS API calls

This means an attacker who fully compromises a build runner gets nothing. No key file to steal. No memory to dump. No credentials that grant signing access outside the controlled environment.

2. Cryptographic Transparency Logs

Every signing operation is automatically logged to a public S3 bucket via a Lambda function that processes CloudTrail events:

{
  "version": "1.0",
  "timestamp": "2024-01-15T12:00:00Z",
  "kms_key_id": "arn:aws:kms:us-east-1:...:key/...",
  "signing_algorithm": "RSASSA_PKCS1_V1_5_SHA_256",
  "message_digest_sha256": "abc123...",
  "signature_base64": "def456...",
  "record_hash": "789ghi..."
}

These logs provide:

  • Public auditability: Anyone can verify what was signed and when
  • Non-repudiation: The signature proves the key owner signed the digest
  • Tamper evidence: S3 versioning ensures logs cannot be silently modified
  • Cryptographic proof: Each log includes the signature that can be verified with the public key

The Lambda function sanitizes the logs to exclude:

  • Source IP addresses
  • IAM identities
  • Request context
  • Any sensitive AWS metadata

Only the cryptographically verifiable facts are published.

The Architecture: Defense in Depth

The system implements multiple security layers that work together:

Network Isolation

Runners operate in a public subnet but with security groups that block all inbound traffic. The runner pulls CI jobs and pushes artifacts.

Note: During early development of the architecture, administrative access is available via AWS Systems Manager Session Manager. The final production deployment is intended to provide no interactive access to the runner.

VPC endpoints keep AWS service traffic within the AWS network:

  • S3 endpoint (Gateway type): No data transfer charges, private S3 access
  • KMS endpoint (Interface type): KMS operations never traverse the public internet
  • Secrets Manager endpoint: Runner tokens retrieved privately
  • CloudWatch Logs endpoint: Monitoring traffic stays private

IAM Least Privilege

Each runner has a dedicated IAM role with minimal permissions:

Code Signing Runner can:

  • Call kms:Sign
  • Retrieve the public key via kms:GetPublicKey
  • Write to the transparency logs S3 bucket
  • Read the Forgejo runner token from Secrets Manager

Immutable Audit Trail

Multiple overlapping audit systems ensure complete accountability:

  1. CloudTrail: Logs every KMS API call with AWS identity, IP, timestamp
  2. CloudWatch Logs: Real-time streaming of signing operations
  3. S3 Transparency Logs: Public, versioned, immutable records
  4. S3 Access Logs: Track who reads the transparency logs
  5. Lambda Execution Logs: Record transparency log publication

All logs are encrypted at rest, and S3 versioning means modifications are visible in the version history.

Auditability and Compliance

The transparency logs enable:

Public Accountability: Anyone can verify that signatures are legitimate by:

  1. Fetching the transparency log entry
  2. Downloading the signed artifact
  3. Computing its SHA-256 hash
  4. Verifying it matches the logged digest
  5. Verifying the signature with the public key

Incident Response: If a compromised binary is discovered:

  1. Find it in the transparency logs (indexed by date)
  2. Identify the exact timestamp
  3. Review CloudTrail for the signing operation
  4. Determine the source (instance ID, IAM role, Forgejo workflow)
  5. Investigate the build that produced the artifact

Compliance: The architecture supports:

  • SOC 2 (audit logging, encryption, access control)
  • ISO 27001 (security controls, monitoring, incident response)
  • FIPS 140-2 Level 2 (KMS hardware-backed keys)
  • Non-repudiation requirements (cryptographic signatures + immutable logs)

Building Trust Through Transparency

Code signing is fundamentally about trust. Users need to trust that the software they run is legitimate and hasn’t been tampered with. But traditional signing approaches require trusting that the private key is kept secure—a trust that’s regularly violated.

This infrastructure shifts the trust model. Instead of “trust that the key is secure,” it’s:

  • Trust the cryptographic impossibility of extracting KMS keys
  • Trust the mathematical proof of signatures verified by public keys
  • Trust the audit trail in public transparency logs
  • Trust the infrastructure-as-code that can be reviewed and reproduced

The security doesn’t depend on secrecy. The entire architecture is public (this repository, the transparency logs, the public keys). Security comes from cryptographic properties and defense in depth.

For anyone building CI/CD infrastructure for security-critical artifacts—whether kernel modules, container images, firmware, or applications—this architecture provides a template for signing without the risk of key compromise.

The code is here: https://codeberg.org/gordonmessmer/signed-code-build-stack

The transparency logs are here: (configured per deployment)

Technical Appendix

References