Secrets We Forgot… Until Automation Saved Us

We All Have That One Secret… That API key that has been sitting in production for ages. The personal access token that was supposed to be rotated 2 months ago. The service key that is about to expire… wait, when does it expire again?

Most developers have experienced working with secrets. We create secrets, use them, and promise ourselves that we will rotate them. But somehow, the secret that was supposed to be rotated after 90 days is still standing strong after 6 months.

Sounds familiar? You’re not alone. According to GitGuardian’s State of Secrets Sprawl study for 2025, 70% of the secrets leaked in 2022 are still valid. This tells us the importance of having a good secrets rotation system, and that the opposite is a security risk.

When it comes to compliance, proving your rotated secrets becomes another headache. Searching through old commits to gather proof for the auditor, trying to reconstruct a timeline that should have been documented automatically.

We decided to tackle both problems at once. Using Kosli’s compliance tracking capabilities, we have created a system that handles secrets rotation and documents evidence, keeping both developers and auditors happy.

What Risks Do We Need to Mitigate?

Long-lived secrets tend to bring both security and operational risks. When credentials sit unchanged for several months to years, they become attractive to attackers that can breach your system and get hands on confidential data. This risk increases for public repositories such as Cyber-Dojo, especially if the secrets are hardcoded into the codebase.

On the operational side, there are also some serious risks. Forgotten secrets expire without warning. This will affect potentially critical and important services that customers use. Handling service outages like this, can take hours compared to the few minutes it would take with a proper rotation system.

From a SOC2 perspective, unmanaged secrets create compliance risks as well. SOC2 requires strong control on security, availability and monitoring. Secrets that are not managed well, can lead to lack of evidence during a SOC2 audit and non-compliance. By implementing automated rotation, alerts and documentation, we can mitigate the risks mentioned above, but also keep a clean audit trail.

An overview of the system

Most secrets management approaches rely on manual tracking and human memory - a recipe for forgotten rotations and expired credentials. We took a different approach: automate the entire lifecycle so the system manages itself.

Our design follows a continuous Monitor → Report → React → Document cycle. The system constantly watches for secrets that need attention, automatically notifies the team when action is required, provides clear guidance on what to fix, and captures compliance evidence as a natural byproduct of the process. This addresses both operational reliability and audit requirements without creating additional overhead for developers.

Our Solution

Over the summer, we have spent time creating a separate github secrets repository with the motive of establishing a proper secrets rotation system. Instead of relying on manual tracking and hoping that developers will remember to rotate secrets, we have made a cron job that automatically sends an alert through one of our slack channels when secrets need attention.

Since GitHub’s Secrets API only tells us when a secret was last updated (not when it expires or should be rotated), we maintain documentation files alongside our secrets. For each secret, we create a simple .txt file containing three key pieces of metadata:

secret-name: SECRET_NAME
secret-expire: 2025-01-15
is-secret: true

These files serve as our source for secret lifecycle information such as expiration dates, rotation schedules, and whether something is actually a secret or not.

This repository contains several scripts that combine and filter the secrets, leaving a json format that contains secrets that are:

• Approaching rotations deadlines (within 30 days of their required annual rotation)
• About to expire (within 30 days of expiration)
• Orphaned (known CI secrets without a corresponding .txt file)
• Documented but missing (has a .txt file but is not a GitHub secret)

The daily ritual: How it works

Every morning, our cron workflow springs into action, keeping our secrets safe and compliant. It gathers data from GitHub and our .txt documentation files, then checks which secrets need attention, notifies the team via Slack, and records attestations for compliance in a Kosli trail. Here’s a closer look at how each of these steps works:

Step 1: Data collection

The system queries the GitHub Secrets API across all our repositories (excluding archived ones) and reads our secret documentation (.txt) files.

Step 2: The blend

Our blend_secrets.py combines data from two sources:

• GitHub API data: which secrets that actually exist
• Documentation files: .txt files containing expiration dates and data for secrets

The script takes these two sources and today’s date as arguments, and adds relevant fields to the output.

For each secret, we create a comprehensive profile with key information:

{
  "name": "SECRET_NAME",
  "updated_at": "2024-08-15",
  "days_since_update": 20,
  "repo": "server",
  "scope": "org",
  "has_github_secret": true,
  "has_txt_file": true,
  "expires_at": "2025-01-15",
  "days_to_expiry": 133
}

Step 3: Updating the JSON using .txt documentation files

In the previous step, we created JSON objects with default values for each secret. Step 3 goes through that JSON and updates the entries that have corresponding .txt files. So a secret’s JSON entry changes from "has_txt_file": false to "has_txt_file": true, "expires_at" changes from the far-future default to the real expiration date, and "is_secret" might change to false if the .txt file marks it as configuration data.

Step 4: Using Kosli for compliance auditing

At this point in the process, we automatically create a Kosli attestation that captures the complete secrets status as compliance evidence. This happens seamlessly - every time our automation runs, we’re not only processing secrets but also documenting that we checked them.

We use custom attestation types in Kosli to define our secrets compliance requirements. Our compliance rules check that every secret has a txt file, isn’t approaching expiry, actually has a github secret and has been rotated within our annual cycle. When a secret fails these checks, it’s automatically flagged as non-compliant. This means when SOC2 auditors ask for evidence, we can provide months of automated compliance data instead of manually reconstructing documentation.

This approach helps developers stay focused on other tasks while maintaining a clean audit trail automatically.

Here is a screenshot of how a secrets check looks in the Kosli app:

Step 5: Filtering and alerting

The filter_secrets.py script takes the output of blend_secrets.py as an argument and filters out only those needing attention. The print_filtered_secrets_summary.py then takes the output of this and prints each secret needing attention in a simple Markdown format.
For each secret, a count is printed of how many times it appears in our workflow files, and a count of zero typically indicates that the secret is dead or that the .txt file has the wrong scope.

The end result is a smart notification system: Slack tells us when we have problems, and GitHub shows us exactly what to fix in a summary like this:

Summary

We can all agree that rotating secrets is crucial for security and reliability. Building a proper system to handle secrets might feel like a daunting task, but as we’ve shown, the investment in automation pays dividends in reduced stress and improved reliability and security. Automated tracking and alerts not only prevents orphaned or expired secrets from causing service outages, but also helps create a clear audit trail.