How I Removed Secrets from Hundreds of Repositories (Without Breaking Pipelines)

The biggest challenge wasn’t finding the secrets… it was removing them safely.

Secrets exposure is one of the most common and dangerous issues in modern DevOps environments.

And it’s rarely intentional.

Developers push code quickly. Pipelines evolve. Credentials get reused. Over time, secrets start spreading across repositories, configs, and logs.

I recently worked on a scenario where we had to identify and remove secrets from hundreds of repositories — without breaking builds, deployments, or developer workflows.

Here’s how I approached it.

Step 1: Discovery — Finding Secrets at Scale

The first step was understanding the scope.

What we found:

• Hardcoded API keys in source code
• Tokens in CI/CD configs
• Credentials inside scripts and environment files
• Secrets leaked in commit history

Tools used:

• Secret scanning tools (like GitHub Advanced Security / Snyk)
• Custom scripts for pattern matching
• Repository-wide scans

Key Insight

Most secrets are not in the latest code. They are buried in commit history.

Step 2: Risk Assessment & Prioritization

Not all secrets are equal.

We categorized them based on:

• Sensitivity (production vs test)
• Exposure (public vs private repo)
• Access level (read vs admin)

Priority order:

1. Production credentials
2. Publicly exposed secrets
3. High-privilege tokens

This helped us focus on high-risk issues first, instead of trying to fix everything at once.

Step 3: Secret Rotation (Before Removal)

Before removing any secret, we rotated it first.

Why this is critical:

If you remove a secret without rotating:

• Pipelines break
• Applications fail
• Teams lose trust in security changes

What we did:

• Generated new credentials
• Updated applications and pipelines
• Validated functionality

Only after that, we removed old secrets.

Step 4: Safe Removal from Code & History

Removing secrets is not just deleting a line of code.

The real challenge:

Secrets exist in:

• Git history
• Old branches
• Forks

Approach used:

• Removed secrets from current code
• Rewrote Git history (where required)
• Cleaned old references

Important Note

Rewriting history can break workflows if not planned properly.

So we:

• Communicated clearly with teams
• Scheduled controlled updates
• Provided instructions to developers

Step 5: Securing CI/CD Pipelines

Pipelines are one of the biggest sources of secret leaks.

Issues we found:

• Secrets printed in logs
• Hardcoded tokens in workflows
• Over-permissive access

Fixes implemented:

• Moved secrets to secure vaults
• Masked sensitive values in logs
• Restricted access using least privilege

Step 6: Centralized Secret Management

To prevent future leaks, we moved towards:

• AWS Secrets Manager / Vault
• Environment-based secret injection
• No secrets in code

Key Rule

Developers should never need to see or handle production secrets.

Step 7: Developer Enablement (Most Important Step)

This step made the biggest difference.

Instead of just fixing issues, we:

• Educated developers on secure practices
• Shared simple guidelines
• Provided ready-to-use templates

Result:

• Fewer future leaks
• Better collaboration
• Faster adoption

Final Outcome

After implementing this approach:

• Secrets were removed from hundreds of repositories
• Pipelines continued to run without disruption
• Security posture improved significantly
• Developer trust was maintained

Key Lessons Learned

• Detection is easy. Safe removal is hard.
• Rotation should always come before removal.
• Communication is as important as technical fixes.
• Prevention is better than cleanup.

Closing Line (High Impact)

Secrets don’t just leak. They stay hidden… until someone finds them.