Self-Healing Infrastructure Security

Autonomous Detection, Response & Recovery on AWS using Wazuh and a SOAR Orchestrator

1. Executive Summary

This document describes the design, build, and validation of a self-healing infrastructure security system deployed on AWS. The system automatically detects security compromises on cloud workloads and initiates an autonomous remediation response — restoring configuration, redeploying compromised compute resources, and rotating credentials — without requiring manual intervention.

The design is inspired by biological immune systems: a sensing layer continuously monitors the environment for signs of compromise, a decision layer interprets those signals, and an effector layer carries out a coordinated response. All core components were individually built and tested, culminating in a fully autonomous end-to-end test in which a simulated file-integrity violation on a monitored instance triggered detection, automatic webhook invocation, and execution of the remediation workflow with no manual step.

A note on terminology: this proof of concept demonstrates self-healing principles through automated detection and remediation. It does not yet verify that a remediation action has actually restored the workload to a trusted, healthy state -a complete self-healing solution would also include a post-remediation validation and health-check step, which is identified as future work in Section 7.

1.1 Outcome Summary

2. System Architecture

The system is composed of four logical layers, mirroring the structure of a biological immune response:

2.1 High-Level Data Flow

1. A Wazuh agent installed on a monitored EC2 instance detects a file integrity or system anomaly (for example, an unauthorized change under /etc).

2. The Wazuh manager correlates the event against detection rules and raises an alert (rule IDs 550 — “Integrity checksum changed” and 554 — “File added to the system”).

3. A custom Wazuh integration script (custom-soar) is invoked automatically for alerts matching those rule IDs. It maps the affected agent to the corresponding AWS EC2 instance ID and sends an HTTPS POST request to a public webhook.

4. The webhook is served by Amazon API Gateway, which forwards the request to an AWS Lambda function (the SOAR orchestrator).

5. For demonstration purposes, this proof of concept has the orchestrator Lambda execute all three remediation actions on every trigger, each independently error-handled:

○ Restore configuration — an AWS Systems Manager Automation runbook re-syncs a known-good configuration baseline from Amazon S3 onto the affected instance.

○ Redeploy compute — the affected instance is terminated from its Auto Scaling Group, which automatically launches a replacement instance from a trusted “golden” AMI.

○ Rotate credentials — AWS Secrets Manager rotates the associated application secret via a dedicated rotation Lambda function.

Running all three actions on every trigger is a simplification made for this proof of concept, so that each remediation path could be exercised and demonstrated. A production implementation would typically be selective: the orchestrator would inspect the type and context of the detected threat (for example, a configuration-only change versus a suspicious process or a credential-exposure indicator) and invoke only the action or combination of actions appropriate to that threat, rather than unconditionally executing restore, redeploy, and rotate together.

6. The orchestrator publishes a summary of the actions taken to an Amazon SNS topic, which delivers an email notification for audit purposes.

7. Every action is logged to Amazon CloudWatch Logs, providing a complete, timestamped audit trail of the automated response.

Rules 550 and 554 were chosen as the trigger because they are Wazuh’s core file-integrity indicators-a checksum change and a newly added file are both strong, low-noise signals of unauthorized modification to a monitored path. In this build, either rule firing on its own is sufficient to invoke the full remediation workflow. In a production deployment, triggering disruptive actions such as instance replacement or credential rotation from a single low-level file-integrity alert is generally considered too aggressive: additional context or correlated detections (for example, an unexpected process, a suspicious outbound connection, or repeated alerts across multiple rules within a short window) would typically be required before an automated response escalates to destructive remediation.

2.2 Component Details

2.2.1 Wazuh (Detection Layer)

● Manager: a dedicated EC2 instance running the all-in-one Wazuh stack (manager, indexer, and dashboard), reachable over HTTPS for analyst access.

● Agent: installed on every monitored workload; configured with real-time File Integrity Monitoring (syscheck) on sensitive directories (/etc, /bin, /sbin, /usr/bin, /usr/sbin, /boot).

● Custom Integration: a Python script registered under <integration> in ossec.conf, scoped to specific rule IDs, which converts a Wazuh alert into an HTTPS call to the SOAR webhook.

2.2.2 API Gateway + Lambda (Orchestration Layer)

● Amazon API Gateway (HTTP API) exposes a single public endpoint that accepts POST requests and forwards them to the orchestrator Lambda.

● The orchestrator Lambda (Python 3.12) parses the incoming payload — whether delivered directly or wrapped inside an API Gateway event body — and extracts the target instance ID before invoking each downstream AWS service.

● Function timeout is configured at 60 seconds to allow all three downstream calls to complete.

2.2.3 Remediation Actions (Effector Layer)

2.3 Identity & Access Management Design

Access is scoped using dedicated, purpose-built IAM roles rather than a single broad role:

An explicit iam: PassRole permission, scoped to the SSM automation role’s ARN, is granted to the orchestrator Lambda’s role — required whenever one role delegates execution to another AWS service.

2.4 Security Considerations & Hardening Notes

● Resource scoping: current IAM policies use “Resource”: “*” for several actions to simplify the proof-of-concept build; production use should scope these to specific ASG, secret, and automation-document ARNs.

● False-positive risk: because the orchestrator automatically terminates and rotates credentials on alert, a false positive can cause unnecessary disruption. A human-approval gate (e.g., via AWS Step Functions) is recommended before destructive actions in production.

● Least privilege: separate, narrowly-scoped roles are used per function rather than a single administrative role.

● Transport security: the webhook between Wazuh and AWS is served over HTTPS via an API Gateway. However, the endpoint currently accepts any request over HTTPS with no additional authentication; a production deployment should add an authorizer (API key, HMAC request signing, mutual TLS, or IAM/SigV4 auth) and, where possible, restrict the endpoint to expected source IPs so that only the Wazuh manager can invoke it.

● Redeploy-compute assumptions: the redeploy action assumes a stateless workload managed by an Auto Scaling Group, where any instance can be replaced by a fresh copy of the golden AMI without data loss. It is not safe to apply as-is to stateful workloads (e.g. instances holding local application data or session state) without first adding a data-persistence or migration strategy.

● Restore-configuration limitations: syncing the S3 baseline back onto the instance repairs known configuration files but does not remove an attacker’s underlying foothold (for example, a planted process, cron job, or credential left outside the restored paths) and does not perform malware scanning. In production, this action is best paired with the redeploy-compute action, or with a broader incident-response process, rather than relied on alone.

● Scaling beyond a static mapping: the custom-soar integration currently maps each Wazuh agent to its EC2 instance ID using a static, hand-maintained mapping. Scaling this to a fleet would require replacing the static mapping with a dynamic lookup (for example, tagging instances with their Wazuh agent ID and querying the EC2 API at alert time) so that new agents and instances are recognized automatically.

● Auditability: every remediation cycle produces both a CloudWatch log entry and an SNS email notification.

3. Deployed Resource Inventory

3.1 Compute

3.2 Identity & Access Management

3.3 Storage, Automation & Secrets

3.4 Orchestration, Messaging & API

3.5 Wazuh Detection Rules Used

4. Implementation & Evidence

This section walks through each build stage in the order it was performed, with supporting screenshots captured directly from the AWS Console and terminal during the build.

4.1 Compute Provisioning -Wazuh Manager & Monitored Agent

The Wazuh manager and the monitored agent instance were provisioned as EC2 instances in the same VPC. The manager runs the all-in-one Wazuh stack; the agent instance runs the monitored Ubuntu workload with the Wazuh agent enrolled against the manager.

4.2 Wazuh Agent Enrollment & Detection Configuration

The agent (“Selfheal01”, agent ID 001) was enrolled with the manager and confirmed Active in the Wazuh Endpoints dashboard. Real-time File Integrity Monitoring (syscheck) was then enabled on system directories.

4.3 Detection Validation

A test file was created under /etc on the monitored instance to simulate an unauthorized change. The Wazuh dashboard confirmed detection via rule 550 (“Integrity checksum changed”) and rule 554 (“File added to the system”).

4.4 IAM Roles & Policies

Three purpose-built IAM roles were created to support the SOAR orchestrator, the SSM automation runbook, and (later) the secret rotation Lambda, each with a narrowly-scoped inline policy.

4.5 Golden Configuration Baseline (S3)

A dedicated S3 bucket (selfheal-golden-config-800153536717) was created to hold the known-good configuration baseline used by the Restore Configuration action.

4.6 Restore Configuration Action (SSM Automation)

An SSM Automation document (selfheal-RestoreConfig) was authored to sync the S3 baseline back onto a target instance. Registering the monitored instance with Systems Manager required attaching an EC2 instance profile (selfheal-ec2-ssm-role) and restarting the SSM agent.

4.7 Redeploy Compute Action (Golden AMI, Launch Template, Auto Scaling Group)

A golden AMI was captured from the clean monitored instance, wrapped in a launch template, and used to create an Auto Scaling Group with a desired capacity of 1.

To validate autonomous redeployment, the ASG-managed instance was manually terminated. The Auto Scaling Group’s activity log confirmed it automatically launched a replacement instance from the golden AMI to restore desired capacity.

4.8 Rotate Credentials Action (Secrets Manager + Rotation Lambda)

A Secrets Manager secret (selfheal/app/db-creds) and a dedicated rotation Lambda (selfheal-rotate-secret) were created. The rotation function was tested directly, generating a new random password and updating the secret.

4.9 Audit & Notification (SNS)

An SNS topic (selfheal-alerts) was created with an email subscription to receive a human-readable summary of every remediation cycle. The subscription confirmation email initially landed in the Spam folder and required manual confirmation.

4.10 SOAR Orchestrator Lambda

The orchestrator Lambda (selfheal-soar-orchestrator) combines all three remediation actions plus the SNS notification step. A manual test invocation with a valid instance ID confirmed all three actions completed successfully in a single run.

4.11 Public Webhook (API Gateway) & Wazuh Integration

The orchestrator was exposed via an API Gateway HTTP API (selfheal-soar-api), producing a public webhook URL. A custom Wazuh integration script was then registered on the manager to call this webhook automatically whenever rule 550 or 554 fires.

5. Testing & Validation

5.1 Detection Test (Wazuh File Integrity Monitoring)

Procedure: On the monitored instance, created a new file under /etc and appended content to it.

sudo touch /etc/test-attack.conf echo “malicious change” | sudo tee -a /etc/test-attack.conf

Result: The Wazuh dashboard’s Security Events view showed alerts for rule 550 and rule 554 within seconds, attributed to agent Selfheal01 (see Figure under Section 4.3).

5.2 Restore Configuration Test (SSM Automation)

Procedure: Manually executed the selfheal-RestoreConfig automation document against the monitored instance.

5.3 Redeploy Compute Test (Auto Scaling Group)

Procedure: Manually terminated an instance that was a member of the Auto Scaling Group (selfheal-app-asg).

Result: The ASG’s activity history recorded a “Terminating” event followed automatically by a “Launching a new EC2 instance” event; a new healthy instance passing 3/3 status checks appeared within approximately two minutes, restoring desired capacity of 1 (evidence in Section 4.7).

5.4 Credential Rotation Test (Secrets Manager)

Procedure: Manually invoked the selfheal-rotate-secret Lambda function against the target secret.

Result: Execution succeeded; retrieving the secret value in the Secrets Manager console confirmed db_password had changed to a newly generated random value (evidence in Section 4.8).

5.5 Full Orchestrator Test (Manual Invocation)

Procedure: Invoked the selfheal-soar-orchestrator Lambda directly with a test event containing a valid instance ID.

Result: All three remediation actions completed successfully in a single invocation, and an SNS email notification was received (evidence in Section 4.10).

5.6 End-to-End Autonomous Test

Procedure: Simulated a compromise on the monitored instance (as in Section 6.1) with no manual triggering of any downstream component.

The CloudWatch log group for the orchestrator function showed a new invocation whose timestamp matched the Wazuh alert time exactly (accounting for the UTC / local time offset), confirming the trigger was fully autonomous — no manual Lambda invocation was made for this test.

Note on the redeploy_compute result: the monitored instance used for this test was intentionally deployed as a standalone EC2 instance, outside the Auto Scaling Group, so that file-integrity monitoring could be validated on the same workload used throughout Sections 4 and 5 without risking disruption to the ASG-managed fleet. One consequence of this design choice is that the autonomous end-to-end test did not validate the redeploy_compute action’s actual execution on that same workload — the Auto Scaling API correctly rejected the termination request with “No managed instance found,” which is expected behavior given the instance’s placement, not a defect. The redeploy action is only applicable to instances under Auto Scaling Group management, and its successful execution was separately validated against a genuinely ASG-managed instance in Section 4.7 / 5.3, rather than as part of this autonomous, same-workload test.

The final confirmation email received for this test:

6. Conclusion

Every stage of the self-healing pipeline — detection, automatic triggering, orchestration, and all three remediation actions — has been individually validated and confirmed to operate correctly in combination during an unassisted, end-to-end test. The system successfully demonstrates autonomous detection-to-recovery behavior consistent with the project’s original design goals.

As a proof of concept, the system demonstrates self-healing principles rather than a complete self-healing solution: it does not yet confirm that a remediated instance is actually back in a trusted, healthy state, and several of its production-readiness gaps are noted in Section 2.4 (single-signal triggering, stateless-workload assumptions, restore-configuration limitations, the static agent-to-instance mapping, and the unauthenticated public webhook). Closing these gaps, most importantly adding a post-remediation health-verification step, correlated multi-signal triggering, and webhook authentication, would be the primary focus of hardening this proof of concept for production use.

About the Author

Maryam Liaqat (Jr. Security Engineer & Wazuh Ambassador) Passionate about simplifying complex deployments and helping others learn DevSecOps practices. Connect with me for consulting, troubleshooting, or collaboration opportunities.

Learn more about Wazuh and the Ambassador Program:

🔗 Learn more about Wazuh here: Wazuh

🔗 Discover the Wazuh Ambassador Program: Wazuh Ambassador