The S3A Alias Gap: Why Your Spark Job Can’t Read What the AWS CLI Just Listed

S3 access points: one bucket, per-consumer policy Three consumers, three access points, one shared bucket. Each access point has its own policy. Based on AWS S3 access point patterns described in the AWS S3 user guide.

A Spark job fails with path-not-found. Dropped into the AWS CLI to check, the same path resolves: aws s3 ls lists the object without complaint. Two clients, two different answers, against the same identifier. The object is either there, or it isn't.

This disagreement between the command line and the application runtime is a recurring shape of failure. When the storage layer says an object isn’t there, but the CLI says otherwise, the cause is almost always in how a client layer translates the same identifier into a wire request. The disagreement is, by itself, useful information: it points away from the storage layer and toward the client stack. The trick is reading it that way at the time of failure.

Why access points exist

Before getting to the failure, it helps to remember why you would be using an access point in the first place. The AWS pattern for sharing a bucket across multiple teams, applications, or partners is to give each consumer its own access point, rather than expand a single bucket policy as new consumers are added. Per the AWS documentation, an access point is a named network endpoint attached to a bucket, with its own access policy that works alongside the bucket policy, its own VPC restrictions, and its own block public access settings. Per-consumer scoping goes down to the prefix: an access-point policy’s resource ARN can use the accesspoint/<name>/object/<prefix>/* format to restrict a consumer to a specific prefix in the bucket, without anyone touching the bucket policy.

Each access point has two primary symbolic identifiers in the data plane: an ARN and an auto-generated alias. AWS designed the alias as a drop-in for the bucket name. aws s3 ls, GetObject, and PutObject all work against the alias the same way they work against a bucket name. From the AWS protocol layer's point of view, the alias is a peer of the bucket name.

Where the connector layer sits

This is where the layering becomes consequential. The S3A connector is not the AWS SDK. S3A sits above the SDK, takes an s3a:// URI, parses the bucket-position string, and builds its own request, which the SDK then sends on the wire. What S3A treats as the bucket-position string, and how it resolves that string into a request, is a property of S3A, not of the AWS data plane.

For S3 access points, the documented S3A configuration is ARN-based. You set a per-bucket property:

fs.s3a.bucket.example-ap.accesspoint.arn = arn:aws:s3:us-east-1:123456789012:accesspoint/my-access-point

The bucket-position segment in the s3a:// URI (here, example-ap) is a logical name S3A uses to look up the configured ARN. As the current connecting guide puts it, the bucket name in the s3a:// URL is not used when connecting to the store; the wire request is built from the ARN. The pattern is ARN-based and was added in Hadoop 3.3.2.

The result is a layering mismatch that is easy to miss. AWS made the alias a first-class identifier; the SDK and CLI honor it. S3A’s documentation does not describe handling the alias the same way. The CLI works against the alias; the S3A job fails. The documentation gap is where the confusion lives.

Where this shows up

The contradiction surfaces in two specific patterns. A third consideration sits next to them.

Pattern 1: the alias in the s3a:// URI. The application's s3a:// URI uses the access point alias in the bucket position (something like s3a://my-access-point-aqfqprnstn7aefdfbarligizwgyfouse1a-s3alias/data/foo). Direct use of the alias in the URI is not part of the documented S3A configuration. The open Hadoop ticket for whether it should be is HADOOP-18273, reported in June 2022 and still unresolved against the current stable release. The job fails with path-not-found; the CLI succeeds against the same alias because the AWS SDK layer treats it as a valid data-plane identifier and the connector does not.

Pattern 2: drift in the configured ARN. The application is properly configured to use a logical name in the URI and a per-bucket ARN property in the Hadoop configuration. The configured ARN, though, does not match the access point the engineer believes it points to. In one investigation, the drift was a one-character typo in the access point binding. Engineers were running the verification command against the alias they thought was configured; Spark was issuing requests through a slightly different access point binding. The CLI test passed against the engineer’s mental model; the job failed against what was actually configured. The error message named the resolved path truthfully and named nothing else.

A version-floor consideration. Two defects in the S3A access-point code paths are worth being past, regardless of which pattern above is in play. The parent ticket for S3 access point support, HADOOP-17198, explicitly flags two follow-up patches as required when backporting: HADOOP-17951 (an existence check that always returned false) and HADOOP-18085 (an SDK-upgrade-induced ARN endpoint mistranslation). The practical floor to pick up both is hadoop-aws 3.4.0 or later with a matching aws-java-sdk bundle.

What better looks like

A few changes would make this category of failure easier to catch, and faster to diagnose when it shows up.

Use the documented ARN-based configuration, not alias as path. S3A’s documented pattern for access points is the per-bucket ARN property. Set fs.s3a.bucket.<logical-name>.accesspoint.arn to the access point ARN, and use <logical-name> as the bucket segment in the s3a:// URI. The <logical-name> is an identifier you choose; the wire request is built from the configured ARN, not from the URL. Direct use of the access point alias in the URL is not part of the documented pattern. Pattern 1 is what comes of that.

Pin hadoop-aws and the AWS SDK to versions past the access-point defects. HADOOP-17198 explicitly flags HADOOP-17951 and HADOOP-18085 as required follow-up patches. The practical floor is hadoop-aws 3.4.0 or later with a matching aws-java-sdk bundle. This pin does not enable alias-URI resolution in S3A; it removes two known defects in the surrounding code paths.

Validate the configured access point identifier across both clients before deploying. If aws s3 ls works against the configured identifier and the application client does not, that disagreement is itself diagnostic. A short pre-deploy check that runs the configured identifier through both clients catches the worst version of this category of failure before it ever reaches production.

Log the resolved request the connector actually issued. For S3A or any client that does URI translation between what the application configured and what the wire request looks like, the resolved bucket-position string, the resolved ARN, the endpoint host the SDK targeted, and the AWS request ID are the most valuable things in the log line. With those fields in the log, the next failure of this shape takes seconds to diagnose, not hours.

Closing

A disagreement between the CLI and S3A on the same identifier is information, not noise. When you see one, the next questions are concrete:

• Is the application’s s3a:// URI using the access point alias directly, or the documented ARN binding through a per-bucket property?
• What identifier is the connector issuing on the wire, compared to what the CLI is issuing?
• Is hadoop-aws past the access-point follow-up patches (HADOOP-17951, HADOOP-18085)?

Answer those at the time of failure, and the cross-layer triage that this category of failure usually triggers becomes a configuration check. Undocumented behavior is fragile behavior; the documented ARN-based per-bucket configuration is the safe path, and the question of whether S3A should resolve the alias the way the SDK does has been open in the Hadoop community since 2022.