Cold Start to First Response: JIT Provisioning on Cloudflare Containers

Waystones Cloud migrated from Fly.io microVMs to Cloudflare Containers about a month ago. The Fly.io architecture had one fundamental advantage: machines knew who they were at boot. Environment variables carried the tenant config. The machine woke up with an identity.

Cloudflare Containers don’t work that way. They boot as generic, blank images. Identity arrives with the first HTTP request. This is what we built to handle that, and what broke along the way.

The 5KB Wall and the Blank Slate Problem

Our first attempt at provisioning was straightforward: inject the pygeoapi YAML config and R2 credentials as environment variables at container start. Cloudflare enforced a hard 5KB limit on environment variables. Our config — collections, layer definitions, credentials, metadata — blew past that immediately. The container was killed before it served a single request.

The deeper problem was architectural. Even if the limit didn’t exist, Cloudflare Containers boot as a shared base image. They don’t know which tenant they belong to until traffic arrives. Env vars at start time solve the wrong problem.

Fix: JIT header provisioning. The Cloudflare Worker handling inbound traffic packs the pygeoapi YAML config as base64 into X-Waystones-Config-B64 and the R2 credentials as JSON into X-Waystones-Config. These headers arrive with every request. The container provisions itself on first contact.

The WSGI interceptor sits in front of pygeoapi’s Flask app and catches this:

def application(environ, start_response): global _CONFIG_LOADED, _pygeoapi_app if not _CONFIG_LOADED: with _lock: if not _CONFIG_LOADED: raw_config = environ.get("HTTP_X_WAYSTONES_CONFIG") if raw_config: _inject_machine_env(raw_config) b64 = environ.get("HTTP_X_WAYSTONES_CONFIG_B64") if b64: config_bytes = base64.b64decode(b64) tmp = CONFIG_PATH + ".tmp" with open(tmp, "wb") as f: f.write(config_bytes) os.replace(tmp, CONFIG_PATH) open(_TENANT_FLAG, "w").close() ...

os.replace() is an atomic rename. The tenant flag file signals to other processes that real config is on disk. Flask never sees a partial write.

The Worker Coordination Problem

Gunicorn runs two workers — separate OS processes, isolated memory, shared disk. A browser opening the map fires multiple concurrent requests. Worker 1 catches the first, starts writing config. Worker 2 catches the second a millisecond later.

The threading lock inside each worker guards against thread-level races within that process. But between workers, both processes independently enter the initialization path.

This is handled by three things in combination:

1. os.replace() - atomic at the filesystem level. Both workers writing identical config bytes is harmless.
2. _TENANT_FLAG - once written by whichever worker gets there first, the other worker's next check sees it and skips the header path entirely, loading from disk instead.
3. _CONFIG_LOADED + _lock - double-checked locking within each worker ensures Flask loads exactly once per process.

The second worker path in the interceptor:

elif os.path.exists(CONFIG_PATH) and os.path.exists(_TENANT_FLAG): print(f"[waystones_wsgi] Using existing config at {CONFIG_PATH}", flush=True) _ensure_openapi_ready()

No sleep timers. No polling. The flag file is the cross-process signal.

The OpenAPI Catch-22

pygeoapi requires an OpenAPI document to serve its collections index. Generating that document requires the config. The config doesn’t exist until the first request. The first request needs the collections index to succeed.

The interceptor resolves this synchronously before handing off to Flask:

def _ensure_openapi_ready() -> None: if not _is_stub_openapi(): return # Fast path: pull from R2 cache subprocess.run(["python3", "/cache_openapi.py", "--download-only"], ...) if not _is_stub_openapi(): return # Slow path: generate synchronously tmp = OPENAPI_PATH + ".tmp" with open(tmp, "w") as f: subprocess.run(["pygeoapi", "openapi", "generate", CONFIG_PATH], stdout=f, check=True) os.replace(tmp, OPENAPI_PATH)

Fast path: the Waystones Cloud backend pre-generates the OpenAPI document from model.json at deploy time - a ~2ms TypeScript function - and uploads it to R2. The interceptor downloads it in ~100ms and skips Python generation entirely.

Slow path: generation runs synchronously and blocks the first request. This only happens on a cache miss. Once generated, a background task uploads it to R2 so the next cold start hits the cache. Atomic rename again — Flask either sees the old stub or the complete document.

The GDAL Blackhole

QGIS Server is a separate container, fronted by its own Python proxy. Same JIT pattern: X-Waystones-Qgis-B64 carries the QGIS project file as base64, X-Waystones-Config carries the R2 credentials.

The proxy writes credentials to three places before spawn-fcgi launches: os.environ (inherited by the child process), /tmp/qgis-env.sh (sourced by the QGIS wrapper script), and nginx's fastcgi_params (passed per-request to QGIS).

The ordering matters. Without credentials in os.environ before spawn-fcgi forks, GDAL boots without AWS keys. When GDAL lacks credentials and needs to authenticate against S3, it assumes it's running on EC2 and tries to reach the instance metadata endpoint at 169.254.169.254. Cloudflare's network drops those packets silently. GDAL waits. The container hangs for exactly 120 seconds before timing out.

Fix:

def _inject_credentials(headers) -> None: os.environ["AWS_EC2_METADATA_DISABLED"] = "true" ...

First line of the function, before anything else. GDAL skips the metadata check and authenticates directly against R2.

The Dual-Mode Contract

Both containers run on Cloudflare for managed deployments and on Docker Compose for self-hosted users. The same image, the same code, different initialization paths.

For the QGIS proxy:

if __name__ == "__main__": mode = os.environ.get("WAYSTONES_MODE", "cloud") if mode == "open-source" and os.path.exists(PROJECT_PATH): _inject_credentials({}) # triggers env var fallback if _start_qgis_stack(): _STARTED = True

OSS mode detects an existing project file and starts eagerly, reading credentials from container environment variables instead of request headers. Cloud mode waits for the first request.

The WSGI interceptor has the same fallback: if PYGEOAPI_CONFIG_B64 is set as an env var, boot.sh decodes and writes it before Gunicorn starts, sets the tenant flag, and the interceptor sees real config on disk from the first request. The same image runs on Railway, Render, or any container host by pointing env vars at the right values.

The fast/slow path logic for OpenAPI stays identical in both modes. The only question is whether the hostname was known when the data was processed — if it was, the cache hits. If it wasn’t, Python generates it and primes the cache for next time.

Result

Container boot → first request arrives → WSGI interceptor catches it → Credentials injected into os.environ → Config written atomically to disk → Tenant flag set → OpenAPI downloaded from R2 (100ms) → Flask loads → Background: warmup, asyncapi, S3 upload → 200 OK

The QGIS container follows the same sequence in parallel, with AWS_EC2_METADATA_DISABLED ensuring GDAL authenticates in milliseconds rather than hanging for two minutes.

The problems were: a 5KB platform constraint, a blank-slate boot identity, a cross-process worker race, a config-before-config catch-22, and a silent network blackhole in GDAL’s authentication fallback. None of them required more hardware. Most of them required knowing exactly when in the boot sequence each piece of state becomes available — and making sure the code respects that order.

Originally published at https://waystones.cloud.