Norway Is a Terrible Country (and Other Lessons From Processing the Entire Planet’s Geodata)

I’m building a geo-processing pipeline for work. The job: extract structured location data from OpenStreetMap for pretty much every country on Earth. Sounds straightforward. It is not.

The Norway Problem

Norway is a terrible country. I mean that technically.

OSM data for Norway maps every single godforsaken fjord, every abandoned rock sitting in the North Atlantic tagged as a dutiful natural=rock node. You parse through hundreds of megabytes of PBF data and what do you extract first? Stones. Many goddamn stones. Gneiss, granite, moraines. Including individual skerries that no human has ever set foot on — but that some Nordic mapping enthusiast with too much free time and a kayak has lovingly digitized.

By the time you get to the actually relevant features, you’ve got the entire Norwegian coastline loaded as a multipolygon in RAM. Speaking of which: the coastline in OSM is such a special case that there’s a dedicated tool just for it — osmcoastline. The coast isn't mapped as closed polygons. It's mapped as directed open ways that you get to assemble into a world polygon yourself. Norway's coastline alone has — roughly — more nodes than Luxembourg has residents.

Relative to its tiny population, the processing overhead for Norway is significantly higher than for any other European country.

The Africa Problem (Same Thing, Inverted)

Similar story in parts of Africa, just the other way around. Someone mapped a hut on the outskirts of a settlement, 25 km from the next one — and your system dutifully parses 25,000 km² of desert tagged landuse=sand, searching for a single school or an amenity=doctors. Data density is thin, but the bounding boxes are mercilessly large.

Why Overpass Won’t Save You

Using the Overpass API for filtering alone would take an estimated eight months — even with dedicated instances (unless you throw a few thousand euros at Hetzner and parallelize across smaller geo-segments). For context: planet.osm.pbf is roughly 75 GB compressed. As XML, that would be around 1.5 TB — which is why nobody uses the XML format anymore. osm2pgsql, the established C++ tool for planet imports, needs a comfortable 500+ GB of Postgres storage and days of runtime with full diff support.

My key learning here: Overpass is built for targeted queries, not for systematically plowing through planetary-scale datasets. So I built a custom OSM processor in Go — an osmium-style pipeline with PBF streaming that filters by tags and writes relevant objects directly into PostGIS. That shrinks the process from months to days.

Fun fact about PBF files: nodes come first, then ways, then relations. Building a way’s geometry means you need all referenced node IDs already in RAM — or you run two passes over the file. That alone turns the streaming approach into its own little puzzle.

Where Things Get Properly Weird

Even with the pipeline humming, you’re still sequencing through gigabytes of geodata, chewing through more or less cleanly typed polygons. Multipolygon relations with role=outer and role=inner — you get to assemble the ways into closed rings yourself. The ring assembly algorithm is its own little adventure, because ways can be arbitrarily oriented and you first have to figure out which endpoint connects to which. On top of that, there are ways that simultaneously serve as road, border, river, AND forest edge — shared geometry. One bad reference and half the relation explodes.

And then, after four days of processing: crash. Because some way in the PBF for Bulgaria wasn’t properly closed. A polygon missing its final node back-reference — a classic OSM data quality moment. So you add a sanitizing step, restart. And then Postgres blows up because PostGIS chokes on some degenerate geometry object during bulk import.

More precisely: it’s not even PostGIS itself — it’s the GEOS C++ library running underneath, throwing a TopologyException: found non-noded intersection with zero context about which of your 300,000 imported polygons is the culprit. So you reach for ST_MakeValid(), the Swiss army knife for broken geometries — which half the time produces broken geometries of its own. Or ST_Buffer(geom, 0) — the classic "make the geometry somehow valid" hack that works until it doesn't. Oh, and coordinates in OSM are internally stored as integers (× 10⁻⁷ degrees) — sounds harmless until you compute areas in projected coordinates and an integer overflow teleports your geometry to Siberia.

I Love This Stuff. Unironically.

A ton of new material to learn: PostGIS topology, PBF internals, Osmium as a C++ library, GEOS error message archaeology, and the wonderful realization that half the world is indeed mapped in OpenStreetMap — but nobody guarantees that the polygons are actually polygons.

The best part of the whole project: opening_hours in OSM is its own grammar. Mo-Fr 08:00-18:00; Sa 09:00-13:00; PH off — there's a dedicated parser for that. You don't need it. But you could.