How We’re Teaching Neural Networks to Anchor AR Objects on Skyscrapers

Or: why standard Augmented Image breaks the moment you try to put a butterfly on the Burj Khalifa

When we started building AR installations on real buildings in Dubai, we ran into a problem no tutorial talks about: every standard Augmented Image tool stops working past 5 meters.

If you want to bring a painting in a gallery to life — perfect. But the moment you want to place an AR object on a skyscraper’s signage, animate a mural on a building facade, or create an AR installation next to the Burj Khalifa — the standard stack breaks.

We spent six months looking for a solution. Here’s where the technology breaks, what we tried, and why neural networks aren’t (yet) the perfect answer.

What Augmented Image is and why it’s everywhere

Augmented Image is one of the most widely used AR modes. The logic is simple:

1. You create a reference image — a picture that becomes an “anchor” in the real world
2. You upload it together with a 3D model into an AR session
3. The app searches for this image in the camera’s video feed
4. Once the image is recognized, a 3D model gets attached to it
5. Every frame, the image’s position is refined

The result: a 3D object looks like it’s physically attached to a real-world surface. A poster comes to life. A museum painting starts telling its own story. A product package turns into an AR presentation.

This works out of the box on both Android (ARCore) and iOS (ARKit). Fast, reliable, no server required.

Perfect solution — except for two limitations.

Limitation #1: 5-meter range

Augmented Image is designed for close distance — roughly up to 5 meters.

Works great for:

• Paintings in galleries
• Posters and prints
• Product packaging
• Restaurant menus

Completely fails for:

• Building signage
• Large murals on facades
• Billboards
• Logos on skyscrapers

We worked on a project with butterflies flying around the Burj Khalifa. Standard Augmented Image physically cannot recognize the building’s logo from 50–100 meters away — a phone camera simply can’t capture enough detail.

Limitation #2: not every image works

The second limitation is more technical: Augmented Image algorithms rely on keypoints — high-contrast areas, sharp edges, distinctive shapes.

If an image is “sparse” in keypoints — a minimalist logo, monochrome composition, smooth gradients — the app just can’t recognize it.

In practice, this means: half of brand materials don’t work with AR out of the box. And it’s a massive problem, because brands specifically invest millions into exactly these kinds of minimalist visual identities.

Our hypothesis: replace computer vision with neural networks

We asked ourselves: what if we use neural networks for recognition instead of classical CV algorithms?

Neural networks:

• Work with the general “understanding” of an image, not just keypoints
• Can recognize objects in complex lighting conditions
• Can be trained for specific projects

But there are obvious downsides:

• Neural networks are slower than classical algorithms
• Most modern models require server-side computation
• You can’t track an image every frame in real time

We accepted the compromise: build not a full Image Tracking, but a “pseudo-version” of it.

Here’s the logic:

1. Neural network recognizes the image (on the server, with some latency)
2. We get the object’s real-world coordinates
3. We place a 3D model at those coordinates inside the AR scene
4. Every few seconds, we re-run recognition to refine the position

This doesn’t give the instant tracking of classical Augmented Image, but it opens up the possibility to work with objects that were previously off-limits.

What we’ve achieved so far

Breakthrough #1: recognizing images with almost no keypoints.

Our network learned to work with minimalist logos and monochrome compositions that ARCore and ARKit simply ignore. This opens up work with brands that have clean visual identities — which is most of the premium segment.

Breakthrough #2: server-side recognition with coordinate mapping into the AR scene.

We built a pipeline: phone sends a frame to the server, the network recognizes the object and its position, coordinates are returned to the device, the 3D model is placed in the AR scene in the correct spot.

It works. Not instantly, but it works.

What’s still on our plate

An honest list of unsolved problems:

1. Recognition speed. Right now, the time between frame upload and coordinate return is too long for genuinely smooth UX. We’re working on pipeline optimization and reducing latency.

2. Recognition accuracy. Still worse than classical computer vision. The neural network sometimes “misses” by several dozen centimeters, which is critical for precise 3D object placement.

Both problems are solvable — it’s a matter of iterations and choosing the right model architectures.

Where we go from here

We’re continuing to experiment with a hybrid approach: classical CV for close-range, speed-critical work, neural networks for distant objects and complex images.

The future of AR probably isn’t about choosing between these approaches — it’s about combining them intelligently depending on the scenario.

If you’re working on similar problems — drop a comment. I’d love to share pipeline details with anyone who’s in the trenches with this stuff.