Mindcraft: Text-Conditioned Infinite Worlds

Infinite worlds are super fun to explore. In particular, I have always loved walking around Minecraft’s endless worlds, where I can find anything from rolling mountains to deep forests to vast deserts.

A beautiful Minecraft forest, all procedurally generated.

But what if I wanted to generate infinite Minecraft worlds from text prompts?

My first instinct is to try using video “world models”, which are often praised for their ability to generate infinitely explorable worlds from prompts. For example, Wan generated this video of “A Minecraft world of jungle cliffs with rivers and sharp cliff drops”:

But not everyone has an NVIDIA H200 to spare! (And video world models have many other drawbacks, including no actual gameplay progression…)

How can you generate playable infinite Minecraft worlds, while adhering to a prompt and running in real-time?

The answer I settled on was simple: LLM designers!

Method

My main implementation question was: “What are the tools I should expose to the LLM to design an expressive infinite procedural world?”

I decided on giving an LLM different types of infinite noise to work with, and a step-by-step process of composing them and assigning biomes. The LLM fills out a JSON whose fields are plugged in as parameters for the terrain generation code during gameplay.

Examples of variations of Perlin and Voronoi noise, which are common infinite noise algorithms.

My spec design process was very informal — mostly vibes-based iteration where I’d describe noise modes and high-level steps to an LLM, have it write the spec, tweak the ordering and CDF behavior until the LLM could reason about sea level correctly, and do a quick eye-test on generated maps.

A broad overview of the process I ended up designing is:

1. LLM makes a name for the world and makes a quick summary
2. LLM picks 1–2 base noise modes and parameters e.g. frequency, octaves, and persistence. It can blend things in different ways for more complex terrain.
3. LLM sets ocean fraction
4. LLM adds some extra “effects”, e.g. rivers, quantization, etc.
5. LLM assigns biome zones for elevation bands
6. LLM overrides rivers with the desired river biome (for frozen vs. not-frozen rivers)

Here are some snapshots of an Claude Opus 4.7’s process of designing a world:

Prompt: “High snowy mountains with sharp peaks, forested foothills, and rivers”

Implementation side tangent: Diffusion?

Originally wanted to do something similar to Terrain Diffusion, but with text conditioning. I would generate a text-conditioned coarse image using my procedural agent, and then run a mini diffusion model trained on real-world terrain data to make it look realistic. I spent a decent amount of time on this, but realized that:

1. Real world terrain data is far too monolithic, so the model smooths everything out regardless of the conditioning. I can get way more expression with a pure LLM designer, even if it’s less “realistic”.
2. Diffusion models are slow and resource-intensive. I took out the model altogether and it was about 20x faster.

The left side is my LLM output. The right side is the diffused-over version. Notice that the diffusion model removes a lot of the expressiveness of the LLM output.

I ended up removing diffusion altogether, which made my terrain generation both more expressive and much faster.

Evaluation

I evaluate Mindcraft on two things: performance and prompt adherence. It was pretty difficult to find open-source, easy-to-use action-conditioned video model baselines, so I just used Wan 2.7 (even though it lacks explicit “action conditioning”).

Performance: Does Mindcraft run in real-time on a Mac M1?

My definition of “real-time” is that the next visible chunks in your walking direction should generate before you exit the chunk you’re on. You walk across a chunk in ~3.7 seconds. Vanilla Minecraft chunk loading takes some time regardless of the chunk generation algorithm, meaning I aimed for less than three seconds to cache new terrain.

Mindcraft far exceeds my standard for “real-time” performance — it takes only an additional 0.5 seconds to generate the next tile, and this tile generation only happens every 128 chunks!

Prompt Adherence: Do Mindcraft worlds adhere to prompts?

Evaluating prompt adherence is difficult. I define two separate categories of “prompt adherence”:

1. How well does Mindcraft adhere to prompts that it should theoretically be able to do?
2. What is the scope of prompts that Mindcraft can adhere to?

To answer these questions, I evaluated on:

1. 10 in-scope prompts. Mindcraft’s prompt scope is limited to landscapes that could reasonably be mistaken for a heavily customized Minecraft seed generated using existing biomes. Here are a few GIFs of results for in-scope prompts; Wan is on the left, while Mindcraft is on the right.

Prompt: “A Minecraft world of endless rolling plains with a wide, slow river cutting a shallow valley through the middle.”

Prompt: “A Minecraft world of giant mushroom islands surrounded by warm tropical seas.”

Prompt: “A Minecraft world of layered desert mesas carved by erosion into grand canyon-like formations with a thin river at the bottom.”

Prompt: “A Minecraft world of sand dunes and tall desert mountains and blazing heat, like the Sahara desert.”

2. 5 out-of-scope prompts. Mindcraft likely can’t do anything with detailed builds, floating things, novel biomes, new game features, or miscellaneous crazy ideas. Again, Wan is on the left, while Mindcraft is on the right.

Prompt: “A Minecraft world with repeated 3D Minecraft builds of Will Smith eating spaghetti.”

Prompt: “A Minecraft world fantasy world on a Switzerland-looking cliff filled with huge beautiful dragons flying around.”

I asked 18 people and 3 VLMs to fill out a form comparing the outputs. For each of the 15 prompts, they were supposed to judge which side of images best matched the prompt. Each question had a sliding scale of 1 (left images match the prompt much more) to 5 (right images match the prompt much more). A 1 meant Wan was better, while a 5 meant Mindcraft was better.

My goal was not to beat Wan, but rather to at least get somewhat close to equal preference on in-scope prompts.

While both humans VLMs slightly preferred Wan’s outputs for in-scope prompts, Mindcraft was able to put up a fight! My goal was satisfied.

As expected, Wan does significantly better for out-of-scope things, showing that while LLM composers are great, they are still very limited by the tools they are given. Despite their huge compute requirements and inconsistencies, video world models still excel in the huge diversity of things they can generate. The benefit of Mindcraft is that it lies in the middle ground — it has decent expressivity, while still producing a playable world that can run very smoothly with shaders on a Mac M1!

If you’d like to try out the project, please check out the GitHub. Stars and feedback are both greatly appreciated!

The initial inspiration for this project came from Alexander Goslin’s amazing Terrain Diffusion Minecraft mod — please check it out!