The Proxy Problem: A Unified Theory of Tokenization

From casino chips to ChatGPT, from banknotes to blockchains — one architectural idea governs them all

There is a question that sits quietly underneath the design of nearly every system that handles value, identity, or sensitive information:

How do you move something without exposing the thing itself?

A merchant doesn’t need to see your bank account number to accept your payment. A language model doesn’t need to “read” English the way a human does to reason about it. A real estate fund doesn’t need to sell an entire building for someone to invest $500 in it. An AI agent executing purchases on your behalf shouldn’t need access to your full financial life to buy you a plane ticket.

In each case, the answer is the same. Not encryption — that only scrambles the original. Not encoding — that only reformats it. The answer is tokenization: the substitution of something heavy, sensitive, or complex with a lightweight symbolic proxy, governed by a controlled mapping that links the two.

This is not a blockchain concept. It is not a payments concept. It is not an AI concept. It is a fundamental primitive of information architecture — and once you see it clearly, you find it everywhere.

What a Token Actually Is

Strip away every domain-specific definition and you find three properties that every genuine token shares.

First: representation. A token stands in for something else. In a language model, a token stands for a word or word-fragment. In a payment system, it stands for a credit card number. In a casino, a chip stands for cash. The token is a compressed, portable symbol — an ambassador for something it doesn’t resemble and can’t become without mediation.

Second: abstraction. The token hides the complexity of the underlying thing from whoever is handling it. A payment terminal processing a token doesn’t need to know your credit history, your bank’s name, or your account balance. It only needs to know whether the token is valid. A language model processing a token ID doesn’t need to parse individual letters — it processes a number. The complexity doesn’t disappear; it’s deferred, moved behind a curtain.

Third: mapping. Every token depends on some form of registry, vault, ledger, vocabulary file, or embedded claim structure that links the token back to what it represents. Without this mapping, the token is a dead symbol — a coat check ticket in a burned-down cloakroom.

This third property is where most discussions of tokenization go wrong, because they assume the mapping must be a centralized database. It doesn’t. The mapping may be a bank’s secure vault, a decentralized blockchain ledger, or claims embedded within the token itself and verified cryptographically. The form of the mapping is an engineering decision. The existence of a controlled mapping relationship is what makes something a token.

The Three-Way Confusion: Tokenization, Encoding, and Encryption

These three mechanisms are routinely conflated in technical writing and system design. They are not interchangeable. They solve different problems using fundamentally different logic.

Encoding is a public, algorithmic translation. When the character “A” becomes 01000001 in binary, or a space becomes %20 in a URL, that's encoding. Any system with the algorithm can reverse it instantly — no secret, no authority, no lookup required. Encoding exists for compatibility and speed, not security. It is a change of shape, not of identity.

Encryption uses mathematical algorithms and secret keys to render data unreadable. The crucial detail: the original data is still there, mathematically embedded in the ciphertext. Given the right key and algorithm, anyone can recover it through computation alone. No external registry required — which also means if someone steals the key, or breaks the math, the original data is exposed in full.

Tokenization operates differently. A surrogate is substituted for the original, which continues to exist independently. There is no mathematical formula that converts the token back into what it represents — only a controlled mapping does that. Steal the token, and you have a dead symbol. The original lives elsewhere, accessible only through the mapping layer.

Encoding Encryption Tokenization Goal Compatibility Confidentiality Risk reduction Reverse how? Public algorithm Algorithm + secret key Controlled mapping Original data Reformatted Scrambled, still present Removed; exists separately If compromised N/A Everything exposed Proxy exposed; original intact

How to Tell If Something Is a Token

Here is a practical three-rule test you can apply to any candidate object.

Rule 1: The Substitution Rule

Does this thing replace an original that continues to exist independently?

If you remove the proxy, does the underlying asset still exist somewhere else? If removing the record is the same as destroying the asset, it is not a token — it is the primary record.

A casino chip passes: the cash you exchanged for it sits in the cage. The chip is a stand-in; the cash persists.

A bank account balance fails: in modern banking, the database record is the money. There is no vault of physical cash corresponding to your specific balance. Delete the record, the money is gone. The balance is an accounting entry, not a proxy.

A hashed password fails: the original password is discarded by design. The hash is a mathematical fingerprint of the input — a terminal representation, not a proxy for something that exists elsewhere.

Rule 2: The Mapping Rule

Is there a controlled relationship — in any form — that links this thing to what it represents?

The key word is controlled. If anyone can derive the meaning using a public algorithm, it is encoding, not tokenization.

A session cookie like ASJ928HFJ passes: its meaning lives in the server's session database. No public algorithm maps that string to User: Alice. Only the server's registry does.

A JWT (JSON Web Token) passes — even though it requires no external database lookup. A JWT embeds its claims (user identity, granted permissions, expiry time) in its payload, signed by the issuer’s private key. The recipient verifies the signature with the issuer’s public key and reads the claims directly. The mapping is controlled by key infrastructure, not derivable by arbitrary parties. JWTs are genuine tokens with a self-contained mapping architecture.

A Base64 string fails: given TUVFVCBBVCBEQVdO, anyone with the Base64 algorithm recovers MEET AT DAWN. The mapping is public and mathematical — encoding, not tokenization.

Rule 3: The Proxy Authority Rule

Does the stand-in carry or reference the authority to act, without requiring re-verification of the underlying identity on every use?

Not all tokens are pure bearer instruments (usable by anyone who possesses them). Some are bound to a specific holder. But what all tokens share is that the authorization context is carried by or referenced in the token itself — eliminating the need to re-authenticate to the identity source on every transaction.

A casino chip is a pure bearer token: the dealer checks chip authenticity, not holder identity. Possession is the authority.

A JWT access token is a bound token: the system verifies the signature and the holder matches the token’s subject claim. But the token carries all authorization context — the resource server needs no roundtrip to the identity provider to know what you’re allowed to do.

A username fails: it is a label for an account. It carries no authorization. You must independently prove who you are before anything is permitted.

Where Tokens Live: The Universal Survey

Language Models

The application of tokenization to human language is, in some ways, the most surprising — and it reveals something important about the concept’s universality.

Computers process numbers, not words. The naïve solution — encode each character as its Unicode number — fails because it destroys semantic structure. The letter sequence A-P-P-L-E has no mathematical relationship to the letters O-R-A-N-G-E, so a model treating them as numbers would have no way to learn that the concepts are related.

The solution is subword tokenization. Algorithms like Byte-Pair Encoding analyze a training corpus and identify an optimal vocabulary of word fragments — typically 32,000 to 100,000 units. The tokenizer consults this vocabulary file (the map) to break text into units and assign each an integer ID. “Tokenization is fundamental” becomes [15496, 1634, 318, 4888, 12093]. That number sequence, not the original text, enters the model.

The map-dependency is evident: the same integer ID means something entirely different in GPT-4’s vocabulary versus LLaMA’s, because they have different vocabulary files. The token has no intrinsic meaning — only map-relative meaning.

What happens next in the pipeline is not tokenization. The model takes those integer IDs and converts them into high-dimensional vectors using a learned mathematical weight matrix — placing “Apple” close to “Orange” and “Banana” in vector space. This is encoding: a mathematical transformation computable from the model’s weights, with no registry required. The two stages serve distinct purposes and must not be confused.

Text → [Vocabulary Map]  → Integer IDs → [Weight Matrix] → Vectors
        TOKENIZATION                        ENCODING

Payment Cards

Payment tokenization is the most commercially mature deployment of the concept, touching hundreds of millions of transactions daily.

The sensitive original is the Primary Account Number (PAN) — the 16-digit number on your card. A Token Service Provider (TSP) — Visa Token Service, Mastercard MDES, or a payment processor — receives the PAN, stores it in a vault, and issues a randomly generated 16-digit surrogate. This token is formatted to look like a card number but carries no account information. There is no mathematical path from the token to the PAN; only authorized vault access provides that.

The architecture includes domain restriction: tokens are issued with constraints limiting their validity to a specific merchant, device, or transaction channel. Intercept a token issued for Amazon transactions and try to use it at an electronics retailer — it fails. The blast radius of any single compromise is bounded by design.

When you tap your phone to pay, the card network’s secure hardware generates a Device Account Number (DAN) — a device-specific token stored in a hardware-isolated chip on your phone. Your real card number never leaves the network’s vault. If the NFC transaction is intercepted, the attacker has a token that only works on your specific device, plus a single-use cryptographic code that has already been consumed.

The contrast with encryption is instructive: if a merchant encrypted and stored card numbers, a sophisticated attacker who compromises both the ciphertext and the encryption keys recovers everything. With tokenization, there is nothing to decrypt. The vault is operated by the TSP — outside the merchant’s environment entirely.

Currency Notes

The banknote may be the oldest widely-held proxy in human existence — and analyzing it precisely reveals an important nuance.

A currency note is a bearer token. It represents, without containing, value. Historically under the gold standard, a note was a claim on gold held in a central reserve. The note was the proxy; the gold was the underlying asset; the central bank was the registry. Under modern fiat systems, the underlying “asset” is sovereign credit — the taxing authority, economic output, and institutional standing of the issuing state.

Here is where precision matters: there is no queryable registry mapping individual banknote serial numbers to specific reserve assets. A central bank does not maintain a database entry for each note in circulation the way a payment TSP maintains vault entries for each token. The “mapping” for a banknote is institutional and legal — the collective agreement of market participants and the legal tender framework of the state.

This makes the banknote a bearer token with a diffuse, institutional map — the proxy relationship holds, the bearer utility holds, but the mapping is of a different character than a payment vault. It’s valid to call a banknote a token; it’s imprecise to describe it as having the same kind of registry as a DAN.

The account-versus-token contrast becomes sharp when examining bank deposits. A deposit balance is not a token. The database record is the money — not a proxy for separately existing cash. This distinction has direct implications for how Central Bank Digital Currencies (CBDCs) are designed: should a digital dollar be a balance in a ledger (account-based, like a bank deposit) or a digital bearer object whose value exists independent of any particular ledger entry (token-based, like a digital banknote)? These are genuinely different architectures with different implications for privacy, programmability, and the nature of money itself.

Casinos and Physical Tokens

The casino chip is the clearest everyday illustration of the tokenization paradigm — which is why it appears in nearly every serious discussion of the concept.

When you buy chips, cash goes into the cage’s float. The chips are bearer tokens: lightweight, domain-restricted, portable proxies for that cash. The cage’s liability ledger is the registry, mapping total outstanding chips to total held cash. The dealer accepts chips at face value without checking your identity or net worth — pure bearer utility.

Domain restriction is enforced physically: Casino A’s chips cannot be redeemed at Casino B. The token lifecycle is visible in real time: issuance at the cage (minting), circulation across tables (active), return to the cage (consumption and settlement). Between minting and settlement, the underlying cash sits immobile in the vault while the proxies move freely.

The coat check ticket is a conceptually identical system, operationally simpler: a random number maps to a coat in a cloakroom. If the cloakroom burns down, the ticket becomes a dead symbol. The original asset is gone; the map is broken. This is the canonical map-failure mode.

Compilers and Lexical Analysis

One of the oldest and most formally rigorous applications of tokenization is in compiler design — predating blockchains by decades.

When a compiler reads source code, its first job is lexical analysis: reading raw character sequences and identifying meaningful units of the programming language. Each identified unit is assigned a lexical token — a structured object encoding its type and value.

The source fragment if (x > 10) { return true; } becomes:

[IF] [LPAREN] [IDENTIFIER: x] [GREATER_THAN] [INTEGER: 10] [RPAREN]
[LBRACE] [RETURN] [BOOLEAN: true] [SEMICOLON] [RBRACE]

The parser operates on this token stream rather than raw characters, reasoning about code structure without being distracted by whitespace, indentation, or comments. The symbol table — a registry tracking every identifier, its type, its scope, its memory allocation — is the map. An identifier token [IDENTIFIER: x] is meaningless without it.

Sessions, JWTs, and OAuth

Web security is built on tokenization at every layer.

When you log into a website, the server generates a random, unpredictable session token and records its association with your authenticated identity in a server-side session store. The token is sent to your browser. Every subsequent request presents the token; the server consults its store and retrieves your context. The token encodes nothing about you — it is a pure opaque proxy, meaningful only through the registry.

OAuth 2.0 access tokens come in two varieties that illustrate the architectural spectrum clearly. Opaque tokens are registry-dependent: the resource server must query the authorization server’s introspection endpoint to validate them. JWT access tokens are self-contained: all claims — user identity, granted scopes, expiry time — are embedded in the token payload and verified against the issuer’s public key. No database call required for validation.

JWTs trade one thing for this performance gain: revocation becomes hard. A JWT passes validation until its expiry, even if the underlying session has been terminated. Production systems address this through short expiry windows (typically 5–60 minutes) combined with refresh token flows, or a lightweight Token Revocation List consulted only for explicitly invalidated tokens.

This tradeoff — self-contained tokens are fast but difficult to revoke; registry-dependent tokens are slower but instantly revocable — is one of the most consequential architectural choices in any tokenized system.

Blockchains and Distributed Ledgers

DLT introduces tokenization into an environment where the registry is not a central database but a ledger maintained by network consensus. No single party controls it; validity is determined by the protocol’s rules. The architecture changes dramatically; the underlying logic does not.

One distinction matters enough to state precisely, because it is routinely collapsed:

ERC-20 tokens — the standard used by most “tokens” in the Ethereum ecosystem — are implemented as an account-based balance system inside a smart contract. The contract maintains a mapping of address → uint256 balance. There are no individual token objects. When you hold 100 USDC, the contract records your address has a balance of 100. This is structurally a ledger, not a bearer token architecture.

ERC-721 tokens (NFTs) are genuinely token-based. Each token has a unique integer ID; the contract maps tokenId → owner address. Discrete token objects exist, can be held, transferred, and burned. The token is the unit of value, not the balance.

Stablecoins like USDC are an interesting hybrid. At the asset-backing layer, they are genuine tokenization: each USDC unit represents a claim on real dollars held in a custodian account — the issuer maintains the registry, redemption is detokenization. But on-chain, USDC is implemented as ERC-20 — account-based. The tokenization is at the asset-backing layer; the on-chain circulation is account-based.

A fragility worth noting for NFTs: many store their actual content via a URI pointing to a file on a centralized server or IPFS. If that server goes offline, the NFT token ID remains on the blockchain — but the mapping to its content is broken. The token becomes a dead symbol. A registry entry pointing to a missing asset. This is the map-failure problem made visible on a public ledger.

Real-World Assets

RWA tokenization applies the same logic to illiquid assets: real estate, private equity, infrastructure, fine art, intellectual property.

A $200 million commercial building is divisible into 200 million tokens at $1 each. Each token is a proxy for a fractional economic interest, enforced by a legal structure — typically a Special Purpose Vehicle (SPV) that holds the actual property. The blockchain records ownership. The SPV holds the asset. Legal agreements define what the token entitles the holder to.

That third layer — the legal agreements — is the one most often underestimated. A token with no enforceable legal backing is a symbol without a genuine map. RWA tokenization is as much legal engineering as technical engineering. The registry exists at multiple layers simultaneously, and all of them must hold.

The Token Lifecycle

Every token, regardless of domain, moves through a defined sequence of states. Formalizing this is the prerequisite for building systems that handle tokens at scale.

NULL → MINTED → ACTIVE → CONSUMED → SETTLED
                    ↓
               EXPIRED / REVOKED

Minting is the moment of creation: the token schema is defined, constraints are injected, the mapping is recorded or embedded, a cryptographic signature is applied. The token exists; it has not been used.

Active is the valid, in-circulation state. The token may be consumed (authorized use), expired (TTL breach), or revoked (explicit cancellation from this state).

Consumed follows a single authorized use. In single-use architectures, no further consumption is possible. The action has been taken.

Settled is the final reconciliation. The action the token authorized has been reflected in the underlying system — funds moved, access logged, asset updated. Settlement closes the gap between the token world, which moves fast, and the asset world, which moves deliberately.

The settlement model matters enormously in practice. Immediate settlement keeps the two worlds synchronized but is expensive — it requires real-time access to the underlying asset system for every transaction. Deferred settlement is how card networks actually work: transactions are authorized in real time via token validation, but funds settle overnight through the clearing system. Streaming settlement is what agentic and high-frequency systems are moving toward: continuous, event-driven reconciliation that updates the underlying system asynchronously.

The Two Critical Invariants

Two design properties separate toy tokenization systems from production-grade ones.

Single-use guarantee: A consumed token must not be spendable again. This requires not just marking tokens consumed but enforcing idempotency — a failed network call might cause a transaction to be retried. The system must distinguish between a legitimate retry of a failed transaction and a malicious replay of a successful one. The standard mechanism: treat (token_id, nonce) as a composite key in the consumption log.

Bounded authority: Every token must carry explicit, narrow constraints on what it authorizes. The distinction between capability (what class of action: payment, read access) and constraints (under what conditions: amount ≤ $50, merchant = “amazon.com”, expiry = tomorrow) is where the security actually lives. A token that authorizes “payment up to $50 at Amazon” cannot be used for $51, at a different merchant, or after expiry — regardless of who holds it or how they obtained it.

The Agentic Frontier

All of this converges on something new.

An AI agent acting on your behalf — browsing, comparing, purchasing, booking — may execute hundreds of transactions per hour. Giving such an agent access to a bank account is dangerous: a compromised agent drains the account. A long-lived payment credential is only marginally better.

The appropriate architecture is purpose-bound token issuance. Before an agent begins a task, a token is minted with explicit capability and constraint parameters:

{
  "issued_to": "agent_instance_xyz",
  "capability": { "type": "payment", "limit": 500, "currency": "USD" },
  "constraints": {
    "merchant_category": ["SaaS", "API_services"],
    "single_merchant_limit": 100,
    "expiry": "2026-03-17T23:59:59Z",
    "max_transactions": 20
  },
  "signature": "issuer_cryptographic_signature"
}

The token encodes not just who is paying but what is authorized, under what conditions, and for this specific task. A merchant receiving this token validates it — checking signature, constraints, registry state — without calling the agent’s home bank or verifying the agent’s identity. The transaction proceeds at signature-verification speed, not API-roundtrip speed.

In multi-agent systems, this composes naturally. An orchestrating agent delegates sub-tasks to specialized agents, each receiving a derived token with further reduced scope. The orchestrator’s token authorizes $500 in spend; the research sub-agent’s token is limited to $50 in search API calls; the writing sub-agent’s token to $30 in language model API costs. Bounded authority is preserved at every level of the hierarchy.

This is what makes agentic commerce structurally possible: the agent doesn’t need your identity. It doesn’t need your account. It needs a token that carries its own authorization and expires when the task is done.

The Objects That Aren’t Tokens

Understanding what tokenization is requires being equally clear about what it isn’t.

Hashed passwords are mathematical fingerprints, not tokens. The original password is discarded by design. There is no underlying original sitting in a vault behind the hash. It fails the substitution rule.

Encrypted data is a mathematical transformation of the original, not a proxy for it. The original data is present in the ciphertext, recoverable by anyone with the key. No controlled proxy relationship. It fails at the conceptual level: encryption hides; tokenization replaces.

Usernames and email addresses are labels for accounts. They carry no authorization. The system must independently verify your identity and retrieve your permissions before anything is permitted. They fail the proxy authority rule.

ERC-20 balances are ledger entries in a smart contract — account-based, not bearer-token-based. There are no discrete token objects, only balances per address. They fail the substitution rule: there are no individual proxy instances; there is only a running tally.

The Unified View

Trace through these domains — LLMs, payment cards, banknotes, casino chips, compilers, blockchains, real estate, session management, AI agents — and a single pattern emerges with striking consistency.

In every case, tokenization solves the same problem: how to transfer capability, authority, or value without exposing the original asset, while maintaining a controlled mapping that preserves the ability to reconcile the proxy with the real.

The surface implementations differ enormously. An LLM vocabulary file and a payment card vault share no code, no infrastructure, no regulatory context. A casino chip and a smart contract token share no physical or digital medium. But the logical architecture — substitute, abstract, map — is identical.

This universality is the argument that tokenization is not a feature of any particular technology. It is a foundational primitive: as basic to the design of systems that handle value and sensitivity as the conditional branch is to the design of systems that handle logic.

The architectural spectrum runs from fully registry-dependent (opaque tokens, no embedded information, maximum opacity) to fully self-contained (JWT-style, all claims embedded and verifiable, no registry required for standard use). Between them lies every practical tokenization system — and the engineering judgment about where to sit on that spectrum is one of the most consequential decisions in system design.

The lifecycle is always the same: mint, activate, consume, settle — with expiry and revocation as the emergency exits. The invariants are always the same: single-use guarantee, bounded authority, observability, revocability.

And the deepest contrast is always account versus token: value in the verified identity versus value in the validated object. Each model has its strengths. Accounts accumulate history. Tokens move fast. The trajectory of modern infrastructure — from payment networks to blockchains to AI agent frameworks — is toward a hybrid of the two: token-based execution that settles against account-based reality.

Fast enough for machines. Secure enough for money. Programmable enough for the intelligent systems increasingly acting on our behalf.

If large language models made language programmable, tokenization in agentic commerce makes value programmable. The bridge between the two is a token that encodes not just representation — but permission, constraint, and intent.

Tags: Tokenization Blockchain Payments Machine Learning Software Architecture Fintech AI Agents Web3