From Candidate Retrieval to Fine-Grained Ranking (Part 3)

Part 2- https://medium.com/p/0894ab3cdb72

In recommendation and ad-click systems, the signal rarely comes from individual features alone. What matters is combinations — a user who is young + mobile + evening behaves differently than the same age group on desktop at noon. These combinations are called feature interactions, and learning them well is the central challenge.

Logistic Regression (LR) was the industry workhorse for years. Fast, interpretable, easy to deploy. But it’s strictly linear — it can only learn interactions if you manually create them as new features (e.g. explicitly engineer an “age × device” column). With hundreds of raw features, the number of possible interactions explodes combinatorially. Human engineers had to guess which ones mattered, which was slow, brittle, and incomplete.

Factorization Machines (FM) were a big step forward — they could automatically learn pairwise (2nd order) interactions by decomposing the interaction weights into low-rank embeddings. But they were limited to 2nd order only. A signal like “young × mobile × evening” (3rd order) was out of reach without extensions that quickly became expensive.

Deep Neural Networks (DNN / MLP) seemed like the answer — stack enough layers with ReLU activations and the network should theoretically approximate any function, including high-order interactions. And they do learn something, but the key problem is that interactions are implicit and inefficient. The network has to accidentally stumble upon feature interactions through gradient descent across millions of parameters. There’s no structural guarantee it will find the right ones, and it wastes a lot of capacity doing so. In practice, vanilla DNNs often underperform on interaction-heavy tasks unless they’re very large.

Wide & Deep (Google, 2016) was an influential hybrid — a linear “wide” component (for memorization) combined with a deep component (for generalization). But the wide side still required manual feature engineering — a human had to decide which cross features to feed in. It didn’t solve the automation problem.

What DCN actually solves

DCN (Google, 2017) was designed to get the best of both worlds without the manual work:

The cross network (left path) addresses the FM/LR problem — it learns bounded, explicit polynomial feature interactions automatically, up to order L+1 where L is the number of cross layers. With 3 layers you get up to 4th-order interactions, and crucially the complexity grows only linearly with depth, not exponentially. No human needs to specify which interactions to look for.

The deep network (right path) addresses the DNN limitation — it handles the implicit, arbitrary non-linear patterns that don’t fit neatly into polynomial form.

Concatenating both means the model can separately optimize structured interaction learning and free-form non-linear learning, then combine them — rather than hoping a single DNN does both well at once.

The practical payoff: better accuracy at lower model size, and no feature engineering pipeline to maintain. For large-scale ad systems serving billions of predictions a day, that combination of accuracy and efficiency was a meaningful industrial advance.

Where We Are in the Pipeline

STAGE 2 — RANKING ← WE ARE HERE
├── FAISS gave us 1,000 candidates for C001
├── We scored them with Wide & Deep
│   └── BUT Wide side still needed manual cross features
│       "someone had to hardcode Denim_AND_evening"
├── DCN v2 is an upgrade to the ranking model
│   └── replaces manual crosses with automatic cross network
└── Same 1,000 candidates, same 100ms budget, better precision

Background — Why DCN v2 Was Needed

Problem 1 — Wide & Deep’s Wide Side Was Still Manual

Wide & Deep (2016):
    Deep side  → automatic ✅
    Wide side  → STILL manual ❌

Someone at your team had to sit down and write:
    "Denim_AND_evening"
    "mobile_AND_sale"
    "CA_AND_Denim_AND_evening"
    ...hundreds of these...

At e-commerce scale with 500 categories × 50M customers
× 10 context features = billions of possible crosses
Human brain cannot enumerate these               ❌

Problem 2 — Deep Network Learns Crosses Implicitly

Wide & Deep Deep side:
    MLP layers DO learn feature interactions
    BUT:
    ├── You can't see WHICH interactions it learned
    ├── You can't control the ORDER of interactions
    ├── It needs many layers and parameters to learn
    │   what could be captured more efficiently
    └── No guarantee it finds the RIGHT crosses

Problem 3 — Original DCN v1 (2017) Was Limited

DCN v1 (2017):
    Introduced cross layers                        ✅
    BUT used vector weights (less expressive)      ❌
    Could not learn feature-specific interactions  ❌
    Treated all features equally                   ❌

DCN v2 (2020):
    Upgraded to MATRIX weights per cross layer     ✅
    Each feature pair gets its own learned weight  ✅
    Much more expressive at same compute cost      ✅

The Solution DCN v2 Introduced

Replace the Wide side's manual crosses (DCN1 - in the Part2 series)
with a Cross Network that:

1. Takes the same x0 input (154 numbers)
2. Automatically computes ALL pairwise interactions (2nd order)
3. Then ALL triple interactions (3rd order)
4. Then ALL quartic interactions (4th order)
5. No human decides which crosses matter
6. Learned entirely from click data

The Math — Built Up Very Slowly

Before touching code, understand the cross layer formula:

x_l+1 = x0 × W_l(x_l) + x_l

Three operations. Let me explain each with 3 numbers first (not 154):

Suppose our entire input is just 3 features:
x0 = [hour=21,  price=79,  CTR=0.04]

Operation 1 — W_l(x_l): Transform current layer input

W is a [3×3] weight matrix:

              out1   out2   out3
in1 (hour)    0.1    0.2    0.3
in2 (price)   0.4    0.1    0.2
in3 (CTR)     0.3    0.4    0.1

W(xl) output1 = (0.1×21) + (0.4×79) + (0.3×0.04)
              = 2.1 + 31.6 + 0.012
              = 33.71

W(xl) output2 = (0.2×21) + (0.1×79) + (0.4×0.04)
              = 4.2 + 7.9 + 0.016
              = 12.12

W(xl) output3 = (0.3×21) + (0.2×79) + (0.1×0.04)
              = 6.3 + 15.8 + 0.004
              = 22.10

W(xl) = [33.71, 12.12, 22.10]

Operation 2 — x0 × W_l(x_l): Element-wise multiply with ORIGINAL input

x0    = [21,    79,    0.04  ]   ← ORIGINAL input, never changes
W(xl) = [33.71, 12.12, 22.10]   ← transformed current layer

Element-wise multiply (NOT matrix multiply):
position 1: 21    × 33.71 = 707.91   ← hour    × (combo of all features)
position 2: 79    × 12.12 = 957.48   ← price   × (combo of all features)
position 3: 0.04  × 22.10 = 0.884    ← CTR     × (combo of all features)

result = [707.91, 957.48, 0.884]

THIS IS THE CROSS.
Each original feature is now multiplied by a learned combination
of ALL other features → automatic pairwise interactions

Operation 3 — + x_l: Residual (blue line in diagram)

x0 × W(xl) = [707.91, 957.48, 0.884]
xl          = [21,     79,     0.04 ]   ← add back current layer input

x_l+1      = [707.91+21, 957.48+79, 0.884+0.04]
           = [728.91,    1036.48,   0.924      ]

Why add xl back?
→ Preserves the signal from previous layer
→ Prevents information loss as we go deeper
→ Makes training more stable (same idea as ResNet)

Now With Real E-commerce Data — All 154 Features

Step 1 — Start With Just ONE Row

Before building the full matrix, understand what ONE row looks like.

C001 viewing P042 (Denim Slim Jeans, $79) at 9pm on mobile:

ONE ROW = 154 numbers side by side:

← dense (10) →← user embedding (64) →← item embedding (64) →← cat (16) →

[32, 85, 0.04, 21, 79, 7, 0.22, 0.18, 8.4, 1, | 0.8, 0.2, 0.9, ...(61 more), 0.3, | 0.7, 0.3, 0.8, ...(61 more), 0.6, | 0.5, 0.1, ...(14 more), 0.4]

Total: 10 + 64 + 64 + 16 = 154 numbers

Now let’s zoom into each section:

Col  Feature               Value   Where it comes from
───────────────────────────────────────────────────────────────
[0]  age                   32      C001's profile table
[1]  avg_order_value       85      C001's order history → mean($)
[2]  item_avg_ctr          0.04    P042's impression logs → clicks/views
[3]  hour_of_day           21      current session timestamp
[4]  item_price            79      P042's product catalog
[5]  days_since_purchase   7       C001's last order date vs today
[6]  item_return_rate      0.22    P042's return logs → returns/purchases
[7]  user_return_rate      0.18    C001's return history → returns/purchases
[8]  session_length        8.4     current session duration in minutes
[9]  device_mobile         1       current device (1=mobile, 0=desktop)

As a row fragment:
position: [0]  [1]  [2]    [3]  [4]  [5]  [6]   [7]   [8]  [9]
value:     [32, 85,  0.04,  21,  79,  7,   0.22, 0.18,  8.4, 1  ]

Section A — Dense Features (Columns 0–9)

These are raw numbers directly about C001 and P042:

The Starting Input x0

x0 for C001 viewing P042 (Denim, $79) at 9pm on mobile:

DENSE FEATURES (positions 0-9):
[0]  age                  = 32
[1]  avg_order_value      = 85
[2]  item_avg_ctr         = 0.04
[3]  hour_of_day          = 21
[4]  item_price           = 79
[5]  days_since_purchase  = 7
[6]  item_return_rate     = 0.22
[7]  user_return_rate     = 0.18
[8]  session_length       = 8.4
[9]  device_mobile        = 1

USER EMBEDDING (positions 10-73):
[10] user_dim_1  = 0.8    ← encodes C001's Denim affinity
[11] user_dim_2  = 0.2    ← encodes C001's price sensitivity
[12] user_dim_3  = 0.9    ← encodes C001's evening activity
...
[73] user_dim_64 = 0.3

ITEM EMBEDDING (positions 74-137):
[74]  item_dim_1  = 0.7   ← encodes P042's Denim-ness
[75]  item_dim_2  = 0.3   ← encodes P042's price tier
[76]  item_dim_3  = 0.8   ← encodes P042's popularity
...
[137] item_dim_64 = 0.6

CATEGORY EMBEDDING (positions 138-153):
[138] cat_dim_1  = 0.5    ← encodes Denim category
...
[153] cat_dim_16 = 0.4

x0 shape: [1000 × 154]   ← all 1,000 candidates stacked

Cross Layer 1–2nd Order (Pairwise Interactions)

class CrossLayer(nn.Module):
    def __init__(self, input_dim: int):
        super().__init__()
        self.W = nn.Linear(input_dim, input_dim, bias=True)

self.W = nn.Linear(154, 154)
Weight matrix shape: [154 × 154]
= 23,716 parameters per cross layer
Each of 154 output dimensions looks at ALL 154 input dimensions

def forward(self, x0: torch.Tensor, xl: torch.Tensor) -> torch.Tensor:
        return x0 * self.W(xl) + xl

At Cross Layer 1, x0 = xl = our 154-number vector:

Step 1 — self.W(xl):
    Weight matrix [154×154] transforms xl

    Output position 0 (will interact with age=32):
    = w0,0×32 + w1,0×85 + w2,0×0.04 + w3,0×21 + w4,0×79
    + w5,0×7  + ... + w10,0×0.8 + w11,0×0.2 + ...
    + w74,0×0.7 + ... + w138,0×0.5 + ...
    = 39.21   ← weighted combination of ALL 154 features

    Output position 3 (will interact with hour=21):
    = w0,3×32 + w1,3×85 + w2,3×0.04 + w3,3×21 + ...
    = 28.43   ← different weights, different combination

    W(xl) = [39.21, 41.83, 30.60, 28.43, 19.62, ...]
             ↑      ↑      ↑      ↑
             will   will   will   will
             cross  cross  cross  cross
             age    order  CTR    hour

Step 2 — x0 * self.W(xl) (element-wise):
    x0    = [32,    85,    0.04,  21,    79,    ...]
    W(xl) = [39.21, 41.83, 30.60, 28.43, 19.62, ...]

    position 0: 32    × 39.21 = 1254.7  ← age    × (ALL features combined)
    position 1: 85    × 41.83 = 3555.6  ← order  × (ALL features combined)
    position 2: 0.04  × 30.60 = 1.224   ← CTR    × (ALL features combined)
    position 3: 21    × 28.43 = 597.0   ← hour   × (ALL features combined)
    position 4: 79    × 19.62 = 1550.0  ← price  × (ALL features combined)
    ...
    position 10: 0.8  × W10   = ?       ← user_dim1 × (ALL features)
    position 74: 0.7  × W74   = ?       ← item_dim1 × (ALL features)

    These are 2nd order crosses — every feature crossed with
    a learned combination of every other feature

Step 3 — + xl (residual):
    [1254.7, 3555.6, 1.224, 597.0, 1550.0, ...]
  + [32,     85,     0.04,  21,    79,     ...]
  = [1286.7, 3640.6, 1.264, 618.0, 1629.0, ...]

x1 = [1286.7, 3640.6, 1.264, 618.0, 1629.0, ...]
Shape: [1000 × 154]   ← SAME shape as input

What x1 now encodes:

x1[0] = 1286.7  encodes: age × (hour+price+CTR+user_embed+item_embed+...)
x1[3] = 618.0   encodes: hour × (age+price+CTR+user_embed+item_embed+...)
x1[10]= ?       encodes: user_dim1 × (age+hour+price+item_embed+...)

In plain English:
"How does C001's age interact with everything else about this moment?"
"How does 9pm interact with everything else about C001 and P042?"
"How does C001's Denim affinity interact with P042's features?"

Cross Layer 2–3rd Order (Triple Interactions)

# Second call: layer(x0, x1) → x2
x_cross = layer(x0, x1)

Now xl = x1 (which already contains 2nd order interactions)
x0 = STILL the original [32, 85, 0.04, 21, 79, ...] ← never changes

Step 1 — self.W(x1):
    x1 already encodes "age × everything", "hour × everything" etc.
    W transforms x1:

    Output position 3 (will interact with hour=21):
    = w0,3×1286.7 + w1,3×3640.6 + w2,3×1.264 + w3,3×618.0 + ...
    = now mixing 2nd order interactions together
    = 8421.3   ← combination of combinations

Step 2 — x0 * self.W(x1):
    position 3: 21 × 8421.3 = 176,847.3
    ← hour × (age×everything + price×everything + ...)
    ← this IS a 3rd order interaction:
       hour × age × (everything)
       hour × price × (everything)

Step 3 — + x1 (residual):
    [..176847.3..] + [..618.0..] = [...177465.3...]

x2 shape: [1000 × 154]   ← still same shape

What x2 encodes:
    "C001's age × hour=9pm × P042's price"
    "C001's Denim affinity × hour=9pm × device=mobile"
    "item_CTR × user_avg_order × item_price"

Deep Network Path — Right Side of Diagram

layers, in_dim = [], self.input_dim   # 154
        for h in deep_hidden_dims:            # [256, 128]
            layers += [nn.Linear(in_dim, h), nn.ReLU(), nn.Dropout(0.1)]
            in_dim = h
        self.deep = nn.Sequential(*layers)

SAME x0 goes into deep network simultaneously:
x0 [1000×154] → Linear(154→256) → ReLU → Dropout
              → Linear(256→128) → ReLU → Dropout
              → x_deep [1000×128]

This is identical to Wide & Deep's deep side.
Learns IMPLICIT interactions — non-linear, harder to interpret
but captures patterns the cross network might miss.

Cross network = explicit, controlled, polynomial
Deep network  = implicit, flexible, non-linear
Together      = best of both                        ✅

Concatenation and Final Score

combined = torch.cat([x_cross, x_deep], dim=1)
        return torch.sigmoid(self.output(combined)).squeeze(1)

x_cross: [1000 × 154]  ← explicit 4th order interactions
x_deep:  [1000 × 128]  ← implicit non-linear interactions

torch.cat → [1000 × 282]

self.output = nn.Linear(282, 1):
    282 numbers → single score per candidate

sigmoid → P(click) for all 1,000 candidates:

Candidate  Cross score  Deep score  Combined  P(click)  Rank
P042       strong Denim  high intent  0.84      0.84      #2
P107       weak cross    medium       0.81      0.81      #4
P334       strong cross  high intent  0.91      0.91      #1  ← cross network
P088       weak cross    low intent   0.43      0.43      #48    found this!

DCN v2 vs Wide & Deep — Final Comparison

WIDE & DEEP:
    Wide side:
        Input:  manually engineered "Denim_AND_evening" strings
        Crosses: only what humans thought to build
        New feature: re-engineer all crosses manually  ❌
        Max order: usually 2nd order (pairs only)      ❌
        Parameters: ~10 (just dense features)

    Deep side:
        154 → 256 → 128 → 64 → 1
        Implicit crosses only

DCN v2:
    Cross network:
        Input:  raw x0 (154 numbers)
        Crosses: ALL combinations learned automatically ✅
        New feature: just add to x0, auto-discovered   ✅
        Max order: 4th order (quartic)                 ✅
        Parameters: 3 × 154 × 154 = 71,148

    Deep network:
        154 → 256 → 128
        Same implicit crosses

    Key win: adding a new feature (e.g. user_loyalty_score)
    Wide & Deep: re-engineer all manual crosses         ❌
    DCN v2:      just add to x0, cross network
                 automatically finds new interactions   ✅