I Always Wanted to Know How Face Recognition Actually Works — So I Built a working Prototype

A beginner-to-intermediate developer’s tour of detection, alignment, embeddings, and matching — with the models industry uses and small code snippets from a real project.

Suggested reading time: ~12 minutes

Audience: Curious developers who’ve used “face unlock” but never looked under the hood

The full working prototype is on GitHub: **https://github.com/adityabbsharma/face-recognition-attendance-marking-app-public**

I didn’t start with a research paper.

I started with a simple itch: phones unlock with a face, offices claim “AI attendance,” and every tutorial either waves a magic API or dumps math with no intuition. I wanted the middle path — build a small working system and learn what each stage is for.

So I built a prototype: capture a face on a phone or webcam, store an identity, then check someone in by matching a new photo. Along the way I learned that “face recognition” is not one model. It’s a pipeline.

This post is the story of that pipeline, in plain language, with the same kinds of models you’d meet in industry (SCRFD, ArcFace / InsightFace buffalo_l) and a few snippets from the Java + ONNX prototype I ended up with.

The big idea in one sentence

A face recognition system turns a photo into a list of numbers (an embedding) so that similar faces land near each other in that number-space — then it measures how close two lists are.

Everything else — detection, cropping, fancy loss functions — is there to make those numbers trustworthy.

What I thought vs what I learned

Those five corrections saved me more time than any library choice.

Stage 0 — Decide what “success” means

Before models, pick the product question:

1. Verification (1:1): “Is this the same person as this enrolled photo?”

2. Identification (1:N): “Which enrolled person is this?” (attendance, access)

My prototype is mostly 1:N: enroll employees, then check-in finds the closest match above a score threshold.

Stage 1 — Find the face (detection)

You can’t recognize what you haven’t located.

Detection answers: Where is the face in this frame? Good detectors also give landmarks (eye corners, nose tip, mouth corners) — five points is common.

What industry uses

My prototype uses SCRFD via ONNX (det_10g.onnx from the buffalo_l-style pack): resize to 640×640, run the network, decode boxes + landmarks, keep the best face.

// Pick the highest-confidence face
Optional<DetectedFace> detected = faceDetector.detectBest(original);

If nothing is found, you have a product decision: reject the capture, or fall back to “use the whole image” (usually worse for recognition).

Beginner takeaway: Recognition quality often dies at detection — wrong box, tiny face, or no face at all.

Stage 2 — Align the face (geometry / warping)

Here’s the part tutorials skip — and the part that finally made recognition click for me.

Why warp at all?

The recognition network (ArcFace) was trained on millions of faces that all share a canonical pose:

• roughly front-facing

• eyes near the same rows

• face filling a fixed 112×112 square

Your selfie is not like that. The person might be tilted, farther from the camera, or off-center. If you feed that raw crop in, the network spends capacity fighting geometry instead of reading identity.

Warping = geometrically rewrite the photo so the face lands on that canonical layout — before the embedding model runs.

Think of it like straightening a scanned document before OCR. Same information, less noise.

What “5 landmarks” give you

After Stage 1, SCRFD doesn’t only return a box. It returns five points on the original image:

1. Left eye

2. Right eye

3. Nose tip

4. Left mouth corner

5. Right mouth corner

In our code those arrive as 10 floats: x0,y0, x1,y1, … x4,y4.

Separately, ArcFace defines a fixed destination for those same five points on a blank 112×112 canvas (the InsightFace template):

// Where eyes/nose/mouth *should* sit after alignment (112×112)
private static final double[][] DST = {
{38.2946, 51.6963}, // left eye
{73.5318, 51.5014}, // right eye
{56.0252, 71.7366}, // nose
{41.5493, 92.3655}, // left mouth
{70.7299, 92.2041} // right mouth
};

So you have:

Source points (src) : Where the face is in the selfie (pixels from the detector)

Destination points (DST) : Where the face should be in the 112×112 training crop |

Warping is the mapping that moves src → DST as well as possible.

What kind of warp? (only three allowed moves)

We don’t allow arbitrary distortion (that could squash a face into looking like someone else). We only allow three moves — like editing a photo with simple tools:

1. Zoom — make the face bigger or smaller so the eyes are about as far apart as in the training template

2. Rotate — straighten a tilted head (like rotating a photo until the eyes are level)

3. Slide — move the whole face left/right/up/down so it sits in the middle of the 112×112 square

We do not stretch the face wider, squash it taller, or bend it. That keeps a face looking like a face.

How the computer picks those three moves: it looks at your five landmarks (where the eyes/nose/mouth are) and the template (where they should be). Then it finds one zoom + one rotation + one slide that places those five points as close as possible to the template. That recipe is often called a Umeyama fit (in OpenCV/Python you’ll see the same idea as *estimateAffinePartial2D*).

You don’t need the matrix algebra. Mentally:

“Rotate and zoom my selfie until the eyes land on the template dots; then nudge it into place.”

static BufferedImage align(BufferedImage source, float[] landmarks10) {
    // 1) Pack detector landmarks into src[5][2]
    // 2) Figure out: zoom + rotate + slide so src ≈ template
    AffineTransform transform = umeyama(src, DST);
    // 3) Redraw the photo with those three moves into a 112×112 image
    BufferedImage out = new BufferedImage(112, 112, TYPE_3BYTE_BGR);
    g.drawImage(source, transform, null);
    return out;
}

Important detail: we apply that zoom/rotate/slide to the full camera frame, not only a cut-out box. The landmarks already know where the face is; after the transform, the right region lands inside the 112×112 canvas. Empty corners (if any) just stay blank/background.

Picture the before / after

Same person, same lighting — but the pixels that enter ArcFace now look like the pixels it saw in training.

Weak fallback: crop without landmarks

If landmarks are missing, we fall back to a padded bounding-box crop resized to 112×112 (cropPadded). That removes background but does not correct tilt. It’s better than full-frame, worse than true alignment.

Beginner takeaway

Warping is not “beauty mode.” It’s coordinate normalization for a neural net. Detect finds the face; warp puts it where ArcFace expects it; then embedding can focus on identity.

Skip Stage 2, and you’ll blame the model for problems that are really geometry.

Stage 3 — Turn the face into a vector (embedding)

Now the recognition network runs — often an ArcFace-trained ResNet (in buffalo_l terms, something like w600k_r50).

Input: aligned 112×112 face Output: typically a 512-float vector — the embedding

That vector is the face’s fingerprint in math form. It is not a compressed photo. You can’t “see” the face by staring at the numbers — but two photos of the same person should produce vectors that point in a similar direction.

Preprocessing detail that bit me:

InsightFace ArcFace ONNX models often expect BGR channels and (pixel - 127.5) / 127.5:

// Channel order: B, G, R — not RGB
tensor[0][0][y][x] = (b - 127.5f) / 127.5f;
tensor[0][1][y][x] = (gVal - 127.5f) / 127.5f;
tensor[0][2][y][x] = (r - 127.5f) / 127.5f;

Get this wrong and similarity scores become random noise. The model isn’t broken — the contract was.

Industry naming you’ll see

You don’t need to derive the ArcFace formula on day one. You do need to know: training loss shaped the space; at inference you only keep the embedding network.

Stage 4 — Normalize (put everyone on the same sphere)

Almost every modern system L2-normalizes the embedding:

// Make the vector length = 1
for (int i = 0; i < vector.length; i++) {
    vector[i] = (float) (vector[i] / norm);
}

After this, comparing faces becomes measuring angles (or equivalently, cosine similarity / dot products). Same person → vectors nearly parallel → score near 1.0. Different people → smaller score.

Beginner takeaway: Store normalized embeddings. Don’t mix model versions in one gallery without re-enrolling — different models speak different “coordinate systems.”

Stage 5 — Compare (similarity + threshold)

Cosine similarity (the score)

// Simplified: how aligned are two vectors?
double score = cosineSimilarity(probeEmbedding, enrolledEmbedding);
// score close to 1.0 → likely same person

Identification in a gallery (1:N)

For check-in: compare the new embedding to every enrolled vector, take the best score, accept only if it’s above a threshold τ:

for (Map.Entry<UUID, float[]> entry : cache.entrySet()) {
    double score = EmbeddingCodec.cosineSimilarity(embedding, entry.getValue());
    if (score > bestScore) {
        bestScore = score;
        bestId = entry.getKey();
    }
}
if (bestScore < similarityThreshold) {
    return Optional.empty(); // no confident match
}

In my prototype, τ around 0.40 worked after detect+align (your number will differ). Threshold is not magic — it’s a business risk knob:

• Lower τ → fewer “unknown” rejections, more false accepts
• Higher τ → safer access, more false rejects

What happens at bigger scale?

Thousands of faces: a loop in memory is fine. Millions: people use vector indexes (FAISS, HNSW, etc.). Same math; faster search.

Putting it together (the loop I run on every photo)

BufferedImage faceImage = prepareFaceImage(original); // detect + align
float[][][][] input = toArcFaceTensor(faceImage);     // 112×112 BGR tensor
float[] vector = runArcFaceOnnx(input);               // 512-D
normalize(vector);                                    // unit length
SearchResult match = search(vector);                  // cosine vs gallery

That’s the whole “miracle” in six lines of intent.

Mistakes I made so you don’t have to

1. Full-frame ArcFace — Running recognition on the whole selfie without detect/align gave weak scores. Adding SCRFD + alignment was the accuracy jump.
2. Assuming one threshold forever — Office webcam ≠ phone selfie in bad light. Recalibrate when the camera changes.

Where the industry is going (short tour)

You don’t need to master these on day one, but names help when you read blogs:

My prototype stays server-side ONNX for simplicity: one model version, consistent embeddings, easier debugging.

What I’d tell past-me

• Learn the pipeline stages, not just “download a .onnx.”
• Treat alignment as mandatory.
• Treat embeddings + cosine as the heart of matching.
• Treat thresholds and re-enrollment as product features, not afterthoughts.
• Build something small that runs end-to-end — that’s when the theory finally sticks.

I always wanted to know how face recognition works. Building a prototype didn’t make me an author of CVPR papers — but it made the black box feel like engineering again: stages, contracts, scores, and tradeoffs.

If you’re a beginner-to-intermediate developer with the same curiosity: start with detect → align → embed → normalize → cosine. Everything else is depth on top of that spine.

The full working prototype is on GitHub: https://github.com/adityabbsharma/face-recognition-attendance-marking-app-public

In that repo you’ll find a Spring Boot backend (JWT auth, identity APIs, ONNX face detect/align/embed), a Dockerized MySQL database, a React (Vite) web app with webcam enroll and check-in, and a React Native Expo Android app for the same attendance flow on a phone.

To run it locally:

1. MySQL — from the repo root: docker compose up -d mysql (host port 3307)
2. API — cd backend && mvn -DskipTests install, then cd api-gateway && mvn spring-boot:run (http://localhost:8080)
3. Web — cd web && npm install && npm run dev (http://localhost:5173)
4. Mobile — cd mobile && npm install && npx expo start --clear, then open in Expo Go on the same Wi‑Fi (EXPO_PUBLIC_API_URL=http://<your-LAN-IP>:8080)

Clone it, enroll a face, watch a cosine score light up — and the next time someone says “AI attendance,” you’ll know exactly which stage they’re talking about.

#FaceRecognition #LearnInPublic #ComputerVision #ONNX #ArcFace #Developers #BuildInPublic