Building an AI-Powered PII Redaction Pipeline in Snowflake (Practical Implementation)

In modern data platforms, one of the biggest responsibilities of a data engineer is protecting sensitive user data.

Every day, organizations collect large volumes of customer interactions such as:

• support conversations
• service logs
• delivery notes
• customer feedback
• call center transcripts

These often contain Personally Identifiable Information (PII) like:

• Names
• Phone numbers
• Emails
• Addresses
• Date of birth

While this data is useful for analytics and operational insights, exposing raw PII to analysts or downstream applications can create serious privacy and compliance risks.

Regulations like GDPR, HIPAA, and PCI DSS require organizations to ensure that sensitive information is properly protected.

Recently, while working on a Snowflake implementation, I explored how Snowflake Cortex AI can automatically detect and redact sensitive information from text data.

In this article, I’ll walk through a practical implementation of AI-powered PII redaction using Snowflake.

The Real-World Problem

Imagine a customer support system storing interaction notes like this:

Customer Rahul Mehta contacted support using email rahul.mehta92@gmail.com and phone number 9876543210.
He reported an issue with his recent order delivered to 42 Park Residency, Andheri East, Mumbai.
Customer confirmed date of birth as 12 July 1992 during verification.

This information contains multiple PII elements:

• Customer name
• Email address
• Phone number
• Address
• Date of birth

If this data flows into analytics dashboards like:

• Tableau
• Power BI
• Sigma
• Streamlit apps

then analysts might accidentally gain access to sensitive personal information.

A better approach is to automatically remove PII while preserving the context of the text.

For example:

Customer [NAME] contacted support using email [EMAIL] and phone number [PHONE_NUMBER].
He reported an issue with his recent order delivered to [ADDRESS].
Customer confirmed date of birth as [DATE_OF_BIRTH] during verification.

This keeps the text useful for analysis while protecting the customer’s identity.

Snowflake Cortex AI_REDACT

Snowflake provides an AI function called **AI_REDACT** that automatically detects and replaces sensitive information in text.

The function identifies different types of PII and replaces them with placeholders such as:

[NAME]
[EMAIL]
[PHONE_NUMBER]
[ADDRESS]
[DATE_OF_BIRTH]

This makes it possible to implement AI-based data privacy directly inside Snowflake using SQL.

Architecture

A typical architecture for PII protection looks like this:

RAW DATA
   │
   │ (Contains PII)
   ▼
AI_REDACT Processing
   │
   ▼
SAFE DATA LAYER
   │
   ▼
Analytics / BI Tools

The idea is simple:

1. Raw data is stored securely.
2. AI redaction removes sensitive information.
3. Only redacted data is shared with analysts and applications.

Step 1 — Create the Raw Table

First, create a table that stores the original support interaction data.

CREATE OR REPLACE TABLE RAW_SUPPORT_INTERACTIONS (
    INTERACTION_ID NUMBER,
    CUSTOMER_NAME STRING,
    PHONE_NUMBER STRING,
    ADDRESS STRING,
    SUPPORT_NOTE STRING
);

Step 2 — Insert Sample Data

INSERT INTO RAW_SUPPORT_INTERACTIONS VALUES
(
1,
'Rahul Mehta',
'9876543210',
'42 Park Residency, Andheri East, Mumbai',
'Customer Rahul Mehta contacted support using email rahul.mehta92@gmail.com and phone number 9876543210. He reported an issue with his recent order delivered to 42 Park Residency, Andheri East, Mumbai. Customer confirmed date of birth as 12 July 1992 during verification.'
);

Step 3 — Apply AI Redaction

Now we can apply the AI_REDACT function.

SELECT
    INTERACTION_ID,
    AI_REDACT(CUSTOMER_NAME) AS CUSTOMER_NAME,
    AI_REDACT(PHONE_NUMBER) AS PHONE_NUMBER,
    AI_REDACT(ADDRESS) AS ADDRESS,
    AI_REDACT(SUPPORT_NOTE) AS SUPPORT_NOTE
FROM RAW_SUPPORT_INTERACTIONS;

Example output:

The AI automatically detects and replaces sensitive information.

Step 4 — Create a Safe Data Layer

In most production pipelines, we should store the redacted data in a separate table.

CREATE OR REPLACE TABLE SAFE_SUPPORT_INTERACTIONS AS
SELECT
    INTERACTION_ID,
    AI_REDACT(CUSTOMER_NAME) AS CUSTOMER_NAME,
    AI_REDACT(PHONE_NUMBER) AS PHONE_NUMBER,
    AI_REDACT(ADDRESS) AS ADDRESS,
    AI_REDACT(SUPPORT_NOTE) AS SUPPORT_NOTE
FROM RAW_SUPPORT_INTERACTIONS;

This table can now safely be used for analytics.

A Challenge I Faced During Implementation

While testing this implementation, I encountered the following error:

The model you requested is unavailable in your region.
Enable cross region inference.

This happens because Snowflake Cortex AI models may not be available in all regions.

The solution was to enable cross-region inference:

ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION';

Once this parameter was enabled, the redaction started working correctly.

Another Issue — NULL Results

Initially, some rows returned NULL values instead of redacted text.

The fix was enabling error-safe batch processing:

ALTER SESSION SET AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR = FALSE;

This prevents a single row error from failing the entire query.

Best Practices

From my experience implementing this pipeline, here are a few key recommendations.

1️⃣ Use AI redaction mainly for unstructured text

Columns like:

• support notes
• chat transcripts
• call center logs

benefit the most from AI detection.

2️⃣ Combine AI with masking policies

For structured PII columns such as:

• SSN
• credit card numbers
• phone numbers

Snowflake masking policies may provide stronger governance.

3️⃣ Never expose raw PII tables

Always implement a layered approach:

RAW DATA
   ↓
AI REDACTION
   ↓
SAFE TABLE
   ↓
ANALYTICS

This ensures sensitive information never reaches downstream tools.

Cost Considerations

One important aspect to understand when using Snowflake Cortex AI functions like AI_REDACT is cost.

Unlike traditional Snowflake queries that are mainly billed based on warehouse runtime, Cortex AI functions use a token-based pricing model.

This means the cost depends on how much text the AI processes.

What is a Token?

A token is a small piece of text processed by the model.

Rough approximation:

• 1 token ≈ 4 characters
• 1 token ≈ ¾ of an English word

So if you process a long paragraph, it will consume more tokens and therefore more cost.

Cost Considerations

One important aspect to understand when using Snowflake Cortex AI functions like AI_REDACT is cost.

Unlike traditional Snowflake queries that are mainly billed based on warehouse runtime, Cortex AI functions use a token-based pricing model.

This means the cost depends on how much text the AI processes.

How Snowflake AI Pricing Works

Snowflake Cortex AI services charge based on:

• Input tokens (text sent to the model)
• Output tokens (AI-generated response)

The total cost depends on the number of tokens processed and the model used.

Example approximate pricing (varies by model):

This means processing large volumes of text across millions of rows can significantly increase cost.

In fact, some large-scale Cortex queries processing billions of records have resulted in thousands of dollars in token charges when not carefully monitored.

Example Cost Scenario

Let’s say you run PII redaction on:

• 100,000 customer support notes
• Each note ≈ 120 tokens

Total tokens processed:

100,000 × 120 = 12,000,000 tokens

If the processing cost is around $0.0003 per 1K tokens, the estimated cost would be roughly:

12,000,000 tokens ≈ $3–$4

This is relatively inexpensive for small datasets, but cost grows quickly with large-scale pipelines.

Best Practices to Control AI Costs

From my experience implementing Cortex AI pipelines, here are a few practical recommendations.

Process only necessary columns

Avoid running AI_REDACT on entire tables unnecessarily.

Good candidates include:

• support notes
• chat transcripts
• free-text comments

Use batch processing

Instead of processing billions of rows in one query, process data in smaller batches.

Monitor token usage

Snowflake provides usage views that help track Cortex AI consumption.

You can monitor usage with queries against:

SNOWFLAKE.ACCOUNT_USAGE

This helps track token usage and credit consumption over time.

Redact once, store results

Instead of redacting data repeatedly in analytics queries, it is better to:

RAW TABLE
   ↓
AI_REDACT PROCESS
   ↓
SAFE TABLE

This prevents the AI function from running repeatedly and increasing cost.

Final Thoughts

AI-powered redaction significantly simplifies the challenge of protecting sensitive data in analytics platforms.

Instead of writing complex regex patterns or custom pipelines, Snowflake Cortex allows us to:

• automatically detect PII
• replace it with placeholders
• keep the text readable
• maintain compliance

And the best part is that it can be implemented using simple SQL directly inside Snowflake.

For data engineers building modern data platforms, this capability can dramatically improve data privacy, governance, and security.

About the Author

Sarvagya Shukla Senior Data Engineer | Snowflake | Data Platforms | AI for Data Engineering