Extracting Data from XFA PDF Forms in Snowflake
Getting Unstructured Data from Legacy Forms — No AI Required
Extracting Data from XFA-Type PDF Forms in Snowflake

Manual data entry is “old school”
(Note: While this article focuses on using Snowflake as the data platform, the code examples contained herein can be adapted to other platforms and use cases related to XFA data extraction)
Introduction and Background
With all the AI hype and hoopla it is easy to get lost in modern data management paradigms and forget about the massive (usually ginormous…yes, it’s a word) volume of legacy data assets that frequently get overlooked in the rush to implement the coolest and most modern data tools and processes. However, for organizations that have used, and continue to use, legacy tools and have amassed large volumes of data assets over time, those assets require additional attention so their valuable, and still useful, information can be leveraged. With that in mind, we have taken a hard look at using a modern data platform to extract data from “old school” data assets. In the following examination and examples, we use Snowflake to extract relevant data out of a library of dynamic XFA PDF forms developed years ago for collecting third party information. Those forms are still being used for data gathering for program and project management and are still very usable for related reporting and analytics.
For those who are unfamiliar with XFA forms, XFA stands for “XML Forms Architecture”. The XML Forms Architecture was developed by JetForm in 1999 and was submitted to the W3C as an XML-based specification for electronic forms. JetForm (which had subsequently been renamed as Accelio) was acquired by Adobe in 2002 and the XFA standard was integrated into Acrobat 6/PDF 1.5 which was introduced in 2003. It provided users with an alternative to the traditional (and more static) AcroForms and gave users the ability to create dynamic, XML-based forms within a PDF wrapper. XFA provided the ability to create “flowing forms” that resized, reformatted, and allowed for complex scripting. Adobe deprecated XFA in 2020 with the introduction of the PDF 2.0 standard (ISO 32000–2:2020).
At the Oregon Department of Transportation (ODOT), transportation and administrative programs that have been going on for years have used (and continue to use) digital documents to collect mission-critical data and information from internal and external partners and for contract management, compliance, and performance management. My office, the Data Solutions Office (DSO), works with internal business units to address the challenges of getting both current and historical data and information from unstructured data assets. This includes multiple formats of PDF documents that have been scanned (including handwritten and typed formats), received as native Adobe Acrobat PDF documents, and, most recently, XFA PDF forms. Historically (and sadly, currently), much, if not most, of the data extraction work from these digital documents has been done by hand by staff combing through documents and manually entering relevant data and information into (primarily) spreadsheets so the data can be captured and utilized for reporting, visualizations, and analyses. There are likely many who are reading this that are nodding their heads right now since this is not an uncommon scenario for most organizations that have been around for more than a decade or two.
ODOT acquired Snowflake as its primary analytics data lake/lakehouse platform in 2023. I’ve written about some of our more impactful work with Snowflake previously in Medium related to utilizing APIs with Workday and Smartsheet (see References at the end of this article). With Snowflake’s initial implementation of Document AI, now decommissioned in favor of the implementation of AI_EXTRACT, we were excited to apply new and modern tools to the challenge of extracting legacy data, not only from standard PDF documents, but from XFA PDF forms. However, the AI tools in Snowflake do not work with XFA forms unless they are “flattened” first. This was attempted, but it added a significant manual process into data acquisition flows. Unfortunately, we also found that flattened XFA forms were unreliable as source data because they often did not show all the data that could be in a form field. This was always true when the data entered went beyond the bounds of that field…only the data visible in the field when it was flattened was able to be seen and extracted using AI_EXTRACT. This was (and is) an unacceptable outcome, so another method was needed.
AI Disclaimer
Just a note before getting into the meat of this article. All the work shown here was done without utilizing Snowflake’s newly implemented Cortex Code capabilities due to Cortex Code not yet being released on Snowflake’s platform at the time this work was done. In this article, the modification of the original xfaTools.py script and the code related to cleaning of XML levels was assisted by prompts from ChatGPT.
The Approach
With a bit of trial and error and some online research, it was decided to use Python stored procedures and UDFs in Snowflake to do XFA Forms ELT…extract the data from the forms, load the resulting extracted XML into a Snowflake stage, and transform the data so they could be loaded into data tables for use in analytics, BI tools, dashboards, and for reporting. The result is a fast, reliable, effective, and efficient process for getting data from XFA PDF forms documents into our analytics lakehouse repository.
As already stated, XFA-based PDF forms have an XML framework at the core. The extraction process needed to look at the form’s “guts” and extract both the form’s architecture and the data that has been entered into each form. Since Snowflake’s implementation of AI_EXTRACT couldn’t work, it was determined from online documentation that the Python pikepdf library would likely provide the best functionality needed to get the XML structure and the embedded data extracted from the forms. This was subsequently tested successfully on a local computer with a small library of XFA PDF files and a few Python scripts. The success of this testing validated the assumption. This was the basis for the decision to use Python stored procedures and UDFs in Snowflake.
The second decision was to use Snowflake managed stages to store the files since we do not have/use AWS S2 buckets or MS Azure storage and the source files are currently stored on local network drives and not in any other cloud storage location.
Thirdly, the decision was made to convert the raw extracted XML to JSON for ease of use in the Snowflake platform and to standardize with our internal DataOps methodologies already in place with previous unstructured data work.
Based on testing and validation of various steps in the process the following workflow was developed.

Workflow for extracting data from XFA PDF forms in Snowflake
In the rest of this article, after getting set up (described in the next section), the remaining descriptions and code examples reference forms already placed in a Snowflake Stage as shown in the workflow diagram. Since each organization will have a different local infrastructure configuration, it is assumed that users will be able to develop their own processes for getting files into a Snowflake Stage or to can set up their configurations to access forms stored in another supported cloud storage instance (AWS, Azure, GCC, etc.).
Getting Set Up
First some “housekeeping” to get the working environment set up.
The initial set up required setting up a Python script to handle key components of the extraction process that would be embedded in the stored procedures. The following script was adapted from the xfaTools.py script from the GITHub pdf-xfa-tools example and is a key primary component that is used inside Snowflake and was placed into the stage so it can be called when needed.
# xfaTools.py
# -- Placed in the SF stage XFA_Forms_Stage/python_libs
import pikepdf
class XfaObj:
def __init__(self, pdf: pikepdf.Pdf):
self.xfaDict = {}
if not hasattr(pdf.Root, "AcroForm"):
return
acro = pdf.Root.AcroForm
if not hasattr(acro, "XFA"):
return
xfa = acro.XFA
for i in range(0, len(xfa), 2):
label = str(xfa[i])
value = xfa[i + 1]
if isinstance(value, pikepdf.Stream):
self.xfaDict[label] = value.read_bytes()
The remaining setup steps were completed using SQL in Snowflake Workspaces. Note that steps 2 & 3 have the database and schema set up on the SQL files’s configurations to run correctly using an existing X-Small Snowflake Warehouse so they are not called out explicitly in the code. As a standard best practice for our work, fully qualified names are used in all SQL coding.
- Create the database and schema for the XFA forms data work
USE ROLE ACCOUNTADMIN;
CREATE DATABASE XFA_DATA;
CREATE SCHEMA XFA_DATA.XFA_FORMS_SCHEMA;
- Create the JSON File Format
-- Create a file format that sets the file type as JSON for the json files in the stages
CREATE or Replace FILE FORMAT xfajsonformat
TYPE = json
STRIP_OUTER_ARRAY = FALSE
;
- Create two Snowflake managed stages and a one stage/subdirectory to contain the xfaTools.py script
--Create XFA Files Stage for loading the original XFA forms PDF files
CREATE OR REPLACE STAGE XFA_DATA.XFA_FORMS_SCHEMA.XFA_FORMS_STAGE
DIRECTORY = (ENABLE = TRUE)
ENCRYPTION = (TYPE = ’SNOWFLAKE_SSE’)
;
--Create the Stage to receive the processed XML and JSON files
CREATE OR REPLACE STAGE XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE
DIRECTORY = (ENABLE = TRUE)
ENCRYPTION = (TYPE = ’SNOWFLAKE_SSE’)
;
--A stage subdirectory cannot be created in a SF stage explicitly. Must load a file into a path that doesn’t exist
--Load the xfaTools.py file into the stage to create the subdirectory
PUT file://<local file path>/xfaTools.py @XFA_FORMS_STAGE/python_libs;
Step 1: Convert XFA Forms to XML Files
The following Python stored procedure works to process all the XFA files in the an input stage and outputs the results as XML files into an output stage:
CREATE OR REPLACE PROCEDURE PROCESS_ALL_XFAPDFS(
INPUT_STAGE STRING,
OUTPUT_STAGE STRING
)
RETURNS VARIANT
LANGUAGE PYTHON
RUNTIME_VERSION = 3.11
PACKAGES = ('snowflake-snowpark-python', 'pikepdf')
IMPORTS = ('@xfa_forms_stage/python_libs/xfaTools.py')
HANDLER = 'run'
AS
$$
import io
from xfaTools import XfaObj
import pikepdf
def read_internal_stage_file(session, input_stage: str, list_name: str) -> bytes:
# expected format of input_stage variable: '@XFA_FORMS_STAGE'
# expected format of list_name variable: 'xfa_forms_stage/file1.pdf'
# Normalize stage name
stage = input_stage.rstrip("/")
# LIST output usually includes stage name prefix
if "/" in list_name:
_, rel_path = list_name.split("/", 1)
else:
rel_path = list_name
full_path = f"{stage}/{rel_path}"
with session.file.get_stream(full_path) as f:
return f.read()
def run(session, input_stage: str, output_stage: str):
results = {
"processed": 0,
"written": 0,
"skipped": 0,
"errors": []
}
# LIST must use the stage name exactly as passed in
files = session.sql(f"LIST {input_stage}").collect()
for row in files:
name = row["name"]
if not name.lower().endswith((".pdf", ".pdf.gz")):
continue
results["processed"] += 1
try:
pdf_bytes = read_internal_stage_file(session, input_stage, name)
pdf = pikepdf.Pdf.open(io.BytesIO(pdf_bytes))
xfa = XfaObj(pdf)
if not xfa.xfaDict:
results["skipped"] += 1
continue
for key, xml_bytes in xfa.xfaDict.items():
output_file = (
name.split("/")[-1]
.rsplit(".", 1)[0]
+ f"_{key}.xml"
)
session.file.put_stream(
io.BytesIO(xml_bytes),
f"{output_stage}/{output_file}",
overwrite=True,
auto_compress=False
)
results["written"] += 1
except Exception as e:
results["errors"].append({
"file": name,
"error": str(e)
})
return results
$$;
To call the procedure to process all of the files in the XFA_FORMS_STAGE and output the resulting XML files to the output stage XFA_XMLOUT_STAGE:
-- Call the procedure PROCESS_ALL_XFAPDFS(input_stage, list_name)
CALL PROCESS_ALL_XFAPDFS('@XFA_DATA.XFA_FORMS_SCHEMA.XFA_FORMS_STAGE', '@XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE');
For a test run of 40 XFA PDF files processed, the output that returned for a fully successful run is:
{
"errors": [],
"processed": 40,
"skipped": 0,
"written": 360
}
Note that for 40 XFA PDF files that are processed, 360 XML files are written to the receiving stage. This is because the stored procedure returns 9 XML files for each PDF. These nine files are shown as follows for a PDF file named File01.pdf:
--List xml files in the stage
LIST @XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE';
File01_</xdp:xdp>.xml
File01_config.xml
File01_datasets.xml
File01_form.xml
File01_localeSet.xml
File01_template.xml
File01_xdp:xdp.xml
File01_xfdf.xml
File01_xmpmeta.xml
It is not the purpose of this discussion to go over the contents of each of these output files. The reader can look at them after doing extractions.
There is only one file that is of interest for acquiring the data from each form. In this example it is the file named File01_datasets.xml since the form data is all contained in that one file.
To keep things tidy, the following code is used to remove all XML files except for the “…datasets.xml” files from the stage,
-- Remove all xml files that are not "datasets" files from the stage OECR_XMLOUT_STAGE
REMOVE @XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE PATTERN='(?i)^(?!.*datasets\.xml).*$';
Step 2: Clean up the top levels of the XML tree and convert XML files to JSON
The next two transforms happen at the same time in the following procedure. The first removes several extraneous top level tree items from each XML file down to the first key node that has more than one child or meaningful structure and makes that node the top level in the XML “tree”. Users may or may not need to adjust the XML “tree” depending on the format of the XML files being processed. In this example, the forms we processed had 2–3 unneeded levels that complicated the downstream code that would require additional “tree” levels that really served no purpose for the extraction of the relevant data.
The second transform in this procedure converts each cleaned XML file to JSON and writes the JSON file to the stage.
CREATE OR REPLACE PROCEDURE XML_TO_JSON_NEWTOPLVL(
stage STRING,
filter_token STRING
)
RETURNS VARIANT
LANGUAGE PYTHON
RUNTIME_VERSION = 3.11
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'run'
AS
$$
import io
import json
import xml.etree.ElementTree as ET
import re
def collapse_single_child(node):
"""
Walk down XML tree until the node has
more than one child or meaningful structure.
"""
children = list(node)
while len(children) == 1:
node = children[0]
children = list(node)
return node
def xml_element_to_value(elem):
children = list(elem)
text = (elem.text or "").strip()
# Leaf node → scalar
if not children:
return text
# Parent node
obj = {}
for child in children:
tag = child.tag.split("}")[-1]
value = xml_element_to_value(child)
if tag in obj:
if not isinstance(obj[tag], list):
obj[tag] = [obj[tag]]
obj[tag].append(value)
else:
obj[tag] = value
return obj
def run(session, stage: str, filter_token: str):
results = {
"processed": 0,
"written": 0,
"skipped": 0,
"errors": []
}
stage = stage.rstrip("/")
files = session.sql(f"LIST {stage}").collect()
for row in files:
name = row["name"]
# Only XML
if not name.lower().endswith(".xml"):
continue
# Optional filter
if filter_token and filter_token.lower() not in name.lower():
results["skipped"] += 1
continue
results["processed"] += 1
try:
# Normalize LIST path
if "/" in name:
_, rel_path = name.split("/", 1)
else:
rel_path = name
full_path = f"{stage}/{rel_path}"
# Read XML
with session.file.get_stream(full_path) as f:
xml_bytes = f.read()
# Parse XML
root = ET.fromstring(xml_bytes)
root = collapse_single_child(root)
xml_filename = rel_path.split("/")[-1].rsplit(".", 1)[0]
# derive normalized document key
xml_key = (
rel_path
.split("/")[-1]
.rsplit(".", 1)[0]
.replace(" ", "-")
)
# optional hardening
xml_key = re.sub(r"[^A-Za-z0-9_-]", "-", xml_key)
# build JSON
json_obj = {
xml_key: {
root.tag.split("}")[-1]: xml_element_to_value(root)
}
}
json_bytes = json.dumps(
json_obj,
indent=2,
ensure_ascii=False
).encode("utf-8")
json_name = rel_path.rsplit(".", 1)[0] + ".json"
session.file.put_stream(
io.BytesIO(json_bytes),
f"{stage}/{json_name}",
overwrite=True,
auto_compress=False
)
results["written"] += 1
except Exception as e:
results["errors"].append({
"file": name,
"error": str(e)
})
return results
$$;
To call the procedure, pass in the stage name and the filter value…in this case the filter is for files with the word ‘datasets’ in the name.
CALL XML_TO_JSON_NEWTOPLVL('@XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE', 'datasets');
While some readers may want to keep or archive the extra XML files, we had no further need for the .xml files so they were removed. To remove them from the stage and keep down the “clutter”, the following code was run leaving only the relevant JSON data files in the stage.
REMOVE @XFA_DATA.XFA_FORMS_SCHEMA.XFA_XMLOUT_STAGE PATTERN='.*.xml';
Step 3: Parse JSON out to tables
In this step, the example will use only one of the XFA forms’ file names as a reference of the process. We had six (6) different XFA forms that were processed and the steps are the same, but the actual extraction code will differ depending on the format of the XFA form itself and the number of fields and tables that are embedded in each XFA form type.
In this work we decided to use the form name as a unique identifier for the data contained in the form. This assisted in joining data extracted into a “parent” table and, because there is often tabular data embedded in an XFA form, and data broken out into one or more “child” tables. The ability to have embedded tables is one of the strengths of XFA type PDF forms as they allow the embedded tables to dynamically grow as additional data are added to the form.
In this example a form that is identified as a “PSR” form is used. The naming convention for the file follows the format shown below:
<contract number> PSR <subcontract ID> <report date> <invoice number> <rev status>.pdf
Using this format the PDF file name may look something like this:
12345 PSR 01 20251130 19 R.pdf
First, and as our standard practice when dealing with JSON data files, a Snowflake VARIANT table was created to capture all the raw JSON in preparation for further processing and to act as a raw data source file for archiving.
-- Create the VARIANT table to hold the raw JSON from each PSR "datasets" file
CREATE OR REPLACE TABLE XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_JSONDataDump
(PSR_JSON_EXTRACT VARIANT)
;
Next, the full JSON contents were copied from each of the PSR files in the stage into the VARIANT table. This results in one variant row per PSR JSON file in the stage.
-- Copy all the json in each file with "PSR" in the name into a variant field in the table
-- Creates one row per file processed
COPY INTO XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_JSONDATADUMP (PSR_JSON_EXTRACT)
FROM (SELECT $1 FROM @XFA_XMLOUT_STAGE)
PATTERN='.*PSR.*.json' -- Optional: use a regex pattern to select specific files
FILE_FORMAT = (FORMAT_NAME = xfajsonformat)
ON_ERROR = 'CONTINUE'
;
Because the PSR XFA form has embedded tables, a parent table is created to hold the primary data elements in the form (e.g., the form fields that hold only one value per form). Then, additional child tables are created to hold variable numbers of rows that come from each embedded table in the XFA form.
The following is representative of the code needed to create the parent table. Note that the first field called FORM_ID is the field that holds the unique value element that is created from the file name mentioned above. Obviously, the number of columns will vary depending on the user’s XFA form structure.
-- Create the PSR ParentTable
CREATE OR REPLACE TABLE XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_PARENT
(
FORM_ID VARCHAR(60),
<FIELD 1 NAME> NUMBER(4.0),
<FIELD 2 NAME> DATE,
<FIELD 3 NAME> VARCHAR(300),
<FIELD 4 NAME> NUMBER(2,0),
…etc...
);
With the parent table created, the data can be loaded from the Variant table. In this example there was a need to adjust for some variability in “DATE” formats submitted in the forms to ensure they imported cleanly in a standard date format regardless of the raw format. Hence the COALESCE function in the code.
Note that the referenced json keys in this example are the ones listed in an actual PSR form and are shown with the value named “doc_body” and the embedded keys in the JSON “tree” listed separated by colons (e.g., doc_body:Page1:TitleBlock:ProgEstInvDate) as is standard for working with JSON in SQL. In most instances the names of the fields in the XFA form were kept as column names in the tables.
-- Load the PARENT PSR Table from the variant data
INSERT INTO XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_PARENT
SELECT
doc_id,
doc_body:Page1:TitleBlock:ProgEstInvoice::number,
COALESCE(
TRY_TO_DATE(doc_body:Page1:TitleBlock:ProgEstInvDate::string, 'yyyy-mm-dd'),
TRY_TO_DATE(doc_body:Page1:TitleBlock:ProgEstInvDate::string, 'mm-dd-yyyy'),
TRY_TO_DATE(doc_body:Page1:TitleBlock:ProgEstInvDate::string, 'yyyymmdd'),
TRY_TO_DATE(doc_body:Page1:TitleBlock:ProgEstInvDate::string, 'mmddyyyy')
),
doc_body:Page1:Line1:ContrName::string,
doc_body:Page1:Line1:RevNo::number,
<….ETC…>
)
FROM (
SELECT
SUBSTRING(k.value::string, 1, LENGTH(k.value::string) - 9) AS doc_id,
v.value AS doc_body
FROM XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_JSONDataDump,
LATERAL FLATTEN(input => PSR_JSON_EXTRACT) v,
LATERAL FLATTEN(input => OBJECT_KEYS(PSR_JSON_EXTRACT)) k
);
Creating and loading the data from one or more embedded tables in the XFA form follows the same process as shown below for creating a child table to the parent table created above. The field SUB_FORM_ID holds the same value as the FORM_ID field in the parent table. Again, the reader’s code will vary depending on the number of fields in their XFA form file(s). Note the use of COALESCE used to get float values correctly imported.
-- Create one of the child tables for the PSR forms
CREATE OR REPLACE TABLE XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_CHILD
(
SUB_FORM_ID VARCHAR(60),
<TABLE 1 FIELD A NAME> VARCHAR(60),
<TABLE 1 FIELD B NAME> VARCHAR(50),
<TABLE 1 FIELD C NAME> FLOAT,
<TABLE 1 FIELD D NAME> NUMBER(4,2),
…etc.
);
-- Load the CHILD Table from the variant data
-- This will load multiple rows depending on how many are in the embedded table in the form
INSERT INTO XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_CHILD
SELECT
doc_id,
d.value:FirmName::string,
d.value:SubIDno::string,
COALESCE(
TRY_TO_DOUBLE(TO_VARCHAR(d.value:SubContractAmt)), NULL
),
d.value:Tasks::number,
-- <input additional json values references here>
)
FROM (
SELECT
SUBSTRING(k.value::string, 1, LENGTH(k.value::string) - 9) AS doc_id,
v.value AS doc_body
FROM XFA_DATA.XFA_FORMS_SCHEMA.XFA_PSRFILE_JSONDataDump,
LATERAL FLATTEN(input => PSR_JSON_EXTRACT) v,
LATERAL FLATTEN(input => OBJECT_KEYS(PSR_JSON_EXTRACT)) k
),
LATERAL FLATTEN(
input => doc_body:Page1:Table1:Row1
) d;
Once tested and validated, the user can set up tasks to automate all the steps in the process, with appropriate data quality and error checking inserted in the automation pathway. This article does not step through setting up tasks as the number and types will likely vary depending on the environments involved as well as the desired error monitoring checkpoints.
Summary and Conclusion
Using native AI tools in Snowflake works very well for extracting unstructured data from standard PDF files but does not work for dynamic XFA PDF forms files unless they are “flattened” which excludes any data that extends past the end of a field boundary in the form. The core of this document outlines a fast, efficient methodology for extracting data from XFA forms into Snowflake tables using a combination of Python and SQL. Our experience is that it is fast and accurate.
Per our best practice standard for ETL/ELT pipelines development, the process converts extracted XML to JSON for improved data ingestion and post-load processing. For those who prefer working with XML files directly, the conversion to JSON can be left out of the examples above and the subsequent data loading scripts adjusted accordingly.
Modernizing and automating the highly manual processes of reading and keying data from legacy unstructured data sources is a game changer. It not only positively impacts data quality and data availability, but ensures that relevant and valuable data are ready for use when needed. Additionally, having the ability to ensure that when new forms are received, the extraction and loading of embedded data occurs without delay and without human intervention moves organizations away from reactive data preparation practices and towards proactive data-informed decision-making.
References
- Three part series on Snowflake and accessing/using Smartsheet data:
- Using Snowflake to access a Workday API for data acquisition
메타데이터
- post_id
- fe84e6bbb5f5
- slug
- extracting-data-from-xfa-pdf-forms-in-snowflake-fe84e6bbb5f5
- url
- https://medium.com/@mrwoodford7/extracting-data-from-xfa-pdf-forms-in-snowflake-fe84e6bbb5f5
- canonical_url
- https://medium.com/@mrwoodford7/extracting-data-from-xfa-pdf-forms-in-snowflake-fe84e6bbb5f5
- author_url
- https://medium.com/@mrwoodford7
- status
- ok
- fetched_at
- 2026-07-08 02:40:31