Evaluating Document AI Frameworks: Spark NLP vs Unstructured for Large-Scale Text Processing

Problem1: Extracting Complete Text Coverage from Complex Documents

In many enterprise pipelines from compliance auditing to enterprise search and knowledge-graph building teams must extract every piece of visible text from large collections of mixed documents (PDFs, HTML pages, and DOCX files). This includes not just paragraphs and headings, but also text found in:

• Navigation menus and footers
• Captions and embedded annotations
• Tables, figure titles, disclaimers, and metadata fields

Capturing all visible text is essential when building traceable, auditable corpora where any omission (even from navigation or footer content) could lead to information loss or compliance gaps.

How Spark NLP Solves It

Spark NLP provides a unified data-processing and NLP pipeline that can read, parse, and clean text from diverse formats at scale using its Readers2X components.

To clean text extracted from HTML using Spark NLP, we leveraged the following annotators:

reader2doc = Reader2Doc() \
    .setContentType('text/html') \
    .setContentPath(directory) \
    .setOutputCol('document')

normalizer = DocumentNormalizer() \
    .setInputCols(['document']) \
    .setOutputCol('normalized') \
    .setAutoMode("HTML_CLEAN") \
    .setPatterns([(":")])

sentence_detector = SentenceDetectorDLModel() \
    .pretrained() \
    .setInputCols(['normalized']) \
    .setOutputCol('sentences') \
    .setExplodeSentences(True)

When processing large volumes of documents, we simply leverage Spark’s native distributed engine to scale efficiently:

pipeline = Pipeline(stages=[reader2doc, normalizer, sentence_detector])
model = pipeline.fit(empty_df)
result_df = model.transform(empty_df)

flat_df = (
    result_df
    .withColumn("sentence", explode("sentences"))
    .select(
        col("filename"),
        col("sentence.result").alias("result")
    )
)

• Complete coverage: Reader2Doc extracts the full visible text layer, including navigation menus, headers, and footers. It does not force semantic filtering or heuristics that skip content.
• Scalable processing: Built on Apache Spark, it can handle millions of files distributed across clusters, ensuring fast ingestion and consistent structure.
• Unified pipeline: The extracted text can flow directly into tokenizers, sentence detectors, embeddings, or downstream NLP models without reformatting.
• Traceability: Every document keeps metadata such as source path, page number, and character offsets, supporting audit and compliance needs.

This makes Spark NLP particularly strong for enterprise-scale ingestion, full-text indexing, and document alignment tasks where completeness and consistency outweigh minimalism.

How Unstructured Handles It

Unstructured’s partition is designed with a different philosophy: to extract semantic content. They segment a document into structured “elements” e.g. Titles, NarrativeText, Tables, etc. while discarding what appears to be boilerplate, such as <nav> menus or repetitive links.

To clean text extracted from HTML using Unstructured, we relied on its built-in cleaning utilities and added a custom function to remove colon characters:

def remove_colons(text: str) -> str:
    return re.sub(r":", "", text)

def clean_element_text(text: str) -> str:
    text = clean_extra_whitespace(text)
    text = replace_unicode_quotes(text)
    text = clean_non_ascii_chars(text)
    text = clean_bullets(text)
    text = remove_colons(text)
    return text.strip()

When processing large volumes of documents, a specialized function is required to iterate over the entire directory and apply this logic to each HTML file:

def ingest_and_clean_unstructured_html(html_path: str):
    elements = partition_html(filename=html_path)
    cleaned_output = []

    for el in elements:
        if hasattr(el, 'text') and el.text:
            cleaned_text = clean_element_text(el.text)
            cleaned_output.append({
                "filename": os.path.basename(html_path),
                "type": el.category if hasattr(el, 'category') else el.__class__.__name__,
                "text": cleaned_text
            })
    return cleaned_output

def process_html_directory(directory_path: str):
    all_results = []

    # Loop through all files in the directory
    for filename in os.listdir(directory_path):
        if filename.lower().endswith(".html"):
            file_path = os.path.join(directory_path, filename)
            print(f"🔍 Processing: {file_path}")

            try:
                output_html = ingest_and_clean_unstructured_html(file_path)
                all_results.extend(output_html)
            except Exception as e:
                print(f"⚠️ Error processing {filename}: {e}")

While this produces cleaner and more human-readable content, it also means that:

• Navigation or meta text is intentionally dropped.
• Structural cues like captions may be separated from their context.
• When completeness is required, downstream users have no direct way to recover filtered text, because Unstructured doesn’t retain the full raw text document stream.
• Processing is file-by-file on CPU, without distributed scaling or Spark integration.

Thus, while Unstructured is ideal for content-centric summarization or LLM preprocessing, it is not appropriate for pipelines that require full document coverage or raw-text fidelity.

[embed]

To illustrate this scenario, we developed a notebook that processes a set of medical records and evaluates the quality of text extraction using a simple Jaccard similarity metric. The results show both frameworks performing closely:

However, as the saying goes, the devil is in the details. A deeper token-level analysis revealed that both frameworks missed the token **dashboard, but Unstructured** also omitted several contextually important words such as:

These missing tokens can be critical in clinical and biomedical contexts, where small lexical gaps may significantly affect the outcomes of downstream NLP tasks. Therefore, despite the seemingly similar similarity scores, these subtle omissions could lead to substantial performance differences in real-world NLP pipelines.

You can review the full notebook for this experiment and the result metrics👉 here

Problem2: Maintaining Structural Context for Data-Rich Documents

In many enterprise domains such as healthcare, finance, insurance, scientific publishing, and legal discovery critical insights critical insights are embedded in structured elements like tables and figures. These are not just blobs of text; their meaning depends heavily on their position within the document, including headers, captions, nearby narrative text, and visual layout.

Without preservation of this structural context, downstream NLP systems struggle to interpret, relate, and reason over the extracted information. For example:

• A clinical lab table needs to be associated with its section heading (“Most Recent Laboratory Results”) so decision support systems know which test belongs to which patient visit.
• A financial table summarizing quarterly results must be tied to the correct caption and date range to feed into a BI dashboard.
• Scientific documents often contain dozens of tables and figures where semantic relationships between text and tables are essential for accurate knowledge extraction and reasoning.

This structural understanding matters not just for content extraction but for semantic NLP tasks such as information extraction, table-aware question answering, contextual reasoning, and knowledge graph construction. Research shows that incorporating structural and layout information significantly improves document understanding and extraction quality because it helps NLP systems interpret data in context, not just as isolated text or table cells [1].

How Spark NLP Solves It

Spark NLP’s Reader2Table orReader2Imageaddresses this challenge by preserving structural and positional metadata during extraction. Each table or image is enriched with information such as its DOM path, nearest header, and section hierarchy, ensuring every piece of extracted data remains tied to its original context.

empty_df = spark.createDataFrame([], 'string').toDF('text')

reader2doc = Reader2Table() \
    .setContentType('text/html') \
    .setContentPath('html_docs/EHR-2025-12-000002.html') \
    .setOutputCol('table') \
    .setExplodeDocs(True)

pipeline = Pipeline(stages=[reader2doc])
model = pipeline.fit(empty_df)
result_df = model.transform(empty_df)

JSON output

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|result                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[{"caption":"","header":["Test","Result","Units","Reference Range","Status"],"rows":[["PSA","0.32","ng/mL","0-4.0","Excellent"],["Testosterone","125","ng/dL","300-1000","Recovering"],["Hemoglobin","14.3","g/dL","13.5-17.5","Normal"],["WBC","7.2","K/uL","4.5-11.0","Normal"],["Creatinine","0.9","mg/dL","0.7-1.3","Normal"],["ALT","22","U/L","7-56","Normal"]]}]                                                                        |
|[{"caption":"","header":["Test","Result","Units","Reference Range","Status"],"rows":[["Testosterone","105","ng/dL","300-1000","Recovering"],["Hemoglobin","12.3","g/dL","13.5-17.5","Normal"],["Creatinine","0.7","mg/dL","0.7-1.3","Normal"]]}]                                                                                                                                                                                               |
|[{"caption":"","header":["Medication","Dose","Frequency","Indication","Status"],"rows":[["Atorvastatin (Lipitor)","10 mg PO","Daily","Hyperlipidemia","Active"],["Aspirin","81 mg PO","Daily","Cardiovascular prophylaxis","Active"],["Vitamin D3","2000 IU PO","Daily","Bone health","Active"],["Calcium carbonate","500 mg PO","BID","Bone health (post-ADT)","Active"],["Multivitamin","1 tab PO","Daily","Nutritional support","Active"]]}]|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

HTML output

reader2doc = Reader2Table() \
    .setContentType('text/html') \
    .setContentPath('html_docs/EHR-2025-12-000002.html') \
    .setOutputCol('table') \
    .setOutputFormat('html-table') \
    .setExplodeDocs(True)

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[<table class="lab-table"><thead><tr><th>Test</th><th>Result</th><th>Units</th><th>Reference Range</th><th>Status</th></tr></thead><tbody><tr><td>PSA</td><td><strong>0.32</strong></td><td>ng/mL</td><td>0-4.0</td><td><span class="status-badge status-active">Excellent</span></td></tr><tr><td>Testosterone</td><td><strong>125</strong></td><td>ng/dL</td><td>300-1000</td><td><span class="status-badge status-completed">Recovering</span></td></tr><tr><td>Hemoglobin</td><td><strong>14.3</strong></td><td>g/dL</td><td>13.5-17.5</td><td><span class="status-badge status-active">Normal</span></td></tr><tr><td>WBC</td><td><strong>7.2</strong></td><td>K/uL</td><td>4.5-11.0</td><td><span class="status-badge status-active">Normal</span></td></tr><tr><td>Creatinine</td><td><strong>0.9</strong></td><td>mg/dL</td><td>0.7-1.3</td><td><span class="status-badge status-active">Normal</span></td></tr><tr><td>ALT</td><td><strong>22</strong></td><td>U/L</td><td>7-56</td><td><span class="status-badge status-active">Normal</span></td></tr></tbody></table>]|
|[<table class="lab-table"><thead><tr><th>Test</th><th>Result</th><th>Units</th><th>Reference Range</th><th>Status</th></tr></thead><tbody><tr><td>Testosterone</td><td><strong>105</strong></td><td>ng/dL</td><td>300-1000</td><td><span class="status-badge status-completed">Recovering</span></td></tr><tr><td>Hemoglobin</td><td><strong>12.3</strong></td><td>g/dL</td><td>13.5-17.5</td><td><span class="status-badge status-active">Normal</span></td></tr><tr><td>Creatinine</td><td><strong>0.7</strong></td><td>mg/dL</td><td>0.7-1.3</td><td><span class="status-badge status-active">Normal</span></td></tr></tbody></table>]                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|[<table class="lab-table"><thead><tr><th>Medication</th><th>Dose</th><th>Frequency</th><th>Indication</th><th>Status</th></tr></thead><tbody><tr><td>Atorvastatin (Lipitor)</td><td>10 mg PO</td><td>Daily</td><td>Hyperlipidemia</td><td><span class="status-badge status-active">Active</span></td></tr><tr><td>Aspirin</td><td>81 mg PO</td><td>Daily</td><td>Cardiovascular prophylaxis</td><td><span class="status-badge status-active">Active</span></td></tr><tr><td>Vitamin D3</td><td>2000 IU PO</td><td>Daily</td><td>Bone health</td><td><span class="status-badge status-active">Active</span></td></tr><tr><td>Calcium carbonate</td><td>500 mg PO</td><td>BID</td><td>Bone health (post-ADT)</td><td><span class="status-badge status-active">Active</span></td></tr><tr><td>Multivitamin</td><td>1 tab PO</td><td>Daily</td><td>Nutritional support</td><td><span class="status-badge status-active">Active</span></td></tr></tbody></table>]                                                                                                                      |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Metadata output:

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|metadata                                                                                                                                                                                                 |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[orderTableIndex -> 1, nearestHeader -> 🔬 Most Recent Laboratory Results (10/22/2016), pageNumber -> 1, domPath -> /html[1]/body[1]/div[1]/div[3]/div[4]/table[1], elementType -> Table, sentence -> 8}]|
|[orderTableIndex -> 2, nearestHeader -> History Laboratory Results (10/22/2016), pageNumber -> 1, domPath -> /html[1]/body[1]/div[1]/div[3]/div[4]/table[2], elementType -> Table, sentence -> 10}]      |
|[orderTableIndex -> 1, nearestHeader -> 💊 Current Medications, pageNumber -> 1, domPath -> /html[1]/body[1]/div[1]/div[3]/div[5]/table[1], elementType -> Table, sentence -> 12}]                       |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

This output captures rich structural information alongside extracted content:

• DOM paths (e.g., /html[1]/body[1]/div[3]/div[5]/table[1]) that identify exactly where in the HTML document a table or image came from.
• Nearest section header context so that a table is semantically linked to its surrounding narrative (“Laboratory Results”, “Current Medications”, etc.).
• Order and hierarchy metadata such as orderTableIndex, allowing precise reconstruction of document structure.
• A structured JSON representation of tables (with headers, rows, captions, and field metadata)
• A HTML representation for visualization, rendering, or further processing.

These enriched representations help downstream NLP tasks such as:

• Table-aware question answering: Models like TAPAS leverage structured table data to answer natural language questions over tables with high accuracy, something that plain text extraction alone cannot support [2].
• Contextual table interpretation: Structural metadata enables models to understand why a table occurs where it does, improving joint inference between narrative text and tabular data, which is known to boost extraction quality when the context is considered [1].
• Semantic integration with knowledge graphs and IE systems: By preserving layout and section cues, extracted table data can be merged into structured knowledge representations with clear provenance.

In practice, this means that Spark NLP pipelines don’t just flatten structured content they provide traceable, semantically rich extractions that downstream models can consume with minimal ambiguity.

How Unstructured Handles It

Unstructured’s partition_html module focuses on extracting semantic content titles, paragraphs, tables, and images but does not preserve the structural layout or positional hierarchy of those elements.

def ingest_and_clean_unstructured_html_tables(html_path: str):
    """
    Extract and clean only HTML table data using Unstructured.
    Returns a list of dicts with text and HTML (if available).
    """
    elements = partition_html(filename=html_path)
    cleaned_output = []

    for el in elements:
        if hasattr(el, "category") and el.category == "Table":
            table_text = getattr(el, "text", None)
            cleaned_entry = {
                "filename": os.path.basename(html_path),
                "type": "Table",
            }

            # clean and add plain text
            if table_text:
                cleaned_entry["text"] = clean_element_text(table_text)

            # look in metadata for HTML, if it exists
            if hasattr(el, "metadata") and isinstance(el.metadata, dict):
                html_content = el.metadata.get("text_as_html")
                if html_content:
                    cleaned_entry["text_as_html"] = html_content

            cleaned_output.append(cleaned_entry)

    return cleaned_output

Output example:

[{'filename': 'EHR-2025-12-000002.html',
  'type': 'Table',
  'text': 'Test Result Units Reference Range Status PSA 0.32 ng/mL 0-4.0 Excellent Testosterone 125 ng/dL 300-1000 Recovering Hemoglobin 14.3 g/dL 13.5-17.5 Normal WBC 7.2 K/uL 4.5-11.0 Normal Creatinine 0.9 mg/dL 0.7-1.3 Normal ALT 22 U/L 7-56 Normal'},
 {'filename': 'EHR-2025-12-000002.html',
  'type': 'Table',
  'text': 'Test Result Units Reference Range Status Testosterone 105 ng/dL 300-1000 Recovering Hemoglobin 12.3 g/dL 13.5-17.5 Normal Creatinine 0.7 mg/dL 0.7-1.3 Normal'},
 {'filename': 'EHR-2025-12-000002.html',
  'type': 'Table',
  'text': 'Medication Dose Frequency Indication Status Atorvastatin (Lipitor) 10 mg PO Daily Hyperlipidemia Active Aspirin 81 mg PO Daily Cardiovascular prophylaxis Active Vitamin D3 2000 IU PO Daily Bone health Active Calcium carbonate 500 mg PO BID Bone health (post-ADT) Active Multivitamin 1 tab PO Daily Nutritional support Active'}]

Unstructured does not support text_as_html field for HTML files.

While the output for other file types may include a text_as_html field containing the visual representation of a table, it lacks:

• The document’s DOM ancestry
• Section or caption linkage (no nearest header tracking)
• Element order within the page layout

As a result, Unstructured’s table output is essentially context-agnostic. While the extracted content may be correct in isolation, the surrounding structural relationships (vital for holistic NLP tasks) are lost. This limitation inhibits the use of Unstructured outputs in workflows that depend on understanding how the table fits into the document narrative or layout.

This notebook showcases the extraction of tabular data across both frameworks and emphasizes Spark NLP’s capability to generate rich DOM-structured output for precise contextual alignment.

[embed]

In real-world enterprise pipelines, accurate understanding of where structured data appears not just what it contains enables advanced NLP use cases such as table question answering, context-aware information extraction, and integration into knowledge graphs. Spark NLP’s DOM-aware, dual JSON/HTML representations provide the structural foundation these tasks require, whereas simpler extraction tools lack the necessary positional fidelity.

Problem3: Processing Millions of Documents Efficiently and Reliably

Modern organizations are often tasked with processing massive volumes of unstructured documents PDFs, HTML pages, contracts, medical reports, or regulatory filings often numbering in the millions. These files arrive daily via ingestion pipelines, enterprise content systems, or compliance workflows.

While sequential or single-machine processing might suffice for small datasets, scaling becomes a serious challenge as data grows. Common bottlenecks include:

• Excessive processing time when files must be handled one by one
• Inconsistent outputs when pipelines fail mid-run and require manual restarts
• Escalating infrastructure costs due to lack of distributed workload handling
• Difficulty scaling NLP pipelines as tokenization, entity recognition, and classification steps are added

This becomes a critical bottleneck for data engineering teams tasked with maintaining real-time compliance, analytics, or document understanding workflows.

How Spark NLP Solves It

Spark NLP is built natively on Apache Spark, bringing distributed data processing to text analytics and NLP workloads.. This means text extraction, normalization, and NLP tasks can be performed in parallel across clusters, allowing millions of documents to be processed efficiently, reproducibly, and at scale.

Key advantages include:

• Scalable architecture: Workloads are automatically partitioned across Spark executors, ensuring linear scalability as cluster resources grow.
• Fault tolerance: Automatic checkpointing and resilient distributed datasets (RDDs) guarantee recovery from node or job failures.
• Unified pipeline integration: Document ingestion, extraction (Reader2Doc, Reader2Table, Reader2Image, ReaderAssembler), tokenization, and NLP inference can all run as a single Spark job no need to move data between tools.
• Operational efficiency: Ideal for enterprise pipelines that process terabytes of data daily.

Here’s a minimal example that ingests all files in a directory using Spark NLP’s ReaderAssembler and saves the results as a Parquet dataset:

reader_assembler = ReaderAssembler() \
    .setContentPath(directory) \
    .setOutputCol("document")

pipeline = Pipeline(stages=[reader_assembler])
model = pipeline.fit(empty_df)

df = model.transform(empty_df)
df.select("document_text.result").write.mode("overwrite").parquet(output)

How Unstructured Handles It

Unstructured, by design, is a single node Python library optimized for lightweight document parsing and LLM preprocessing, not distributed workloads. While it provides easy-to-use functions like partition_html and partition_pdf, each document must be processed individually on a single CPU core.

def extract_text_from_file(filepath: Path) -> str:
   try:
        elements = partition(filename=str(filepath))
    except Exception as e:
        print(f"⚠️ Failed to read {filepath.name}: {e}")
        return ""

    text_content = []
    for element in elements:
        try:
            txt = getattr(element, "text", None)
            if txt:
                text_content.append(txt)
        except Exception:
            continue

    return "\n".join(text_content)

for idx, file_path in enumerate(files, start=1):
    file_t0 = time.perf_counter()

    text = extract_text_from_file(file_path)
    print(f"✔ [{idx}/{len(files)}] {file_path.name} processed in {file_t1 - file_t0:.2f}s")

This approach works well for small or ad-hoc datasets but faces clear limitations at enterprise scale:

• No built-in parallel or distributed processing across clusters
• Requires external orchestration tools (like Dask or Ray) to scale horizontally
• Limited integration with Spark-based ETL or NLP workflows
• Higher latency and I/O overhead when processing millions of files sequentially

Thus, for large-scale ingestion pipelines, Unstructured’s simplicity becomes a constraint increasing operational complexity and total runtime.

Experiment Results: Spark NLP vs Unstructured

To evaluate ingestion performance, we processed 60 mixed-format documents using both frameworks under the same conditions.

All experiments were executed on a single machine no Spark cluster, no distributed environment to ensure a fair comparison.

Average processing time for 60 documents. Spark NLP achieves ~2× faster throughput than Unstructured.

Even in this single-node setup, Spark NLP achieved nearly a 2x speedup, completing the full pipeline in roughly half the time of Unstructured.

This improvement comes primarily from Spark NLP’s ability to automatically parallelize work across all available CPU cores, distributing file reads and transformations efficiently under the hood.

Reproduce the benchmark and explore the full pipeline setup here.

Scaling Beyond a Single Machine

While this benchmark ran on one machine, Spark NLP’s real advantage emerges at larger scales:

• The same pipeline can run unchanged across a Spark cluster, leveraging multiple nodes for linear scalability.
• As document volume grows to thousands or millions, Spark’s distributed scheduler automatically partitions the workload each executor handling its own batch of documents in parallel.
• This architecture ensures both speed and fault tolerance, something single-threaded Python tools can’t easily replicate.

[embed]

When processing scales from hundreds to millions of documents, architecture becomes the differentiator. Spark NLP’s distributed design allows organizations to scale text extraction and NLP workloads horizontally, maintaining both speed and reliability something single node solutions like Unstructured simply aren’t built to achieve.

Whether for compliance auditing, enterprise search, or large scale document intelligence, Spark NLP ensures that scale doesn’t compromise consistency.

References:

1. https://aclanthology.org/P13-2116/
2. https://www.johnsnowlabs.com/tapas-question-answering-from-tables-in-spark-nlp/