Hierarchical Semantic Chunking

Hierarchical Semantic Chunking is a repository indexing strategy that organizes code into multiple semantic levels using syntax analysis and embeddings. It enables efficient retrieval by combining broad contextual understanding from coarse chunks with precise code-level retrieval from fine-grained chunks.

Introduction

Hierarchical Semantic Chunking is a document segmentation algorithm designed to preserve the semantic structure of large codebases and technical documents before they are indexed for Retrieval-Augmented Generation (RAG). Instead of dividing a repository into fixed-size token windows, the algorithm constructs a hierarchy of semantically meaningful chunks that closely follows the logical organization of the source code. Small syntactic units such as functions, methods, classes, and interfaces become the lowest level of the hierarchy, while larger semantic regions are formed by merging closely related neighboring units into higher-level chunks.

This hierarchical organization addresses one of the most fundamental problems in retrieval systems: the conflict between retrieval precision and contextual completeness. Small chunks provide highly precise retrieval because they contain little irrelevant information, but they often omit surrounding context necessary for understanding the retrieved code. Large chunks preserve context but frequently include unrelated implementations that reduce retrieval precision and waste valuable transformer context tokens.

Hierarchical Semantic Chunking resolves this trade-off by maintaining multiple levels of abstraction simultaneously. Coarse chunks capture high-level semantic regions suitable for efficient retrieval, while fine-grained chunks preserve detailed program structure for precise re-ranking. Rather than forcing retrieval to operate at a single granularity, the algorithm allows the retrieval pipeline to move progressively from broad semantic regions toward highly specific code fragments.

Unlike traditional text chunking algorithms that rely solely on token counts or line numbers, Hierarchical Semantic Chunking combines syntax-aware parsing, semantic embeddings, similarity analysis, hierarchical clustering, and multi-stage retrieval into a unified indexing strategy. This architecture has become increasingly important for repository-scale code retrieval because software systems naturally exhibit hierarchical organization that simple fixed-window chunking fails to capture

Why Hierarchical Semantic Chunking Exists

Large language models cannot process entire software repositories simultaneously. Every repository must therefore be divided into smaller retrieval units before indexing. The quality of this segmentation directly influences retrieval accuracy because the language model can only reason over the chunks returned by the retrieval system.

A straightforward approach divides the repository into fixed-size token windows. Although computationally simple, this strategy frequently splits functions across chunk boundaries, separates related implementations, and merges unrelated code solely because they happen to occupy adjacent lines within the source file. As a result, retrieval often returns incomplete implementations or large amounts of irrelevant surrounding code.

An alternative approach uses syntactic boundaries such as functions and classes as retrieval units. This preserves program structure but introduces a different problem. Individual functions often depend heavily on neighboring implementations, helper methods, or associated class members. Retrieving only a single function may therefore omit the contextual information required for accurate reasoning.

Hierarchical Semantic Chunking addresses both limitations by introducing multiple semantic levels. Fine-grained chunks preserve syntactic precision, while semantically related neighboring chunks are merged into larger regions that retain broader contextual relationships. Retrieval first identifies the most relevant semantic region before selecting the most informative fine-grained components within that region.

Rather than treating chunking as a preprocessing convenience, the algorithm treats segmentation as a representation learning problem whose objective is to preserve the semantic organization of the repository.

• Why do fixed-size token chunks often produce poor retrieval quality?
• Why are syntax-aware chunks still insufficient for repository understanding?
• Why is hierarchical organization more effective than using a single chunk size?
• Could future retrieval systems eliminate chunking entirely?
• Is semantic hierarchy an inherent property of software or an artifact of programming practices?
• How much retrieval performance depends on chunk quality compared to embedding quality?

References

Kamradt, G. 5 Levels of Text Splitting.

LangChain Documentation — Semantic Chunking.

Microsoft GraphRAG Technical Report.

AST-Based Fine-Grained Chunk Construction

FUNCTION construct_fine_grained_chunks_from_ast(source_document_text, language_parser, minimum_chunk_token_length, maximum_chunk_token_length):

    abstract_syntax_tree = language_parser.parse(source_document_text)

    candidate_ast_nodes = EMPTY_LIST()

    FUNCTION collect_semantic_unit_nodes(current_node):
        IF current_node.type IN [FUNCTION_DEFINITION, CLASS_DEFINITION, METHOD_DEFINITION, TOP_LEVEL_STATEMENT_BLOCK]:
            APPEND(candidate_ast_nodes, current_node)
        FOR EACH child_node IN current_node.children:
            collect_semantic_unit_nodes(child_node)

    collect_semantic_unit_nodes(abstract_syntax_tree.root_node)

    fine_grained_chunk_collection = EMPTY_LIST()

    FOR EACH semantic_unit_node IN candidate_ast_nodes:

        raw_node_text = extract_source_text(source_document_text, semantic_unit_node.start_byte, semantic_unit_node.end_byte)
        node_token_length = count_tokens(raw_node_text)

        IF node_token_length < minimum_chunk_token_length:
            CONTINUE

        IF node_token_length > maximum_chunk_token_length:
            sub_chunks = recursively_split_oversized_node(semantic_unit_node, source_document_text, maximum_chunk_token_length)
            FOR EACH sub_chunk IN sub_chunks:
                APPEND(fine_grained_chunk_collection, sub_chunk)
            CONTINUE

        fine_grained_chunk = BUILD_CHUNK_OBJECT(
            chunk_text = raw_node_text,
            node_type = semantic_unit_node.type,
            start_line = semantic_unit_node.start_line,
            end_line = semantic_unit_node.end_line,
            parent_node_reference = semantic_unit_node.parent,
            enclosing_scope_name = resolve_enclosing_scope_name(semantic_unit_node)
        )

        APPEND(fine_grained_chunk_collection, fine_grained_chunk)

    fine_grained_chunk_collection = sort_chunks_by_document_order(fine_grained_chunk_collection)

    RETURN fine_grained_chunk_collection

The first stage of the algorithm partitions the repository into fine-grained syntactic units using Abstract Syntax Tree (AST) analysis. Rather than splitting documents according to line numbers or token counts, the parser identifies language constructs such as functions, methods, classes, interfaces, enums, and other syntactic elements that naturally represent independent program components.

Operating on the AST provides several advantages over textual segmentation. Every chunk corresponds to a complete syntactic unit, ensuring that no function or class is divided across chunk boundaries. This preserves program correctness, maintains variable scope, and prevents incomplete implementations from appearing within the retrieval index.

The algorithm also enforces a maximum token limit for each chunk. Extremely large functions are recursively subdivided into smaller syntactic regions while preserving structural integrity whenever possible. This prevents oversized chunks from dominating retrieval while remaining compatible with transformer context constraints.

These fine-grained chunks form the lowest level of the hierarchical index. At this stage, every chunk represents a precise implementation unit that can later be retrieved independently if required.

• Why is AST parsing preferred over line-based chunking?
• Why should chunk boundaries align with programming language syntax?
• How should extremely large functions be partitioned without losing structural meaning?
• Could compiler intermediate representations produce better chunk boundaries than ASTs?
• How should anonymous functions, lambdas, and nested classes be represented?
• Should chunk boundaries differ across programming languages?

References

Aho, Lam, Sethi, & Ullman. Compilers: Principles, Techniques, and Tools.

Tree-sitter Documentation.

Semantic Embedding Generation

FUNCTION generate_semantic_embeddings_for_chunks(fine_grained_chunk_collection, embedding_model, embedding_batch_size):

    chunk_embedding_collection = EMPTY_LIST()

    total_chunk_count = length(fine_grained_chunk_collection)

    FOR batch_start_index IN RANGE(0, total_chunk_count, embedding_batch_size):

        batch_end_index = MINIMUM(batch_start_index + embedding_batch_size, total_chunk_count)
        current_chunk_batch = fine_grained_chunk_collection[batch_start_index : batch_end_index]

        contextualized_chunk_texts = EMPTY_LIST()
        FOR EACH chunk IN current_chunk_batch:
            contextualized_text = PREPEND(chunk.enclosing_scope_name, chunk.chunk_text)
            APPEND(contextualized_chunk_texts, contextualized_text)

        batch_embedding_vectors = embedding_model.encode(contextualized_chunk_texts)

        FOR index IN RANGE(0, length(current_chunk_batch)):
            current_chunk_batch[index].embedding_vector = normalize_vector(batch_embedding_vectors[index])
            APPEND(chunk_embedding_collection, current_chunk_batch[index])

    RETURN chunk_embedding_collection

Once fine-grained chunks have been constructed, each chunk is converted into a dense semantic embedding using a code embedding model. Unlike lexical token representations, embeddings capture semantic relationships between code fragments, allowing structurally similar implementations to occupy nearby locations within the embedding space even when they differ syntactically.

These embeddings become the primary representation used for semantic analysis throughout the remainder of the algorithm. Rather than comparing source code directly, similarity computations operate entirely within the embedding space.

Generating embeddings before hierarchical construction is essential because subsequent stages depend upon measuring semantic continuity between neighboring chunks. Without embedding representations, the algorithm would have no reliable mechanism for determining whether adjacent functions belong to the same conceptual topic or represent independent implementation regions.

The embedding model therefore transforms syntactic program structures into numerical representations suitable for similarity analysis while preserving much of the semantic information contained within the original code.

• Why are semantic embeddings computed before chunk merging?
• Why are embeddings more informative than lexical similarity for repository organization?
• How does embedding quality influence the resulting hierarchy?
• Could task-specific embedding models outperform general-purpose code embeddings?
• Should embeddings represent implementation semantics or API behavior?
• How stable are embedding relationships across different programming languages?

References

OpenAI Embeddings Documentation.

Voyage AI Documentation.

CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

Adjacent Similarity Analysis and Topic Boundary Detection

FUNCTION detect_topic_boundaries_from_adjacent_similarity(embedded_chunk_collection, similarity_drop_threshold, smoothing_window_size):

    total_chunk_count = length(embedded_chunk_collection)

    raw_adjacent_similarity_scores = EMPTY_LIST()

    FOR index IN RANGE(0, total_chunk_count - 1):
        current_vector = embedded_chunk_collection[index].embedding_vector
        next_vector = embedded_chunk_collection[index + 1].embedding_vector
        similarity_score = compute_cosine_similarity(current_vector, next_vector)
        APPEND(raw_adjacent_similarity_scores, similarity_score)

    smoothed_similarity_scores = apply_moving_average_smoothing(raw_adjacent_similarity_scores, smoothing_window_size)

    detected_boundary_indices = EMPTY_LIST()

    FOR index IN RANGE(0, length(smoothed_similarity_scores)):
        IF smoothed_similarity_scores[index] < similarity_drop_threshold:
            APPEND(detected_boundary_indices, index + 1)

    RETURN detected_boundary_indices

After embedding every fine-grained chunk, the algorithm measures semantic similarity between neighboring chunks. Rather than comparing every possible chunk pair, it focuses exclusively on adjacent units because neighboring functions are most likely to belong to the same logical subsystem.

Cosine similarity provides a quantitative estimate of semantic continuity between consecutive chunks. High similarity indicates that neighboring implementations discuss closely related concepts, whereas a significant decrease in similarity suggests a transition to a different topic or subsystem.

The algorithm scans the sequence of similarity scores searching for valleys that fall below a predefined threshold. These valleys become semantic boundaries separating one conceptual region from the next. Instead of imposing arbitrary segmentation intervals, the algorithm allows the semantic content itself to determine where larger repository regions begin and end.

Selecting the similarity threshold represents an important design decision. Higher thresholds create many small semantic regions, improving retrieval precision while reducing contextual completeness. Lower thresholds merge increasingly diverse implementations into larger regions, preserving more context at the expense of retrieval specificity.

This stage transforms an unordered collection of syntactic chunks into semantically coherent groups whose boundaries emerge naturally from the embedding space.

• Why are only adjacent chunks compared during similarity analysis?
• Why do similarity valleys indicate semantic topic boundaries?
• How does the similarity threshold influence chunk hierarchy?Could learned boundary detection outperform fixed similarity thresholds?
• Should topic boundaries consider repository structure in addition to embedding similarity?
• How sensitive is boundary detection to embedding model quality?

References

Hearst, M. TextTiling: Segmenting Text into Multi-Paragraph Subtopic Passages.

LangChain Documentation — Semantic Chunking.

Hierarchical Chunk Construction

FUNCTION construct_hierarchical_chunk_tree(embedded_chunk_collection, detected_boundary_indices, maximum_merged_chunk_token_length, summarization_model):

    mid_level_chunk_groups = EMPTY_LIST()
    current_group = EMPTY_LIST()

    FOR index IN RANGE(0, length(embedded_chunk_collection)):

        APPEND(current_group, embedded_chunk_collection[index])

        IF (index + 1) IN detected_boundary_indices OR index == length(embedded_chunk_collection) - 1:
            APPEND(mid_level_chunk_groups, current_group)
            current_group = EMPTY_LIST()

    mid_level_chunk_collection = EMPTY_LIST()

    FOR EACH group IN mid_level_chunk_groups:

        combined_text = concatenate_chunk_texts_in_order(group)
        combined_token_length = count_tokens(combined_text)

        IF combined_token_length > maximum_merged_chunk_token_length:
            group_summary_text = summarization_model.summarize(combined_text, maximum_merged_chunk_token_length)
        ELSE:
            group_summary_text = combined_text

        mid_level_chunk = BUILD_CHUNK_OBJECT(
            chunk_text = group_summary_text,
            child_chunk_references = group,
            start_line = group[0].start_line,
            end_line = group[LAST_INDEX].end_line
        )

        APPEND(mid_level_chunk_collection, mid_level_chunk)

    top_level_document_summary_text = summarization_model.summarize(
        concatenate_chunk_texts_in_order(mid_level_chunk_collection),
        maximum_merged_chunk_token_length
    )

    top_level_chunk = BUILD_CHUNK_OBJECT(
        chunk_text = top_level_document_summary_text,
        child_chunk_references = mid_level_chunk_collection
    )

    RETURN BUILD_HIERARCHY_OBJECT(
        top_level_node = top_level_chunk,
        mid_level_nodes = mid_level_chunk_collection,
        leaf_level_nodes = embedded_chunk_collection
    )

Once semantic boundaries have been identified, neighboring fine-grained chunks belonging to the same semantic region are merged into larger coarse chunks. Rather than discarding the original fine-grained units, the algorithm preserves them as child nodes within the newly constructed hierarchy.

Each coarse chunk therefore represents a complete semantic region rather than an arbitrary collection of functions. The merged text becomes the parent representation, while the original functions remain accessible for later retrieval refinement.

The algorithm also computes a representative embedding for every coarse chunk by aggregating the embeddings of its constituent child nodes. Mean pooling provides a simple yet effective approximation of the semantic content contained within the larger region.

Maintaining both parent and child representations is one of the defining characteristics of Hierarchical Semantic Chunking. Coarse chunks provide efficient repository navigation, while fine-grained chunks preserve retrieval precision. Neither representation replaces the other; instead, both cooperate throughout the retrieval pipeline.

• Why are fine-grained chunks merged instead of discarded?
• Why is a representative embedding computed for every coarse chunk?
• Why does the hierarchy preserve both parent and child nodes?
• Could graph-based aggregation outperform mean pooling?
• Should hierarchy depth adapt automatically according to repository complexity?
• Is there an optimal number of hierarchy levels for repository retrieval?

References

Microsoft GraphRAG Technical Report.

CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

Two-Stage Hierarchical Retrieval

FUNCTION retrieve_using_two_stage_hierarchical_search(query_text, chunk_hierarchy, embedding_model, mid_level_top_k, leaf_level_top_k):

    query_embedding_vector = normalize_vector(embedding_model.encode(query_text))

    mid_level_similarity_scores = EMPTY_LIST()

    FOR EACH mid_level_chunk IN chunk_hierarchy.mid_level_nodes:
        mid_level_chunk_embedding = compute_or_retrieve_cached_embedding(mid_level_chunk, embedding_model)
        similarity_score = compute_cosine_similarity(query_embedding_vector, mid_level_chunk_embedding)
        APPEND(mid_level_similarity_scores, BUILD_SCORE_PAIR(mid_level_chunk, similarity_score))

    ranked_mid_level_chunks = sort_by_score_descending(mid_level_similarity_scores)
    selected_mid_level_chunks = ranked_mid_level_chunks[0 : mid_level_top_k]

    candidate_leaf_chunks = EMPTY_LIST()

    FOR EACH scored_mid_level_chunk IN selected_mid_level_chunks:
        FOR EACH leaf_chunk IN scored_mid_level_chunk.chunk.child_chunk_references:
            APPEND(candidate_leaf_chunks, leaf_chunk)

    leaf_level_similarity_scores = EMPTY_LIST()

    FOR EACH leaf_chunk IN candidate_leaf_chunks:
        similarity_score = compute_cosine_similarity(query_embedding_vector, leaf_chunk.embedding_vector)
        APPEND(leaf_level_similarity_scores, BUILD_SCORE_PAIR(leaf_chunk, similarity_score))

    ranked_leaf_chunks = sort_by_score_descending(leaf_level_similarity_scores)
    selected_leaf_chunks = ranked_leaf_chunks[0 : leaf_level_top_k]

    final_retrieval_results = EMPTY_LIST()

    FOR EACH scored_leaf_chunk IN selected_leaf_chunks:
        result_with_context = attach_parent_context(scored_leaf_chunk.chunk, chunk_hierarchy)
        APPEND(final_retrieval_results, result_with_context)

    RETURN final_retrieval_results

The completed index stores both coarse semantic regions and their associated fine-grained children. Retrieval proceeds in two distinct stages. During the first stage, the user’s query is embedded and compared only against the coarse-level embeddings using an Approximate Nearest Neighbor index such as HNSW. This rapidly identifies the semantic regions most relevant to the query while avoiding expensive comparisons against every individual function in the repository.

The second stage performs fine-grained re-ranking within each retrieved semantic region. Rather than searching the entire repository again, the algorithm compares the query only against the child chunks belonging to the retrieved parents. This significantly reduces computational cost while improving retrieval precision because the search is restricted to semantically coherent neighborhoods.

The final ranking is therefore produced from the fine-grained children rather than the coarse parents. The coarse hierarchy acts as an efficient routing mechanism, whereas the fine-grained level provides precise retrieval. This hierarchical search strategy combines the scalability of large semantic regions with the accuracy of function-level retrieval.

The two-stage design explains why Hierarchical Semantic Chunking has become increasingly popular in modern repository-scale Retrieval-Augmented Generation systems. Instead of forcing retrieval to choose between efficiency and precision, the hierarchy provides both. Broad semantic organization accelerates repository navigation, while detailed child-level re-ranking preserves the exact contextual information required by transformer-based code generation.

• Why does retrieval begin with coarse semantic regions instead of fine-grained chunks?
• Why is child-level re-ranking necessary after coarse retrieval?
• How does hierarchical retrieval improve both efficiency and retrieval precision?
• Could multiple retrieval stages outperform the current two-stage architecture?
• Should hierarchy traversal depend on query complexity?
• As transformer context windows continue to expand, will hierarchical retrieval remain necessary?

References

Microsoft GraphRAG Technical Report.

Malkov, Y. A., & Yashunin, D. A. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.

LangChain Documentation — Semantic Chunking.

name: ast_hierarchical_semantic_chunking_and_retrieval_pipeline

on:
  document_ingestion_trigger:
    description: Fires once per source document added to the corpus
  query_time_trigger:
    description: Fires once per user query against the already-built hierarchy

env:
  minimum_chunk_token_length: smallest allowed size for a leaf-level AST chunk, prevents trivially small fragments
  maximum_chunk_token_length: largest allowed size for a leaf-level AST chunk before forced sub-splitting
  embedding_batch_size: number of chunks embedded per model call
  similarity_drop_threshold: cosine similarity below which two adjacent chunks are considered topically different
  smoothing_window_size: width of moving average applied to similarity scores before boundary detection
  maximum_merged_chunk_token_length: largest allowed size for a mid-level or top-level merged chunk before summarization is forced
  mid_level_top_k: number of mid-level chunks retrieved in retrieval stage 1
  leaf_level_top_k: number of leaf-level chunks retrieved in retrieval stage 2

jobs:

  stage_1_ast_fine_grained_chunk_construction:
    description: Parse the source document into an AST and extract fine-grained, semantically coherent chunks
    depends_on: none, this is the entry point for document_ingestion_trigger
    inputs:
      source_document_text: raw file content
      language_parser: grammar-aware AST parser for the document's language
    steps:
      - id: parse_ast
        run: abstract_syntax_tree = language_parser.parse(source_document_text)
      - id: walk_tree_recursively
        run: collect_semantic_unit_nodes(root_node)
        loop_type: recursive tree traversal, not a fixed-count loop
        why_recursive_and_not_flat: >
          Semantic units (functions, classes, methods) are nested arbitrarily deep in real
          source files. A flat loop over top-level nodes would miss methods inside classes,
          or nested functions inside functions. Recursion is required because the depth of
          nesting is unknown ahead of time.
      - id: filter_and_split_by_size
        run: FOR EACH candidate_node → discard if too small, recursively split if too large
        condition_explained:
          why_minimum_length_check_exists: >
            Extremely short nodes (e.g. a single-line getter) carry almost no distinguishing
            semantic signal on their own. Embedding them separately wastes retrieval budget
            and adds noise, since their embeddings tend to cluster meaninglessly close together.
          why_maximum_length_check_exists: >
            Embedding models have a finite effective context window; chunks near or beyond it
            get compressed into a lossy average, degrading downstream similarity comparisons.
          effect_of_changing_these_thresholds: >
            Too narrow a range yields either a flood of near-duplicate tiny chunks (bad recall
            precision) or overly coarse chunks that mix unrelated logic (bad retrieval precision).
            This is the single biggest lever on final retrieval granularity.
    outputs:
      fine_grained_chunk_collection: ordered list of leaf-level chunks with line ranges and scope metadata

  stage_2_semantic_embedding_generation:
    description: Convert each fine-grained chunk into a dense vector representation
    depends_on: stage_1_ast_fine_grained_chunk_construction
    inputs:
      fine_grained_chunk_collection: from stage_1
      embedding_model: pretrained text/code embedding model
      embedding_batch_size: from env
    steps:
      - id: batch_chunks
        run: FOR batch_start_index IN RANGE(0, total_chunk_count, embedding_batch_size)
        loop_type: fixed-stride iteration over the chunk list
        why_batched_and_not_one_at_a_time: >
          Embedding models run far more efficiently on batched input due to parallel
          hardware utilization. Calling the model once per chunk would be correct but
          orders of magnitude slower for large documents.
      - id: prepend_scope_context
        run: contextualized_text = PREPEND(chunk.enclosing_scope_name, chunk.chunk_text)
        why_this_step_exists: >
          A chunk's raw text alone (e.g. a method body) often loses meaning without knowing
          its enclosing class or module. Prepending scope name anchors the embedding in its
          actual context, which directly affects the quality of stage_3's similarity signal.
      - id: encode_and_normalize
        run: batch_embedding_vectors = embedding_model.encode(contextualized_chunk_texts), then normalize
        why_normalization_matters: >
          Cosine similarity in stage_3 assumes unit-length vectors. Skipping normalization
          silently biases similarity scores toward chunks with larger raw vector magnitude,
          independent of actual semantic closeness.
    outputs:
      embedded_chunk_collection: fine-grained chunks, each now carrying an embedding_vector

  stage_3_adjacent_similarity_analysis_and_boundary_detection:
    description: Measure how semantically related each chunk is to its immediate neighbor, and flag where topics shift
    depends_on: stage_2_semantic_embedding_generation
    inputs:
      embedded_chunk_collection: from stage_2
      similarity_drop_threshold: from env
      smoothing_window_size: from env
    steps:
      - id: compute_pairwise_adjacent_similarity
        run: FOR index IN RANGE(0, total_chunk_count - 1) → cosine_similarity(chunk[index], chunk[index+1])
        loop_type: sequential, order-dependent iteration
        why_order_matters_here: >
          Unlike stage_2's batching, this loop cannot be parallelized arbitrarily — each
          comparison depends on document order, because the entire point is to detect where
          consecutive content diverges. Shuffling chunks before this stage would produce
          meaningless boundaries.
      - id: smooth_similarity_curve
        run: apply_moving_average_smoothing(raw_adjacent_similarity_scores, smoothing_window_size)
        why_smoothing_exists: >
          Raw adjacent similarity is noisy — a single unusually-worded chunk can create a
          false dip even within a coherent topic. Smoothing prevents over-segmentation from
          one-off outliers.
        effect_of_window_size: >
          Too small a window leaves noise-driven false boundaries, fragmenting the hierarchy
          in stage_4 into too many tiny mid-level groups. Too large a window over-smooths and
          erases genuine topic shifts, merging unrelated sections into one mid-level chunk.
      - id: threshold_check
        run: IF smoothed_similarity_scores[index] < similarity_drop_threshold THEN mark boundary at index+1
        condition_explained:
          why_a_threshold_and_not_a_fixed_chunk_count: >
            Document structure is unpredictable — some documents have five natural topics,
            others fifty. A fixed count of boundaries would force unrelated content together
            in short documents or fragment single topics in long uniform ones. A threshold
            adapts to actual content structure.
          effect_of_threshold_value: >
            A high threshold (closer to 1.0) creates many boundaries, producing many small,
            highly focused mid-level groups in stage_4. A low threshold creates few boundaries,
            producing fewer, broader mid-level groups. This directly trades off retrieval
            precision against retrieval recall at the mid-level stage.
    outputs:
      detected_boundary_indices: positions in the chunk sequence where topic shifts occur

  stage_4_hierarchical_chunk_construction:
    description: Merge fine-grained chunks between detected boundaries into mid-level groups, then summarize upward into a top-level document node
    depends_on: stage_3_adjacent_similarity_analysis_and_boundary_detection, stage_1_ast_fine_grained_chunk_construction
    inputs:
      embedded_chunk_collection: from stage_2
      detected_boundary_indices: from stage_3
      maximum_merged_chunk_token_length: from env
      summarization_model: model used to compress oversized merged text
    steps:
      - id: group_chunks_between_boundaries
        run: FOR index IN RANGE(0, total_chunk_count) → accumulate into current_group, close group at each boundary_index
        loop_type: sequential accumulation with a reset condition
        why_this_pattern_and_not_simple_batching: >
          Unlike stage_2's fixed-size batching, group sizes here are variable and determined
          entirely by stage_3's output. This is intentional: mid-level groups must reflect
          actual topic boundaries, not an arbitrary fixed count of chunks.
      - id: summarize_oversized_groups
        run: IF combined_token_length > maximum_merged_chunk_token_length THEN summarization_model.summarize(...) ELSE keep as-is
        condition_explained:
          why_this_condition_exists: >
            A mid-level chunk used directly in stage_5's first-pass retrieval must itself fit
            within embedding and reasoning limits. Without this check, large topic groups
            would either fail to embed properly or dilute their own embedding vector.
          effect_of_skipping_summarization: >
            Oversized un-summarized mid-level chunks produce averaged-out embeddings that
            match almost every query moderately well and no query strongly, degrading
            stage_5's first-pass filtering accuracy.
      - id: build_top_level_summary
        run: top_level_document_summary_text = summarization_model.summarize(all mid-level text)
        why_a_top_level_node_exists: >
          Provides a single whole-document representation usable for corpus-level filtering
          (e.g. "which documents are relevant at all") before even entering stage_5's
          mid-level search, though stage_5 as defined here operates at mid/leaf level.
    outputs:
      chunk_hierarchy: three-tier structure — leaf_level_nodes, mid_level_nodes, top_level_node — with explicit parent/child references

  stage_5_two_stage_hierarchical_retrieval:
    description: Given a query, first narrow down to relevant topic groups, then find the precise chunks within them
    depends_on: stage_4_hierarchical_chunk_construction
    trigger: query_time_trigger, runs independently and repeatedly against an already-built chunk_hierarchy
    inputs:
      query_text: user's search query
      chunk_hierarchy: from stage_4
      mid_level_top_k: from env
      leaf_level_top_k: from env
    steps:
      - id: embed_query
        run: query_embedding_vector = normalize(embedding_model.encode(query_text))
      - id: stage_5a_mid_level_search
        run: FOR EACH mid_level_chunk IN chunk_hierarchy.mid_level_nodes → score against query
        why_this_first_pass_exists: >
          Comparing the query against every single leaf chunk directly would be accurate but
          expensive at scale, and worse, leaf-level chunks out of their topic context can
          score deceptively high on narrow lexical/semantic overlap alone. Filtering by topic
          group first constrains the search space to plausible regions before fine comparison.
      - id: select_top_mid_level_groups
        run: selected_mid_level_chunks = ranked_mid_level_chunks[0 : mid_level_top_k]
        effect_of_mid_level_top_k: >
          Too small: relevant leaf chunks in a topic group that scored just outside the cutoff
          are permanently excluded, hurting recall. Too large: stage_5b searches over nearly
          the whole document again, eroding the efficiency and precision benefit of having
          a hierarchy at all.
      - id: stage_5b_leaf_level_search_within_selected_groups
        run: FOR EACH scored_mid_level_chunk → FOR EACH leaf_chunk IN its children → score against query
        loop_type: nested loop, outer over selected groups, inner over each group's children
        why_nested_and_not_flat_over_all_leaves: >
          This is the entire point of the two-stage design — the inner loop only ever runs
          over leaves belonging to a group that already passed the coarse relevance filter.
          A flat single-stage search over all leaves would be more accurate in isolation but
          loses the topic-context signal that mid-level filtering provides, and does not scale.
      - id: select_top_leaf_chunks
        run: selected_leaf_chunks = ranked_leaf_chunks[0 : leaf_level_top_k]
      - id: attach_context
        run: FOR EACH selected leaf chunk → attach_parent_context using chunk_hierarchy
        why_this_exists: >
          A leaf chunk returned in isolation often lacks enough surrounding information for
          a downstream reader or LLM to fully use it. Reattaching the parent mid-level summary
          gives just enough context without returning the entire document.
    outputs:
      final_retrieval_results: ranked, context-attached leaf chunks answering the query

Conclusion

Hierarchical Semantic Chunking provides a structured approach for preparing large codebases for Retrieval-Augmented Generation by preserving both local implementation details and broader semantic relationships. Unlike fixed-size or purely syntax-based chunking methods, it creates a multi-level representation where fine-grained code units and larger semantic regions coexist, enabling more accurate and efficient retrieval.

By combining AST-based segmentation, semantic embeddings, similarity-based boundary detection, hierarchical chunk construction, and two-stage retrieval, the algorithm aligns repository organization with the way software systems are naturally structured. This reduces the trade-off between retrieval precision and contextual completeness by allowing retrieval systems to first identify relevant semantic regions and then refine results to the most relevant code fragments.

As code repositories continue to grow in scale and complexity, chunking should be treated as a fundamental component of retrieval architecture rather than a simple preprocessing step. Hierarchical Semantic Chunking demonstrates that preserving semantic structure can significantly improve how AI systems understand, search, and reason over large software systems.