Chunking Lab: the full record of breaking RAG chunking on purpose

Step 0

Set up the environment

Why

Once only. The virtual environment lives in labs/rag-lab/.venv, apart from the system Python.

The pip install -e . line lets you write python -m rag_lab afterwards without setting PYTHONPATH each time.

If your machine does not have 3.11, replace python3.11 on the second line with your own (3.10 or later is fine, usually python3). All the output below is from 3.11.15.

Command

cd labs/rag-lab
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

Read these lines

Every dependency version is pinned in requirements.txt — install again in six months and you get the same wheels and compute the same vectors.
The model is not downloaded yet. It downloads during the first command that needs it, about 470 MB.

Change something yourself

Every command after this assumes the virtual environment is active. In a new terminal, run source labs/rag-lab/.venv/bin/activate first.

Step 1

Read the Knowledge Base

Why

First confirm that the knowledge base and the evaluation questions are themselves right: no duplicate section ids, and each question’s Ground Truth points at a section that exists.

This step loads no model and uses no network; it returns in under a second.

Command

python -m rag_lab validate

What you should see

knowledge base  zh  7 documents, 61 sections
  allergens.md              7 sections   1296 chars  过敏原与交叉接触说明
  delivery.md               9 sections   1243 chars  外卖配送规则
  membership.md            10 sections   1154 chars  会员制度问答
  menu.md                   8 sections   1430 chars  菜单与可调整项
  promotion-policy.md       8 sections   1252 chars  优惠与券规则
  refund-policy.md         11 sections   1777 chars  退款与补偿政策
  store-operations.md       8 sections   1407 chars  门店操作手册
query set       11 queries, schema rag-lab.queries.v1

every evidence section resolves; ids unique; concepts agree. OK

Read these lines

7 documents, 61 sections. A “section” is a stable semantic section — each part of the knowledge base has an id that does not change with the cutting, for example refund.missing.combo-core-item.
Ground Truth is tied to these ids, not to chunks. So however the cutting changes below, the yardstick does not move.

Change something yourself

In content/rag-lab/evaluation/queries.json, change one evidence_sections entry to an id that does not exist and run it again — it names the culprit.

content/rag-lab/knowledge/zh/Seven documents in Chinese, with the section ids written in invisible HTML comments
content/rag-lab/evaluation/queries.json11 questions, expected answers, evidence sections, distractor sections

Step 2

Measure the documents: where chunk size comes from

Why

“How big to cut” should not be a feeling. First measure the real documents with this model’s own tokenizer, then decide the candidate values.

This step downloads the model (about a minute the first time); after that everything is local.

Command

python -m rag_lab stats

What you should see


tokenizer: XLMRobertaTokenizer (intfloat/multilingual-e5-small)
model window: 512 tokens, 384 dimensions

document                     chars   tokens  sections    section tokens (min/med/max)
allergens.md                  1296      976         7                  73 / 131 / 183
delivery.md                   1243      864         9                  63 / 102 / 120
membership.md                 1154      803        10                   65 / 76 / 108
menu.md                       1430     1102         8                  90 / 125 / 256
promotion-policy.md           1252      896         8                  75 / 116 / 144
refund-policy.md              1777     1308        11                  86 / 116 / 156
store-operations.md           1407     1025         8                  92 / 128 / 160

61 sections, 6974 tokens total
section tokens: min 63, median 111, mean 114, max 256
chars per token: 1.37

Read these lines

The median rule is 111 tokens, the shortest 63, the longest 256 (the menu table).
That is how the candidate values below were picked: 64 is clearly smaller than one rule, 192 is about one and a half, 448 is about four and sits close to the model’s 512 window. Not “the usual 500”.
1.37 characters per token — Chinese is roughly one token per character, but “roughly” is not a unit, so everything is counted in tokens.

Change something yourself

This knowledge base is small. Run stats on your own documents and the median will be quite different, and so will the candidate values.

Step 3

Fixed chunking: cut the documents

Why

Only cut for now, no retrieval. See where a ruler of fixed length lands.

--document only filters what is shown; the index is still the whole knowledge base.

Command

python -m rag_lab chunk --strategy fixed --chunk-size 64 --overlap 0 --document refund

What you should see


strategy fixed-64-0  {"id": "fixed-64-0", "splitter": "fixed", "unit": "tokens", "chunk_size": 64, "overlap": 0}
113 chunks, tokens min 2 / median 65 / max 66
chunks holding more than one section: 52

fixed-64-0:refund-policy#000                   64t  ## 适用范围
                                                   sections: refund.scope
fixed-64-0:refund-policy#001                   65t  规定"退多少钱、由谁发起"。配送环节本身的延误、取消与补偿，按《外卖配送规则》处理；因原
                                                   sections: refund.scope
fixed-64-0:refund-policy#002                   65t  对应规则判断是否成立，再按本政策计算金额，不重复退款。
                                                   sections: refund.scope, refund.missing.standard
fixed-64-0:refund-policy#003                   65t  付金额。实付金额指该商品在本单中实际承担的价格，已含平摊后的优惠。顾客可以选择退款，也可
                                                   sections: refund.missing.standard
fixed-64-0:refund-policy#004                   65t  ，订单中其余已送达的商品不退。顾客要求把整单退掉时，按下文"整单退款"判断是否成立。

Read these lines

The whole knowledge base becomes 113 chunks, of which 52 span more than one rule.
Look at chunk 3 (#002): it starts with half a sentence of the previous rule and ends inside the next. The cut does not know about rules.
Add --full to print each chunk in full.

Change something yourself

Change --chunk-size to 448, and the same file goes from 21 chunks to 3.

python -m rag_lab chunk --strategy fixed --chunk-size 448 --overlap 0 --document refund

labs/rag-lab/src/rag_lab/chunking.pyTwo splitters, under two hundred lines in all

Step 4

Embedding + retrieval: four things in one command

Why

The next command does four things in turn, and the code keeps them as four separate parts:

① each chunk gets the passage: prefix and goes into the model, giving a 384-dimensional vector (embedding.py); ② the question gets the query: prefix and also becomes a vector; ③ all vectors are L2-normalised, so cosine similarity is a single dot product (retrieval.py); ④ sort, and take the top 3.

The prefixes are not decoration; they are the input format the e5 family of models requires. Vectors are cached in results/.cache/, so changing a chunk size does not recompute text that has not changed.

labs/rag-lab/src/rag_lab/embedding.pyModel, tokenizer, prefixes, cache, provenance
labs/rag-lab/src/rag_lab/retrieval.pyOne numpy matrix and one dot product, no vector database
labs/rag-lab/configs/experiment.jsonThe half that stays fixed: model, revision, top_k

Step 5

The first Top-K: what cut-up evidence looks like

Why

The question is Scenario 1 from the game: the combo’s main drink was left out; can the whole order be refunded? Answering it needs two rules — the general case and the combo exception.

Command

python -m rag_lab retrieve --strategy fixed --chunk-size 64 --overlap 0 --query-id Q01-zh

What you should see


strategy   fixed-64-0  {"id": "fixed-64-0", "splitter": "fixed", "unit": "tokens", "chunk_size": 64, "overlap": 0}
index      113 chunks
query      Q01-zh  套餐里的主饮没送到，小吃都收到了，能不能把整个套餐退掉？
top_k      3

#1  0.9393  fixed-64-0:refund-policy#005  (65 tokens)
    sections: refund.missing.combo-core-item
    主饮漏送时，不按单件漏送处理。主饮是套餐的核心商品，缺少主饮后剩余附件不构成一份可交付的套餐，因此顾客有权要求退还整个套餐的实付金额，已送达的附件无需退回。顾客也可以选择只补做

#2  0.9278  fixed-64-0:store-operations#007  (65 tokens)
    sections: ops.stockout.combo, ops.stockout.no-confirm
    差价。顾客不接受任何方案的，整份套餐按《退款与补偿政策》的整单情形退款，已制作完成的附件不要求退回。
    
    套餐主饮本身售罄的，不做替换，直接进入退款流程。
    
    ## 联系不上顾客时

#3  0.9275  fixed-64-0:refund-policy#004  (65 tokens)
    sections: refund.missing.standard, refund.missing.combo-core-item
    ，订单中其余已送达的商品不退。顾客要求把整单退掉时，按下文"整单退款"判断是否成立。
    
    ## 套餐主饮漏送
    
    套餐按整体定价，其中的饮品称为主饮，小吃、配料杯等称为附件。
    
    套餐

evidence needed:  refund.missing.standard, refund.missing.combo-core-item
  ✗ refund.missing.standard                        22% in top-3   (not complete in any chunk)
  ✓ refund.missing.combo-core-item                 61% in top-3   (not complete in any chunk)

evidence coverage@3 50%   complete False   retrieved 195 tokens   redundancy 0%   signal 66%

Read these lines

Rank 1 reads up to “顾客也可以选择只补做” (“the customer may also choose to have it remade only”) and stops — that is the end of 64 tokens, not the end of the sentence.
The last three lines are the verdict: the base rule is only 22% retrieved, the exception 61%, complete False.
signal is the share of the retrieved text that belongs to evidence sections; redundancy is the share of it that repeats. There is no overlap this time, so it is 0%.

Change something yourself

Try another question: --query-id Q06-zh (the peanut allergy one, which needs three pieces of evidence). You can also ask your own: --query "生日券过期了还能补发吗" (“Can an expired birthday coupon be reissued?”), though then there is no Ground Truth to score it against.

Step 6

Change the chunk size

Why

Change one number and leave everything else alone — the same question, the same model, the same top-k.

Command

python -m rag_lab retrieve --strategy fixed --chunk-size 192 --overlap 0 --query-id Q01-zh

What you should see

The text of the top three comes before this; only the final verdict is shown here.

evidence needed:  refund.missing.standard, refund.missing.combo-core-item
  ✓ refund.missing.standard                        69% in top-3   (first complete at rank 2)
  ✓ refund.missing.combo-core-item                100% in top-3   (first complete at rank 2)

evidence coverage@3 100%   complete True   retrieved 579 tokens   redundancy 0%   signal 45%

Read these lines

This time both rules land in the same chunk (rank 2), complete True.
The cost is on the same line: the context goes from 195 tokens to 579, and signal drops from 66% to 45%.

Change something yourself

Try 448. complete is still True, the context keeps growing and signal keeps falling. No one number is best for every question — step 10 runs all 11 questions together.

python -m rag_lab retrieve --strategy fixed --chunk-size 448 --overlap 0 --query-id Q01-zh

Step 7

Add overlap

Why

Another question: a delivery is 40 minutes late and the rider has not picked it up; can it be cancelled? The answer takes two sentences — a definition (over 30 minutes counts as a severe delay) and a conclusion (it can be cancelled before pickup).

Run it once with no overlap, then once with 32 tokens of overlap.

Command

python -m rag_lab retrieve --strategy fixed --chunk-size 96 --overlap 0 --query-id Q03-zh

What you should see

The text of the top three comes before this; only the final verdict is shown here.

evidence needed:  delivery.delay.severe, delivery.delay.cancel-before-pickup
  ✗ delivery.delay.severe                          45% in top-3   (not complete in any chunk)
  ✓ delivery.delay.cancel-before-pickup           100% in top-3   (not complete in any chunk)

evidence coverage@3 50%   complete False   retrieved 291 tokens   redundancy 0%   signal 50%

Read these lines

complete False: the definition section is only 45% retrieved, because the cut falls in the middle of it.

Change something yourself

Change --overlap to 32 and run it again:

python -m rag_lab retrieve --strategy fixed --chunk-size 96 --overlap 32 --query-id Q03-zh

evidence needed:  delivery.delay.severe, delivery.delay.cancel-before-pickup
  ✓ delivery.delay.severe                          84% in top-3   (first complete at rank 1)
  ✓ delivery.delay.cancel-before-pickup            68% in top-3   (first complete at rank 4)

evidence coverage@3 100%   complete True   retrieved 291 tokens   redundancy 0%   signal 49%
also retrieved (listed as not-the-answer): delivery.delay.minor

complete True. The chunk at rank 1 now holds both the definition and the conclusion, and its similarity rises from 0.9361 to 0.9399. The context did not grow — it is still 291 tokens, just filled with the right things.

Step 8

Overlap, overdone

Why

The last step’s conclusion sounds like “the more overlap the better”. Add a little more, from 32 to 48, and change nothing else.

Command

python -m rag_lab retrieve --strategy fixed --chunk-size 96 --overlap 48 --query-id Q03-zh

What you should see

The text of the top three comes before this; only the final verdict is shown here.

evidence needed:  delivery.delay.severe, delivery.delay.cancel-before-pickup
  ✗ delivery.delay.severe                           0% in top-3   (first complete at rank 5)
  ✓ delivery.delay.cancel-before-pickup            92% in top-3   (first complete at rank 3)

evidence coverage@3 50%   complete False   retrieved 292 tokens   redundancy 18%   signal 41%

Read these lines

complete False — the same question cannot be answered again.
redundancy 18%: rank 1 and rank 3 are two neighbouring chunks, and more than half of them is the same passage. The top three places hold only two different pieces of content.
The definition section has dropped to rank 5. It was in the knowledge base all along; it just did not make the top three.
The index grows too: for the same 61 sections, overlap 0 gives 76 chunks, 32 gives 108, and 48 gives 141.

Change something yourself

Now try 64. On this question it goes back to complete True — overlap is not a monotonic knob. Run all four values, 0 / 32 / 48 / 64, and the point is clearer than any one sentence of conclusion.

python -m rag_lab retrieve --strategy fixed --chunk-size 96 --overlap 64 --query-id Q03-zh

Step 9

structure-aware: let the document decide the cuts

Why

A different way of cutting: do not count tokens; cut at Markdown headings, and keep a table together with the note right after it.

This splitter cannot read section ids. It gets exactly the same input as the fixed-length splitter — the body text with the id comments removed. Otherwise it would be cutting along the answer.

Command

python -m rag_lab chunk --preset structure-aware --document menu

What you should see


strategy structure-aware  {"id": "structure-aware", "splitter": "structure", "unit": "tokens", "label": "按结构切", "role": "boundary", "max_tokens": 480, "min_tokens": 40, "note": "Boundaries come from the Markdown — headings, paragraphs, a table and its note. The ceiling is the model's window, not a target size."}
60 chunks, tokens min 63 / median 111 / max 402
chunks holding more than one section: 1

structure-aware:menu#000                       90t  ## 适用范围
                                                   sections: menu.scope
structure-aware:menu#001                      402t  ## 主要饮品
                                                   sections: menu.drinks.table, menu.drinks.table-notes
structure-aware:menu#002                      123t  ## 甜度档位

Read these lines

The menu is cut into 7 chunks, and chunk 2 is 402 tokens — the whole price table and the note below it, in one chunk.

Change something yourself

Ask this way of cutting “can the brown-sugar pearl milk be made with no sugar at all?”:

python -m rag_lab retrieve --preset structure-aware --query-id Q09-zh

evidence needed:  menu.drinks.table, menu.drinks.table-notes
  ✓ menu.drinks.table                             100% in top-3   (first complete at rank 1)
  ✓ menu.drinks.table-notes                       100% in top-3   (first complete at rank 1)

evidence coverage@3 100%   complete True   retrieved 653 tokens   redundancy 0%   signal 63%
also retrieved (listed as not-the-answer): menu.sugar-levels

complete True; the table and its note are both 100% retrieved. The same question under fixed 192-token cutting is complete False — the cut fell in the middle of the table.

labs/rag-lab/configs/strategies.jsonEight ways of cutting, in three controlled comparisons

Step 10

Run all 11 questions

Why

One question’s result is easily fooled by a coincidence. Run all 8 ways of cutting × 11 questions; it takes about ten seconds.

Command

python -m rag_lab evaluate

What you should see

fixed-64-0                 113 chunks   complete 1/11   coverage 27%   194 tokens   redundancy 0%   signal 42%
fixed-192-0                 40 chunks   complete 6/11   coverage 71%   578 tokens   redundancy 0%   signal 27%
fixed-448-0                 18 chunks   complete 10/11   coverage 91%   1209 tokens   redundancy 0%   signal 18%
fixed-96-0                  76 chunks   complete 2/11   coverage 41%   280 tokens   redundancy 0%   signal 43%
fixed-96-32                108 chunks   complete 4/11   coverage 55%   290 tokens   redundancy 10%   signal 41%
fixed-96-48                141 chunks   complete 2/11   coverage 36%   290 tokens   redundancy 18%   signal 40%
fixed-96-64                207 chunks   complete 5/11   coverage 59%   290 tokens   redundancy 22%   signal 45%
structure-aware             60 chunks   complete 6/11   coverage 71%   396 tokens   redundancy 0%   signal 42%

wrote labs/rag-lab/results/manifest.json
wrote labs/rag-lab/results/evaluation.json
wrote labs/rag-lab/results/chunks.json
wrote labs/rag-lab/results/retrieval.json
wrote labs/rag-lab/results/evaluation.md

Read these lines

There is no overall score. Each column goes its own way, which is exactly what we want to see: the large chunks retrieve completely most often (10/11) and have the lowest signal (18%); the overlap group is not monotonic (2 → 4 → 2 → 5).
A readable version is written to labs/rag-lab/results/evaluation.md, one table per question.
Every output carries the same manifest: the knowledge base sha256, the question set sha256, the model id and revision, dependency versions, the time. Change one character of the knowledge base and the hash changes.

Change something yourself

Run only some of them: --preset can be repeated. Then results/ holds only those, and the next step’s export will fail and name them (it needs every way of cutting the game uses) — just run a full evaluate. You can also change --top-k, but after that you can no longer compare with the numbers above; it is another variable.

python -m rag_lab evaluate --preset fixed-192-0 --preset structure-aware

labs/rag-lab/results/evaluation.mdA readable report, per question and per way of cutting
labs/rag-lab/results/manifest.jsonProvenance only: hashes, model, versions, time

Step 11

Regenerate the results on the site

Why

The game pages read static artifacts; no model runs in the browser. This command is the only road from results/ to the site.

Command

python -m rag_lab export

What you should see

wrote src/data/rag02/generated/manifest.json
wrote src/data/rag02/generated/game.json
wrote src/data/rag02/generated/evaluation-matrix.json

Read these lines

configs/site.json only decides which runs are shown; every score, rank and passage is read back from results/.
After it runs, git diff src/data/rag02/generated/: if you changed no parameter, the only difference should be the timestamp in the manifest.

Change something yourself

In configs/strategies.json, change the chunk_size of fixed-192-0 to 256, re-run evaluate and export, then npm run build — the site fails at build time, because the numbers the walkthrough quotes no longer match. That is deliberate.

After the run

The results should be exactly the same as this page: the same similarities, the same ranks, the same token counts. The model revision is pinned, the vectors are deterministic, and ties in the sorting are broken by chunk order. The only thing that changes is the timestamp in the manifest.
If they differ, first check that the dependency versions and the model revision in results/manifest.json match.
This Lab was not written for the chunking topic alone. The later Query Rewrite changes the query part, Hybrid Search adds a second retrieval path, and Rerank adds a layer after retrieval — the knowledge base, the question set and the metrics stay put.

What this round did not do

There is no generation step. Handing the top-k to a local LLM to write answers is doable, but this round did not do it: generation would stack another variable on top of the one being measured, and the question “did the evidence come back” already has its answer before any model writes a sentence.

Reference

The half that stays fixed

Only the cutting changes. Everything below is the same in every run — otherwise a change in the results would have two possible causes, and the experiment would show nothing.

Embedding model: intfloat/multilingual-e5-small @ 614241f622f53c4eeff9890bdc4f31cfecc418b3(MIT · 384 dimensions · window 512 tokens · XLMRobertaTokenizer)
Prefixes: query: / passage: — the input format the e5 family of models requires
Similarity: cosine (dot product of L2-normalized vectors)
top_k: 3
Coverage threshold: at least 60% of a rule’s characters must fall inside the top-k for it to count as “retrieved”
Knowledge base: 7 documents · 61 sections · sha256 46d3bdea87debd98
Question set: 11 questions · sha256 bdd2a597778ca588
Environment: Python 3.11.15 · torch 2.14.0 · sentence-transformers 5.7.0 · numpy 2.4.6

The half that changes — 8 ways of cutting, in three controlled comparisons:

fixed-64-064 tokens a piece, no overlap113 chunks · 2–66 tokens
fixed-192-0192 tokens a piece, no overlap40 chunks · 17–193 tokens
fixed-448-0448 tokens a piece, no overlap18 chunks · 81–449 tokens
fixed-96-096 tokens a piece, no overlap76 chunks · 17–98 tokens
fixed-96-3296 tokens a piece, neighbours overlap by 32 tokens108 chunks · 36–98 tokens
fixed-96-4896 tokens a piece, neighbours overlap by 48 tokens141 chunks · 61–98 tokens
fixed-96-6496 tokens a piece, neighbours overlap by 64 tokens207 chunks · 66–98 tokens
structure-awarecut by document structure, up to 480 tokens60 chunks · 63–402 tokens

Set up the environment

Read the Knowledge Base

Measure the documents: where chunk size comes from

Fixed chunking: cut the documents

Embedding + retrieval: four things in one command

The first Top-K: what cut-up evidence looks like

Change the chunk size

Add overlap

Overlap, overdone

structure-aware: let the document decide the cuts

Run all 11 questions

Regenerate the results on the site

After the run

The half that stays fixed

Site analytics