The cut, on the text
| rank | chunk | score | holds |
|---|
Three chunking policies over the same eight Wikipedia articles, English and Greek, same target size, same BM25. All three return a passage holding most of the query for at least 99% of them. What changes is whether the passage arrives whole.
- a fixed cut truncates
- —
- of queries
- sentence aligned
- —
- of queries return the whole answer, at the same index size
- the overlap trick costs
- —
- more index, for the same answers
The leaderboard
Every policy, both query families, over — queries. The whole column is the share of queries whose top result contains the entire query. The majority column is the share containing more than half of it, and it is 1.000 everywhere but one cell, which is the point: the boundary is barely deciding what you find. The exception is named under the table, because a table of 1.000s with one 0.990 in it is the kind of thing prose quietly rounds away.
| policy | chunks | indexed | whole, sentence | whole, keywords | majority | characters lost, median / worst |
|---|
Method
The task is self retrieval: a sentence is taken out of an article and used as the query, so the passage that contains it is the right answer by construction and nothing had to be labelled. It is a weak task on purpose, because it flatters every method, so a difference that survives it is real. Every query is asked twice, once as the sentence and once as its longest words in the order they appear, which is closer to what a person types. The two families give the same answer.
A result is right when it holds the text somebody asked for, measured as the fraction of the query's characters inside the returned chunk. Scoring by "the chunk the sentence started in" was the first rule written, and it is unfair to any policy whose chunks overlap: it put the overlap policy at 0.823 against 0.938 for fixed, a difference that was entirely an artefact of the rule.