## Summary
On Japanese (`lang="ja"`) sites, searching for a common compound nou…n such as 新幹線 ("Shinkansen" / bullet train) returns unrelated pages — ranked *above* the pages that actually contain the word — while searching for a fragment of it (幹線) returns exactly the right pages.
The root cause appears to be that the index and the query are segmented by different tokenizers that disagree on compounds:
- **Index time** (extended binary): 新幹線 is split into 新 + 幹線, so the index contains the words `新` and `幹線` but **not** `新幹線`.
- **Query time** (`Intl.Segmenter` with ICU's Japanese dictionary): 新幹線 is kept as a **single word** (`[...new Intl.Segmenter('ja', {granularity:'word'}).segment('新幹線')]` → one segment).
The query word 新幹線 therefore matches no indexed word, and the fallback appears to shorten the term from the end (新幹線 → 新幹 → 新) until something matches — landing on the extremely generic single-kanji word `新` ("new"), which exists on many pages via other splits (e.g. 新千歳空港 → 新 + 千歳 + 空港). Those unrelated pages then match, and can outrank the pages that genuinely contain 新幹線. The discriminative part of the query (幹線) — which *is* in the index — is exactly the part that gets discarded.
The [v1.5.0 release notes](https://github.com/Pagefind/pagefind/releases/tag/v1.5.0) describe the query-side `Intl.Segmenter` as segmenting queries the same way the content was indexed, but for Japanese the two sides use different dictionaries (Lindera vs ICU), so they frequently disagree on compounds.
## Reproduction
`site/shinkansen.html` — the page a user searching 新幹線 wants:
```html
<!doctype html>
<html lang="ja">
<head><title>東京から大阪への移動手段</title></head>
<body data-pagefind-body><h1>東京から大阪への移動手段</h1><p>東京から大阪への移動は新幹線が便利です。新幹線の切符は駅の窓口で購入できます。</p></body>
</html>
```
`site/chitose.html` — an unrelated page (about 新千歳空港 airport, no mention of 新幹線):
```html
<!doctype html>
<html lang="ja">
<head><title>札幌の空港アクセス</title></head>
<body data-pagefind-body><h1>札幌の空港アクセス</h1><p>新千歳空港から札幌駅まではバスで移動できます。新千歳空港は北海道の玄関口です。</p></body>
</html>
```
`site/search.html` — a minimal Default UI page:
```html
<!doctype html>
<html lang="ja">
<head>
<title>検索</title>
<link rel="stylesheet" href="/pagefind/pagefind-ui.css" />
<script src="/pagefind/pagefind-ui.js"></script>
</head>
<body>
<div id="search"></div>
<script>
window.addEventListener('DOMContentLoaded', () => {
new PagefindUI({ element: '#search' });
});
</script>
</body>
</html>
```
```sh
npx [email protected] --site site --serve
# open http://localhost:1414/search.html
```
| Query | Expected | Actual (1.5.2, Chromium) |
|---|---|---|
| `新幹線` | only 東京から大阪への移動手段 | **札幌の空港アクセス ranked first**, then 東京から大阪への移動手段 |
| `幹線` | 東京から大阪への移動手段 | 東京から大阪への移動手段 ✅ |
So a fragment of the compound finds the right page, while the full, natural query surfaces an unrelated page above it. (Via the JS API the scores are 0.28 for the unrelated page vs 0.26 for the correct one.)
Decoding the generated `pagefind/index/*.pf_index` chunks confirms the vocabulary: the index contains `新`, `幹線`, `千歳`, `空港`, … but no `新幹線`.
## Real-world impact
On our 189-page Japanese blog, the query 新幹線 returns 33 results led by pages about 新千歳空港 and 新宿 (which merely contain the word `新` after segmentation); the handful of pages that actually discuss 新幹線 are only findable by typing 幹線. An exact-phrase query `"新幹線"` returns 0 results, since no indexed word can match. Any Japanese compound that ICU keeps whole but Lindera splits behaves this way; the failure is invisible when the leftover prefix happens to be discriminative (駐車場 → 駐車 works fine) and severe when it is generic (新幹線 → 新).
## Environment
- Pagefind 1.5.2 (extended, via `npx pagefind`)
- Observed in Chromium 140 (Playwright) and reproduced in Node.js 22 (both use ICU for `Intl.Segmenter`)
- The production site is a static Astro 6 build with Pagefind run directly via the CLI, but Astro is not involved — the repro above uses plain HTML files and `npx pagefind` only
## Workaround we're shipping
> **Edit 3:** the workaround now also handles the **katakana direction** reported by @oharu121 in the comments (Lindera keeps a loanword whole, ICU shreds it into non-word fragments), which we confirmed bites hard on our corpus: of 231 katakana compounds that ICU splits, **66 were broken**, several returning zero or only wrong results. Two findings shaped the fix. First, as noted in the comments, a quoted-exact probe can't detect the whole word — the quoted phrase is assembled from the same shredded fragments — and no wrapper-level rewrite can prevent the shredding, because it happens inside `pagefind.js` (and, since 1.5.0, inside `pagefind-worker.js`'s own copy). The workaround therefore now includes a small **build-time patch to the generated bundle**: spans wrapped in U+2060 (word joiner) sentinels bypass query-side segmentation; unwrapped queries behave byte-identically to stock. Second, even with exact probes available, *"is the whole run an indexed word?"* is **not a sufficient gate**: we hit katakana compounds where Lindera and ICU both split, just differently — the katakana version of Edit 2's context-dependence — so the whole word is absent from the index *and* the plain query is junk. The reliable signal is the inverse: **some ICU fragment of the run is not an indexed word**, in which case the AND match can never succeed.
> **Edit 2:** earlier versions of this section rewrote the query by AND-ing the compound's leading/trailing bigrams onto it (新幹線 → `新幹線 新幹 幹線`), first via `processTerm`, then with a zero-result fallback. Both lost recall, and measuring why surfaced a fact that makes any query-side rewrite harder than it looks: **Lindera segments the *same* compound differently depending on context, within the same index**. In our index 国内線 ("domestic flights") is sometimes 国内+線 and sometimes 国+内線, so requiring both bigrams drops every page that only contains one of the splits (国内線: 70 → 50 hits, 最寄駅: 10 → 2). Meanwhile compounds that Lindera keeps whole (日暮里) don't contain their internal bigrams at all, which zeroed the query entirely. The version below no longer replaces the result set — it only reorders it.
Current approach, per kanji-only query word of 3+ characters:
1. **Probe with a quoted exact search** (`"日暮里"`) whether the compound exists verbatim as an indexed word. If it does, the plain query already behaves well — leave it untouched.
2. If it doesn't (新幹線, 国内線), run both the bigram-expanded query and the plain query, and return the **union**: expanded results first, then the plain query's remaining results deduped. Recall is now always identical to plain Pagefind; the expansion only promotes the pages that contain the compound's fragments (for 新幹線, the 7 genuine pages now rank above the 40 新* prefix matches).
And per katakana run that ICU splits into multiple fragments (using マイレージ, "mileage", as the running example — ICU yields マイ+レ+ージ):
3. **Probe each ICU fragment — and the run as a whole — with sentinel-quoted exact searches** (see the bundle patch below for why plain quoted probes can't work). The run is broken iff **some fragment is not an indexed word** (マイ and レ are indexed, ージ is not, so the AND over fragments can never match) or **the whole run is indexed verbatim** (Lindera kept マイレージ whole, so the pages that contain it don't contain the fragments either way).
4. For broken runs, additionally search with the run **sentinel-wrapped so it reaches the index whole**, and return the union: sentinel results first, then the plain query's remaining results. Healthy runs — where both tokenizers agree, e.g. シャトルバス → シャトル+バス on both sides — are left untouched.
The sentinel mechanism is the part that can't live in a wrapper. Query-side segmentation happens inside the generated bundle, so a build step patches it — **both `pagefind.js` and `pagefind-worker.js`, which carry independent copies of the segmentation code** (patching only one silently changes nothing, as noted in the comments). The patch is opt-in: only spans wrapped in U+2060 bypass segmentation, everything else is byte-identical to stock, and the build fails loudly if a Pagefind upgrade changes the minified target:
```js
// scripts/patch_pagefind_bundle.mjs — run after the `pagefind` CLI in the build
import { readFile, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
const BUNDLE_DIR = 'dist/pagefind'; // wherever the pagefind CLI writes your bundle
const FILES = ['pagefind.js', 'pagefind-worker.js'];
const TARGET = 'for(const{segment:word}of wordSegmenter.segment(term))';
const PATCHED =
'for(const{segment:word}of __sentinelAwareSegment(wordSegmenter,term))';
// Spans between U+2060 pairs (odd split indices) bypass the word segmenter;
// everything else is segmented exactly as stock Pagefind does.
const HELPER = `function __sentinelAwareSegment(segmenter,text){if(!text.includes("\\u2060"))return segmenter.segment(text);const out=[];const parts=text.split("\\u2060");for(let i=0;i<parts.length;i++){if(i%2===1){if(parts[i])out.push({segment:parts[i]});}else{for(const s of segmenter.segment(parts[i]))out.push(s);}}return out;}
`;
let failed = false;
for (const file of FILES) {
const path = join(BUNDLE_DIR, file);
const source = await readFile(path, 'utf8');
if (source.includes(PATCHED)) continue; // already patched
const occurrences = source.split(TARGET).length - 1;
if (occurrences !== 1) {
console.error(
`[patch-pagefind] ${file}: expected exactly 1 occurrence of the ` +
`segmentation call, found ${occurrences} — Pagefind's minified ` +
`code has changed, update TARGET.`,
);
failed = true;
continue;
}
await writeFile(path, HELPER + source.replace(TARGET, PATCHED));
}
if (failed) process.exit(1);
```
Since the search logic needs result counts and merging, it can't live in `processTerm`. We wrap the Pagefind API module and point the Default UI at the wrapper via its `bundlePath` option. The wrapper is a static file served at `/pagefind-mod/pagefind.js` alongside the real bundle (in an Astro project: `public/pagefind-mod/pagefind.js`; the real bundle stays at `/pagefind/` — the wrapper imports it by absolute URL, and since the real module resolves its assets from its own `import.meta.url`, the index still loads from `/pagefind/`):
```js
// /pagefind-mod/pagefind.js
// Only search(), debouncedSearch() and preload() are wrapped; everything else
// (createInstance, mergeIndex, …) is re-exported from the real bundle as-is.
export * from '/pagefind/pagefind.js';
import {
search as baseSearch,
preload as basePreload,
} from '/pagefind/pagefind.js';
const SENTINEL = '\u2060'; // word joiner; understood by the patched bundle
const segmenter =
typeof Intl !== 'undefined' && typeof Intl.Segmenter !== 'undefined'
? new Intl.Segmenter('ja', { granularity: 'word' })
: null;
// Skip queries we can't or shouldn't rewrite: no Intl.Segmenter, no kanji or
// katakana, or quote/sentinel characters anywhere (rebuilding the term would
// break phrase syntax / the sentinel protocol).
function isRewritable(term) {
return (
segmenter !== null &&
typeof term === 'string' &&
!term.includes('"') &&
!term.includes(SENTINEL) &&
/\p{Script=Han}|\p{Script=Katakana}/u.test(term)
);
}
function segmentWords(term) {
const words = [];
for (const { segment } of segmenter.segment(term)) {
const word = segment.trim();
if (word) words.push(word);
}
return words;
}
function isExpandableWord(word) {
return Array.from(word).length >= 3 && /^\p{Script=Han}+$/u.test(word);
}
function bigrams(word) {
const chars = Array.from(word); // code points, not UTF-16 units
return [chars.slice(0, 2).join(''), chars.slice(-2).join('')];
}
// ー (U+30FC) is Script=Common, so include it explicitly to keep the long
// vowel mark inside the run (スーツケース must not break at ー).
const KATAKANA_RUN = /[\p{Script=Katakana}ー]+/gu;
// Maximal katakana runs that ICU would split into multiple fragments — the
// only runs whose behavior our rewrite could change.
function splitKatakanaRuns(term) {
return (term.match(KATAKANA_RUN) ?? []).filter(
(run) => segmentWords(run).length > 1,
);
}
// Wraps the given runs in sentinels so the patched bundle keeps them whole.
function withWholeKatakana(term, runsToKeepWhole) {
if (runsToKeepWhole.size === 0) return term;
return term.replace(KATAKANA_RUN, (run) =>
runsToKeepWhole.has(run) ? SENTINEL + run + SENTINEL : run,
);
}
export async function search(term, options) {
if (!isRewritable(term)) return baseSearch(term, options);
const words = segmentWords(term);
const compounds = words.filter(isExpandableWord);
const kanaRuns = [...new Set(splitKatakanaRuns(term))];
if (compounds.length === 0 && kanaRuns.length === 0) {
return baseSearch(term, options);
}
const exactProbe = async (word) => {
const { results } = await baseSearch(`"${SENTINEL}${word}${SENTINEL}"`);
return results.length > 0;
};
const [kanjiProbes, kanaProbes] = await Promise.all([
Promise.all(compounds.map((w) => baseSearch(`"${w}"`))),
// A run is broken when some ICU fragment is missing from the index, or
// when the whole run is itself an indexed word (see Edit 3).
Promise.all(
kanaRuns.map(async (run) => {
const [whole, ...fragments] = await Promise.all(
[run, ...segmentWords(run)].map(exactProbe),
);
return whole || fragments.some((indexed) => !indexed);
}),
),
]);
// Kanji compounds are worth fixing when ABSENT from the index (ICU kept
// whole, Lindera split).
const missing = new Set(
compounds.filter((_, i) => kanjiProbes[i].results.length === 0),
);
const brokenKana = new Set(kanaRuns.filter((_, i) => kanaProbes[i]));
if (missing.size === 0 && brokenKana.size === 0) {
return baseSearch(term, options);
}
// One combined fixed query: katakana runs sentinel-wrapped in place, kanji
// bigrams appended (word order doesn't affect Pagefind's AND matching).
const fixedTerm = [
withWholeKatakana(term, brokenKana),
...[...missing].flatMap(bigrams),
].join(' ');
const [expanded, plain] = await Promise.all([
baseSearch(fixedTerm, options),
baseSearch(term, options),
]);
if (expanded.results.length === 0) return plain;
const promoted = new Set(expanded.results.map((r) => r.id));
return {
...plain,
results: [
...expanded.results,
...plain.results.filter((r) => !promoted.has(r.id)),
],
};
}
let lastDebounceId = 0;
export async function debouncedSearch(term, options, debounceTimeoutMs = 300) {
const id = ++lastDebounceId;
await new Promise((resolve) => setTimeout(resolve, debounceTimeoutMs));
if (id !== lastDebounceId) return null; // superseded by a newer call
return search(term, options);
}
export async function preload(term, options) {
const tasks = [basePreload(term, options)];
if (isRewritable(term)) {
// Preload the variants search() might issue; wrapping every split run is
// fine here (worst case we warm an unused index chunk).
const expanded = [
withWholeKatakana(term, new Set(splitKatakanaRuns(term))),
...segmentWords(term).filter(isExpandableWord).flatMap(bigrams),
].join(' ');
if (expanded !== term) tasks.push(basePreload(expanded, options));
}
await Promise.all(tasks);
}
```
```js
new PagefindUI({ element: '#search', bundlePath: '/pagefind-mod/' });
```
Measured on our (now ~200-page) index: マイレージ appears on 7 pages; stock returns **0 results** (and `"マイレージ"` also returns 0 — the quoted phrase is built from the same fragments), the patched setup returns **exactly those 7 pages**. Progressive typing no longer loses the results mid-word. All kanji-side numbers from Edit 2 are unchanged, and every query where the gates don't fire returns the identical result set to unpatched Pagefind.
## Possible directions
A few ideas, in case they're useful:
1. **Segment queries against the index's own vocabulary** (longest-match over indexed words) instead of — or as a fallback after — `Intl.Segmenter`, so the query can only produce words that can actually match.
2. **When shortening an unmatched CJK term, keep searching the remainder** rather than discarding it: 新幹線 → 新 + 幹線 instead of just 新. The discarded tail is often the discriminative part.
3. **Emit ICU-style segmentation at index time as well** (e.g. via `icu_segmenter`), so both vocabularies coexist in the index. ICU4X and browser ICU4C dictionaries still differ slightly, but far less than Lindera vs ICU.
Happy to provide the full repro as a tarball or test against a branch. Thanks for Pagefind — aside from this, it's been great to work with on a bilingual (ja/en) static blog.
---
*The investigation (index decoding, segmenter comparison, minimal repro) and this write-up were done together with Claude (Anthropic's Fable 5); the recall regression in the earlier workaround was caught by a code review from OpenAI's GPT-5.6 (Sol). Everything was reviewed and reproduced by a human before filing. The katakana extension in Edit 3 was likewise built with Claude, prompted by @oharu121's reports below.*
