Files are read and split on this device. Nothing is uploaded.
Choosing a chunk size
For search and retrieval, chunks of about 200 to 800 tokens work well: big enough to hold a complete thought, small enough that a match points to the right place. Overlap of 10 to 20 percent keeps a sentence that falls on a boundary readable in both chunks. For summarising a long document in parts, use chunks close to what the model can read at once, minus room for its answer.
Natural breaks or exact size
Natural breaks keep paragraphs and sentences whole whenever they fit, so each chunk reads cleanly. Exact size fills every chunk as close to the limit as it can, which gives more even chunks at the cost of cutting mid-sentence. With Markdown headings turned on, each section starts a fresh chunk and records its heading path, such as Guide > Install.
Questions
What is in the JSONL file?
One line per chunk with an id, the text, its number, the start and end position in the original text, its size in tokens or characters, the heading path for Markdown, the page numbers for a PDF, and the file name.
Which token counts are used?
With exact counts on, tokens are counted with OpenAI's o200k tokenizer, used by GPT-4o and later. Other models count a little differently, so leave some headroom if you're close to a hard limit.
Can it read scanned PDFs?
It reads the text layer of a PDF. A scanned PDF has no text layer; run it through the searchable PDF tool first, then open the result here.
How big a file can it handle?
Long books and reports are fine. Very large files take a few seconds, and the preview shows the first chunks with a button to show more; the download always has every chunk.