# Text Chunker — Split Text for RAG & Embeddings

> Split long text into token-sized chunks for RAG and embeddings pipelines — exact token counts, configurable overlap, export as JSONL. Runs locally.

Source: https://toolpantry.app/text-chunker

## About

Split long documents into token-sized chunks with overlap — the first step of every RAG and embeddings pipeline. Export as JSONL and feed it straight to your embedding script.

## Quick answer

How do I chunk text for embeddings? Paste the document, choose a chunk size (512 tokens is the common default) and an overlap (50–100 tokens), and press Split. Review the chunks and download them as JSONL or plain text.

## How to chunk text for RAG

1. **Paste the document** — or a section of it; very long books are better processed chapter by chapter.
2. **Choose size and overlap** — 512 tokens with 64 overlap is the standard starting point.
3. **Export and embed** — download chunks.jsonl and run your embedding pipeline over each line.

## Choosing the right chunk size

- **128–256 tokens** — best when queries are precise and answers live in a paragraph or two (FAQ bots, docs search).
- **512 tokens** — the all-round default; good balance of context and retrieval precision.
- **1,024–2,048 tokens** — for narrative or legal text where an answer spans pages.
- **Overlap 10–20%** — keeps boundary sentences intact in both chunks.

## Good to know

Chunking is the step most RAG quality problems trace back to: chunks that are too small lose context, too large dilute the embedding, and no overlap cuts sentences in half. After splitting, embed each line of the JSONL and store it with its source reference. Related: [LLM token calculator](https://toolpantry.app/llm-token-calculator.html) to size your context budget and [token counter](https://toolpantry.app/token-counter.html) for plain counting.

## Notes

Chunking is token-exact: the text is tokenized with the real GPT tokenizer, sliced by token count, and decoded back — so every chunk lands within a token or two of your target size. The JSONL export has one {"text": …, "tokens": …} object per line.

## FAQ

**What chunk size should I use for RAG?**

512 tokens with 50–100 tokens of overlap is the common default. Smaller chunks (128–256) give more precise retrieval but more embeddings to store; larger chunks (1,024+) keep more context per hit but retrieve less precisely. Try 512 first and adjust by testing retrieval quality.

**Why do chunks overlap?**

Overlap keeps sentences that fall on a boundary readable in both neighboring chunks. Without overlap, a key sentence can be cut in half and retrieved in a useless state. 10–20% of the chunk size is a good overlap.

**What is the JSONL export for?**

Each line is a JSON object with the chunk text and its token count — the shape embedding pipelines expect. You can feed the file straight to your embedding script or load it in a notebook without reformatting.