How Client-Side OCR Reads Text From Images

Optical character recognition sounds like it should need a server. You have an image of text, you want the text back, and that feels like a job for some heavy model running on a GPU somewhere. But the entire pipeline runs perfectly well inside a browser tab, and that is exactly how the OCR tool on this site works. The image you drop never leaves your machine.

The engine doing the work is Tesseract, compiled to WebAssembly and wrapped by the tesseract.js library (version ^6.0.1 in our package.json). Tesseract began as a research project at Hewlett-Packard, was open-sourced in 2005, and its modern recognition core is a neural network rather than the older pattern-matching approach. Running it in the browser means the WASM binary, the engine, and the per-language model files all download once and then execute locally against the pixels of your image.

This post walks through what actually happens between "here is a PNG" and "here is the text," using the real code from our ImageTextExtractor component as the reference. No part of this is hand-waving about how OCR works in general: every parameter mentioned is one the tool actually sets.

The worker model

The first thing the component does is spin up a Tesseract worker. Workers matter because OCR is CPU-heavy and you do not want it blocking the UI thread that is painting your progress bar.

import { createWorker, PSM, OEM } from 'tesseract.js';

const langs = secondaryLanguage
  ? `${selectedLanguage}+${secondaryLanguage}`
  : selectedLanguage;

const worker = await createWorker(langs, OEM.LSTM_ONLY, {
  logger: (m) => {
    if (m.status === 'recognizing text') {
      setProgress(Math.round(m.progress * 100));
    }
    // ...other status branches
  },
});

Three things are decided right here. The langs string tells the worker which language model files to fetch. The second argument, OEM.LSTM_ONLY, picks the OCR Engine Mode. And the logger callback is how the worker reports progress back, which is why the UI can show statuses like "Loading OCR core," "Loading language traineddata," and a percentage during the recognition phase.

OEM.LSTM_ONLY is the important one. OEM stands for OCR Engine Mode, and Tesseract historically shipped two engines: a "legacy" engine based on character-shape pattern matching, and an LSTM engine, which is a recurrent neural network. By selecting LSTM only, the tool uses the neural recognizer exclusively. That choice changes what the model files are (they contain trained network weights, not just glyph templates) and generally produces better results on real-world fonts.

The recognition pipeline, stage by stage

When you call worker.recognize(imageFile), Tesseract runs a sequence of stages. Understanding them explains why some images read cleanly and others come back as garbage.

Binarization

OCR does not reason about color. The first step is to convert the image to pure black and white, a process called binarization. Tesseract uses Otsu's method by default, which picks a threshold by looking at the histogram of pixel intensities and choosing the cutoff that best separates the two clusters (dark ink, light background).

This is the single biggest reason image quality matters. If your text is dark gray on a slightly-less-dark gray background, the histogram has no clean valley to split on, and binarization either drops faint strokes or floods the page with noise. High contrast gives Otsu a clean threshold; low contrast does not. The same goes for uneven lighting: a photo of a receipt with a shadow across half of it can binarize the lit half correctly and turn the shadowed half into a solid block. This is exactly why the tool's own tips panel tells you to ensure good lighting and contrast.

Page layout analysis and segmentation

Once the image is binary, Tesseract figures out where the text is. It detects connected components (blobs of adjacent black pixels), groups them into lines, groups lines into regions, and then splits lines into words by looking at gaps. This is layout analysis.

How aggressively it does this is controlled by the Page Segmentation Mode (PSM), which the tool exposes directly:

await worker.setParameters({
  tessedit_pageseg_mode: pageMode as unknown as typeof PSM.AUTO,
});

The component offers five modes:

These are not cosmetic. If you feed a single-line receipt total into Auto mode, the layout analyzer can misjudge the structure and waste effort hunting for columns that are not there. Telling it "this is one line" (PSM 7) removes that guesswork. The tool's tips even call this out: use Single Line mode for receipts. Conversely, Sparse Text (11) is the right call for something like a screenshot with scattered labels, where there is no flowing paragraph structure at all.

Character classification

This is the LSTM's job. For each text line, Tesseract feeds a normalized strip of the image into the recurrent network, which scans left to right and emits a sequence of character probabilities. Because it is an LSTM rather than a per-glyph classifier, it has context: it sees a span of the line at once, so it can disambiguate characters that look identical in isolation. The classic example is distinguishing the letter "l", the digit "1", and the pipe "|", which often differ only by the surrounding letters.

The network outputs a probability distribution over characters at each step, and a decoding pass collapses that into the most likely string. This is also where the per-word confidence values come from. The component reads them straight out of the result:

const words = pageData.words?.map(w => ({
  text: w.text,
  confidence: w.confidence,
  bbox: w.bbox,
})) || [];

Each word comes back with its recognized text, a confidence percentage, and a bbox (bounding box giving the pixel coordinates x0, y0, x1, y1 of where that word sat in the image). The tool surfaces the overall page confidence in the UI and color-codes it: green at 80 or above, yellow at 60 or above, red below, with a "review recommended" nudge for low scores. That number is the model telling you how sure it is, and it is a genuinely useful signal. A clean, high-contrast scan tends to score well; a blurry or low-contrast photo drags the score down, and the bounding boxes let you see which specific words it struggled with.

The language model

The LSTM does not work in a vacuum. Tesseract pairs it with a language model: a dictionary and character n-gram statistics packed into the traineddata file for each language. When the raw network output is ambiguous, the language model nudges decoding toward real words and plausible letter sequences. That is why selecting the correct language is not optional politeness; it changes the result.

Pick English for a French document and two things go wrong. First, the character set is off: French uses accented glyphs (é, à, ç) that the English model is not trained to emit confidently. Second, the dictionary is wrong, so the decoder cannot lean on it to fix borderline characters. The tool ships eighteen languages, and crucially it lets you combine two of them with that + syntax shown earlier, so eng+fra loads both models for a mixed-language document.

Why it can run entirely in the browser

There is no server round trip anywhere in the extractText function. worker.recognize(imageFile) takes the File object straight from the drop zone and hands its bytes to the WASM engine running in the page. The image is read into memory, processed, and discarded. Nothing is uploaded.

The cost of that privacy is the initial download. The first time you run a recognition, the browser fetches the WASM core and the traineddata for your chosen language, which is why the progress logger has distinct phases for loading the OCR core, initializing the engine, and loading language traineddata before it ever reaches the recognizing-text phase. Language files are not tiny (an LSTM traineddata file is typically a few megabytes), so the very first scan in a session feels slower than the rest. After that, the recognition itself is pure local computation.

When the job is done, the component is explicit about cleanup:

const { data } = await worker.recognize(imageFile);
// ...process data...
await worker.terminate();

Calling worker.terminate() tears down the worker and frees the WASM memory it allocated. Spinning a fresh worker per extraction keeps memory from accumulating across multiple scans, which matters on a long-lived tab.

What you get back, and how the tool shapes it

The raw recognition result is richer than a single text blob. The component pulls three structured pieces out of it: the full text, an array of paragraphs, and the per-word data with confidence and bounding boxes. From there it applies plain string transformations that have nothing to do with OCR but everything to do with usability:

if (trimWhitespace) {
  processedText = processedText.split('\n').map(line => line.trim()).join('\n');
  processedText = processedText.replace(/\n{3,}/g, '\n\n');
}

if (removeLineBreaks) {
  processedText = processedText.replace(/\n+/g, ' ').replace(/\s+/g, ' ').trim();
}

Trimming whitespace collapses the ragged spacing that binarization-plus-segmentation tends to leave behind. The "remove line breaks" option flattens everything into one paragraph, which is what you want when you OCR a column of text and intend to reflow it. Case transforms (upper, lower, title) and the export formats (plain text, JSON, Markdown, HTML) are all post-processing on the recognized string. The JSON export, for instance, bundles the confidence score, the detected paragraphs, and word and character counts alongside the text, which is handy if you are piping the output into another tool.

Practical takeaways for getting good results

Everything above collapses into a few rules that follow directly from the pipeline:

• Feed it contrast. Binarization is the first gate. Dark text on a light background, evenly lit, gives Otsu's method a clean threshold. Low contrast and shadows are where OCR quality dies.
• Resolution helps the classifier. The LSTM normalizes line height, but it cannot recover detail that was never captured. Small, low-resolution text leaves the network guessing.
• Match the language. The dictionary and character set in the traineddata file actively correct ambiguous characters. Use eng+fra style combinations for mixed text.
• Match the page mode to the layout. Single Line for receipts, Sparse Text for scattered screenshots, Auto for full documents. The wrong PSM makes the segmenter solve a problem you do not have.
• Read the confidence score. It is the model's own honesty about how sure it is. A low score on a clean-looking image usually points to a binarization or language mismatch you can fix and rerun.

Try it

All of this is wired into the Image Text Extractor tool on loopaloo. Drop in an image, pick a language (or two), choose a page segmentation mode if your layout is unusual, and the recognition runs locally in your browser via Tesseract.js. You get the extracted text, a confidence score, and one-click export to plain text, JSON, Markdown, or HTML, with the original image never leaving your device.