Tika 4972

.skills/devs/development/SKILL.md

@@ -133,6 +133,27 @@ personal configuration does not override it.
   -Dmaven.repo.local=$(pwd)/.local_m2_repo
 ```
 
+## Building the Docs
+
+`docs/` is not in the default reactor; build it from its own pom. Antora
+renders the checked-out branch from the worktree, so uncommitted `.adoc`
+edits are included.
+
+```bash
+./mvnw -f docs/pom.xml clean package     # site in docs/target/site/tika/<version>/
+```
+
+- **Broken links don't fail the build** (`--log-failure-level=fatal`). Check
+  the log for `ERROR (asciidoctor)` lines such as `target of xref not found`.
+- **Parent-directory npm leaks in.** The Antora plugin runs `npx` in `docs/`,
+  which has no `package.json` or `node_modules`, so npm and Node search
+  upward. A `package.json` or `node_modules` in any parent directory, often
+  `$HOME` after an `npm install` run there, becomes the npm project root and
+  supplies packages. The plugin warns `Detected the existence of
+  $HOME/node_modules`. `npm prefix` in `docs/` should print `docs/`; if it
+  prints a parent, move that install elsewhere (e.g. `npm install -g`).
+- For a release build, use `update-site-for-release`.
+
 ## Common Module Paths
 
 | Module | Path |

.skills/users/file-forensics/SKILL.md

@@ -106,7 +106,7 @@ java -jar tika-app.jar --config=file-forensics-config.json -Z --extract-dir=evid
 # mkdir -p evidence/suspect-embedded && unzip -q suspect-embedded.zip -d evidence/suspect-embedded
 
 jq 'map(del(."tk:content"))' suspect.rmeta.json > suspect.meta.json
-ls -lR evidence/suspect-embedded && sha256sum evidence/suspect-embedded/*/unpacked/* 2>/dev/null || sha256sum evidence/suspect-embedded/unpacked/*
+ls -lR evidence/suspect-embedded && sha256sum evidence/suspect-embedded/unpacked/*
 ```
 
 `suspect.rmeta.json` is the full evidence record — a JSON array where entry
@@ -339,20 +339,18 @@ files is incomplete.
   back visually corrupted (`VB\_Name`). The `-z`/`-Z` extracted files are
   pristine bytes. (Alternative: capture with `-J -t` for unescaped plain
   text per entry.)
-- Extracted files are **renumbered** (`00000001.jpg`, ...); original names
-  live in each rmeta entry's `tk:resource-name`, and `-z`/`-Z` writes a
-  sidecar `<name>.json` metadata dump for mapping numbers back to names.
-- The output is a Frictionless data package (Tika 4.2+): the files sit under
-  `<container>/unpacked/`, and `<container>/datapackage.json` already carries a
-  SHA-256 per file. Digest what you extracted
-  (`sha256sum evidence/suspect-embedded/*/unpacked/*`) and check it against the
+- Extracted files are **renumbered** (`1.jpg`, `2.pdf`, ...); each
+  `datapackage.json` resource carries the original name (`tk:resource-name`)
+  and `tk:embedded-id`, the key to the file's row in `rmeta.json`.
+- The output is a Frictionless data package (Tika 4.2+): for a single input
+  file the extract dir is the package — files under `unpacked/`, and
+  `datapackage.json` already carries a SHA-256 per file (and for
+  `rmeta.json`). Digest what you extracted
+  (`sha256sum evidence/suspect-embedded/unpacked/*`) and check it against the
   manifest, so each artifact is pinned the same way the container file is.
 - Mechanics and gotchas (Pipes-mode delay, chatty stderr) are in
   `file-to-markdown`; the server-side equivalent is `/unpack`, which returns
-  the same package as one zip over HTTP. **`/unpack` names differ from
-  `-z`/`-Z`:** plain sequential names (`unpacked/1.jpg`, `unpacked/2.pdf`, ...)
-  and the rmeta rows live in the package's `metadata.json` rather than a
-  sidecar — map names back via each row's `tk:resource-name`. Against the
+  the same package as one zip over HTTP, laid out the same way. Against the
   server started above (forensics config loaded):
 
   ```bash

.skills/users/file-to-markdown/SKILL.md

@@ -151,11 +151,13 @@ embedded content in: email attachments (`.eml`, `.msg`), archive members
 (`.zip`), PDF attachments, embedded objects/images in any office format
 (an inline pasted picture counts), and so on.
 
-What lands on disk: a `<name>/` Frictionless package — `datapackage.json`
-(SHA-256 per file), `metadata.json`, and the embedded files **renumbered**
-under `unpacked/` (`00000001.jpg`, ...). Original names are not preserved on
-disk; they are in each rmeta entry's `tk:resource-name`, so keep the sibling
-`<name>.json` (an rmeta-shaped dump `-z` also writes) to map numbers back.
+What lands on disk: the extract dir (default `<name>-unpacked/`) is a
+Frictionless package — `datapackage.json` (SHA-256 per file),
+`rmeta.json` (the rmeta list), and the embedded files **renumbered**
+under `unpacked/` (`1.jpg`, ...). Original names are not preserved on disk;
+each `datapackage.json` resource has `tk:resource-name` and `tk:embedded-id`
+(its row in `rmeta.json`). An input
+directory gets one such package per file, under its relative path.
 
 `-z`/`-Z` route through Tika Pipes mode rather than the fast synchronous
 path the other flags use — expect several seconds and a burst of

CHANGES.txt

@@ -1,5 +1,24 @@
 Release 4.2.0 - unreleased
 
+   * Frictionless unpack: the package's metadata.json is renamed rmeta.json, is listed in
+     datapackage.json as a resource (size, sha256), and now gets the metadata filters and
+     user metadata the emitted list gets; resources are in embedded-id order. A document
+     with no attachments is a package of datapackage.json and rmeta.json: tika-server
+     /unpack returns it with 200 instead of 204 (unpack-config.emptyPackage=SKIP, which the
+     thumbnail preset sets, restores the 204). The package name is the caller's filename
+     on tika-server, never the server's spool file. tika-app -z/-Z on a single file makes
+     the output dir the package (default <name>-unpacked/), names files unpacked/1.<ext>
+     as tika-server /unpack does, and no longer writes a <name>.json next to it. New
+     unpack-config.packageRoot and packageName set where a DIRECTORY-mode package goes
+     and its name. Packages pass frictionless validate: each resource's name is derived
+     from its path, the original filename moves to tk:resource-name, and tk:embedded-id
+     keys it to its rmeta.json row (TIKA-4971).
+
+   * Image checks (byte-size, min-image-size, blank-image-check; new tika-image-processing
+     module) make it less likely that blank or tiny images reach text recognizers. VLM
+     output gets tk:ocr-warnings (truncated, repetition, empty); VLM parsers take
+     temperature; dropped HTTP connections are retried (TIKA-4972).
+
    * Config properties whose value reaches the host (paths, executables, class names,
      URLs, credentials, operator-sized limits) are marked @OperatorOnly and refused in
      per-request JSON before any setter runs. Newly locked: pdf-parser's

docs/modules/ROOT/examples/jina-ocr-basic.json

@@ -0,0 +1,16 @@
+{
+  "engines": {
+    "vlm": {
+      "openai-vlm-parser": {
+        "baseUrl": "https://api.jina.ai",
+        "model": "jina-ocr-v1",
+        "apiKey": "${env:JINA_API_KEY}"
+      }
+    }
+  },
+  "text-recognizers": [
+    {
+      "engine": "vlm"
+    }
+  ]
+}

docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc

@@ -338,7 +338,7 @@ ls -R /tmp/tika-app-test/frictionless-out
 ----
 
 *Expected:* `test_recursive_embedded.docx.json` plus a `test_recursive_embedded.docx/`
-directory holding `datapackage.json`, `metadata.json` and `unpacked/`.
+directory holding `datapackage.json`, `rmeta.json` and `unpacked/`.
 
 === Test 24: Unpack zipped, and the pre-4.2 flat layout
 

docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc

@@ -158,7 +158,7 @@ curl -s -X PUT -T test_recursive_embedded.docx http://localhost:9998/unpack/all
 unzip -l /tmp/unpack.zip
 ----
 
-*Expected:* A Frictionless package: `datapackage.json`, `metadata.json`, and the embedded
+*Expected:* A Frictionless package: `datapackage.json`, `rmeta.json`, and the embedded
 files plus the original container (`0.docx`) under `unpacked/`.
 
 === Test 13: GET /parsers

docs/modules/ROOT/pages/configuration/index.adoc

@@ -158,18 +158,63 @@ stays loaded, and `image-parser` and `pdf-parser` call it:
 }
 ----
 
-An entry naming an engine takes only `engine`, `_mime-include`, `_mime-exclude`, `_min-width` and
-`_min-height`; the engine's own settings live under `engines`, where an
+An entry naming an engine takes only `engine`, `_mime-include`, `_mime-exclude`, `min-width`,
+`min-height` and `checks`; the engine's own settings live under `engines`, where an
 xref:configuration/inference.adoc[inference binding] can name the same engine. The 4.0 form,
 the engine inline, still works: `{ "tesseract-ocr-parser": { "language": "eng" } }`, and takes
-the same underscore keys.
+the same keys. `_mime-include`/`_mime-exclude` keep their underscore because every component list
+uses the one spelling; `min-width`, `min-height` and `checks` belong to this list alone, so they
+need none, and the older `_min-width`/`_min-height` still mean the same (setting both spellings of
+one key fails the load). In the inline form these three are removed from the engine's settings.
 
-`_min-width`/`_min-height` keep an image whose recorded size (`tiff:ImageWidth`,
+`min-width`/`min-height` keep an image whose recorded size (`tiff:ImageWidth`,
 `tiff:ImageLength`) falls short in either dimension away from the engine, in pixels; an image of
 unknown size is sent. Whatever the entry says, an image under 2 x 2 (a spacer in an email
 signature) never reaches any engine; the engines' own byte gates (`minFileSizeToOcr`) are
 unchanged.
 
+[#image-checks]
+*Send the engine fewer images that hold no text.* An entry's `checks` lists
+image checks by name, run in order, cheapest first; the first to refuse keeps the image from the
+engine, whatever the engine is (Tesseract, a VLM, ...), and records `skipped:<reason>` in
+`tk:ocr-warnings` (on the parent document too, for an inline image). A check that cannot read the
+image admits it. The checks live in the `tika-image-processing` module (JDK only, no native code):
+
+[source,json]
+----
+{
+  "engines": { "jina": { "openai-vlm-parser": { "baseUrl": "https://api.jina.ai", "model": "jina-ocr-v1" } } },
+  "text-recognizers": [
+    { "engine": "jina",
+      "checks": [
+        { "byte-size": { "minBytes": 100, "maxBytes": 52428800 } },
+        { "min-image-size": { "minTotalPixels": 10000, "minShortSidePixels": 16 } },
+        { "blank-image-check": { "maxInk": 0.0001, "tolerance": 40 } }
+      ] }
+  ]
+}
+----
+
+`byte-size` refuses an image shorter than `minBytes` or longer than `maxBytes` by the length the
+stream already knows, without reading it, so it belongs first; 0 turns a limit off and an image of
+unknown length is admitted. `min-image-size` refuses an image under `minTotalPixels` in area or whose
+shorter side is under `minShortSidePixels`, by the dimensions in its header (it reads only the start of a
+large file, and the whole file only when the dimensions are not there); `minWidthPixels` and `minHeightPixels`
+ask for more of one dimension on top of `minShortSidePixels`, and 0 turns a test off. The `min-image-size`
+values shown are its defaults, and `byte-size` has no limit until set: below 100 x 100 an image holds a few
+characters at most, and a strip such as 11 x 1000 passes the area test but holds no line of text.
+`blank-image-check` refuses an image whose
+share of pixels differing from its most common gray level by more than `tolerance` is at most
+`maxInk` (0.0001 is 0.01 %, below a page with a line of text), measured on a copy of at most 1,000
+pixels a side. With `despeckle` (on by default) a pixel counts only if most of its 3 x 3 neighbourhood
+differs from the background, so scattered specks, as on a noisy scan, do not hide a blank page: 0.1 % to 5 %
+salt-and-pepper noise comes out blank, and a stroke of text does not.
+
+A check's settings need no underscore. `min-width` and `min-height` on the entry are a separate,
+metadata-only size minimum that runs before any check. The checks do not tell a photo from a page of
+text, and do not detect rotation. Nor do they, or the `tk:ocr-warnings` codes, catch fluent hallucinations
+(xref:configuration/parsers/vlm-parsers.adoc#fluent-hallucinations[what they do not catch]).
+
 *Turn OCR off.* An empty list is authoritative:
 
 [source,json]

docs/modules/ROOT/pages/configuration/inference.adoc

@@ -60,7 +60,9 @@ A hosted engine caps the requests in flight per key and answers the excess with
 usually with no `Retry-After`; a saturated one answers 502, 503 or 504. Those four answers are
 retried, `maxRetries` times (4 unless set, 0 to fail at once), with a short jittered backoff
 from 250 ms doubling to 4 s, or the `Retry-After` the server sends, and never past the budget
-the parse granted the call. Any other answer, a 401 or a 400, fails at once. The retries cover
+the parse granted the call. A connection that drops before a full response arrives is retried
+the same way, sharing the `maxRetries` count. Any other answer, a 401 or a 400, fails at once,
+as do an unknown host, a refused connection and a TLS failure. The retries cover
 collisions between a few parse workers; they do not turn more workers than the key allows
 into throughput. Keep `numClients` (tika-server, Pipes) at or below the key's concurrency,
 counting every engine the workers call, and raise the key's tier before raising the workers.

docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc

@@ -38,7 +38,7 @@ xref:configuration/index.adoc[Configuration]. Naming it under `parsers` is depre
 |Parser |Endpoint |Config key
 
 |`OpenAIVLMParser`
-|Any OpenAI-compatible chat completions endpoint (vLLM, Ollama, local FastAPI, OpenAI)
+|Any OpenAI-compatible chat completions endpoint (vLLM, Ollama, local FastAPI, OpenAI, Jina AI)
 |`openai-vlm-parser`
 
 |`ClaudeVLMParser`
@@ -143,6 +143,30 @@ endpoint. The API key is passed as a `key` query parameter.
 
 Change `baseUrl` if you are using Vertex AI or a proxy.
 
+== Jina OCR
+
+https://jina.ai[Jina AI]'s `jina-ocr-v1` model is a document-OCR model that outputs
+clean Markdown. Its API is OpenAI-compatible, so `OpenAIVLMParser` drives it with no
+extra dependencies.
+
+=== Basic Configuration
+
+[source,json]
+----
+include::example$jina-ocr-basic.json[]
+----
+
+Set the `JINA_API_KEY` environment variable to your key.
+`${env:NAME}` syntax is supported everywhere in Tika JSON config
+(xref:configuration/index.adoc#env[Secrets from the environment]).
+
+Jina OCR accepts images only (PNG, JPEG, GIF, TIFF, WebP, BMP). It does not accept
+PDFs or video. The MIME type declared in the `data:` URI is informational — Jina
+auto-detects the format from the bytes, so a mismatched label does not cause an error.
+
+NOTE: `jina-ocr-v1` tables are emitted as raw HTML inside the model's Markdown output.
+Tika's Markdown-to-XHTML converter handles this correctly — see <<_markdown_to_xhtml_conversion>>.
+
 == Using a VLM parser for PDF parsing (deprecated)
 
 Claude and Gemini can process entire PDFs with their vision capabilities.
@@ -197,7 +221,8 @@ All three parsers share the same configuration POJO (`VLMOCRConfig`):
 
 |`maxRetries`
 |`4`
-|Retries of a 429, 502, 503 or 504 answer, with a jittered backoff inside the parse budget;
+|Retries of a 429, 502, 503 or 504 answer, or of a connection that dropped before a full
+response arrived, with a jittered backoff inside the parse budget;
 `0` fails at once. A hosted service caps requests in flight per key, so see
 xref:configuration/inference.adoc#concurrency[Hosted engines and concurrency].
 
@@ -223,6 +248,12 @@ text into the parent document's content stream. Mirrors
 captioning or tagging prompt: the parser still runs, but the PDF parser's `AUTO` OCR never replaces
 a page's extracted text with its output.
 
+|`temperature`
+|unset
+|Sampling temperature sent with each request (0 to 2; Claude accepts 0 to 1). Unset sends none and the
+service's default applies. `0` makes repeated requests for the same image return the same text on
+services that honor it, which makes a wrong answer reproducible; it does not make the answer correct.
+
 |`minFileSizeToOcr`
 |`0`
 |Minimum input file size in bytes.
@@ -247,6 +278,28 @@ trusted callers. See <<_per_request_configuration,Per-request configuration>>.
 |HTTP path appended to `baseUrl` for the chat-completions endpoint. Used by the OpenAI-compatible parser only. Claude and Gemini hardcode their own API paths (`/v1/messages` and `/v1beta/models/\{model}:generateContent` respectively) and ignore this field.
 |===
 
+When a file is not sent because of `minFileSizeToOcr`, `maxFileSizeToOcr` or
+`maxImagePixels`, the parser sets `vlm:skipped-reason` to `file-too-small`, `file-too-large` or
+`image-too-large`; the last also sets `vlm:image-pixels`.
+xref:configuration/index.adoc#image-checks[Image checks] can screen out images too small or
+blank to be worth sending, before any text recognizer sees them.
+
+The returned text is never changed or dropped. When it looks doubtful, `tk:ocr-warnings` (multi-valued)
+gets one of: `truncated` (the model stopped at `maxTokens`), `repetition` (a repeated line or sentence;
+output of at least 1,000 bytes that deflates to under 12% of its size), or `empty`. For an
+`INLINE` image, whose text goes into the parent document, the same
+codes are added to the parent's `tk:ocr-warnings`, and a skip appears there as `skipped:<reason>`. That
+needs the recursive parser wrapper, which supplies the parent's metadata.
+
+[#fluent-hallucinations]
+=== Limits of the checks and warnings
+
+The image checks screen what is sent, and `tk:ocr-warnings` describes the shape of what comes
+back: a loop, a cut-off answer or no answer. Neither judges accuracy, so neither identifies fluent
+hallucinations, text that reads well but is not what the image says. Like any OCR engine, a VLM
+can misread an image, and output with no warning has not been verified. Where accuracy matters,
+spot-check a sample against the source.
+
 == Markdown-to-XHTML conversion
 
 The VLM's text response is expected to be markdown. Tika parses it using
@@ -296,7 +349,7 @@ rejected with an exception:
 |May be lowered, never raised above the initialization-time value — that ceiling is what bounds
 cost on a paid endpoint.
 
-|`maxImagePixels`, `maxRetries`, `allowRuntimePrompt`
+|`maxImagePixels`, `temperature`, `maxRetries`, `allowRuntimePrompt`
 |Always blocked.
 
 |`prompt`, `textRecognizer`

docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc

@@ -310,7 +310,7 @@ change with the aliases: page renders are emitted at the end of each page whiche
 and a page the parser does not read (`maxPages`) is not rendered. Two floors are new: a page
 narrower or shorter than 2 pixels at the target dpi is not rendered, and an image under 2 x 2
 pixels reaches no text recognizer or inference binding (`pages.render.minWidth`/`minHeight`,
-`_min-width`/`_min-height` on a recognizer entry, `minWidth`/`minHeight` on a binding). A binding's
+`min-width`/`min-height` (or the older `_min-width`/`_min-height`) on a recognizer entry, `minWidth`/`minHeight` on a binding). A binding's
 `minWidth: 0` restores 4.0; the 2-pixel floor before the recognizers stays whatever the entry says. A per-request `ocr` block now changes only the fields it names (4.0 replaced the whole
 block). In Java, `PDFParserConfig.getOcr()` and `getImageStrategy()` are gone: `getPages()` is
 the overlay, the aliases write to it, the deprecated setters stay.