TIKA-4766: minimal typed Document parse contract for tika-grpc (stage 1)

[krickert]

First slice of the typed Document contract — stage 1 of the sequence proposed by @nddipiazza in #2921: the Document envelope, Dublin Core DocumentMetadata, the lossless tagged metadata tail, and ParseStatus. Generic transformer only; no blocks/markdown, no embedded recursion, no FormatCategory; ParseBytes is deferred to its own JIRA.

The reply is additive. FetchAndParseReply.fields/status are untouched and document is a new field alongside — no client breaks (the "stage the break" approach from #2916). The legacy fields map keeps carrying the flat content until the content tree lands in a later stage.

Tail values mirror Tika's String[]-backed metadata model: every entry is a typed array, tagged by the declared Property element type, so declared integer sequences (pdf:charsPerPage, tiff:BitsPerSample) arrive as int64s per element and dates as Timestamps. Untyped keys stay strings, never guessed; if any element refuses its declared type the whole entry falls back to strings. No reflection anywhere — the mapping is an explicit compile-time table (TikaCoreProperties.TITLE → metadata.title), so reviewing it is reading one file.

Coverage: 31 mapper unit tests (tagging contract, DC mapping, builder edge cases, a registered-transformer test demonstrating the stage-2 mechanism), 10 in-process tika-grpc tests including a unary round trip through the real forked pipes worker, and a live-server e2e case asserting typed booleans/integers/strings over the wire.

Deferred to follow-up PRs per the staged plan: per-format transformers, the structured content tree, embedded-document recursion.

[krickert]

@nddipiazza new try -

Minimal scope, good document shape, no reflection. Followed your feedback - most of the code is tests.

The mapping is a greppable, compile-time-checked table (TikaCoreProperties.TITLE → metadata.title)

Let's give it another look?

[tballison]

From a quick read, this looks great. I'm struggling with how to migrate from what we have to a more structured approach given what I'm hoping is a 4.0.0 release fairly soon.

I personally don't want to hold 4.0.0 for this, but I do think this is important.

Apologies if you've done this in previous iterations, but would it be possible to keep grpc as is and then work on a tika.v2 service that covers this work and the potential follow on work. We can then release tika.v2 when this is finalized enough.

Is that too complicated?

@krickert and @nddipiazza wdyt?

[krickert]

From a quick read, this looks great. I'm struggling with how to migrate from what we have to a more structured approach given what I'm hoping is a 4.0.0 release fairly soon.

I personally don't want to hold 4.0.0 for this, but I do think this is important.

Apologies if you've done this in previous iterations, but would it be possible to keep grpc as is and then work on a tika.v2 service that covers this work and the potential follow on work. We can then release tika.v2 when this is finalized enough.

Is that too complicated?

@krickert and @nddipiazza wdyt?

The v2 idea is spot on. The strange lesson I had by transitioning from a java kingdom to a serialized schema IDL is that the models should never break. I used to purposefully break my APIs to force customers to write the data how I wanted them to. Now I see the light.

Protobufs provide an "out" for most of it, and the ultimate out you get is when you break the contract and just add a "v2" so your "v1" shapes can still serialize. Then it'll be up to you to map it. Otherwise data structures are always additive and you deprecate or retire old fields.

So your suggestion is exactly what the "v{n}" package name is for. We can release a new point version of the tika engine separately from the model version since they play well together.

And I don't mind as many iterations as we need. The API shape is something we get stuck with, and the Tika team is the one that defines the shape after parsing.

There are two schools of data structuring that come out of parsing: data shape (the physical look of it like bounding boxes) and then the data surface (NLP analysis, tokens, the chunks plus body, and governed metadata). I've found in parser-land that it's best to separate these two concerns. Docling falls more into the first category (layout-first, NLP second) while tika seems to follow the second (NLP first, layout second). Both sorta flirt with each other as far as data goes, but that tends to be where each parser aims to be.

There are ways to more accurately combine them, but don't ever try to pull an Icarus and fly too close to a unified single model. If you try, you'll end up just angering both camps to use it. The idea is that you share metadata or link between the two data views.

A lot of theory there, but hope this makes sense..

BTW - if you want instant docling-looking data using diskless OCR, this is a grpc service I slapped together that's faster than docling (nvidia/IBM gfx compatible) but returns the same data and this is a grpc service that wraps poi into grpc.

[nddipiazza]

@krickert i agree a v2 is needed. we had people using v1 so that was going to be needed regardless because actually tika v3 had a grpc contract that a few groups are using which is compatible with the contract we have in 4.x

[krickert]

@krickert i agree a v2 is needed. we had people using v1 so that was going to be needed regardless because actually tika v3 had a grpc contract that a few groups are using which is compatible with the contract we have in 4.x

I'll go ahead and make the protos in a new package. I'd encourage you to Mark the old ones as deprecated but I'll leave that up to you.

Are you okay if that's what I do with this pass or is there any additional things you'd like me to change before I start.