Skip to content

dotAI: embed Block Editor (Story Block) fields as Markdown instead of HTML-stripped text #36003

Description

@fmontes

Description

When dotAI generates embeddings for content, Story Block (Block Editor) fields are currently extracted by converting the field to HTML and then running it through Tika to strip the markup down to plain text.

In com.dotcms.ai.util.ContentToStringUtil:

private Optional<String> parseBlockEditor(@NotNull String val) {
    final StoryBlockMap storyBlockMap = new StoryBlockMap(val);
    return parseHTML(storyBlockMap.toHtml());   // → Tika strips all markup to plain text
}

The problem: Tika (and the JSoup/regex fallbacks in parseHTML) flatten the content to plain text, which destroys structure that matters for embedding quality — tables, code blocks, ordered/unordered lists, headings. The model gets a worse, less-structured representation of the content than it should.

What we want

Embed Story Block fields as Markdown instead of HTML-stripped-to-text. Markdown is already clean (no markup to strip) and preserves the structure — tables, code fences, lists, and headings survive into the text we embed, giving the model a far better representation.

How this is now feasible

PR #35728 (issue #35727) adds StoryBlockMap.toMarkdown() (and $markdownTool.blockToMarkdown(json)), a Tiptap-JSON → Markdown converter that handles paragraphs, headings, blockquotes, lists, code blocks (with language), tables, images, links, etc. This gives us a direct JSON → Markdown path that doesn't lose structure.

The fix is essentially: in parseBlockEditor, use storyBlockMap.toMarkdown() and return that markdown directly (no Tika/HTML round-trip needed, since markdown is already plain text).

Related work

Depends on #35728 being merged (or StoryBlockMap.toMarkdown() being available).

Acceptance Criteria

  • dotAI embeds Story Block fields as Markdown (via StoryBlockMap.toMarkdown()), not HTML-stripped-via-Tika.
  • Structure is preserved in the embedded text: tables, code blocks, ordered/unordered lists, and headings are no longer flattened away.
  • No Tika/HTML round-trip is performed for Story Block fields (markdown is returned directly).
  • Conversion never throws on unknown/extended Tiptap nodes or marks (consistent with feat(tiptap): convert Story Block content to Markdown (#35727) #35728's graceful degradation).
  • Covered by a test asserting that a Story Block containing a table and a code block embeds with that structure intact.

Priority

Medium

Additional Context

ContentToStringUtil already imports MarkdownTool and StoryBlockMap, so the wiring is local to parseBlockEditor. The existing HTML path (toHtml() + Tika) can remain for other field types; only the Block Editor branch changes.

Activity

  1. fmontes commented on Jun 5, 2026

    @fmontes
    MemberAuthor

    This is a one line change:

    Tika.parse(storyBlockMap.toMarkdown())
  2. added theissue type on Jun 9, 2026
  3. self-assigned this
    on Jun 22, 2026
  4. github-actions commented on Jun 30, 2026

    @github-actions
    Contributor
  5. hmoreras commented on Jul 10, 2026

    @hmoreras
    Member

    ✅ QA — passed on modernization build (2026-07-10)

    Setup: local dotCMS on localhost:8081, image dotcms/dotcms:modernization (built 2026-07-08). Content type BlockContent with a body field of type Block Editor.

    Method — fire a contentlet whose Block Editor body has every structure the AC cares about, then invoke the same StoryBlockMap.toMarkdown() call that ContentToStringUtil.parseBlockEditor now makes (per
    ContentToStringUtil.java lines 140-142). If the extracted string is proper Markdown with the structure preserved, the AC is met.

    Reproduction — two curl calls

    1. Seed a contentlet with H1 + paragraph + table + bullet list + fenced code block:

    IDENT=$(curl -s -u admin@dotcms.com:admin -X PUT
    "http://localhost:8081/api/v1/workflow/actions/default/fire/PUBLISH?indexPolicy=WAIT_FOR"
    -H 'Content-Type: application/json'
    -d '{"contentlet":{"contentType":"BlockContent","title":"qa-36003","body":"{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"type":"text","text":"Embedding
    Structure Test"}]},{"type":"paragraph","content":[{"type":"text","text":"Prose paragraph before the
    table."}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":
    "text","text":"Feature"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"State"}]}]}]},{"type":"t
    ableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"markdown
    extraction"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"under
    test"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Bullet
    section"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"first
    item"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"second
    item"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Code block
    section"}]},{"type":"codeBlock","attrs":{"language":"java"},"content":[{"type":"text","text":"public class Hello { }"}]}]}"}}'
    | jq -r '.entity.identifier')

    1. Extract the field via the same code path parseBlockEditor invokes:

    curl -s -u admin@dotcms.com:admin -X POST http://localhost:8081/api/vtl/dynamic
    -H 'Content-Type: application/json' -H 'Accept: text/plain'
    -d "{"id":"$IDENT","velocity":"#set($c=$dotcontent.find($bodyMap.id))$c.body.toMarkdown()"}"

  6. added a commit that references this issue on Jul 16, 2026
    eb15ac3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions