You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
dotAI: embed Block Editor (Story Block) fields as Markdown instead of HTML-stripped text #36003
When dotAI generates embeddings for content, Story Block (Block Editor) fields are currently extracted by converting the field to HTML and then running it through Tika to strip the markup down to plain text.
In com.dotcms.ai.util.ContentToStringUtil:
privateOptional<String> parseBlockEditor(@NotNullStringval) {
finalStoryBlockMapstoryBlockMap = newStoryBlockMap(val);
returnparseHTML(storyBlockMap.toHtml()); // → Tika strips all markup to plain text
}
The problem: Tika (and the JSoup/regex fallbacks in parseHTML) flatten the content to plain text, which destroys structure that matters for embedding quality — tables, code blocks, ordered/unordered lists, headings. The model gets a worse, less-structured representation of the content than it should.
What we want
Embed Story Block fields as Markdown instead of HTML-stripped-to-text. Markdown is already clean (no markup to strip) and preserves the structure — tables, code fences, lists, and headings survive into the text we embed, giving the model a far better representation.
How this is now feasible
PR #35728 (issue #35727) adds StoryBlockMap.toMarkdown() (and $markdownTool.blockToMarkdown(json)), a Tiptap-JSON → Markdown converter that handles paragraphs, headings, blockquotes, lists, code blocks (with language), tables, images, links, etc. This gives us a direct JSON → Markdown path that doesn't lose structure.
The fix is essentially: in parseBlockEditor, use storyBlockMap.toMarkdown() and return that markdown directly (no Tika/HTML round-trip needed, since markdown is already plain text).
Covered by a test asserting that a Story Block containing a table and a code block embeds with that structure intact.
Priority
Medium
Additional Context
ContentToStringUtil already imports MarkdownTool and StoryBlockMap, so the wiring is local to parseBlockEditor. The existing HTML path (toHtml() + Tika) can remain for other field types; only the Block Editor branch changes.
Setup: local dotCMS on localhost:8081, image dotcms/dotcms:modernization (built 2026-07-08). Content type BlockContent with a body field of type Block Editor.
Method — fire a contentlet whose Block Editor body has every structure the AC cares about, then invoke the same StoryBlockMap.toMarkdown() call that ContentToStringUtil.parseBlockEditor now makes (per
ContentToStringUtil.java lines 140-142). If the extracted string is proper Markdown with the structure preserved, the AC is met.
Reproduction — two curl calls
Seed a contentlet with H1 + paragraph + table + bullet list + fenced code block:
IDENT=$(curl -s -u admin@dotcms.com:admin -X PUT
"http://localhost:8081/api/v1/workflow/actions/default/fire/PUBLISH?indexPolicy=WAIT_FOR"
-H 'Content-Type: application/json'
-d '{"contentlet":{"contentType":"BlockContent","title":"qa-36003","body":"{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"type":"text","text":"Embedding
Structure Test"}]},{"type":"paragraph","content":[{"type":"text","text":"Prose paragraph before the
table."}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":
"text","text":"Feature"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"State"}]}]}]},{"type":"t
ableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"markdown
extraction"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"under
test"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Bullet
section"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"first
item"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"second
item"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Code block
section"}]},{"type":"codeBlock","attrs":{"language":"java"},"content":[{"type":"text","text":"public class Hello { }"}]}]}"}}'
| jq -r '.entity.identifier')
Extract the field via the same code path parseBlockEditor invokes:
Activity
fmontes commentedon Jun 5, 2026
This is a one line change:
feat(dotai): embed Story Block fields as Markdown instead of HTML-str…
github-actions commentedon Jun 30, 2026
PRs linked to this issue
hmoreras commentedon Jul 10, 2026
✅ QA — passed on modernization build (2026-07-10)
Setup: local dotCMS on localhost:8081, image dotcms/dotcms:modernization (built 2026-07-08). Content type BlockContent with a body field of type Block Editor.
Method — fire a contentlet whose Block Editor body has every structure the AC cares about, then invoke the same StoryBlockMap.toMarkdown() call that ContentToStringUtil.parseBlockEditor now makes (per
ContentToStringUtil.java lines 140-142). If the extracted string is proper Markdown with the structure preserved, the AC is met.
Reproduction — two curl calls
IDENT=$(curl -s -u admin@dotcms.com:admin -X PUT
"http://localhost:8081/api/v1/workflow/actions/default/fire/PUBLISH?indexPolicy=WAIT_FOR"
-H 'Content-Type: application/json'
-d '{"contentlet":{"contentType":"BlockContent","title":"qa-36003","body":"{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"type":"text","text":"Embedding
Structure Test"}]},{"type":"paragraph","content":[{"type":"text","text":"Prose paragraph before the
table."}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":
"text","text":"Feature"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"State"}]}]}]},{"type":"t
ableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"markdown
extraction"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"text","text":"under
test"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Bullet
section"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"first
item"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"text","text":"second
item"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Code block
section"}]},{"type":"codeBlock","attrs":{"language":"java"},"content":[{"type":"text","text":"public class Hello { }"}]}]}"}}'
| jq -r '.entity.identifier')
curl -s -u admin@dotcms.com:admin -X POST http://localhost:8081/api/vtl/dynamic
-H 'Content-Type: application/json' -H 'Accept: text/plain'
-d "{"id":"$IDENT","velocity":"#set($c=$dotcontent.find($bodyMap.id))$c.body.toMarkdown()"}"
36003 dotai embed block editor story block fields as markdown instead…