Skip to content

Block Editor: lossless Markdown round-trip for all stored content (dotcms-* fences for rich nodes, custom-node fallback, table attrs, marks) on the fire/save path #36658

Description

@hassandotcms

Description

Since #36002 (PR #36253), the workflow fire endpoints convert a Block Editor field supplied as Markdown into Tiptap/ProseMirror JSON server-side. But Markdown has no syntax for the rich nodes the editor persists (dotContent embedded contentlets, dotVideo, youtube, gridBlock, aiContent, or customer-supplied custom blocks), so today:

  • A Markdown write to a document containing rich blocks is ignored (existing document kept, warning logged). The save returns 200 and the caller has no signal that nothing was applied.
  • The overwrite guard's whitelist is missing youtube and image, so a single YouTube embed wrongly blocks all Markdown writes to that document.
  • Constructs the guard does not inspect are lost silently: merged table cells (colspan/rowspan/colwidth), text alignment, underline/sub/sup, link target/rel.
  • The JSON-to-Markdown direction flattens rich nodes to plain text (dotContent emits its title), so a JSON -> MD -> JSON round trip destroys them.

This blocks non-interactive clients (AI agents via MCP, headless imports) from updating any content that contains an embed, which is exactly the content Markdown-driven agents are expected to maintain.

What we want

Give every rich node a Markdown form so the round trip is lossless, using fenced code blocks with dotcms-* info strings. This is standard CommonMark, preserved verbatim by every parser and by LLMs, and parseable with the fence handling TiptapMarkdown already has:

```dotcms-content
{"identifier": "2d5d1c4c-...", "languageId": 1, "title": "Q3 Report"}
```
  • Thin payloads: dotContent/dotImage/dotVideo need only identifier (plus languageId); the server already rehydrates attrs.data from the identifier on every read (StoryBlockAPI.allowedTypes). title is emitted for readability and ignored on parse. gridBlock carries its node subtree verbatim (children preserved, not Markdown-editable). Unknown/custom node types (remote customBlocks extensions) are carried by a generic dotcms-node fence with the raw node JSON.
  • Parsing is always on: any inbound Markdown may contain these fences. Payloads are untrusted: strict per-label schema, hard-coded node type per label (no type smuggling), size caps. Any invalid fence degrades to an ordinary codeBlock and never throws (consistent with feat(tiptap): convert Story Block content to Markdown (#35727) #35728's graceful-degradation contract).
  • Emission is opt-in: toMarkdown() output stays byte-identical, since dotAI embeddings (dotAI: embed Block Editor (Story Block) fields as Markdown instead of HTML-stripped text #36003) and $storyBlock.toMd template output must not change. A new round-trip mode emits the fences, exposed via $markdownTool / StoryBlockMap overloads.
  • The guard becomes honest and shrinks: whitelist youtube/image (bugfix); each fence added removes its type from the "unrepresentable" set; when a write still cannot be applied or would degrade content (e.g. merged table cells), say so in the existing response envelope (ResponseEntityView.messages). This is additive with no contract change: the endpoint keeps accepting writes exactly as today.
  • Marks/alignment (textAlign, underline, link target) carried via block-level <!-- dotcms:attrs {...} --> comment decorations (block comments already survive the parser today).
  • Update the MCP server guidance (execute.ts) that still describes pre-Workflow fire: convert Block Editor Markdown to ProseMirror JSON on save (server-side) #36002 behavior.

Related work

Acceptance Criteria

  • Firing with Markdown containing each dotcms-* fence (dotcms-content, dotcms-image, dotcms-video, dotcms-youtube, dotcms-grid, dotcms-ai, dotcms-node) stores a valid Tiptap document; embedded contentlets read back fully hydrated via GraphQL/REST with no human re-save.
  • JSON -> toMarkdown(round-trip mode) -> toTiptap is a fixed point for documents composed of all supported node types (asset-bound nodes compared by identifier).
  • Default toMarkdown() output is byte-identical to current behavior for the existing test corpus (no dotAI embedding or template regression).
  • Invalid, oversized, or type-smuggling fence payloads degrade to a plain code block; conversion never throws.
  • A document containing only supported rich nodes accepts Markdown writes; when a write is still ignored or degrades content, the fire response carries a message saying what and why (response envelope only; no new rejections, no status-code changes).
  • Documents containing a youtube or image node accept Markdown writes (whitelist fix).
  • Merged table cells and mark/alignment decorations survive the round trip (or, at minimum for cells, are detected and reported rather than silently lost).
  • Server-converted JSON opens and re-saves unchanged in the Block Editor (golden test).
  • Security matrix covered by tests: payload schema validation, node-type allow-list, payload size caps, fence content containing backticks or -->.

Activity

  1. self-assigned this
    on Jul 21, 2026
  2. moved this from New to In Progress in dotCMS - Product Planningon Jul 23, 2026
  3. github-actions commented on Jul 23, 2026

    @github-actions
    Contributor
  4. moved this from In Progress to In Review in dotCMS - Product Planningon Jul 27, 2026
  5. moved this from In Review to QA in dotCMS - Product Planningon Jul 31, 2026
  6. fmontes commented on Sep 13, 2026

    @fmontes
    Member

    Documentation impact — 4 pages on dev.dotcms.com need updating

    The dotcms-* vocabulary from this issue (plus the Markdown/HTML save path from #36002 and the HTML leg in #36659) isn't documented anywhere public. The feature targets AI agents and headless clients — exactly the users who can only discover a capability if it's written down. Today the only way to learn the vocabulary is to read TiptapMarkdown.java.

    Pages

    Page Problem Size
    Block Editor ⚠️ Contains an inaccuracy (below). No mention of Markdown as a save format, the dotcms-* fences, <dotcms-*> elements, <!-- dotcms:attrs -->, or the overwrite behavior change Largest
    Markdown Viewtool Documents only parse / parseFile — blockToMarkdown is missing entirely, both forms Smallest, highest value
    Block Editor JSON Schema Omits aiContent, gridBlock and custom/remote blocks — the three that now have a Markdown form. Block Types table also lacks system names for dotContent and the AI block Medium
    Saving Content via API No Story Block row in the field-type table; no mention that a 200 can carry advisory messages Small

    Two things to flag to the writer

    The Block Editor page is now wrong, not just incomplete. It frames saving raw HTML as something you get "by leveraging the ability to convert a WYSIWYG field to a Block Editor field." HTML — and Markdown — are accepted on the save path generally, with no relationship to WYSIWYG conversion.

    Terminology trap: PR #36709's description says $markdownTool.blockToMarkdown(...). The viewtool is registered under the key markdown, so the working reference is $markdown.blockToMarkdown(json[, "roundtrip"]); $markdownTool renders as literal text. Don't copy the PR prose. (The same typo is in TiptapMarkdownReadableParityTest.java:13.)

    What the docs need to answer

    Three questions, nothing more: what can I send, what do I get back, what happens when I get it wrong (invalid payload → plain code block, save still returns 200, never throws). Plus the behavior change: a Markdown/HTML write to a document holding embeds now applies as a full replace and names what it replaced in messages — it previously returned 200 and changed nothing.

    Where the answers live

    • TiptapMarkdown.java → DotcmsFences: labels, required fields, size/depth caps, the two output flavors
    • TiptapHtml.java: same payloads as HTML attributes — note the hyphenated spellings (language-id, mime-type, text-align) differ from the JSON keys
    • MapToContentletPopulator.java: JSON-vs-Markdown-vs-HTML routing and the exact wording of every advisory message
    • MarkdownTool.java / StoryBlockMap.java: the two read surfaces; toolbox.xml:321 confirms the context key
    • TiptapMarkdownFenceVocabularyTest.java: worked examples of every valid and degrading payload

    Decide before drafting

    1. Is the fence vocabulary public API? Stability commitment + full reference treatment, or a smaller note about an internal transport that happens to be reachable. Everything else follows from this.
    2. Which release carries it? Both PRs merged to main (2026-07-31, 2026-08-04); the user-visible version needs confirming.
    3. Should STORY_BLOCK_MARKDOWN_RICH_OVERWRITE_PROTECT be public? Useful to a team caught out by the change, and a setting nobody should casually enable.
    4. Does Block Editor v3 inherit this? Separate page, needs confirming.
    5. Do the headless/GraphQL docs need a note that a thin identifier-only fence reads back fully hydrated? Non-obvious and reassuring for exactly that audience.
  7. fmontes commented on Sep 13, 2026

    @fmontes
    Member

    Recipe: see the new behaviour working (REST only)

    For whoever writes the docs — a happy-path walkthrough against a demo instance, no edge cases. Every command and output below was run end-to-end on a build carrying the fix. Needs curl and jq; uses the demo site's Blog type, whose blogContent field is a Story Block.

    0. Get a token

    TOKEN=$(curl -s -X POST http://localhost:8080/api/v1/authentication/api-token \
      -H 'Content-Type: application/json' \
      -d '{"user":"admin@dotcms.com","password":"admin","expirationDays":1,"label":"doc-recipe"}' \
      | jq -r .entity.token)

    1. Pick an existing contentlet to embed

    REF=$(curl -s -H "Authorization: Bearer $TOKEN" \
      "http://localhost:8080/api/content/query/+contentType:Blog/limit/1" | jq -r '.contentlets[0].identifier')

    2. Write a blog post as Markdown, embeds included

    The two fenced blocks are the new part — a YouTube embed and an embedded contentlet, neither of which plain Markdown can express.

    cat > body.md <<'MD'
    # Written as Markdown
    
    Ordinary **prose**, a list:
    
    - one
    - two
    
    Then two blocks Markdown has no syntax for:
    
    ```dotcms-youtube
    {"src": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}
    ```
    
    ```dotcms-content
    {"identifier": "REPLACE_ME", "languageId": 1}
    ```
    MD
    
    jq -n --arg body "$(sed "s/REPLACE_ME/$REF/" body.md)" \
      '{contentlet:{contentType:"Blog",title:"Markdown recipe",urlTitle:"markdown-recipe-demo",
        contentHost:"demo.dotcms.com",postingDate:"2026-09-13",blogContent:$body}}' > create.json
    
    curl -s -X PUT http://localhost:8080/api/v1/workflow/actions/default/fire/PUBLISH \
      -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
      -d @create.json > create-res.json
    
    jq -r '.entity.blogContent.content[].type' create-res.json
    ID=$(jq -r .entity.identifier create-res.json); INODE=$(jq -r .entity.inode create-res.json)

    Markdown went in, a real Block Editor document came out:

    heading
    paragraph
    bulletList
    paragraph
    youtube      ← from the dotcms-youtube fence
    dotContent   ← from the dotcms-content fence
    

    Open the post in the Block Editor at this point — it renders and re-saves like any hand-authored document.

    3. Read it back — the embed is fully populated

    We only sent an identifier. The server fills in the rest on every read, so the embed can never go stale.

    curl -s -H "Authorization: Bearer $TOKEN" "http://localhost:8080/api/content/id/$ID" \
      | jq '.contentlets[0].blogContent.content[] | select(.type=="dotContent") | .attrs.data
            | {identifier, title, urlTitle, contentType}'
    {
      "identifier": "e7ceee89-3cc5-4e35-9795-5927ac88ad89",
      "title": "10 Surfing spots to checkout in Tahiti",
      "urlTitle": "10-surfing-spots-to-checkout-in-tahiti",
      "contentType": "Blog"
    }

    4. Read it back as Markdown

    curl -s -X POST http://localhost:8080/api/vtl/dynamic \
      -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
      -d "$(jq -n --arg v "#set(\$c = \$dotcontent.find(\"$INODE\"))\${c.blogContent.toMarkdown(\"roundtrip\")}" '{velocity:$v}')" \
      > roundtrip.md

    roundtrip.md comes back as what went in, embeds and all:

    # Written as Markdown
    
    Ordinary **prose**, a list:
    
    - one
    - two
    
    Then two blocks Markdown has no syntax for:
    
    ```dotcms-youtube
    {"src":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}
    ```
    
    ```dotcms-content
    {"identifier":"e7ceee89-3cc5-4e35-9795-5927ac88ad89","languageId":1,"title":"10 Surfing spots to checkout in Tahiti"}
    ```

    Drop "roundtrip" and you get the old human-readable output instead, where the embeds flatten to a link and a title. That output is unchanged from before and is the one to use for reading, not for writing back.

    5. Edit the prose and write it straight back

    This is the whole point of the feature. Edit roundtrip.md however you like — change a sentence, add a paragraph — then send it back.

    sed 's/Ordinary \*\*prose\*\*/Ordinary **prose**, now edited by an agent/' roundtrip.md > edited.md
    
    jq -n --arg body "$(cat edited.md)" --arg id "$ID" \
      '{contentlet:{identifier:$id,contentType:"Blog",blogContent:$body}}' > edit.json
    
    curl -s -X PUT http://localhost:8080/api/v1/workflow/actions/default/fire/PUBLISH \
      -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d @edit.json \
      | jq -c '{messages, blocks: [.entity.blogContent.content[].type]}'
    {"messages":[],"blocks":["heading","paragraph","bulletList","paragraph","youtube","dotContent"]}

    Prose edited, both embeds intact, nothing to report. Before this change, that write would have been discarded entirely.

    6. The new warning — what happens if you don't carry them over

    jq -n --arg id "$ID" \
      '{contentlet:{identifier:$id,contentType:"Blog",blogContent:"Just a plain paragraph now.\n"}}' > drop.json
    
    curl -s -X PUT http://localhost:8080/api/v1/workflow/actions/default/fire/PUBLISH \
      -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d @drop.json \
      | jq -r '.messages[].message'

    HTTP 200, the write applies, and the response says exactly what it cost:

    Story Block field [blogContent]: 2 rich block(s) in the stored document are not present in the
    submitted Markdown and were replaced (youtube https://www.youtube.com/watch?v=dQw4w9WgXcQ,
    dotContent e7ceee89-3cc5-4e35-9795-5927ac88ad89). Carry them over as dotcms-* fences to preserve them.
    

    This is the behaviour change worth documenting most clearly: it is a warning on a successful save, not an error, and it arrives in messages — which most clients ignore today.

    Clean up

    curl -s -X PUT http://localhost:8080/api/v1/workflow/actions/default/fire/DESTROY \
      -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
      -d "{\"contentlet\":{\"identifier\":\"$ID\",\"contentType\":\"Blog\"}}"

    Notes

    • Steps 2–6 all work identically with HTML instead of Markdown, using <dotcms-youtube src="…"> and <dotcms-content identifier="…" language-id="1"> elements (TiptapHtml: accept the same rich-node vocabulary on HTML input (parity with the Markdown leg) #36659). Note the hyphenated attribute names.
    • The same flow through GraphQL: query the field as blogContent { json } and the embed comes back populated exactly as in step 3.
    • $dotcontent.find() takes an inode, not an identifier — worth calling out, it's an easy first-try mistake.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions