Skip to content

Perf regression from #2312: queries returning nested rows are 14–45× slower to hydrate #2355

Closed

Description

@nathan-muir

#2312 (fix(db): enforce functional callback metadata boundaries, first released in 1.0.0-rc.1.2) made live queries whose result rows hold nested objects much slower. For the nested-document query below, cold hydrate is 14 to 45 times slower, and per-write latency is 4 to 28 times slower. The existing benchmark doesn't catch it, because every query in it selects a few scalar fields from flat rows under limit(50).

Cause

Before #2312, finalizeRow in packages/db/src/query/compiler/index.ts called stripInternalRouteMetadata only in includes mode (if (parentKeyStream)). Since #2312, every output row of every query goes through transformPublicContainers up to three times:

  • stripInternalRouteMetadata(authoredResults);
  • the callback-view transformPublicContainers(selectedValue, …);
  • stripInternalRouteMetadata(callbackSelected).

transformPublicContainers calls Object.getOwnPropertyDescriptors on every nested object and array before it can tell that nothing changed. So the cost scales with the size of every result row, even when there's nothing to strip.

Benchmark

The patch below adds an open issues + documents case to scripts/bench/incremental-update.ts. It joins every open issue to a rich-text document of 20 blocks (about 80 nested objects), selects the whole document, and has no limit.

I ran it with the benchmark workflow, comparing #2312 (a3c4d067) against its parent (035a5260). Both sides run the same scripts/bench. Full run and per-configuration tables: nathan-muir#2 (comment)

Writes

Query Δ median write time (geomean) Best case Worst case Flags
list: newest 50 open +20.2% -32.9% +165.9% 3 🔴
list + author +22.5% -38.8% +205.1% 1 🔴 1 🟢
list + comment count +13.1% -7.2% +55.3% 1 🔴
list + 3 recent comments +16.4% -5.8% +108.5% 1 🔴
issue detail + comments +13.8% -51.3% +128.3% 1 🔴 1 🟢
open issues + documents +978.5% +316.8% +2742.1% 12 🔴

Cold hydrate

Query Δ cold hydrate time (geomean) Best case Worst case Flags
list: newest 50 open +26.5% -29.8% +199.7% 1 🔴
list + author +71.3% -30.4% +136.5% 3 🔴
list + comment count +25.7% -48.4% +395.2% 1 🔴
list + 3 recent comments +11.5% -7.6% +33.0% —
issue detail + comments +2.0% -32.3% +55.7% —
open issues + documents +2796.9% +1324.5% +4442.1% 12 🔴

The open issues + documents query is the only one with a large change, and it shows in all 12 configurations. At 10,000 issues, cold hydrate went from 163 ms to 5,535 ms, and a synced write from 0.058 ms to 1.54 ms.

diff --git a/scripts/bench/incremental-update.ts b/scripts/bench/incremental-update.ts
index 9d146db6..dafcf11a 100644
--- a/scripts/bench/incremental-update.ts
+++ b/scripts/bench/incremental-update.ts
@@ -49,6 +49,27 @@ type Comment = {
   updateTick: number
 }
 
+// A rich-text body: a row whose value is a deep tree rather than a few scalars.
+type IssueDocument = {
+  id: number
+  issueId: number
+  blocks: Array<DocumentBlock>
+  updateTick: number
+}
+
+type DocumentBlock = {
+  id: string
+  kind: `paragraph` | `heading` | `list`
+  text: string
+  marks: Array<{
+    type: `bold` | `link`
+    from: number
+    to: number
+    attrs: { href?: string }
+  }>
+  attrs: { indent: number; align: `left` | `center` }
+}
+
 type ManualSyncUtils<T extends object, TKey extends string | number> = {
   begin: Parameters<SyncConfig<T, TKey>[`sync`]>[0][`begin`]
   write: (message: ChangeMessageOrDeleteKeyMessage<T, TKey>) => void
@@ -67,9 +88,11 @@ type Fixture = {
   issues: ManualCollection<Issue, number>
   users: ManualCollection<User, number>
   comments: ManualCollection<Comment, number>
+  documents: ManualCollection<IssueDocument, number>
   issueById: Map<number, Issue>
   userById: Map<number, User>
   commentById: Map<number, Comment>
+  documentById: Map<number, IssueDocument>
   visibleIssueId: number
   selectedIssueId: number
   nextCommentId: number
@@ -164,6 +187,9 @@ const defaultCustomScale: FixtureScale = {
 
 const options = parseArgs(process.argv.slice(2), defaultOptions)
 
+const BLOCKS_PER_DOCUMENT = 20
+const BLOCK_KINDS = [`heading`, `paragraph`, `paragraph`, `list`] as const
+
 const queryCases: Array<QueryCase> = [
   {
     name: `list: newest 50 open`,
@@ -303,6 +329,27 @@ const queryCases: Array<QueryCase> = [
       ) as unknown as BenchmarkCollection,
     write: insertSelectedIssueComment,
   },
+  {
+    name: `open issues + documents`,
+    scenario: `visible document update`,
+    // Every open issue with its whole document and no limit, so each output row
+    // carries a nested tree into result finalization.
+    createQuery: ({ issues, documents }) =>
+      createLiveQueryCollection((q) =>
+        q
+          .from({ issue: issues })
+          .where(({ issue }) => eq(issue.status, `open`))
+          .join({ document: documents }, ({ issue, document }) =>
+            eq(issue.id, document.issueId),
+          )
+          .select(({ issue, document }) => ({
+            id: issue.id,
+            title: issue.title,
+            document,
+          })),
+      ) as unknown as BenchmarkCollection,
+    write: updateVisibleDocument,
+  },
 ]
 
 const scales = resolveScales(options)
@@ -468,6 +515,13 @@ function createFixture(options: BenchmarkRunOptions): Fixture {
     })
   }
 
+  const documentsData: Array<IssueDocument> = issuesData.map((issue) => ({
+    id: issue.id,
+    issueId: issue.id,
+    blocks: createDocumentBlocks(issue.id, random),
+    updateTick: 0,
+  }))
+
   const fixture = {
     seed: options.seed,
     issues: createManualCollection(
@@ -491,6 +545,15 @@ function createFixture(options: BenchmarkRunOptions): Fixture {
     issueById: new Map(issuesData.map((issue) => [issue.id, issue])),
     userById: new Map(usersData.map((user) => [user.id, user])),
     commentById: new Map(commentsData.map((comment) => [comment.id, comment])),
+    documents: createManualCollection(
+      `bench-documents`,
+      documentsData,
+      (document) => document.id,
+      options.sourceIndexMode,
+    ),
+    documentById: new Map(
+      documentsData.map((document) => [document.id, document]),
+    ),
     visibleIssueId,
     selectedIssueId,
     nextCommentId: options.commentCount + 1,
@@ -514,6 +577,70 @@ function applySourceIndexes(
   fixture.users.createIndex((user) => user.id)
   fixture.comments.createIndex((comment) => comment.issueId)
   fixture.comments.createIndex((comment) => comment.createdAt)
+  fixture.documents.createIndex((document) => document.issueId)
+}
+
+function createDocumentBlocks(
+  issueId: number,
+  random: () => number,
+): Array<DocumentBlock> {
+  const blocks: Array<DocumentBlock> = []
+  for (let index = 0; index < BLOCKS_PER_DOCUMENT; index++) {
+    const length = 40 + Math.floor(random() * 80)
+    blocks.push({
+      id: `${issueId}-${index}`,
+      kind: BLOCK_KINDS[index % BLOCK_KINDS.length]!,
+      text: `Block ${index} of issue ${issueId}`.padEnd(length, `.`),
+      marks: [
+        { type: `bold`, from: 0, to: 5, attrs: {} },
+        {
+          type: `link`,
+          from: 6,
+          to: 12,
+          attrs: { href: `https://example.com/${issueId}/${index}` },
+        },
+      ],
+      attrs: { indent: index % 3, align: index === 0 ? `center` : `left` },
+    })
+  }
+  return blocks
+}
+
+function updateVisibleDocument(
+  fixture: Fixture,
+  iteration: number,
+  mutationMode: MutationMode,
+): WriteResult {
+  const document = fixture.documentById.get(fixture.visibleIssueId)!
+  const [first, ...rest] = document.blocks
+  const next = {
+    ...document,
+    blocks: [{ ...first!, text: `Edited at tick ${iteration}` }, ...rest],
+    updateTick: iteration,
+  }
+  fixture.documentById.set(document.id, next)
+
+  if (mutationMode === `synced`) {
+    writeSync(fixture.documents, {
+      type: `update`,
+      value: next,
+    })
+    return
+  }
+
+  const transaction = writeOptimistic(() => {
+    fixture.documents.update(document.id, (draft) => {
+      draft.blocks = next.blocks
+      draft.updateTick = next.updateTick
+    })
+  })
+
+  return {
+    cleanup: () => {
+      transaction.rollback()
+      fixture.documentById.set(document.id, document)
+    },
+  }
 }
 
 function updateVisibleIssue(
@@ -706,6 +833,7 @@ async function cleanupFixture(
     fixture.issues.cleanup(),
     fixture.users.cleanup(),
     fixture.comments.cleanup(),
+    fixture.documents.cleanup(),
   ])
 }

The patch applies to main: scripts/bench hasn't changed since #2312.

Possible direction

I've tried a fast path in transformPublicContainers. It runs a non-allocating scan first, and if no container under the value holds an omitted key and no leaf transforms, it returns the value as-is. It also skips the callback-view walk in finalizeRow when sourceViews is empty. That brings the 10,000-issue cold hydrate from 6,272 ms to 638 ms on current main, but it's still about 4 times slower than before #2312, because the scan still visits every nested object. Recovering all of it probably needs to avoid visiting source-row data at all, since collection rows never carry the compiler's symbols. Branch: nathan-muir#1

Activity

  1. KyleAMathews commented on Nov 5, 2026

    @KyleAMathews
    Collaborator

    Hmm that's bad — looking into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions