Skip to content

refactor(ios): bind surface geometry and synthesis orientation - #4189

Merged
thymikee merged 22 commits into
mainfrom
refactor/surface-geometry
Nov 4, 2026
Merged

thymikee merged 22 commits into
mainfrom
refactor/surface-geometry

Conversation

@thymikee

@thymikee thymikee commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Bind Apple window, screen and digitizer coordinates to measured geometry. Preserve lit-display admission and refuse repeated presses after orientation changes, including 20-step and time-budget chunk boundaries, with completed-action disclosure and global indices.

37 files, +969/−429. Base is main at 7d42395ff7e1c03b643c2cf56683808bf5427cf3; the parent #4122 is merged as 0d5e5d6ae913deef546fbf1c75cfda3a94dd7513. ElementSearch and surface-policy layers remain held.

Validation

The branch is rebased onto current main. The rebase changed one file: main's #4187 renamed RunnerAlert.buttons to actions, so the alert test now reads alert.actions.first and keeps this branch's synthesis stub. Local git range-diff and per-file diff comparison against both the pre-rebase head and 2a775092b4e2934f4713c51999870a73194be075 report no lost hunk on either side.

At 2b9940af2a9e3e91c44e14535c026f96a2c2ac41 the local check:affected --run --head gate passed with exactCommit=true and checkoutChanged=false (receipt .tmp/validation/run-oXhSEW). Hosted coverage and integration checks are running at this head; draft smoke lanes stay skipped.

Device evidence, recorded in docs/evidence/surface-geometry-4189-2026-11-03.md:

Open limitation

The early windowQueryRejected route is now proved on a sole-display iPhone, iPhone 17 Pro, iOS 26.2, through back --in-app: the rejected applicationDefinedResizableWindows query short-circuits before any hit-test, the sole-screen marker is the only screen log, synthesis logs point=(32.2,84.0) referenceFrame=(0,0,402,874) interfaceOrientation=1 displayId=1, the navigation canary moves Liked posts: 0 to 1, and VISUAL_VERIFICATION changed=yes. Coordinate tap, hold and a sampled gesture pan deliver on the same window with fresh canaries. The probe is a natural rejection, not a patch: the heavy-feed snapshot exceeds the XCTest serializable depth, so the public query raises kAXErrorIllegalArgument while private AX still answers, which is why snapshots keep working. Confirmed only on that one simulator, device shape and feed shape; no foldable, physical device or scoped set, and not a global AX outage.

Default inner portrait refusal, scoped unsupported-device-scope and the new route are device-level limitations, not release blockers.

Earlier native proof and receipts remain attributed to fea9dbd85668d2d72be2f526eb350917c73f8d0d; prior b45 full gate, native methods, and live evidence retain their original attribution.

@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agent-device-website Ready Ready Preview Nov 4, 2026 2:41pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.81 MB 5.81 MB +1.7 kB
Package (unpacked) 5.81 MB 5.81 MB +1.7 kB
Package (download) 1.76 MB 1.76 MB +881 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.9 ms 27.9 ms -1.1 ms
CLI --help 50.0 ms 51.9 ms +1.9 ms

@thymikee

thymikee commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

At a76b6bf a repeated press can use a stale orientation, two regressions no longer test the production boundary, and live evidence is missing. The branch also conflicts with its base now, so please rebase.

Could each sequence step refresh the complete surface geometry when orientation changes, or refuse a stale surface, and pass that same orientation to native record creation? A repeated press can rotate the app on its first step, while tap conversion and long-press conversion retain the initial orientation and the event record reads the current one. Later presses or holds can hit the wrong control or miss while synthesis reports success. Cover a production sequence whose first action changes orientation.

Please restore the production rotation-basis assertion for a prepared scroll with a keyboard-clipped viewport. This test now rotates using its own viewport constant, so it can pass when dispatch uses the clipped band. Capture the synthesized endpoint through the native boundary and assert the full-panel basis and clipped planning and evidence band independently.

Please exercise production surface geometry construction with a placed-window single-display iPad fixture and assert that the emitted landscape point uses the measured whole display. The moved test supplies the expected panel size directly, so it cannot detect a missing sole display or a window-sized basis.

The changes to live geometry and dispatch still need live evidence: the reported Duo and iPhone runs did not complete, and the composed tap test substitutes synthesis. After rebuilding the runner and restarting the daemon, please provide source-SHA-attributed commands, display and point logs, and fresh visible outcomes for Duo closed, half-open and open in default and scoped simulator sets, including hinge readback and lit-panel capture. On an offset inner-panel window, cover prepared selector and coordinate taps through synthesized and XCTest dispatch, a hold, landscape keyboard-clipped scroll, a sampled gesture, and an orientation-changing repeated press. On a single-display iPhone, cover normal interaction and AX-unavailable window-less synthesized tap, hold and sampled routes, with the sole-screen marker and visible outcomes. Preserve the placed-window iPad coverage and rerun the composed native harness at this head.

Could the existing measured-context boundary own the resolved window and surface geometry, with tagged-point adapters and native records consuming the same orientation, one sole-display size reader shared with the window-less route, and clipping confined to the planner? Would changing the native synthesis interface first, defining refresh or refusal at sequence boundaries, and retargeting the two regressions keep the existing window and full-panel invariant without a new ADR?

The packet reports all 13 hosted checks passing, while the exact-head affected gate failed and its unit stage did not run. The substituted providers in the reported timeout test do not execute the edited Swift, and without receipt logs its cause remains unproven. Before merge, fix the sequence behavior and restore both production-boundary regressions, then provide current-head native and live evidence and a passing exact-head affected receipt including the unit stage.

@thymikee
thymikee force-pushed the refactor/surface-geometry branch from a76b6bf to b45b4c5 Compare November 3, 2026 14:54
@thymikee thymikee changed the title refactor(ios): bind surface coordinates to measured geometry refactor(ios): bind surface geometry and synthesis orientation Nov 3, 2026
@thymikee

thymikee commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Review update at b45b4c5a2171f1c8a376fe142f3303ae1278c07a, rebased onto published parent ec80eecb1e0251d51d93aa68071c88577f1ea7ba.

The sequence now refuses after an orientation change before a later action dispatches. Conversion and native record construction use the same orientation. Prepared keyboard-clipped scroll and measured placed-iPad dispatch assertions now cross the production native boundary. The parent's explicit litDisplayID: query API, per-query budgets, exception refusal, tests and ADR are preserved.

Current-head validation: unit-enabled iOS build and incremental isolation scan passed; 15 method-selected native tests passed, including retargeted fallback and alert stubs. swift test --package-path apple/snapshot-presentation passed 50 tests (unchanged library source since 451ba0e7). Full unit results: 1,920 files / 17,156 passed, one skipped; platform E2E contracts: 76 passed, six explicit live skips. Related integration: 136 files / 1,059 passed. Both gates passed with an unchanged checkout and exactCommit=true.

Sanitized local receipt results
[
  {
    "status": "passed",
    "exactCommit": true,
    "refs": {
      "base": "origin/main",
      "head": "HEAD",
      "resolvedHead": "b45b4c5a2171f1c8a376fe142f3303ae1278c07a"
    },
    "checkoutChanged": false,
    "receipt": "run-bBnaUs",
    "results": [
      {
        "id": "check:tooling",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "check:fallow",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "check:unit",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      }
    ]
  },
  {
    "status": "passed",
    "exactCommit": true,
    "refs": {
      "base": "origin/main",
      "head": "b45b4c5a2171f1c8a376fe142f3303ae1278c07a",
      "resolvedBase": "c8ad8fb1257787ce0973cf74271cb9a4fafb51d9",
      "resolvedHead": "b45b4c5a2171f1c8a376fe142f3303ae1278c07a",
      "mergeBase": "c8ad8fb1257787ce0973cf74271cb9a4fafb51d9"
    },
    "checkoutChanged": false,
    "receipt": "run-gwvX1M",
    "results": [
      {
        "id": "format",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "lint",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "typecheck",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "layering",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "fallow",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "native-dupes",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "build",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "ios-snapshot-differential",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Apple / Smoke Tests)"
      },
      {
        "id": "vitest-related",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "unit",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "coverage",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Coverage)"
      },
      {
        "id": "provider-integration",
        "status": "deduplicated",
        "exitCode": null,
        "reason": "covered by related tests"
      },
      {
        "id": "swift-runner-ios",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Apple / Smoke Tests, Conformance Differential / iOS Conformance Differential, Perf Nightly / iOS Command Perf, XCTest Nightly / iOS Runner Full XCTest Suite (${{ matrix.leg.name }}))"
      },
      {
        "id": "swift-runner-macos",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Apple / Smoke Tests)"
      },
      {
        "id": "web-smoke",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Apple / Smoke Tests, Integration Tests, Lint, Typecheck & Package, Perf Nightly / Android Command Perf, Perf Nightly / iOS Command Perf)"
      },
      {
        "id": "command-docs",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "docs-links",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "docs-prose",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "xctest-selection",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "packaged-runner-swift",
        "status": "passed",
        "exitCode": 0,
        "reason": null
      },
      {
        "id": "replay-ios",
        "status": "skipped",
        "exitCode": null,
        "reason": "parked, workflow_dispatch only (Replay Manual / iOS Replay Suite)"
      },
      {
        "id": "replay-ios-device",
        "status": "skipped",
        "exitCode": null,
        "reason": "parked, workflow_dispatch only (Replay Manual / iOS Replay Suite)"
      },
      {
        "id": "replay-macos",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Apple / Smoke Tests)"
      },
      {
        "id": "replay-linux",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Linux / Smoke Tests)"
      },
      {
        "id": "linux-command-evidence",
        "status": "skipped",
        "exitCode": null,
        "reason": "GitHub-authoritative (jobs: Linux / Smoke Tests)"
      },
      {
        "id": "replay-android",
        "status": "skipped",
        "exitCode": null,
        "reason": "parked, workflow_dispatch only (Replay Manual / Android Full Emulator Suite)"
      }
    ]
  }
]

The initial full run run-Hczxoj failed ten new native duplicate blocks; API/fixture consolidation removed them. An incorrect explicit SHA invocation was rejected before checks; run-gwvX1M uses the actual head. Neither is represented as a pass.

Final-head live: the exact device_open-returned clients connected to this worktree's restarted 0.21.37-dev daemon, isolated state /private/tmp/agent-device-surface-4189, and rebuilt private derived data. Native source fingerprint: ef9e8d7a8a3331f99f7e7ba22e2239ff12ea14510b968243db7fe985a06cdd43.

Duo commands: open com.callstack.agentdevicelab --metro-host 127.0.0.1 --metro-port 8199; fold open; press 'label="Form"'; fill 'id="field-name"' 'Geometry b45b4c5a'; get text 'id="field-name"'; snapshot; screenshot; scroll down --pixels 200. Hinge readback was 180. The field visibly contained and read back Geometry b45b4c5a; snapshot and lit capture succeeded. The scroller visibly moved while the keyboard stayed open; response movement=moved, keyboardAvoided=true, keyboard top 449, planning/evidence height 437.

Duo tap: point=(617,515) reference=(0,0,871,669) orientation=4
record: name=agent-device-tap displayID=3 interfaceOrientation=4
Duo scroll: start=(319,475) end=(119,475) reference=(0,0,951,669) orientation=4
record: name=agent-device-controlled-scroll displayID=3 interfaceOrientation=4
iPhone tap: point=(220,319) reference=(0,0,440,956) orientation=1
record: name=agent-device-tap displayID=1 interfaceOrientation=1

On the assigned iPhone, open com.apple.Preferences, snapshot, press 'label="General"' succeeded. Earlier visible confirmation showed General. Final cleanup read Duo hinge 0, shut down both assigned simulators, released both daemon claims cleanly and stopped Metro.

Still draft: the requested verified orientation/default/scoped/half-open matrix, explicit XCTest fallback, hold, sampled gesture, rotation-changing repeat and live AX-unavailable routes are incomplete. Initial rotation attempts did not consistently match the application's observed orientation; they are not credited as three-orientation proof. No extra scoped simulator or iPad was used. Native/mutation and earlier live evidence is attributed to its original SHA.

Range-diff compared old 06843017..a76b6bf0 with ec80eecb..b45b4c5a2171f1c8a376fe142f3303ae1278c07a. Cold self-review found no further correctness issue; an independent reviewer was unavailable in this child. Size: 27 files, +716/−422; production net −38. Later layers remain held. No CI watch or Apex reply was started.

@thymikee

thymikee commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

The code at b45b4c5 addresses the earlier design concern: geometry is shared and measured, native orientation is explicit, there is one measurement reader, and a sequence refuses to continue after rotation. I found no new defect. All 16 checks pass and there are no conflicts. The live device matrix still needs completion before merge.

The evidence record leaves changed native delivery routes unvalidated, so offset and multi-panel delivery, fallback, holds, sampled contacts, rotation during repetition, and AX-unavailable sole-screen dispatch remain unproven. After rebuilding the current runner and restarting its daemon, please post SHA-attributed commands, verified app orientations, hinge readback, lit-panel captures, native display/point/orientation logs, and fresh visible outcomes for Duo closed, half-open, and open in default and scoped sets. For an offset inner-panel window, cover prepared selector and coordinate taps through synthesized and XCTest dispatch, hold, sampled gesture, and a repeated press that visibly performs its first action and refuses further dispatch after rotation. On a single-display iPhone, cover AX-unavailable windowless synthesized tap, hold, and sampled routes with the sole-screen marker and visible outcomes. Please retain the clipped landscape scroll, placed-iPad native regression, and current-head composed-harness evidence; substituted synthesis confirms geometry arguments but does not prove contact delivery.

@thymikee

thymikee commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Publication evidence for fea9dbd85668d2d72be2f526eb350917c73f8d0d, retaining base fix/foldable-springboard-surface at ec80eecb1e0251d51d93aa68071c88577f1ea7ba. Keep draft. ElementSearch and surface-policy layers remain held.

The accepted P2 spans both the 20-step and time-budget chunk boundaries. The first response captures orientation, the host pins it onto remaining chunks, and native execution checks it before dispatch. Missing/invalid bindings refuse with completed-action disclosure and global indices. The production request drive now goes through real createAppleInteractor in packages/platform-apple/src/__tests__/interactions-press-series-runner-requests.test.ts; the older multi-chunk fixture was updated. The independent production review of b45b4c5a..f749 was clear and accepted the smaller design; helper extraction addressed complexity, and dedicated request-drive ownership addressed R84.

Exact-head gate

I read and verified the parent's run-5lG7kL/receipt.json; it names full head fea9dbd85668d2d72be2f526eb350917c73f8d0d, status=passed, exactCommit=true, checkoutChanged=false, with clean initial/final status. Same-commit evidence applies across checkouts, so I did not repeat that gate.

  • 14 checks passed, 11 skipped, one provider-integration deduplication.
  • Related tests: 1,066 passed. Unit tests: 17,157 passed, one skipped.
  • Format, lint, typecheck, layering, Fallow, native duplicates, build, command docs, docs links/prose, XCTest selection and packaged Swift passed.
  • GitHub-authoritative and parked checks remain skips, including coverage, device/native lanes and compatibility obligations.

Earlier complexity (f749), three-unit-failure (0a755) and test-name/R84 failures were failures. They are resolved by subsequent changes and the exact fea9 receipt; they are not relabeled as passes.

Same-method native regression proof

Only my tool-provided isolated checkout was changed. I recorded clean fea9, ran pnpm install --frozen-lockfile && pnpm build, detached this checkout at b45b4c5a2171f1c8a376fe142f3303ae1278c07a, repeated setup, and applied only the supplied native-regression.patch to RunnerRequestPanelTests.swift. The patch SHA-256 is ed8f19ff733e63ce636fb7df8980df262a1ab2d32176fb550aa1dcfd7f3bd2c3. The probe method is byte-for-byte identical to the committed head method.

Device: host local, 8F0B51BD-1B55-4B5E-A579-92959A027C2F, iPhone Duo (fixtures), iOS 27.1 (24A94401). It was opened in the Device panel before native runs. No other device was used.

Fresh unit-enabled base and head builds both passed, including isolation scans:

AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1 \
AGENT_DEVICE_XCUITEST_DESTINATION='platform=iOS Simulator,id=8F0B51BD-1B55-4B5E-A579-92959A027C2F' \
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH='<private lane-specific DerivedData>' \
pnpm build:xcuitest:ios

For each lane I found the actual AgentDeviceRunner_AgentDeviceRunnerUITests_iphonesimulator27.1-arm64.xctestrun under its fresh private build products. The same selected method ran on base and head:

xcodebuild test-without-building -xctestrun '<actual lane xctestrun>' \
  -destination 'platform=iOS Simulator,id=8F0B51BD-1B55-4B5E-A579-92959A027C2F' \
  -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRepeatedPressCarriesOrientationAcrossStepAndBudgetChunkBoundaries \
  -collect-test-diagnostics never -resultBundlePath '<unique lane xcresult>'
  • Base b45: build passed; selected test exited 65, 1 failed test / 10 assertion failures / 0 unexpected failures. Each chunk size (1 and 20) produced five assertion failures: absent response orientation, continuation completed 1 instead of 0, absent failed index, absent UNSUPPORTED_OPERATION, and dispatch counts 2 vs 1 / 21 vs 20. This is measured regression detection, not a setup failure.
  • I reversed only my patch, verified the base clean, restored fea9, repeated setup and rebuilt from fresh separate DerivedData.
  • Head fea9: identical selected method exited 0, 1 passed / 0 assertion failures / 0 unexpected failures / 0 skipped.
  • Additional head selection exited 0: 3 passed, 0 failed, 0 skipped: testRepeatedPressStopsWhenTheFirstActionChangesOrientation, testProductionRunnerRequestsDecodeWithoutDroppingAKey, testEveryRunnerRequestFieldHasAProductionRequest.

A third fresh, private DerivedData build at head, with AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS unset, passed. Its recorded production compiler settings contain no AGENT_DEVICE_RUNNER_UNIT_TESTS; the full isolation scan passed. pnpm check:xctest-selection passed: 606 declared methods, zero unreachable by every lane (the intentional testCommand skip remains).

Cleanup: device_close(hostId=local, deviceId=8F0B51BD-1B55-4B5E-A579-92959A027C2F, shutdown=true) succeeded. No manual CLI session or daemon was opened. My checkout is restored clean at fea9; logs, command arrays, SHA records, patch, receipts, xcresults and drafts are retained under .tmp/4189-native-final. No source fixes, other worktree/ref changes, Apex replies/resolutions, CI watch or later-stack publication were performed. Size verified: 37 files, +940/−429; this round 12 files, +225/−8.

Attribution and remaining gaps

Prior b45 evidence owns the earlier full gate, 15 native methods, snapshot-presentation results and live Duo/iPhone evidence. Those results are not credited to fea9.

The new native fixture intercepts synthesis. It proves the dispatch/refusal contract, not live hit delivery or natural app rotation. The verified orientation/default/scoped/half-open matrix, forced XCTest fallback, hold, sampled gestures, natural rotation-changing repeat and live AX-unavailable routes remain incomplete. CI coverage/device/compatibility checks and approval remain pending. This draft is published, not merge-ready.

@thymikee

thymikee commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

At fea9dbd the continuation binding and missing-binding validation address the earlier review, and I found no new defect. All checks pass and there are no conflicts.

The current runner still needs the live delivery matrix described in the evidence. The cross-chunk fixture changes a stub orientation and intercepts synthesis, so it cannot prove natural rotation or actual hit delivery. Offset-window, multi-panel, fallback, and AX-unavailable delivery remain unproven. After rebuilding the current runner and restarting its daemon, could you post SHA-attributed commands and fresh visible outcomes for Duo closed, half-open, and open in default and scoped sets, with verified app orientations, hinge readback, and lit captures? Please cover offset inner-window selector and coordinate taps through synthesis and XCTest fallback, hold, clipped landscape scroll, sampled gesture, and a repeated press that performs its first action then refuses after natural rotation across both chunk boundaries. On a sole-display iPhone, please cover normal and AX-unavailable windowless tap, hold, and sampled delivery with sole-screen markers. Include native display, point, and orientation logs, and retain the placed-iPad, clipped-scroll, and composed-harness regression evidence.

@thymikee
thymikee added this pull request to stack #4197 November 3, 2026 17:54
Base automatically changed from fix/foldable-springboard-surface to main November 3, 2026 17:55
@thymikee
thymikee force-pushed the refactor/surface-geometry branch from fea9dbd to 2a77509 Compare November 3, 2026 17:55
@thymikee

thymikee commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

Partial live evidence at 2a775092b4e2934f4713c51999870a73194be075; keep draft. This does not complete the requested matrix.

On local Duo 9542F4F5-51D1-4F5D-B3C9-02D78A69F0D9, the isolated checkout ran frozen install, pnpm build, full pnpm format, and AGENT_DEVICE_XCUITEST_DESTINATION='platform=iOS Simulator,id=9542F4F5-51D1-4F5D-B3C9-02D78A69F0D9' AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH="$PWD/.tmp/4189-live/derived" pnpm build:xcuitest:ios, all successful. The private daemon reported checkout code 0.21.37-dev; the device-open-bound client was configured to that daemon. Its runner log shows xcodebuild test-without-building using that private DerivedData xctestrun and Duo destination. Build cache fingerprint: 3101bd882cff045600941d7c02bb3008986594ffd706b12961f40fe3cd71b8f4. Tester used this checkout's Metro on port 8199. All device commands below used the device-open-bound CLI and the private daemon config.

Default set pose Hinge readback Observed result
fold closed 0° Fresh snapshot -i, press 'label="Catalog"', snapshot -i changed home-title to catalog-title. Lit capture 466×678. Native synthesized tap (128,626), reference 386×678, display 1, orientation 1.
fold half-open 130° Fresh Home snapshot, press 'role="button" label="Catalog"', fresh snapshot changed home-title to catalog-title. Lit capture 951×669. Native tap (617,590), reference 871×669, display 3, orientation 4.
fold open 180° Same fresh Home→Catalog selector transition and lit 951×669 capture. Native tap (617,590), reference 871×669, display 3, orientation 4. press 287 617 then returned to home-title; native point (617,664), display 3, orientation 4.

In open landscape, open com.callstack.agentdevicelab 'agent-device-test-app:///automation', scroll down --until 'id="automation-longpress"', and longpress 'id="automation-longpress"' 900 changed Long presses: 0 to Long presses: 1. Native long-press point (501,515), reference 871×669, display 3, orientation 4. The automation screen independently displayed Window: landscape.

A sampled gesture pan 350 400 100 0 700 returned success and logged an agent-device-gesture-plan record on display 3/orientation 4, but the subsequent fixture label still read pan changed no. This does not establish a hit or a product defect without further target/gesture diagnosis. A landscape fill 'id="field-name"' 'Geometry 2a775092' visibly populated the field and scroll down --pixels 200 dispatched with the keyboard visible, but returned movement: unobserved; delivery is not proved. Native drag was (435,475) to (235,475), reference 951×669, display 3/orientation 4.

Still unproved: the scoped-set pose matrix, explicit forced XCTest fallback, verified offset inner-window selector/coordinate routes across orientations, successful sampled gesture, natural rotation refusal at both chunk boundaries, and sole-display iPhone normal/AX-unavailable routes. Current help fold explicitly refuses scoped iOS sets with unsupported-device-scope; no scoped clone or sole-display iPhone was authorized for this run. No controlled failure injection was represented as natural. Earlier placed-iPad, clipped-scroll and composed-harness proof remains attributed to prior evidence, not this head.

Exact-head affected gate run-uVpghQ failed, exactCommit=true: vitest-related had 1,090 passes and one failure in test/integration/provider-scenarios/apple-platform-output-guard.test.ts, where fake-harness record exceeded 20 seconds. Functional device work and other host activity overlapped; contention is possible, not established. A same-command unchanged-base comparison is still needed before attribution. The prior fea9dbd pass does not validate this head.

Cleanup: reopened Preferences, left Duo open and landscape, closed this session, stopped its private daemon with --clean (no orphaned claims), stopped this checkout's Metro, and closed the panel with shutdown=false. DEVICE RELEASED, Duo remains booted. No code, labels, draft state, or other devices changed. Logs, command outputs, captures, and receipt are retained in this checkout under .tmp/4189-live and .tmp/validation/run-uVpghQ.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Continuation at 2a775092b4e2934f4713c51999870a73194be075 on the current-checkout daemon and the previously rebuilt production runner (source fingerprint 3101bd882cff045600941d7c02bb3008986594ffd706b12961f40fe3cd71b8f4). This supplements the earlier partial matrix; the requested matrix is still incomplete. No production source was changed.

On default Duo 9542F4F5-51D1-4F5D-B3C9-02D78A69F0D9, a fresh snapshot measured gesture-target-image at x=315.7..555.7, y=336.3..496.3. The Expo developer-menu tutorial was covering the fixture during initial pan/pinch attempts; after dismissing it and visually checking the fixture, gesture pinch 1.5 480 420 changed the app canary from pinch changed no to pinch changed yes. The native gesture-plan record used display 3 and orientation 4. This resolves the earlier uncalibrated gesture observation, but is not a proof of all requested orientations.

In landscape, after filling the form with the SHA marker, scroll down --pixels 200 reported keyboardAvoided: true, keyboard top 449, planned height 437, and movement: unobserved. On a second scroll, fresh snapshots independently showed the email field move from y=186.3 to y=32.3 with the keyboard still present. Native controlled-scroll dispatch used display 3/orientation 4. The response did not itself verify movement; the comparison is the evidence.

Two natural-rotation attempts did not prove refusal. press 'id="automation-press"' --count 25 --interval-ms 500 completed all 25 with sequence orientation 4 before a concurrent orientation landscape-right took effect. --count 3 --interval-ms 10000 completed all 3 with sequence orientation 2 before orientation landscape-left took effect. The same-session commands serialized; neither result is a rotation-during-sequence pass. Offset-window tap coverage across independently verified orientations and forced live XCTest fallback remain outstanding. The only force-synthesis seam located is in unit-test code; this run added no production fault flag and does not call intercepted synthesis a live fallback hit.

On sole-display iPhone 18 Pro BDA9B772-A0A9-46A4-AC91-C9107FAE0277, ordinary window-backed gesture pinch 1.5 201 545 changed the canary to pinch changed yes; a Catalog selector tap changed home-title to catalog-title; and longpress 'id="automation-longpress"' 900 changed its counter 0→1. Native records identified display 1/orientation 1. These are normal delivery results, not windowless or AX-unavailable proof: no sole-screen marker was recorded. The attempted agent-device-test-app:///heavy-feed deep link displayed an unmatched route; the documented Settings → Open heavy feed route was not found after scroll down --until 'id="open-heavy-feed"' returned typed scroll_until_edge_reached. No controlled AX failure was claimed.

A private scoped set was provisioned at .tmp/4189-matrix/continuation/scoped-set, Duo 68A62309-2026-4678-91E0-61325F3ECB3F. T3 device_open returned “No device 68A62309-2026-4678-91E0-61325F3ECB3F on 'This machine' (hostId local). Call device_list for current ids.” The advised device_list did not include the scoped ID, and one retry returned the same error. The scoped UI route was stopped; no typed scoped fold refusal or scoped capture was observed. The private simulator was shut down after recording its set path and ID.

All UI used each device_open-bound client with this checkout's private daemon config. The default Duo and 18 Pro sessions and panels were closed separately, both remain booted and DEVICE RELEASED. The private daemon was stopped cleanly and this run's Metro stopped. Command outputs, screenshots, runner logs, scoped ID and cleanup records are retained in .tmp/4189-matrix/continuation; the complete handoff is .tmp/4189-matrix/HANDOFF.md. The previously reported exact-head affected gate run-uVpghQ remains failed; its diagnosis is assigned separately, and no gate was rerun here. Draft, labels, and branch publication were unchanged.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Live evidence at 2a775092b4e2934f4713c51999870a73194be075; prior partial evidence and its original attribution: first matrix, continuation.

  • Default Duo, private UIKit v2 fixture: on the closed display, press 'id="canary-target"' --count 21 --interval-ms 600 delivered 20 hits, changed scene orientation 1→3, and refused the next chunk with completedSteps=20. --count 3 --interval-ms 10000 delivered one hit, changed 3→1, and refused the next chunk with completedSteps=1. Scene transitions completed before refusal; no concurrent CLI rotation was used. One earlier open-panel attempt delivered 21 and UIKit rejected the rotation request, so that attempt is inconclusive. On closed/half-open/open at 0°/130°/180°, landscape selector and coordinate taps, hold, and pan changed independent fixture counters; display IDs were 1/3/3. Separately, a controlled primary-HID-unavailable runner delivered selector and coordinate single taps via xctest-coordinate-tap in all three poses. This injected failure is not natural HID failure.
  • 18 Pro, normal exact-head runner, portrait: coordinate tap, hold, and pan changed canary counts from 0/0/0 to 1/1/1. With a separate private controlled missing-window resolver patch (shared resolver returns nil after lit-panel policy), the same actions changed counts from 1/1/1 to 2/2/2. Runner logs recorded AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE with (0,0,402,874) and display 1/orientation 1; tap and hold point (201,507), plus a sampled gesture record. Snapshots still used AX; this is neither natural AX failure nor global AX unavailability. The patch was reverted and is not a production change.
  • The requested Duo offset-window and inner-portrait rows remain unproved. A fresh device_open returned this thread's prior session ID; the current instruction forbade reusing it, so no Duo UI was driven in this phase. A new offset-window fixture was built but not run. Private scoped Duo remains blocked at T3 binding. Natural AX failure and the early windowQueryRejected route remain unproved. Existing prior iPad/composed and clipped-scroll evidence keeps its original SHA attribution. These gaps mean this is not a readiness claim.
  • After both devices were released and the private patch reversed, pnpm check:affected --run --head 2a775092b4e2934f4713c51999870a73194be075 passed all runnable checks with exactCommit=true; local receipt .tmp/validation/run-XAlS9a/receipt.json. GitHub-authoritative/parked checks were skipped, not passed locally. The related selection that timed out in the earlier local run passed here; this does not establish the earlier timeout's cause.

Raw command/exit/timestamp logs, captures, private patch/source/binary hashes, runner identities and cleanup records are retained in .tmp/4189-v2/HANDOFF.md and .tmp/4189-final/HANDOFF.md in the isolated worktree. Both devices were left booted and released; the controlled experiment used separate derived data and daemons.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Follow-up to the prior SHA-attributed matrix, still at 2a775092b4e2934f4713c51999870a73194be075: with an unchanged runner and a private UIKit fixture that creates an actually inset UIWindow, default Duo selector and fresh-coordinate taps delivered real canary hits in all three poses.

  • Open 180° and half-open 130°: scene orientation 4, lit native display 3 (669×951); UIKit screen-space window and XCTest-resolved window/reference both (45,60,861,549) in the rotated 951×669 screen. Open selector/coordinate advanced hits 0→1→2; half-open advanced 2→3→4. Native open points (377,495) and (384,495), reference (45,60,861,549). Lit captures show the inset window and unobstructed target.
  • Closed 0°: after relaunching the fixture on the smaller panel, UIKit and XCTest-resolved window/reference were (45,60,376,558) on display 1, orientation 1, 466×678; selector and fresh coordinate (214,395) advanced hits 0→1→2. The initial post-fold window was clipped, so it was not credited. The fixture uses real UIKit geometry; no runner or orientation result was overridden. Snapshots used XCTest because the window did not fill the screen.
  • Inner portrait remains unproved: the CLI orientation request returned success while the actual scene remained landscape. An app-driven portrait geometry request then returned UIKit's The current windowing mode does not allow for programmatic changes to interface orientation. The observed restriction is specific to that mode, not a claim about all modes. Private scoped binding and natural AX failure remain unproved. The prior controlled resolver result is not a natural AX failure.

Raw commands, JSON, captures, window/panel logs, fixture hashes and cleanup are retained in .tmp/4189-offset/HANDOFF.md in the isolated worktree. Duo was restored to open/Preferences and released booted. The runner source was unchanged; the earlier exact-head check:affected receipt run-XAlS9a remains applicable and was not rerun for this private fixture experiment. This is not a readiness claim.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Combined inset-window + XCTest fallback evidence at 2a775092b4e2934f4713c51999870a73194be075, supplementing the earlier inset measurements. This uses the real private UIKit inset-window fixture and a separately built controlled primary-HID-unavailable runner: its single-tap synthesis entry returns the existing failed status before contact; fallback policy and actual XCTest delivery are unchanged. It is not a natural HID failure or an unmodified production runner.

  • Open 180° and half-open 130°: actual UIKit and XCTest-resolved window/reference (45,60,861,549) in the 951×669 landscape screen; display 3, orientation 4. Single selector and fresh coordinate press 456 384 each returned xctest-coordinate-tap and advanced the canary 0→1→2 open, 2→3→4 half-open. XCTest logged actual window taps; planned pre-contact native points were (377,495)/(384,495) open and (384,495) twice half-open.
  • Closed 0°: after fixture relaunch on the smaller screen, actual UIKit and XCTest-resolved window/reference (45,60,376,558) on 466×678 display 1, orientation 1. Selector then fresh press 214 395 returned the same fallback and advanced canary 0→1→2; planned pre-contact native points (214,388) and (214,395). Captures show the inset and visible target. The runner logs retain the injected marker, allowed fallback policy, and XCTest Tap Window/event path; the canary changes independently confirm delivery.

Inner portrait remains unproved after the prior mode-specific UIKit refusal; scoped T3 binding remains blocked and scoped SSH authorization pending; natural AX failure and early rejected-tree route remain unproved. The previous exact-head gate receipt run-XAlS9a passed and was not rerun for this private experiment. Raw commands, responses, captures, patch/build fingerprints and cleanup are in .tmp/4189-combined/HANDOFF.md in the isolated worktree. The temporary patch was reversed; Duo was restored to open/Preferences and released booted. This is not a readiness claim.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Scoped Duo live follow-up at 2a775092b4e2934f4713c51999870a73194be075, supplementing prior default-set evidence. With explicit authorization, a temporary task-owned loopback-only SSH server was used to run this checkout's CLI against the existing private scoped set and UUID 68A62309-2026-4678-91E0-61325F3ECB3F; its own daemon and runner were bound to that exact set and device. No T3 binding retry or default-device substitution occurred.

Current help fold says scoped folding is supported, and the actual target confirmed it: closed 0°, half-open 130°, open 180°. On the real UIKit inset fixture, selector and fresh coordinate taps advanced canary hits 0→1→2 closed, 0→1→2 half-open, and 2→3→4 open. Closed window/reference (45,60,376,558) on display 1/orientation 1; half/open (45,60,861,549) on display 3/orientation 4. Lit captures and runner logs retained; the fixture was relaunched after folding to the inner panel. No scoped forced-fallback scenario was run.

Scoped orientation portrait returned UNSUPPORTED_OPERATION / unsupported-device-scope, stating CoreDevice cannot verify rotation for a foldable simulator in a scoped set. Its response says dispatch unknown; the following snapshot still showed orientation 4 and unchanged hits. This is an orientation refusal, not a fold refusal or a completed portrait row. Natural AX failure and the early rejected-tree route remain unproved; previous controlled fallback is not natural failure. The earlier exact-head gate passed (run-XAlS9a) and was not rerun.

Raw SSH-scoped commands, JSON, captures, runner identity and cleanup are retained in .tmp/4189-loopback/HANDOFF.md in the isolated worktree. The owned session and daemon were closed, the exact scoped UUID was returned to Shutdown, and the temporary loopback server and keys were removed. No readiness claim is made.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Scoped inset-window + controlled XCTest fallback at 2a775092b4e2934f4713c51999870a73194be075, supplementing scoped normal delivery: using the actual scoped Duo and real UIKit inset fixture, a separate private runner injected primary-HID unavailability before contact; fallback policy and actual XCTest delivery were unchanged. This is not a natural HID failure or the unmodified production runner.

  • Closed 0°: UIKit/XCTest/reference (45,60,376,558), display 1/orientation 1. Selector and fresh press 214 395 each reported xctest-coordinate-tap and advanced hits 0→1→2.
  • Half-open 130°: (45,60,861,549), display 3/orientation 4. Selector and fresh press 456 384 each reported fallback and advanced 0→1→2.
  • Open 180°: same inner reference/display/orientation; selector and fresh press 456 384 advanced 2→3→4 with the same fallback. Lit captures, injected markers, policy decisions, actual XCTest Tap Window paths, native planned points and fresh canary snapshots are retained. Planned primary points alone are not contact evidence.

The scoped portrait command remains refused UNSUPPORTED_OPERATION / unsupported-device-scope; default inner portrait had a mode-specific UIKit refusal. Natural AX failure and early windowQueryRejected remain unproved; controlled resolver-nil coverage does not establish either. The exact-head gate run-XAlS9a passed previously and was not rerun. No waiver or readiness claim is implied. Raw commands, captures, provenance and cleanup are in .tmp/4189-scoped-fallback/HANDOFF.md; the scoped device was returned to Shutdown, task-owned loopback SSH and keys removed, and private patch reversed.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Readiness assessment requested for 2a775092b4e2934f4713c51999870a73194be075: please decide whether the remaining limitations below satisfy the requested validation or need further supported evidence.

  • Default Duo: rotation boundaries, delivery and sole-display evidence, real inset-window delivery, and inset-window plus controlled XCTest fallback.
  • Scoped Duo: normal inset-window delivery and inset-window plus controlled XCTest fallback cover closed, half-open, and open. The primary-HID failure was injected before contact; XCTest delivered the recorded hits.
  • Remaining limits: default inner portrait was not observed because UIKit refused the geometry request in that windowing mode. Scoped portrait returned UNSUPPORTED_OPERATION / unsupported-device-scope; the follow-up showed no additional canary hit and unchanged scene orientation, while the response reported dispatch unknown. The controlled missing-window resolver covers downstream sole-screen delivery only: it does not establish natural AX failure, global AX unavailability, or the early windowQueryRejected route.
  • Local pnpm check:affected --run --head 2a775092b4e2934f4713c51999870a73194be075 passed with exactCommit=true (receipt run-XAlS9a). GitHub-authoritative and parked lanes were skipped locally, not passed by that receipt.

Please assess whether these explicit limitations are acceptable or require further evidence. No waiver, readiness designation, draft change, or merge is assumed.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

At 2a77509, the code looks good: no defect or design issue. On the readiness question, two of the three limitations are fine and one still needs a short run.

Accepted: the default inner portrait that UIKit refuses in that windowing mode, and the scoped portrait that returns a typed unsupported-device-scope refusal. Neither needs another unsupported attempt. The Duo matrix (default and scoped, closed, half-open and open), the rotation boundaries, the inset-window delivery, the controlled XCTest fallback and the sole-display iPhone runs cover the rest of what I asked for.

Could you validate the early rejected-query route before merge? The controlled resolver returns nil after surface policy while AX still answers, so it does not establish that navigation reaches windowQueryRejected=true or avoids another windows query before native delivery. On one sole-display iPhone, use the exact-head runner and a documented minimal fault patch to reject the AX query through daemon back; show a sole-screen native tap and an independent visible navigation outcome without another windows query. In that same unavailable state, run coordinate tap, hold, and sampled gesture with fresh canary outcomes, sole-screen markers, and display, point, and orientation logs. Retain source, patch, and build provenance and restore the patch afterward. A natural reproducible AX failure covering these routes would also suffice. Resolver

The controlled resolver and the injected primary failure show downstream delivery and the XCTest fallback, not a global AX failure. All 21 checks pass, the exact-head check:affected run passed, and no conflicts are known.

@thymikee
thymikee force-pushed the refactor/surface-geometry branch from 2a77509 to 2b9940a Compare November 4, 2026 14:40
@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

The remaining gap from this request is closed on 2b9940af2a9e3e91c44e14535c026f96a2c2ac41, with limits recorded.

Natural route proved. On a sole-display iPhone (iPhone 17 Pro, iOS 26.2), the heavy Tester feed exceeds the depth the XCTest serializable AX bridge can encode, so the public applicationDefinedResizableWindows query raises kAXErrorIllegalArgument while private AX still answers — that is why snapshots keep working. No production code was patched to get there. back --in-app then takes the early route: windowQueryRejected=true is logged before any hit-test, the sole-screen marker is the only screen log, synthesis logs point=(32.2,84.0) referenceFrame=(0,0,402,874) interfaceOrientation=1 displayId=1, and the outcome is independent of the logs — Liked posts: 0 → 1 plus VISUAL_VERIFICATION changed=yes. Coordinate tap, hold and a sampled gesture pan deliver on the same window with fresh canaries (Last action: unliked post 0, heavy-post-2-like y 1001.3 → 798.3). Command counts, log windows, the daemon signature, the repacked-bundle hashes and cleanup are in docs/evidence/surface-geometry-4189-2026-11-03.md.

Confirmed limits. The early route is proved only for that one simulator, device shape and feed shape. No foldable, no physical device, no scoped set, and this is not global AX unavailability.

Default inner portrait and scoped device. I treated your statement that neither is a release blocker as acceptance and did not re-run either. The recorded behaviour stands: UIKit kept landscape after a mode-specific setFacePortait-style refusal, and the scoped set returned UNSUPPORTED_OPERATION / unsupported-device-scope with no additional canary hit on the follow-up. Say the word if you want either re-verified at this head.

Two things to note about the branch. The branch was rebased onto current main (7d42395ff7); the only adaptation was main's #4187 renaming RunnerAlert.buttons to actions, and both touched alert tests were rebuilt and run green on that head. And I have not run the readiness assessment you asked for, so I am not calling this release-ready — this comment reports the evidence and its limits.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Exact-head test evidence for 2b9940af2a9e3e91c44e14535c026f96a2c2ac41 (base 7d42395):

  • The reported run-oXhSEW receipt was unavailable locally, so one fresh pnpm check:affected --run --head 2b9940af2a9e3e91c44e14535c026f96a2c2ac41 ran after frozen install/build. run-12Ix6M passed with exactCommit=true and checkoutChanged=false. GitHub-owned and parked lanes were skipped locally; provider integration was deduplicated into related tests.
  • Default Duo, open hinge 180°, observed scene orientation 4: selector and fresh coordinate taps moved a UIKit canary from 0 to 2 hits; 900 ms hold and sampled pan each incremented their own canary. Native records used display 3 and orientation 4. Its windowed snapshot fell back to XCTest; on the separate 18 Pro, snapshots reported simulator-ax-bridge.
  • On iPhone 18 Pro iOS 27.0, current-source repacked Heavy feed naturally produced kAXErrorIllegalArgument. Coordinate tap changed liked count 0→1, hold changed it 1→0, and sampled pan moved a post 203 points. Sole-screen records used display 1, orientation 1. back --in-app returned to Settings; visual verification reported changed=yes and a fresh bridge snapshot showed the Settings control.
  • For that back command, the runner trace shows the Button query rejection and suppression count 4→5, followed by sole-screen resolution and native tap (32.2,84) before completion. The same XCTest trace positively logs successful Window enumerations on earlier commands and rejected Window enumerations on the gesture commands; it records no Window enumeration in this back segment. Together with the exact built source's changed-counter predicate and short-circuit, this supports no second XCTest windows query in this back coordinate context. UIKit's windows=1 inset log is a separate scene-window read.

The private build used runner source fingerprint de3d47a2…, test executable SHA-256 b7eab840…, and a checkout-origin authenticated daemon; both session xctestrun paths and raw logs were retained. 84f's original raw artifacts remain unavailable, so its independent no-second-query assertion is unverified. The full prior matrix remains attributed to 2a77509; this is a focused 2b run, not a matrix replay or a readiness/waiver decision.

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

At 2b9940a, the code looks good, and the recorded rejected-query run settles my last request: on a sole-display iPhone, navigation reaches windowQueryRejected and delivers natively without another windows query, and coordinate tap, hold and sampled gesture land in that state. All 21 checks pass, and no conflicts are known.

Not blocking, take or leave: soleScreenReferenceFrame(portraitSize:interfaceOrientation:) in RunnerTests+SynthesizedInteraction.swift is now a one-line pass-through to SurfaceGeometry.panelFrame(nativeSize:orientation:). Calling panelFrame directly at line 474 and in the two unit tests would let you delete the wrapper.

The PR is still a draft, so it waits for you to mark it ready.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 4, 2026
@thymikee
thymikee marked this pull request as ready for review November 4, 2026 16:37
@thymikee
thymikee merged commit 6643864 into main Nov 4, 2026
20 of 21 checks passed
@thymikee
thymikee deleted the refactor/surface-geometry branch November 4, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant