Skip to content

fix(mobilerun): report keyboard focus, not focus-engine focus, in /mobilerun/state (DRO-3220) - #44

Merged
Timo972 merged 2 commits into
masterfrom
timo/dro-3220-wda-focus-state
Oct 23, 2026
Merged

Timo972 merged 2 commits into
masterfrom
timo/dro-3220-wda-focus-state

Conversation

@Timo972

@Timo972 Timo972 commented Oct 22, 2026

Copy link
Copy Markdown

Fixes the focus fields of GET /mobilerun/state (Linear DRO-3220).

Root cause

fb_a11yNodeForSnapshot: (WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m, master lines 482 and 492-499) set every node's isFocused from snapshot.hasFocus, and filled phone_state.focusedElement / isEditable from the first node in pre-order with hasFocus. hasFocus is focus-engine focus, not keyboard focus:

  1. Web fields all focused. WebKit sets it on every <input> of a page, focused or not. Every web input reported isFocused: true, and focusedElement was whichever input came first.
  2. No focusedElement in Settings → Wi-Fi → Other…. On a touch-driven iPhone, UIKit text fields never set hasFocus, even while they are the first responder with the keyboard up. No node matched, so focusedElement was missing and isEditable stayed false.

Fix

  • +fb_mobilerunFocusedSnapshotInTree: picks the one focused node from the snapshot already taken. It picks the deepest node with hasKeyboardFocus (the first responder). If there is none, it picks the deepest node with the private IsEditing trait (1 << 21), the same fallback the native RemoteXPC tree in mobilerun-ios uses. On iOS, hasFocus is no longer read. tvOS still checks it first.
  • Only that node reports isFocused: true. phone_state.focusedElement describes the same node, so the tree and phone_state cannot disagree.
  • isEditable is true for the four XCTest text input types (TextField, SearchField, SecureTextField, TextView), as before. It is now also true for a node carrying the TextEntry (1 << 18) or IsEditing trait.
  • The fix needs no extra snapshot or AX round trip, only one more walk over the snapshot that is already in memory. The wire shape is unchanged.

Tests

The new FBMobilerunFocusTests (UnitTests) runs the real tree walk over snapshot doubles. It covers web inputs with hasFocus and no keyboard focus, one keyboard-focused web input, a keyboard-focused field in a separate sheet window, the deepest node beating an ancestor, the IsEditing fallback, and keyboard focus beating IsEditing.

  • xcodebuild test -scheme WebDriverAgentLib -only-testing:UnitTests/FBMobilerunFocusTests -only-testing:UnitTests/FBMobilerunA11yCommandsTests on an iPhone 17 Pro simulator: 11/11 passed.
  • xcodebuild build-for-testing -scheme WebDriverAgentRunnerTunnel -destination 'generic/platform=iOS', signed (team 5KTKV6FRNZ): TEST BUILD SUCCEEDED.

Hardware validation: pending

This has not been run on a device yet. Still to check on the iPhone 15 Pro (iOS 26.5, WDA mode):

  • A Safari page with several inputs and nothing focused: no node has isFocused, and there is no focusedElement. After tapping one input, only that input is focused.
  • Settings → Wi-Fi → Other… with the keyboard up: focusedElement is the active field and isEditable is true.

Assumption to confirm: XCTest fills hasKeyboardFocus and/or the IsEditing trait in the standard snapshot for the Settings sheet field. WDA's own typing path already relies on hasKeyboardFocus in standard snapshots. If the sheet's content were missing from the app snapshot entirely (a remote view outside the snapshot), this fix would not be enough. The hardware check will show which case applies.

🤖 Generated with Claude Code

…bilerun/state

/mobilerun/state set every node's isFocused, and picked
phone_state.focusedElement/isEditable, from the snapshot's hasFocus.
That is focus-engine focus: WebKit sets it on every <input> of a page,
so all web fields read as focused with nothing focused, and UIKit text
fields never set it on a touch-driven iPhone, so a Settings sheet with
the keyboard up had no focusedElement and isEditable false (DRO-3220).

Pick the one focused node up front from the same snapshot: the deepest
node with hasKeyboardFocus, else the deepest with the IsEditing trait
(1 << 21). Only that node reports isFocused, and phone_state describes
it. isEditable also accepts the TextEntry/IsEditing traits besides the
four text input types. tvOS keeps hasFocus as its first signal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Timo972

Timo972 commented Oct 22, 2026

Copy link
Copy Markdown
Author

Hardware validation status (head 993948ff): blocked, not yet run

Built unsigned the same way as Scripts/ci/build-real.sh (Tunnel scheme). Re-signed on the macmini with mobilerun-ios setup --wda --reinstall --backend asc (team 5KTKV6FRNZ) and installed on two supervised iOS 26.5.x iPhone 15 Pros that are registered in the team. Neither phone could be driven:

  • 00008130-0009555C0E2A001C (iOS 26.5): the runner launched and /mobilerun/state answered. The phone was stuck in StandBy, though (locked, on the charger, physically landscape). Home, /wda/unlock ("Timed out while waiting until the screen gets unlocked"), swipe-ups and the orientation API ("Unable To Rotate Device") all failed to leave it, so Safari and Settings were unreachable.
  • 00008130-0004143621D3803A (iOS 26.5.2): also in StandBy, and it showed "Unable to Verify App" for the "Created via API (XBG52QLFDF)" developer. The phone has no internet, so iOS won't validate the profile and the runner launch is refused.

Both need someone at the rig: put the phone in portrait or off the charger, and connect it to Wi-Fi with internet. The fixtures are ready: https://httpbin.org/forms/post with nothing focused and then one field tapped, and Settings → Wi-Fi → Other… with the keyboard up.

For comparison, native mode on iOS 27.0.1 (no WDA) gets both focused cases right: one focused input, correct focusedElement/isEditable, also in the Other… sheet. With nothing focused, it marks every web node as isFocused (27 nodes on the form). That half is a mobilerun-ios bug and will be fixed there.

@Timo972

Timo972 commented Oct 22, 2026

Copy link
Copy Markdown
Author

Hardware QA: passed (head 993948ff)

Device: iPhone 15 Pro 00008130-0009555C0E2A001C, iOS 26.5, macmini lab, 2026-10-22. Unlocked, Auto-Lock off.

Setup:

  • Runner built unsigned like Scripts/ci/build-real.sh (Tunnel scheme), then signed and installed with mobilerun-ios setup --wda --reinstall --backend asc (team 5KTKV6FRNZ).
  • Portal: mobilerun-ios --local, WDA mode. phone_state and the tree come from this runner's /mobilerun/state.
Case (DRO-3220 Q05/Q06) Focused nodes phone_state Result
Safari, httpbin.org/forms/post loaded, nothing focused 0 (all 4 web inputs isFocused:false) no focusedElement, isEditable:false, keyboard hidden ✅ (before: every web input focused)
Safari, "Customer name" tapped 1: that TextField focusedElement TextField, isEditable:true, keyboard visible ✅
… then keyboard/input "wda44" (clear) 1: same field focusedElement.text = "wda44" ✅ input lands in the reported field
Safari address bar tapped 1: Address TextField isEditable:true, keyboard visible ✅
Settings → Wi-Fi → Other… sheet, Name active 1: Name TextField focusedElement TextField "Network Name", isEditable:true, keyboard visible ✅ (before: no focusedElement, isEditable:false)
same sheet, Password tapped 1: Password SecureTextField focusedElement SecureTextField, isEditable:true ✅ focus follows
Settings join-network sheet, Password + keyboard/input "abc" 1: Password focusedElement.text = "•••" ✅
Settings lists (Display & Brightness, Wi-Fi) 0 no focusedElement ✅

Notes, not caused by this PR:

  • Secure fields in screenshots: with a SecureTextField focused, the broadcast screenshot shows neither the keyboard nor the typed bullets. iOS leaves the keyboard and secure text out of screen captures; the tree had the right state.
  • Launching Safari: a deep link to Safari failed (CoreDeviceError 10002) and a dock tap didn't open it on this phone. The portal's POST /action/app launched it.
  • Phone left with this PR's runner installed.

Earlier blocker: this phone was in StandBy. It's fixed now (someone unlocked it and set Auto-Lock to Never).

🤖 Generated with Claude Code

@Timo972
Timo972 marked this pull request as ready for review October 22, 2026 22:03
niko-droidrun
niko-droidrun previously approved these changes Oct 23, 2026
Resolve the conflict with #43 (bounded state snapshot): the focused node is
picked from the snapshot master now passes into
fb_mobilerunA11yStateWithSnapshot:scale:truncated:, and the tree keeps
master's SpringBoard-sized fallback root when no snapshot could be taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Timo972
Timo972 merged commit 8ce9d6f into master Oct 23, 2026
37 of 50 checks passed
@Timo972
Timo972 deleted the timo/dro-3220-wda-focus-state branch October 23, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants