Repository navigation
fix(mobilerun): report keyboard focus, not focus-engine focus, in /mobilerun/state (DRO-3220) - #44
Conversation
…bilerun/state /mobilerun/state set every node's isFocused, and picked phone_state.focusedElement/isEditable, from the snapshot's hasFocus. That is focus-engine focus: WebKit sets it on every <input> of a page, so all web fields read as focused with nothing focused, and UIKit text fields never set it on a touch-driven iPhone, so a Settings sheet with the keyboard up had no focusedElement and isEditable false (DRO-3220). Pick the one focused node up front from the same snapshot: the deepest node with hasKeyboardFocus, else the deepest with the IsEditing trait (1 << 21). Only that node reports isFocused, and phone_state describes it. isEditable also accepts the TextEntry/IsEditing traits besides the four text input types. tvOS keeps hasFocus as its first signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hardware validation status (head
|
Hardware QA: passed (head
|
| Case (DRO-3220 Q05/Q06) | Focused nodes | phone_state |
Result |
|---|---|---|---|
| Safari, httpbin.org/forms/post loaded, nothing focused | 0 (all 4 web inputs isFocused:false) |
no focusedElement, isEditable:false, keyboard hidden |
✅ (before: every web input focused) |
| Safari, "Customer name" tapped | 1: that TextField | focusedElement TextField, isEditable:true, keyboard visible |
✅ |
… then keyboard/input "wda44" (clear) |
1: same field | focusedElement.text = "wda44" |
✅ input lands in the reported field |
| Safari address bar tapped | 1: Address TextField | isEditable:true, keyboard visible |
✅ |
| Settings → Wi-Fi → Other… sheet, Name active | 1: Name TextField | focusedElement TextField "Network Name", isEditable:true, keyboard visible |
✅ (before: no focusedElement, isEditable:false) |
| same sheet, Password tapped | 1: Password SecureTextField | focusedElement SecureTextField, isEditable:true |
✅ focus follows |
Settings join-network sheet, Password + keyboard/input "abc" |
1: Password | focusedElement.text = "•••" |
✅ |
| Settings lists (Display & Brightness, Wi-Fi) | 0 | no focusedElement |
✅ |
Notes, not caused by this PR:
- Secure fields in screenshots: with a SecureTextField focused, the broadcast screenshot shows neither the keyboard nor the typed bullets. iOS leaves the keyboard and secure text out of screen captures; the tree had the right state.
- Launching Safari: a deep link to Safari failed (
CoreDeviceError 10002) and a dock tap didn't open it on this phone. The portal'sPOST /action/applaunched it. - Phone left with this PR's runner installed.
Earlier blocker: this phone was in StandBy. It's fixed now (someone unlocked it and set Auto-Lock to Never).
🤖 Generated with Claude Code
Resolve the conflict with #43 (bounded state snapshot): the focused node is picked from the snapshot master now passes into fb_mobilerunA11yStateWithSnapshot:scale:truncated:, and the tree keeps master's SpringBoard-sized fallback root when no snapshot could be taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes the focus fields of
GET /mobilerun/state(Linear DRO-3220).Root cause
fb_a11yNodeForSnapshot:(WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m, master lines 482 and 492-499) set every node'sisFocusedfromsnapshot.hasFocus, and filledphone_state.focusedElement/isEditablefrom the first node in pre-order withhasFocus.hasFocusis focus-engine focus, not keyboard focus:<input>of a page, focused or not. Every web input reportedisFocused: true, andfocusedElementwas whichever input came first.hasFocus, even while they are the first responder with the keyboard up. No node matched, sofocusedElementwas missing andisEditablestayed false.Fix
+fb_mobilerunFocusedSnapshotInTree:picks the one focused node from the snapshot already taken. It picks the deepest node withhasKeyboardFocus(the first responder). If there is none, it picks the deepest node with the private IsEditing trait (1 << 21), the same fallback the native RemoteXPC tree in mobilerun-ios uses. On iOS,hasFocusis no longer read. tvOS still checks it first.isFocused: true.phone_state.focusedElementdescribes the same node, so the tree and phone_state cannot disagree.isEditableis true for the four XCTest text input types (TextField, SearchField, SecureTextField, TextView), as before. It is now also true for a node carrying the TextEntry (1 << 18) or IsEditing trait.Tests
The new
FBMobilerunFocusTests(UnitTests) runs the real tree walk over snapshot doubles. It covers web inputs withhasFocusand no keyboard focus, one keyboard-focused web input, a keyboard-focused field in a separate sheet window, the deepest node beating an ancestor, the IsEditing fallback, and keyboard focus beating IsEditing.xcodebuild test -scheme WebDriverAgentLib -only-testing:UnitTests/FBMobilerunFocusTests -only-testing:UnitTests/FBMobilerunA11yCommandsTestson an iPhone 17 Pro simulator: 11/11 passed.xcodebuild build-for-testing -scheme WebDriverAgentRunnerTunnel -destination 'generic/platform=iOS', signed (team 5KTKV6FRNZ): TEST BUILD SUCCEEDED.Hardware validation: pending
This has not been run on a device yet. Still to check on the iPhone 15 Pro (iOS 26.5, WDA mode):
isFocused, and there is nofocusedElement. After tapping one input, only that input is focused.focusedElementis the active field andisEditableis true.Assumption to confirm: XCTest fills
hasKeyboardFocusand/or the IsEditing trait in the standard snapshot for the Settings sheet field. WDA's own typing path already relies onhasKeyboardFocusin standard snapshots. If the sheet's content were missing from the app snapshot entirely (a remote view outside the snapshot), this fix would not be enough. The hardware check will show which case applies.🤖 Generated with Claude Code