tree 42bbfaeeb6f1723c1b944d78c914ff88e4dd1562
parent 348072c93430e6f7fb171507e636963f9a4b1e84
author drousso@apple.com <drousso@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc> 1557909422 +0000
committer drousso@apple.com <drousso@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc> 1557909422 +0000

Web Automation: elements larger than the viewport have incorrect in-view center point
https://bugs.webkit.org/show_bug.cgi?id=195696
<rdar://problem/48737122>

Reviewed by Simon Fraser.

Original patch by Brian Burg <bburg@apple.com>.

Source/WebCore:

Some conversion methods do not exist for `FloatRect`/`FloatPoint`. Fill them in as needed,
and export some symbols used by WebDriver code to compute an element's in-view center point
in various coordinate systems.

* dom/TreeScope.h:
* dom/TreeScope.cpp:
(WebCore::TreeScope::elementsFromPoint): Added.
* page/FrameView.h:
* page/FrameView.cpp:
(WebCore::FrameView::absoluteToLayoutViewportPoint const): Added.
(WebCore::FrameView::layoutViewportToAbsoluteRect const): Added.
(WebCore::FrameView::absoluteToLayoutViewportRect const): Added.
* platform/ScrollView.h:
* platform/ScrollView.cpp:
(WebCore::ScrollView::viewToContents const): Added.
(WebCore::ScrollView::contentsToView const): Added.
(WebCore::ScrollView::contentsToRootView const): Added.
* platform/Widget.h:
* platform/Widget.cpp:
(WebCore::Widget::convertToRootView const): Added.
(WebCore::Widget::convertFromRootView const): Added.
(WebCore::Widget::convertToContainingView const): Added.
(WebCore::Widget::convertFromContainingView const): Added.

Source/WebKit:

This seems to be an omission in the specification. While it does mention that the in-view
center point (IVCP) must be within the viewport, the algorithm never intersects the element
bounding box with the viewport rect.

* WebProcess/Automation/WebAutomationSessionProxy.cpp:
(WebKit::WebAutomationSessionProxy::computeElementLayout):
This code is incorrect. For `CoordinateSystem::LayoutViewport`, coordinates should be in
root view coordinates so that it can be later converted to screen and synthesized as a HID
event in screen coordinates. Intersect the element rect and the viewport rect before finding
the center point of the part of the element that's visible in the viewport.

(WebKit::convertRectFromFrameClientToRootView): Added.
(WebKit::convertPointFromFrameClientToRootView): Added.
Added helpers to properly account for scroll contents position on iOS.

* UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::viewportInViewCenterPointOfElement):
Now that we determine whether the element is inside the viewport much earlier, if the
element has no `inViewCenterPoint`, we can return a `TargetOutOfBounds` instead of a more
"generic" `ElementNotInteractable`.

(WebKit::WebAutomationSession::simulateMouseInteraction):
Rename `locationInView` -> `locationInViewport`.

(WebKit::WebAutomationSession::simulateTouchInteraction):
This code is incorrect. The `unobscuredContentRect` is in screen coordinates, but
we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0).
Grab the visual viewport rect and see if the location exceeds the viewport size.

* UIProcess/Automation/ios/WebAutomationSessionIOS.mm:
(WebKit::operator<<):
Add logging helper for `TouchInteraction` enum.

(WebKit::WebAutomationSession::platformSimulateTouchInteraction):
Move local variable.

* UIProcess/Automation/SimulatedInputDispatcher.cpp:
(WebKit::SimulatedInputDispatcher::transitionInputSourceToState):
Fix a typo in logging.

* UIProcess/Automation/Automation.json:
Simplify enum name.

* Platform/Logging.h:
Add logging channel to dump fully resolved interaction details.


git-svn-id: http://svn.webkit.org/repository/webkit/trunk@245320 268f45cc-cd09-0410-ab3c-d52691b4dbfc