Data detectors sometimes show up in the wrong place when resizing images with Live Text
https://bugs.webkit.org/show_bug.cgi?id=235598
rdar://88032375
Reviewed by Dean Jackson.
Source/WebCore:
On macOS, Live Text data detectors sometimes show up in the wrong place when images are resized; this can happen
in the case where image analysis injects data detection results into the image, but then the image is resized
from underneath the user's mouse cursor. To fix this, add some logic to invalidate ImageOverlayController's
cached data detector highlight information in the case where the image overlay layout has been changed.
Test: fast/images/text-recognition/mac/image-overlay-data-detectors.html
* WebCore.xcodeproj/project.pbxproj:
Have libWebCoreTestSupport additionally link against libPAL, so that we can use PAL's DataDetectorsCore soft-
linking utilities.
* dom/ImageOverlay.cpp:
(WebCore::ImageOverlay::updateSubtree):
Drive-by fix in adjacent code: when installing image overlays in media elements, make sure that we install them
inside the media controls root container by calling `ensureUserAgentShadowRoot()` before inserting the overlay
content; this ensures that we don't end up with a redundant image overlay in the shadow root. Tests for this
will be added in #235623.
(WebCore::ImageOverlay::updateWithTextRecognitionResult):
* page/ImageOverlayController.cpp:
(WebCore::ImageOverlayController::textRecognitionResultsChanged):
(WebCore::ImageOverlayController::hasActiveDataDetectorHighlightForTesting const):
* page/ImageOverlayController.h:
* page/mac/ImageOverlayControllerMac.mm:
(WebCore::ImageOverlayController::textRecognitionResultsChanged):
Add a hook to inform ImageOverlayController when image overlay content changes. If the image overlay host
matches the currently active host element showing data detector highlights, then invalidate the highlights;
these highlights will be recomputed once the user hovers over the data detector elements again.
(WebCore::ImageOverlayController::hasActiveDataDetectorHighlightForTesting const):
Add a testing-only helper method to query whether or not there is an active data detector highlight. See below
for more details.
* testing/Internals.cpp:
Add some more WebCore testing support to make it possible to test data detector highlights in Live Text on
macOS.
(WebCore::makeDataForLine):
(WebCore::Internals::installImageOverlay):
Add an optional argument to provide a list of data detector quads to inject into the overlay host. For now, each
data detector element simply corresponds to a dummy `DDScannerResult` returned by the static
`fakeDataDetectorResultForTesting()` helper below.
(WebCore::Internals::hasActiveDataDetectorHighlight const):
Add an internal testing hook to query whether or not ImageOverlayController is tracking an active data detector
highlight.
* testing/Internals.h:
* testing/Internals.idl:
* testing/Internals.mm:
(WebCore::Internals::fakeDataDetectorResultForTesting):
Source/WebCore/PAL:
Move some soft-linked DataDetectorsCore API out of the iOS-specific define, so that we can call them on macOS.
See WebCore/ChangeLog for more details.
* pal/cocoa/DataDetectorsCoreSoftLink.h:
* pal/cocoa/DataDetectorsCoreSoftLink.mm:
LayoutTests:
Add a layout test to exercise (some) of the changes. This new layout test consists of 4 steps:
1. Hover over a data detector in Live Text and confirm that a data detector highlight is activated.
2. Resize the image (via script) such that the cursor is no longer over a data detector; confirm that the data
detector highlight is cleared.
3. Move over the data detector highlight in the resized image, and confirm that the highlight is once again
activated.
4. Move out of the image altogether and confirm that the highlight is deactivated.
* fast/images/text-recognition/mac/image-overlay-data-detectors-expected.txt: Added.
* fast/images/text-recognition/mac/image-overlay-data-detectors.html: Added.
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@288621 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/LayoutTests/ChangeLog b/LayoutTests/ChangeLog
index 628911b..1242b30 100644
--- a/LayoutTests/ChangeLog
+++ b/LayoutTests/ChangeLog
@@ -1,3 +1,22 @@
+2022-01-26 Wenson Hsieh <wenson_hsieh@apple.com>
+
+ Data detectors sometimes show up in the wrong place when resizing images with Live Text
+ https://bugs.webkit.org/show_bug.cgi?id=235598
+ rdar://88032375
+
+ Reviewed by Dean Jackson.
+
+ Add a layout test to exercise (some) of the changes. This new layout test consists of 4 steps:
+ 1. Hover over a data detector in Live Text and confirm that a data detector highlight is activated.
+ 2. Resize the image (via script) such that the cursor is no longer over a data detector; confirm that the data
+ detector highlight is cleared.
+ 3. Move over the data detector highlight in the resized image, and confirm that the highlight is once again
+ activated.
+ 4. Move out of the image altogether and confirm that the highlight is deactivated.
+
+ * fast/images/text-recognition/mac/image-overlay-data-detectors-expected.txt: Added.
+ * fast/images/text-recognition/mac/image-overlay-data-detectors.html: Added.
+
2022-01-26 Antti Koivisto <antti@apple.com>
[iOS] imported/w3c/web-platform-tests/css/cssom/getComputedStyle-detached-subtree.html is failing
diff --git a/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors-expected.txt b/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors-expected.txt
new file mode 100644
index 0000000..edc2134
--- /dev/null
+++ b/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors-expected.txt
@@ -0,0 +1,12 @@
+PASS internals.hasActiveDataDetectorHighlight became true
+PASS observed active data detector highlight
+PASS internals.hasActiveDataDetectorHighlight became false
+PASS cleared active data detector highlight
+PASS internals.hasActiveDataDetectorHighlight became true
+PASS observed active data detector highlight (again)
+PASS internals.hasActiveDataDetectorHighlight became false
+PASS cleared active data detector highlight (again)
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
diff --git a/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors.html b/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors.html
new file mode 100644
index 0000000..82c6989
--- /dev/null
+++ b/LayoutTests/fast/images/text-recognition/mac/image-overlay-data-detectors.html
@@ -0,0 +1,75 @@
+<!DOCTYPE html>
+<html>
+<head>
+<script src="../../../../resources/js-test.js"></script>
+<script src="../../../../resources/ui-helper.js"></script>
+<style>
+body, html {
+ margin: 0;
+}
+
+img {
+ position: absolute;
+ top: 0;
+ left: 0;
+ width: 100px;
+ height: 100px;
+}
+</style>
+</head>
+<body>
+<img src="../../resources/green-400x400.png"></img>
+<script>
+jsTestIsAsync = true;
+
+addEventListener("load", async () => {
+ image = document.querySelector("img");
+
+ eventSender.mouseMoveTo(400, 400);
+ internals.installImageOverlay(image, [
+ {
+ topLeft : new DOMPointReadOnly(0, 0.5),
+ topRight : new DOMPointReadOnly(1, 0.5),
+ bottomRight : new DOMPointReadOnly(1, 1),
+ bottomLeft : new DOMPointReadOnly(0, 1),
+ children: [{
+ text : "webkit.org",
+ topLeft : new DOMPointReadOnly(0, 0.5),
+ topRight : new DOMPointReadOnly(1, 0.5),
+ bottomRight : new DOMPointReadOnly(1, 1),
+ bottomLeft : new DOMPointReadOnly(0, 1),
+ }],
+ }
+ ], [], [
+ {
+ topLeft : new DOMPointReadOnly(0, 0.5),
+ topRight : new DOMPointReadOnly(1, 0.5),
+ bottomRight : new DOMPointReadOnly(1, 1),
+ bottomLeft : new DOMPointReadOnly(0, 1),
+ }
+ ]);
+
+ eventSender.mouseMoveTo(49, 74);
+ eventSender.mouseMoveTo(50, 75);
+ await shouldBecomeEqual("internals.hasActiveDataDetectorHighlight", "true");
+ testPassed("observed active data detector highlight");
+
+ image.style.width = "400px";
+ image.style.height = "400px";
+
+ await shouldBecomeEqual("internals.hasActiveDataDetectorHighlight", "false");
+ testPassed("cleared active data detector highlight");
+
+ eventSender.mouseMoveTo(199, 299);
+ eventSender.mouseMoveTo(200, 300);
+ await shouldBecomeEqual("internals.hasActiveDataDetectorHighlight", "true");
+ testPassed("observed active data detector highlight (again)");
+
+ eventSender.mouseMoveTo(200, 510);
+ await shouldBecomeEqual("internals.hasActiveDataDetectorHighlight", "false");
+ testPassed("cleared active data detector highlight (again)");
+ finishJSTest();
+});
+</script>
+</body>
+</html>
\ No newline at end of file
diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog
index 2a34130..4a22bcf 100644
--- a/Source/WebCore/ChangeLog
+++ b/Source/WebCore/ChangeLog
@@ -1,3 +1,70 @@
+2022-01-26 Wenson Hsieh <wenson_hsieh@apple.com>
+
+ Data detectors sometimes show up in the wrong place when resizing images with Live Text
+ https://bugs.webkit.org/show_bug.cgi?id=235598
+ rdar://88032375
+
+ Reviewed by Dean Jackson.
+
+ On macOS, Live Text data detectors sometimes show up in the wrong place when images are resized; this can happen
+ in the case where image analysis injects data detection results into the image, but then the image is resized
+ from underneath the user's mouse cursor. To fix this, add some logic to invalidate ImageOverlayController's
+ cached data detector highlight information in the case where the image overlay layout has been changed.
+
+ Test: fast/images/text-recognition/mac/image-overlay-data-detectors.html
+
+ * WebCore.xcodeproj/project.pbxproj:
+
+ Have libWebCoreTestSupport additionally link against libPAL, so that we can use PAL's DataDetectorsCore soft-
+ linking utilities.
+
+ * dom/ImageOverlay.cpp:
+ (WebCore::ImageOverlay::updateSubtree):
+
+ Drive-by fix in adjacent code: when installing image overlays in media elements, make sure that we install them
+ inside the media controls root container by calling `ensureUserAgentShadowRoot()` before inserting the overlay
+ content; this ensures that we don't end up with a redundant image overlay in the shadow root. Tests for this
+ will be added in #235623.
+
+ (WebCore::ImageOverlay::updateWithTextRecognitionResult):
+ * page/ImageOverlayController.cpp:
+ (WebCore::ImageOverlayController::textRecognitionResultsChanged):
+ (WebCore::ImageOverlayController::hasActiveDataDetectorHighlightForTesting const):
+ * page/ImageOverlayController.h:
+ * page/mac/ImageOverlayControllerMac.mm:
+ (WebCore::ImageOverlayController::textRecognitionResultsChanged):
+
+ Add a hook to inform ImageOverlayController when image overlay content changes. If the image overlay host
+ matches the currently active host element showing data detector highlights, then invalidate the highlights;
+ these highlights will be recomputed once the user hovers over the data detector elements again.
+
+ (WebCore::ImageOverlayController::hasActiveDataDetectorHighlightForTesting const):
+
+ Add a testing-only helper method to query whether or not there is an active data detector highlight. See below
+ for more details.
+
+ * testing/Internals.cpp:
+
+ Add some more WebCore testing support to make it possible to test data detector highlights in Live Text on
+ macOS.
+
+ (WebCore::makeDataForLine):
+ (WebCore::Internals::installImageOverlay):
+
+ Add an optional argument to provide a list of data detector quads to inject into the overlay host. For now, each
+ data detector element simply corresponds to a dummy `DDScannerResult` returned by the static
+ `fakeDataDetectorResultForTesting()` helper below.
+
+ (WebCore::Internals::hasActiveDataDetectorHighlight const):
+
+ Add an internal testing hook to query whether or not ImageOverlayController is tracking an active data detector
+ highlight.
+
+ * testing/Internals.h:
+ * testing/Internals.idl:
+ * testing/Internals.mm:
+ (WebCore::Internals::fakeDataDetectorResultForTesting):
+
2022-01-26 Youenn Fablet <youenn@apple.com>
[MacOS] Set kAudioOutputUnitProperty_CurrentDevice on CoreAudioSharedUnit outputBus
diff --git a/Source/WebCore/PAL/ChangeLog b/Source/WebCore/PAL/ChangeLog
index 619afe8..8d755bb 100644
--- a/Source/WebCore/PAL/ChangeLog
+++ b/Source/WebCore/PAL/ChangeLog
@@ -1,3 +1,17 @@
+2022-01-26 Wenson Hsieh <wenson_hsieh@apple.com>
+
+ Data detectors sometimes show up in the wrong place when resizing images with Live Text
+ https://bugs.webkit.org/show_bug.cgi?id=235598
+ rdar://88032375
+
+ Reviewed by Dean Jackson.
+
+ Move some soft-linked DataDetectorsCore API out of the iOS-specific define, so that we can call them on macOS.
+ See WebCore/ChangeLog for more details.
+
+ * pal/cocoa/DataDetectorsCoreSoftLink.h:
+ * pal/cocoa/DataDetectorsCoreSoftLink.mm:
+
2022-01-25 Eric Carlson <eric.carlson@apple.com>
[macOS] Add new screen and window capture backend
diff --git a/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.h b/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.h
index 3fa1f1b..de9c34c 100644
--- a/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.h
+++ b/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.h
@@ -35,10 +35,6 @@
#if PLATFORM(MAC)
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, DataDetectorsCore, DDBinderPhoneNumberKey, CFStringRef)
#elif PLATFORM(IOS_FAMILY)
-SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerCreate, DDScannerRef, (DDScannerType type, DDScannerOptions options, CFErrorRef * errorRef), (type, options, errorRef))
-SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerScanQuery, Boolean, (DDScannerRef scanner, DDScanQueryRef query), (scanner, query))
-SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScanQueryCreate, DDScanQueryRef, (CFAllocatorRef allocator), (allocator))
-SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerCopyResultsWithOptions, CFArrayRef, (DDScannerRef scanner, DDScannerCopyResultsOptions options), (scanner, options))
SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDResultGetRange, CFRange, (DDResultRef result), (result))
SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDResultGetType, CFStringRef, (DDResultRef result), (result))
SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDResultGetCategory, DDResultCategory, (DDResultRef result), (result))
@@ -67,5 +63,10 @@
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, DataDetectorsCore, DDScannerCopyResultsOptionsForPassiveUse, DDScannerCopyResultsOptions)
SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerEnableOptionalSource, void, (DDScannerRef scanner, DDScannerSource source, Boolean enable), (scanner, source, enable))
#endif // PLATFORM(IOS_FAMILY)
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerCreate, DDScannerRef, (DDScannerType type, DDScannerOptions options, CFErrorRef * errorRef), (type, options, errorRef))
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerScanQuery, Boolean, (DDScannerRef scanner, DDScanQueryRef query), (scanner, query))
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScanQueryCreate, DDScanQueryRef, (CFAllocatorRef allocator), (allocator))
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScanQueryCreateFromString, DDScanQueryRef, (CFAllocatorRef allocator, CFStringRef string, CFRange range), (allocator, string, range))
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, DataDetectorsCore, DDScannerCopyResultsWithOptions, CFArrayRef, (DDScannerRef scanner, DDScannerCopyResultsOptions options), (scanner, options))
#endif // ENABLE(DATA_DETECTION)
diff --git a/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.mm b/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.mm
index 2b6447f..adb80ba 100644
--- a/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.mm
+++ b/Source/WebCore/PAL/pal/cocoa/DataDetectorsCoreSoftLink.mm
@@ -36,10 +36,6 @@
#if PLATFORM(MAC)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, DataDetectorsCore, DDBinderPhoneNumberKey, CFStringRef, PAL_EXPORT)
#elif PLATFORM(IOS_FAMILY)
-SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerCreate, DDScannerRef, (DDScannerType type, DDScannerOptions options, CFErrorRef * errorRef), (type, options, errorRef))
-SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerScanQuery, Boolean, (DDScannerRef scanner, DDScanQueryRef query), (scanner, query))
-SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScanQueryCreate, DDScanQueryRef, (CFAllocatorRef allocator), (allocator))
-SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerCopyResultsWithOptions, CFArrayRef, (DDScannerRef scanner, DDScannerCopyResultsOptions options), (scanner, options))
SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDResultGetRange, CFRange, (DDResultRef result), (result))
SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDResultGetType, CFStringRef, (DDResultRef result), (result))
SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDResultGetCategory, DDResultCategory, (DDResultRef result), (result))
@@ -68,5 +64,9 @@
SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerCopyResultsOptionsForPassiveUse, DDScannerCopyResultsOptions)
SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerEnableOptionalSource, void, (DDScannerRef scanner, DDScannerSource source, Boolean enable), (scanner, source, enable))
#endif // PLATFORM(IOS_FAMILY)
-
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerCreate, DDScannerRef, (DDScannerType type, DDScannerOptions options, CFErrorRef * errorRef), (type, options, errorRef))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerScanQuery, Boolean, (DDScannerRef scanner, DDScanQueryRef query), (scanner, query))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScanQueryCreate, DDScanQueryRef, (CFAllocatorRef allocator), (allocator))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScanQueryCreateFromString, DDScanQueryRef, (CFAllocatorRef allocator, CFStringRef string, CFRange range), (allocator, string, range))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, DataDetectorsCore, DDScannerCopyResultsWithOptions, CFArrayRef, (DDScannerRef scanner, DDScannerCopyResultsOptions options), (scanner, options))
#endif // ENABLE(DATA_DETECTION)
diff --git a/Source/WebCore/WebCore.xcodeproj/project.pbxproj b/Source/WebCore/WebCore.xcodeproj/project.pbxproj
index 413426a..9a3cbad 100644
--- a/Source/WebCore/WebCore.xcodeproj/project.pbxproj
+++ b/Source/WebCore/WebCore.xcodeproj/project.pbxproj
@@ -5514,6 +5514,7 @@
F46C447E234654540039A79D /* ClipboardItemBindingsDataSource.h in Headers */ = {isa = PBXBuildFile; fileRef = F46C447C234654540039A79D /* ClipboardItemBindingsDataSource.h */; };
F46D5386273D7E460009FA80 /* ImageOverlay.h in Headers */ = {isa = PBXBuildFile; fileRef = F46D5385273D7E3F0009FA80 /* ImageOverlay.h */; settings = {ATTRIBUTES = (Private, ); }; };
F46D53A1273EEFA00009FA80 /* ImageAnalysisQueue.h in Headers */ = {isa = PBXBuildFile; fileRef = F46D539D273EECF70009FA80 /* ImageAnalysisQueue.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ F471137527A0AFA6001BC7CA /* libPAL.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 1C09D0501E31C32900725F18 /* libPAL.a */; };
F473845825DDE9FB006DE8DD /* DataOwnerType.h in Headers */ = {isa = PBXBuildFile; fileRef = F473845725DDE9FB006DE8DD /* DataOwnerType.h */; settings = {ATTRIBUTES = (Private, ); }; };
F47A09D120A93A9700240FAE /* DisabledAdaptations.h in Headers */ = {isa = PBXBuildFile; fileRef = F47A09CF20A939F600240FAE /* DisabledAdaptations.h */; settings = {ATTRIBUTES = (Private, ); }; };
F47A5E3E195B8C8A00483100 /* StyleScrollSnapPoints.h in Headers */ = {isa = PBXBuildFile; fileRef = F47A5E3B195B8C8A00483100 /* StyleScrollSnapPoints.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -18283,6 +18284,7 @@
AA5F3B9116CC5BEB00455EB0 /* CoreFoundation.framework in Frameworks */,
A15E6BF11E212A6A0080AF34 /* Foundation.framework in Frameworks */,
41230913138C42FF00BCCFCA /* JavaScriptCore.framework in Frameworks */,
+ F471137527A0AFA6001BC7CA /* libPAL.a in Frameworks */,
4123081B138C429700BCCFCA /* WebCore.framework in Frameworks */,
);
runOnlyForDeploymentPostprocessing = 0;
diff --git a/Source/WebCore/dom/ImageOverlay.cpp b/Source/WebCore/dom/ImageOverlay.cpp
index 0f993fa..69d30af 100644
--- a/Source/WebCore/dom/ImageOverlay.cpp
+++ b/Source/WebCore/dom/ImageOverlay.cpp
@@ -37,6 +37,7 @@
#include "HTMLDivElement.h"
#include "HTMLMediaElement.h"
#include "HTMLStyleElement.h"
+#include "ImageOverlayController.h"
#include "MediaControlsHost.h"
#include "Page.h"
#include "Quirks.h"
@@ -207,20 +208,30 @@
bool hadExistingElements = false;
Elements elements;
RefPtr<HTMLElement> mediaControlsContainer;
- if (RefPtr shadowRoot = element.shadowRoot()) {
#if ENABLE(MODERN_MEDIA_CONTROLS)
- if (is<HTMLMediaElement>(element)) {
- if (RefPtr controlsHost = downcast<HTMLMediaElement>(element).mediaControlsHost()) {
- auto& containerClass = controlsHost->mediaControlsContainerClassName();
- for (auto& child : childrenOfType<HTMLDivElement>(*shadowRoot)) {
- if (child.hasClass() && child.classNames().contains(containerClass)) {
- mediaControlsContainer = &child;
- break;
- }
- }
- }
+ mediaControlsContainer = ([&]() -> RefPtr<HTMLElement> {
+ RefPtr mediaElement = dynamicDowncast<HTMLMediaElement>(element);
+ if (!mediaElement)
+ return { };
+
+ Ref shadowRoot = mediaElement->ensureUserAgentShadowRoot();
+ RefPtr controlsHost = mediaElement->mediaControlsHost();
+ if (!controlsHost) {
+ ASSERT_NOT_REACHED();
+ return { };
}
-#endif
+
+ auto& containerClass = controlsHost->mediaControlsContainerClassName();
+ for (auto& child : childrenOfType<HTMLDivElement>(shadowRoot.get())) {
+ if (child.hasClass() && child.classNames().contains(containerClass))
+ return &child;
+ }
+ ASSERT_NOT_REACHED();
+ return { };
+ })();
+#endif // ENABLE(MODERN_MEDIA_CONTROLS)
+
+ if (RefPtr shadowRoot = element.shadowRoot()) {
if (hasOverlay(element)) {
RefPtr<ContainerNode> containerForImageOverlay;
if (mediaControlsContainer)
@@ -511,6 +522,12 @@
// FIXME: We should come up with a way to coalesce the bounding quads into one or more rotated rects with the same angle of rotation.
fitElementToQuad(dataDetectorContainer.get(), convertToContainerCoordinates(firstQuad));
}
+
+ if (!result.dataDetectors.isEmpty()) {
+ auto* page = document->page();
+ if (auto* overlayController = page ? page->imageOverlayControllerIfExists() : nullptr)
+ overlayController->textRecognitionResultsChanged(element);
+ }
#endif // ENABLE(DATA_DETECTION)
constexpr float minScaleForFontSize = 0;
diff --git a/Source/WebCore/page/ImageOverlayController.cpp b/Source/WebCore/page/ImageOverlayController.cpp
index 8f89c62..9dadf09 100644
--- a/Source/WebCore/page/ImageOverlayController.cpp
+++ b/Source/WebCore/page/ImageOverlayController.cpp
@@ -211,6 +211,19 @@
{
}
+#if ENABLE(DATA_DETECTION)
+
+void ImageOverlayController::textRecognitionResultsChanged(HTMLElement&)
+{
+}
+
+bool ImageOverlayController::hasActiveDataDetectorHighlightForTesting() const
+{
+ return false;
+}
+
+#endif // ENABLE(DATA_DETECTION)
+
#endif // !PLATFORM(MAC)
} // namespace WebCore
diff --git a/Source/WebCore/page/ImageOverlayController.h b/Source/WebCore/page/ImageOverlayController.h
index cfe49cf..2e1f786 100644
--- a/Source/WebCore/page/ImageOverlayController.h
+++ b/Source/WebCore/page/ImageOverlayController.h
@@ -60,6 +60,11 @@
void selectionQuadsDidChange(Frame&, const Vector<FloatQuad>&);
void elementUnderMouseDidChange(Frame&, Element*);
+#if ENABLE(DATA_DETECTION)
+ WEBCORE_EXPORT bool hasActiveDataDetectorHighlightForTesting() const;
+ void textRecognitionResultsChanged(HTMLElement&);
+#endif
+
void documentDetached(const Document&);
private:
diff --git a/Source/WebCore/page/mac/ImageOverlayControllerMac.mm b/Source/WebCore/page/mac/ImageOverlayControllerMac.mm
index da36882..aed58b9 100644
--- a/Source/WebCore/page/mac/ImageOverlayControllerMac.mm
+++ b/Source/WebCore/page/mac/ImageOverlayControllerMac.mm
@@ -183,6 +183,20 @@
m_activeDataDetectorHighlight = nullptr;
}
+void ImageOverlayController::textRecognitionResultsChanged(HTMLElement& element)
+{
+ if (m_hostElementForDataDetectors != &element)
+ return;
+
+ clearDataDetectorHighlights();
+ uninstallPageOverlayIfNeeded();
+}
+
+bool ImageOverlayController::hasActiveDataDetectorHighlightForTesting() const
+{
+ return !!m_activeDataDetectorHighlight;
+}
+
void ImageOverlayController::elementUnderMouseDidChange(Frame& frame, Element* elementUnderMouse)
{
if (m_activeDataDetectorHighlight)
diff --git a/Source/WebCore/testing/Internals.cpp b/Source/WebCore/testing/Internals.cpp
index 239d85a..54ecf5c 100644
--- a/Source/WebCore/testing/Internals.cpp
+++ b/Source/WebCore/testing/Internals.cpp
@@ -112,6 +112,7 @@
#include "IDBRequest.h"
#include "IDBTransaction.h"
#include "ImageOverlay.h"
+#include "ImageOverlayController.h"
#include "InspectorClient.h"
#include "InspectorController.h"
#include "InspectorDebuggableType.h"
@@ -5764,6 +5765,7 @@
Internals::ImageOverlayLine::~ImageOverlayLine() = default;
Internals::ImageOverlayText::~ImageOverlayText() = default;
Internals::ImageOverlayBlock::~ImageOverlayBlock() = default;
+Internals::ImageOverlayDataDetector::~ImageOverlayDataDetector() = default;
#if ENABLE(IMAGE_ANALYSIS)
@@ -5783,7 +5785,7 @@
return {
getQuad<Internals::ImageOverlayLine>(line),
line.children.map([](auto& textChild) -> TextRecognitionWordData {
- return { textChild.text, getQuad<Internals::ImageOverlayText>(textChild), textChild.hasLeadingWhitespace };
+ return { textChild.text, getQuad(textChild), textChild.hasLeadingWhitespace };
}),
line.hasTrailingNewline
};
@@ -5813,7 +5815,7 @@
#endif // ENABLE(IMAGE_ANALYSIS)
-void Internals::installImageOverlay(Element& element, Vector<ImageOverlayLine>&& lines, Vector<ImageOverlayBlock>&& blocks)
+void Internals::installImageOverlay(Element& element, Vector<ImageOverlayLine>&& lines, Vector<ImageOverlayBlock>&& blocks, Vector<ImageOverlayDataDetector>&& dataDetectors)
{
if (!is<HTMLElement>(element))
return;
@@ -5824,18 +5826,30 @@
return makeDataForLine(line);
})
#if ENABLE(DATA_DETECTION)
- , Vector<TextRecognitionDataDetector>()
-#endif
+ , dataDetectors.map([] (auto& dataDetector) -> TextRecognitionDataDetector {
+ return TextRecognitionDataDetector { fakeDataDetectorResultForTesting(), { getQuad(dataDetector) } };
+ })
+#endif // ENABLE(DATA_DETECTION)
, blocks.map([] (auto& block) {
- return TextRecognitionBlockData { block.text, getQuad<ImageOverlayBlock>(block) };
+ return TextRecognitionBlockData { block.text, getQuad(block) };
})
});
#else
UNUSED_PARAM(blocks);
+ UNUSED_PARAM(dataDetectors);
UNUSED_PARAM(lines);
#endif
}
+bool Internals::hasActiveDataDetectorHighlight() const
+{
+#if ENABLE(DATA_DETECTION) && ENABLE(IMAGE_ANALYSIS)
+ if (auto* controller = contextDocument()->page()->imageOverlayControllerIfExists())
+ return controller->hasActiveDataDetectorHighlightForTesting();
+#endif
+ return false;
+}
+
bool Internals::isSystemPreviewLink(Element& element) const
{
#if USE(SYSTEM_PREVIEW)
diff --git a/Source/WebCore/testing/Internals.h b/Source/WebCore/testing/Internals.h
index 6a3388a..303e472 100644
--- a/Source/WebCore/testing/Internals.h
+++ b/Source/WebCore/testing/Internals.h
@@ -47,6 +47,8 @@
#include "MediaUniqueIdentifier.h"
#endif
+OBJC_CLASS DDScannerResult;
+
namespace WebCore {
class AbstractRange;
@@ -938,7 +940,17 @@
~ImageOverlayBlock();
};
- void installImageOverlay(Element&, Vector<ImageOverlayLine>&&, Vector<ImageOverlayBlock>&& = { });
+ struct ImageOverlayDataDetector {
+ RefPtr<DOMPointReadOnly> topLeft;
+ RefPtr<DOMPointReadOnly> topRight;
+ RefPtr<DOMPointReadOnly> bottomRight;
+ RefPtr<DOMPointReadOnly> bottomLeft;
+
+ ~ImageOverlayDataDetector();
+ };
+
+ void installImageOverlay(Element&, Vector<ImageOverlayLine>&&, Vector<ImageOverlayBlock>&& = { }, Vector<ImageOverlayDataDetector>&& = { });
+ bool hasActiveDataDetectorHighlight() const;
#if ENABLE(IMAGE_ANALYSIS)
void requestTextRecognition(Element&, RefPtr<VoidCallback>&&);
@@ -1259,6 +1271,10 @@
ExceptionOr<RenderedDocumentMarker*> markerAt(Node&, const String& markerType, unsigned index);
ExceptionOr<ScrollableArea*> scrollableAreaForNode(Node*) const;
+#if ENABLE(DATA_DETECTION)
+ static DDScannerResult *fakeDataDetectorResultForTesting();
+#endif
+
#if ENABLE(MEDIA_STREAM)
// RealtimeMediaSource::Observer API
void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
diff --git a/Source/WebCore/testing/Internals.idl b/Source/WebCore/testing/Internals.idl
index 0282c29..d731f2a 100644
--- a/Source/WebCore/testing/Internals.idl
+++ b/Source/WebCore/testing/Internals.idl
@@ -306,6 +306,16 @@
required DOMPointReadOnly bottomLeft;
};
+[
+ ExportMacro=WEBCORE_TESTSUPPORT_EXPORT,
+ JSGenerateToJSObject,
+] dictionary ImageOverlayDataDetector {
+ required DOMPointReadOnly topLeft;
+ required DOMPointReadOnly topRight;
+ required DOMPointReadOnly bottomRight;
+ required DOMPointReadOnly bottomLeft;
+};
+
typedef (FetchRequest or FetchResponse) FetchObject;
[
@@ -967,7 +977,8 @@
[Conditional=IMAGE_ANALYSIS] readonly attribute Element? textRecognitionCandidate;
[Conditional=IMAGE_ANALYSIS] undefined requestTextRecognition(Element element, VoidCallback callback);
- undefined installImageOverlay(Element element, sequence<ImageOverlayLine> lines, optional sequence<ImageOverlayBlock> blocks = []);
+ undefined installImageOverlay(Element element, sequence<ImageOverlayLine> lines, optional sequence<ImageOverlayBlock> blocks = [], optional sequence<ImageOverlayDataDetector> dataDetectors = []);
+ readonly attribute boolean hasActiveDataDetectorHighlight;
boolean usingAppleInternalSDK();
boolean usingGStreamer();
diff --git a/Source/WebCore/testing/Internals.mm b/Source/WebCore/testing/Internals.mm
index 414e3f5..0b7e385 100644
--- a/Source/WebCore/testing/Internals.mm
+++ b/Source/WebCore/testing/Internals.mm
@@ -42,6 +42,7 @@
#import <Metal/Metal.h>
#endif
#import <pal/spi/cocoa/NSAccessibilitySPI.h>
+#import <wtf/cf/TypeCastsCF.h>
#import <wtf/cocoa/NSURLExtras.h>
#import <wtf/spi/darwin/SandboxSPI.h>
@@ -49,7 +50,9 @@
#import <pal/ios/UIKitSoftLink.h>
#endif
-
+#if ENABLE(DATA_DETECTION)
+#import <pal/cocoa/DataDetectorsCoreSoftLink.h>
+#endif
namespace WebCore {
@@ -172,4 +175,26 @@
}
#endif
+#if ENABLE(DATA_DETECTION)
+
+DDScannerResult *Internals::fakeDataDetectorResultForTesting()
+{
+ static NeverDestroyed result = ([]() -> RetainPtr<DDScannerResult> {
+ auto scanner = adoptCF(PAL::softLink_DataDetectorsCore_DDScannerCreate(DDScannerTypeStandard, 0, nullptr));
+ auto stringToScan = CFSTR("webkit.org");
+ auto query = adoptCF(PAL::softLink_DataDetectorsCore_DDScanQueryCreateFromString(kCFAllocatorDefault, stringToScan, CFRangeMake(0, CFStringGetLength(stringToScan))));
+ if (!PAL::softLink_DataDetectorsCore_DDScannerScanQuery(scanner.get(), query.get()))
+ return { };
+
+ auto results = adoptCF(PAL::softLink_DataDetectorsCore_DDScannerCopyResultsWithOptions(scanner.get(), DDScannerCopyResultsOptionsNoOverlap));
+ if (!CFArrayGetCount(results.get()))
+ return { };
+
+ return { [[PAL::getDDScannerResultClass() resultsFromCoreResults:results.get()] firstObject] };
+ })();
+ return result->get();
+}
+
+#endif // ENABLE(DATA_DETECTION)
+
}
