Introduction to WebKit

What is WebKit?

WebKit is an open-source Web browser engine. It’s a framework in macOS and iOS, and used by many first party and third party applications including Safari, Mail, Notes, Books, News, and App Store.

The WebKit codebase is mostly written in C++ with bits of C and assembly, primarily in JavaScriptCore, and some Objective-C to integrate with Cocoa platforms.

It primarily consists of the following components, each inside its own directory in Source:

• bmalloc - WebKit’s malloc implementation as a bump pointer allocator. It provides an important security feature, called IsoHeap, which segregates each type of object into its own page to prevent type confusion attacks upon use-after-free.
• WTF - Stands for Web Template Framework. WebKit’s template library. The rest of the WebKit codebase is built using this template library in addition to, and often in place of, similar class templates in the C++ standard library. It contains common container classes such as Vector, HashMap (unordered), HashSet, and smart pointer types such as Ref, RefPtr, and WeakPtr used throughout the rest of WebKit.
• JavaScriptCore - WebKit’s JavaScript engine; often abbreviated as JSC. JSC parses JavaScript and generates byte code, which is then executed by one of the following four tiers. Many tiers are needed to balance between compilation time and execution time. Also see Phil's blog post about Speculation in JavaScriptCore.
  • Interpreter - This tier reads and executes instructions in byte code in C++.
  • Baseline JIT - The first Just In Time compiler tier serves as the profiler as well as a significant speed up from the interpreter.
  • DFG JIT - Data Flow Graph Just In Time compiler uses the data flow analysis to generate optimized machine code.
  • FTL JIT - Faster than Light Just In Time compiler which uses B3 backend. It’s the fastest tier of JSC. JavaScriptCode also implements JavaScriptCore API for macOS and iOS applications.
• WebCore - The largest component of WebKit, this layer implements most of the Web APIs and their behaviors. Most importantly, this component implements HTML, XML, and CSS parsers and implements HTML, SVG, and MathML elements as well as CSS. It also implements CSS JIT, the only Just In Time compiler for CSS in existence. It works with a few tree data structures:
  • Document Object Model - This is the tree data structure we create from parsing HTML.
  • Render Tree - This tree represents the visual representation of each element in DOM tree computed from CSS and also stores the geometric layout information of each element.
• WebCore/PAL and WebCore/platform - Whilst technically a part of WebCore, this is a platform abstraction layer for WebCore so that the rest of WebCore code can remain platform independent / agnostic across all the platforms WebKit can run on: macOS, iOS, Windows, Linux, etc... Historically, most of this code resided in WebCore/platform. There is an ongoing multi-year project to slowly migrate code to PAL as we remove the reverse dependencies to WebCore.
• WebKitLegacy (a.k.a. WebKit1) - This layer interfaces WebCore with the rest of operating systems in single process and implements WebView on macOS and UIWebView on iOS.
• WebKit (a.k.a. WebKit2) - This layer implements the multi-process architecture of WebKit, and implements WKWebView on macOS and iOS. WebKit’s multi-process architecture consists of the following processes:
  • UI process - This is the application process. e.g. Safari and Mail
  • WebContent process - This process loads & runs code loaded from websites. Each tab in Safari typically has its own WebContent process. This is important to keep each tab responsive and protect websites from one another.
  • Networking process - This process is responsible for handling network requests as well as storage management. All WebContent processes in a single session (default vs. private browsing) share a single networking session in the networking process.
• WebInspector / WebDriver - WebKit’s developer tool & automation tool for Web developers.

Contributing to WebKit

There are many ways to get involved and contribute to the WebKit Project. Filing a new bug, fixing a bug, or adding a new feature.

There are three different kinds of contributors in the WebKit project.

• Contributor - This category encompasses everyone. Anyone who files a bug or contributes a code change or reviews a code change is considered as a contributor
• Committer - A committer is someone who has write access to WebKit's subversion repository.
• Reviewer - A reviewer is someone who has the right to review and approve code changes other contributors proposed.

See Commit and Review Policy for more details on how to become a committer or a reviewer.

Staying in Touch

Before getting in touch with WebKit developers using any of the avenues below, make sure that you have checked our page on how to ask questions about WebKit.

You can find WebKit developers, testers, and other interested parties on the #WebKit Slack workspace. Join the WebKit slack, and stay in touch.

Bug tracking in WebKit

bugs.webkit.org hosted is the primary bug tracking tool we use. When making a code change, we post a code change (patch) on this website.

Filing a bug and editing bugs

To file a new WebKit bug, see reporting bugs.

To edit an existing bug, you may need editbug-bits.

Code Reviews in bugs.webkit.org

We also use bugs.webkit.org to upload & review code changes to WebKit. You can post a code change with Tools/Scripts/webkit-patch upload. Note that the webkit-patch script only looks for changes below current directory, so generally you should change the current directory to the top-level directory of a WebKit first.

When a patch is posted on bugs.webkit.org requesting a formal code review (r? flag is set), The Early Warning System (a.k.a. EWS) will automatically build and run tests against your code change. This allows contributors to find build or test failures before committing code changes to the WebKit’s primary Subversion repository.

Once a patch is approved by a reviewer (r+ flag is set), then the patch can be either committed directly into the Subversion repository by a WebKit committer, who has write access to the Subversion repository, or via the commit queue which can be requested by setting cq? flag and approved by any WebKit committer by setting cq+.

The Subvesion commit message should be created by Tools/Scripts/commit-log-editor based on the change log entries. Tools/Scripts/webkit-patch land does this automatically.

Security Bugs in bugs.webkit.org

Security bugs have their own components in bugs.webkit.org. We’re also working on a new policy to delay publishing tests for security fixes until after the fixes have been widely deployed.

Do not post a patch or describe a security bug in a bug that is not in security component of bugs.webkit.org.

Getting started with WebKit

Getting Code

See Getting the Code

Adding Tools to PATH

For convenience, you can add Tools/Scripts/ to your path as follows in ~/.zshrc like so:

export PATH=$PATH:/Volumes/Data/webkit/Tools/Scripts/

where /Volumes/Data/webkit is the path to a WebKit checkout.

This will allow you to run various tools you by name instead of typing the full path of the script.

Updating checkouts

There is a script to update a WebKit checkout: Tools/Scripts/update-webkit. This script will automatically merge change logs mentioned below.

Building WebKit

See Building WebKit

Fixing mysterious build or runtime errors after Xcode upgrades

If you see mysterious build failures or if you’ve switched to a new version of macOS or Xcode, delete the WebKitBuild directory. make clean may not delete all the relevant files, and building after doing that without deleting the WebKitBuild directory may result in mysterious build or dyld errors.

Building with Address Sanitizer to investigate memory corruption bugs

To build Address Sanitizer or ASan builds to analyze security bugs, run Tools/Scripts/set-webkit-configuration --asan --release. This will enable ASan build. If want to attach a debugger, you can also specify --debug instead of --release. Once you don’t need to build or run ASan anymore, you can specify --no-asan in place of --asan to disable ASan. Note that this configuration is saved by creating a file called Asan in the WebKitBuild directory, so if you are trying to do a clean Asan build by deleting the build directory you need to rerun this command.

Using Xcode

You can also use Xcode to build & debug WebKit. Open WebKit.xcworkspace at the top level directory.

In order to make Xcode use build files built by make command above, go to File > Workspace Settings... > Advanced... > Custom > Relative to Workspace and adjust the relative paths of Products and Intermediates to point to WebKitBuild directory. Note that debugging WebCore code typically requires attaching to the relevant WebContent process, not the application process, which is mostly running code in Source/WebKit/UIProcess. Depending on what you’re debugging, you’d have to attach & debug different processes in the coalition.

You may find it useful to use the debug helpers under Tools/lldb/lldb_webkit.py. This can be added to ~/.lldbinit for automatic loading into LLDB on launch by adding the line command script import {Path to WebKit}/Tools/lldb/lldb_webkit.py. For more details, see the Wiki article on lldb formatters.

When debugging a debug build in LLDB, there are also a few functions that can be called on objects that will dump debugging info.

• RenderObject
  • showNodeTree()
  • showLineTree()
  • showRenderTree()
• Node
  • showTree()
  • showNodePath()
  • showTreeForThis()
  • showNodePathForThis()

Correctness Testing in WebKit

WebKit is really big on test driven development, we have many types of tests.

• JavaScript tests - Resides in top-level JSTests directory. This is the primary method of testing JavaScriptCore. Use Tools/Scripts/run-javascriptcore-tests to run these tests.
• Layout tests - Resides in top-level LayoutTests directory. This is the primary method of testing WebCore. If you’re making code changes to WebCore, you typically run these tests. Use Tools/Scripts/run-webkit-tests to run these. Pass -1 to run tests using WebKitLegacy (a.k.a. WebKit1). WebKitTestRunner is used to run these tests for WebKit2, and DumpRenderTree is used to these tests for WebKit1. There are a few styles of layout tests but all of them have a test file and expected result (ends with -expected.txt), and the test passes if the test file’s output matches that of the expected result.
• API tests - Reside in Tools/TestWebKitAPI, these are GTests that test APIs exposed by JavaScriptCore, WebKitLegacy, and WebKit layers as well as unit tests for selected WTF classes. WebKit does not use XCTests. Use Tools/Scripts/run-api-tests to run these tests. Because these API tests are sequentially, it’s preferable to write layout tests when possible.
• Bindings tests - Reside in Source/WebCore/bindings/scripts/test, these are tests for WebCore’s binding code generator. Use Tools/Scripts/run-bindings-tests to run these tests.
• webkitpy tests - Tests for WebKit’s various Python scripts in Tools/Scripts/webkitpy. Use Tools/Scripts/test-webkitpy to run these tests.
• webkitperl tests - Tests for WebKit’s various Perl scripts in Tools/Scripts/webkitperl. Use Tools/Scripts/test-webkitperl to run these tests.

Performance Testing in WebKit

The WebKit project has a “no performance regression” policy. We maintain the performance of the following of the benchmarks and are located under PerformanceTests. If your patch regresses one of these benchmarks even slightly (less than 1%), it will get reverted.

• JetStream2 - Measures JavaScript and WASM performance.
• MotionMark - Measures graphics performance.
• Speedometer 2 - Measures WebKit’s performance for complex web apps.

The following are benchmarks maintained by Apple‘s WebKit team but not available to other open source contributors since Apple doesn’t have the right to redistribute the content. If your WebKit patch regresses one of these tests, your patch may still get reverted.

• RAMification - Apple's internal JavaScript memory benchmark.
• ScrollPerf - Apple's internal scrolling performance tests.
• PLT - Apple's internal page load time tests.
• Membuster / PLUM - Apple's internal memory tests. Membuster for macOS and PLUM for iOS and iPadOS.

Contributing code to WebKit

WebKit has a rigorous code contribution process and policy in place to maintain the quality of code.

Coding style

Code you write must follow WebKit’s coding style guideline. You can run Tools/Scripts/check-webkit-style to check whether your code follows the coding guidelines or not (it can report false positives or false negatives). If you use Tools/Scripts/webkit-patch upload to upload your patch, it automatically runs the style checker against the code you changed so there is no need to run check-webkit-style separately.

Some older parts of the codebase do not follow these guidelines. If you are modifying such code, it is generally best to clean it up to comply with the current guidelines.

Convenience Tools

Tools/Scripts/webkit-patch provides a lot of utility functions like applying the latest patch on bugs.webkit.org (apply-from-bug) and uploading a patch (upload --git-commit=<commit hash>) to a bugs.webkit.org bug. Use --all-commands to the list of all commands this tool supports.

Licensing

Much of the code we inherited from KHTML is licensed under LGPL. New code contributed to WebKit will use the two clause BSD license. When contributing new code, update the copyright date. When moving the existing code, you need to include the original copyright notice for the moved code and you should also not change the license, which may be BSD or LGPL depending on a file, without the permission of the copyright holders.

Regression Tests

Once you have made a code change, you need to run the aforementioned tests (layout tests, API tests, etc...) to make sure your code change doesn’t break existing functionality. These days, uploading a patch on bugs.webkit.org triggers the Early Warning System (a.k.a. EWS)

For any bug fix or a feature addition, there should be a new test demonstrating the behavior change caused by the code change. If no such test can be written in a reasonable manner (e.g. the fix for a hard-to-reproduce race condition), then the reason writing a tests is impractical should be explained in the accompanying change log.

Any patch which introduces new test failures or performance regressions may be reverted. It’s in your interest to wait for the Early Warning System to fully build and test your patch on all relevant platforms.

ChangeLog Files

ChangeLogs are simple text files which provide historical documentation for all changes to the WebKit project. All patches require an entry to the ChangeLog.

The first line contains the date, your full name, and your email address. Use this to write up a brief summary of the changes you’ve made. Don’t worry about the “Reviewed by NOBODY (OOPS!)” line, the person landing your patch will fill this in. There is one ChangeLog per top-level directory. If you changed code and tests you will need to edit at least two ChangeLogs. Tools/Scripts/prepare-ChangeLog script will create stub entries for ChangeLog files based on code changes you made in your Git or Subversion checkouts.

You should edit these stubs to describe your change, including the full URL to the bug (example entry, note that you can use --bug flag). (You should set EMAIL_ADDRESS and CHANGE_LOG_NAME in your environment if you will be running this script frequently.) A typical change log entry before being submitted to bugs.webkit.org looks like this:

2012-10-04  Enrica Casucci  <e•••••@apple.com>

        Font::glyphDataAndPageForCharacter doesn't account for text orientation when using systemFallback on a cold cache.
        https://bugs.webkit.org/show_bug.cgi?id=98452.

        Reviewed by NOBODY (OOPS!).

        The text orientation was considered only when there is a cache hit.
        This change moves the logic to handle text orientation to a separate
        inline function that is called also when the glyph is added to the cache.

        Test: fast/text/vertical-rl-rtl-linebreak.html

        * platform/graphics/FontFastPath.cpp:
        (WebCore::applyTextOrientationForCharacter): Added.
        (WebCore::Font::glyphDataAndPageForCharacter): Modified to use the new function in
        both cases of cold and warm cache.

The “No new tests. (OOPS!)” line appears if prepare-ChangeLog did not detect the addition of new tests. If your patch does not require test cases (or test cases are not possible), remove this line and explain why you didn’t write tests. Otherwise all changes require test cases which should be mentioned in the ChangeLog.

WebKit’s Build System

Apple’s macOS, iOS, watchOS, and tvOS ports use Xcode and the rest use CMake to build WebKit. There is an ongoing effort to make Apple's ports also use CMake.

In order to reduce the compilation time, which used to take 40+ minutes on the fully loaded 2018 15“ MacBook Pro, we bundle up multiple C++ translation units (.cpp files) and compile them as a single translation unit. We call this mechanism Unified Sources or Unified Builds.

Unified sources are generated under WebKitBuild/X/DerivedSources where X is the name of build configuration such as Debug and Release-iphonesimulator. For example, WebKitBuild/Debug/DerivedSources/WebCore/unified-sources/UnifiedSource116.cpp may look like this:

#include "dom/Document.cpp"
#include "dom/DocumentEventQueue.cpp"
#include "dom/DocumentFragment.cpp"
#include "dom/DocumentMarkerController.cpp"
#include "dom/DocumentParser.cpp"
#include "dom/DocumentSharedObjectPool.cpp"
#include "dom/DocumentStorageAccess.cpp"
#include "dom/DocumentType.cpp"

How to add a new .h or .cpp file

To add a new header file or a translation unit (e.g. .cpp, .m, or .mm), open WebKit.xcworkspace and add respective files in each directory.

Make sure to uncheck the target membership so that it’s not compiled as a part of the framework in xcodebuild. Instead, add the same file in Sources.txt file that exists in each subdirectory of Source. e.g. Source/WebCore/Sources.txt for WebCore. This will ensure the newly added file is compiled as a part of unified sources. When a header file in WTF is used in WebCore, or a header file in WebCore is used in WebKit or WebKitLegacy, we need to export the file to those projects. To do that, turn on the target membership in respective framework as set the membership to “Private” as seen below. This will ensure the relevant header file is exported from WTF / WebCore to other downstream projects like WebKitLegacy.

FIXME: Mention WTF_EXPORT_PRIVATE and WEBCORE_EXPORT.

FIXME: Add instructions on how to add files to CMake.

Build Failures with Unified Sources

Because of Unified Sources, it’s possible that adding a new file will cause a new build failure on some platform. This happens because if UnifiedSource1.cpp contains a.cpp, b.cpp, c.cpp, then #include in a.cpp could have pulled in some header files that c.cpp needed. When you add b2.cpp, and c.cpp moves to UnifiedSource2.cpp, c.cpp no longer benefits from a.cpp “accidentally” satisfying c.cpp’s header dependency. When this happens, you need to add a new #include to c.cpp as it was supposed to be done in the first place.

Conditional Compilation

Every translation unit in WebKit starts by including “config.h”. This file defines a set of C++ preprocessor macros used to enable or disable code based on the target operating system, platform, and whether a given feature is enabled or disabled.

For example, the following #if condition says that the code inside of it is only compiled if SERVICE_WORKER feature is enabled:

#if ENABLE(SERVICE_WORKER)
...
#endif

Similarly, the following #if condition will enable the in-between code only on macOS:

#if PLATFORM(MAC)
...
#endif

For code which should be enabled in iOS, watchOS, tvOS, and Mac Catalyst we use PLATFORM(IOS_FAMILY). For each specific variant of iOS family, we also have PLATFORM(IOS), PLATFORM(WATCHOS), PLATFORM(APPLETV), and PLATFORM(MACCATALYST).

The following #if condition will enable the in-between code only if CoreGraphics is used:

#if USE(CG)
...
#endif

Finally, if a certain piece of code should only be enabled in an operating system newer than some version, we use __IPHONE_OS_VERSION_MIN_REQUIRED or __MAC_OS_X_VERSION_MIN_REQUIRED. For example, the following #if enables the in-between code only on macOS 10.14 (macOS Mojave) or above:

#if PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101400
...
#endif

WebKit’s Continuous Integration Infrastructure

WebKit’s CI (continuous integration) infrastructure is located at build.webkit.org).

build.webkit.org will build and test commits from WebKit in the chronological order and report test results to results.webkit.org. Due to the chronological ordering, results could be a few hours behind during the work week.

We also have a dashboard to monitor the health of build.webkit.org at build.webkit.org/dashboard. If you observe that some bots are offline, or otherwise not processing your patch, please notify webkit-dev@webkit.org.

This dashboard isn't great for investigating individual test failures, results.webkit.org is a better tool for such investigations. It keeps track of individual test status by configuration over time. You can search individual tests by name or look at the historical results of entire test suites. These results will link back to the test runs in buildbot which are associated with a specific failure. See layout tests section for more details on how to use these tools to investigate test failures observed on bots.

FIXME: Add a section about downloading build products from build.webkit.org.

Memory Management in WebKit

In WebKit, when an object is owned by another object, we typically use std::unique_ptr to express that ownership. WebKit uses two primary management strategies when objects in other cases: garbage collection and reference counting.

Garbage collection in WebKit

FIXME: Write this.

Reference counting in WebKit

Overview

Most of WebCore objects are not managed by JavaScriptCore’s garbage collector. Instead, we use reference counting. We have two referencing counting pointer types: RefPtr and Ref. RefPtr is intended to behave like a C++ pointer whereas Ref is intended to behave like a C++ reference, meaning that the former can be set to nullptr but the latter cannot.

Ref<A> a1; // This will result in compilation error.
RefPtr<A> a2; // This is okay.
Ref<A> a3 = A::create(); // This is okay.
a3->f(); // Calls f() on an instance of A.
A* a4 = a3.ptr();
a4 = a2.get();

Unlike C++‘sstd::shared_ptr, the implementation of referencing counting is a part of a managed object. The requirements for an object to be used with RefPtr and Ref is as follows:

• It implements ref() and deref() member functions
• Each call to ref() and deref() will increment and decrement its internal reference counter
• The initial call to ref() is implicit in new, after the object had been allocated and the constructor has been called upon; i.e. meaning that the reference count starts at 1.
• When deref() is called when its internal reference counter reaches 0, “this” object is destructed and deleted.

There is a convenience super template class, RefCounted<T>, which implements this behavior for any inherited class T automatically.

How to use RefPtr and Ref

When an object which implements the semantics required by RefPtr and Ref is created via new, we must immediately adopt it into Ref type using adoptRef as follows:

class A : public RefCounted<T> {
public:
    int m_foo;

    int f() { return m_foo; }

    static Ref<A> create() { return adoptRef(*new A); }
private:
    A() = default;
};

This will create an instance of Ref without calling ref() on the newly created object, avoiding the unnecessary increment from 0 to 1. WebKit’s coding convention is to make the constructor private and add a static create function which returns an instance of a ref counted object after adopting it.

Note that returning RefPtr or Ref is efficient thanks to copy elision in C++11, and the following example does not create a temporary Ref object using copy constructor):

Ref<A> a = A::create();

When passing the ownership of a ref-counted object to a function, use rvalue reference with WTFMove (equivalent to std::move with some safety checks), and use a regular reference when there is a guarantee for the caller to keep the object alive as follows:

class B {
public:
    void setA(Ref<A>&& a) { m_a = WTFMove(a); }
private:
    Ref<A> m_a;
};

...

void createA(B& b) {
    b.setA(A::create());
}

Note that there is no WTFMove on A::create due to copy elision.

Forwarding ref and deref

As mentioned above, objects that are managed with RefPtr and Ref do not necessarily have to inherit from RefCounted. One common alternative is to forward ref and deref calls to another object which has the ownership. For example, in the following example, Parent class owns Child class. When someone stores Child in Ref or RefPtr, the referencing counting of Parent is incremented and decremented on behalf of Child. Both Parent and Child are destructed when the last Ref or RefPtr to either object goes away.

class Parent : RefCounted<Parent> {
public:
    static Ref<Parent> create() { return adoptRef(*new Parent); }

    Child& child() {
        if (!m_child)
            m_child = makeUnique<Child>(*this);
        return m_child
    }

private:
    std::unique_ptr<Child> m_child;    
};

class Child {
public:
    ref() { m_parent.ref(); }
    deref() { m_parent.deref(); }

private:
    Child(Parent& parent) : m_parent(parent) { }
    friend class Parent;

    Parent& m_parent;
}

Reference Cycles

A reference cycle occurs when an object X which holds Ref or RefPtr to another object Y which in turns owns X by Ref or RefPtr. For example, the following code causes a trivial memory leak because A holds a Ref of B, and B in turn holds Ref of the A:

class A : RefCounted<A> {
public:
    static Ref<A> create() { return adoptRef(*new A); }
    B& b() {
        if (!m_b)
            m_b = B::create(*this);
        return m_b.get();
    }
private:
    Ref<B> m_b;
};

class B : RefCounted<B> {
public:
    static Ref<B> create(A& a) { return adoptRef(*new B(a)); }

private:
    B(A& a) : m_a(a) { }
    Ref<A> m_a;
};

We need to be particularly careful in WebCore with regards to garbage collected objects because they often keep other ref counted C++ objects alive without having any Ref or RefPtr in C++ code. It’s almost always incorrect to strongly keep JS value alive in WebCore code because of this.

ProtectedThis Pattern

Because many objects in WebCore are managed by tree data structures, a function that operates on a node of such a tree data structure can end up deleting itself (this object). This is highly undesirable as such code often ends up having a use-after-free bug.

To prevent these kinds of bugs, we often employ a strategy of adding protectedThis local variable of Ref or RefPtr type, and store this object as follows:

ExceptionOr<void> ContainerNode::removeChild(Node& oldChild)
{
    // Check that this node is not "floating".
    // If it is, it can be deleted as a side effect of sending mutation events.
    ASSERT(refCount() || parentOrShadowHostNode());

    Ref<ContainerNode> protectedThis(*this);

    // NotFoundError: Raised if oldChild is not a child of this node.
    if (oldChild.parentNode() != this)
        return Exception { NotFoundError };

    if (!removeNodeWithScriptAssertion(oldChild, ChildChangeSource::API))
        return Exception { NotFoundError };

    rebuildSVGExtensionsElementsIfNecessary();
    dispatchSubtreeModifiedEvent();

    return { };
}

In this code, the act of removing oldChild can execute arbitrary JavaScript and delete this object. As a result, rebuildSVGExtensionsElementsIfNecessary or dispatchSubtreeModifiedEvent might be called after this object had already been free’ed if we didn’t have protectedThis, which guarantees that this object’s reference count is at least 1 (because Ref’s constructor increments the reference count by 1).

This pattern can be used for other objects that need to be protected from destruction inside a code block. In the following code, childToRemove was passed in using C++ reference. Because this function is going to remove this child node from this container node, it can get destructed while the function is still running. To prevent from having any chance of use-after-free bugs, this function stores it in Ref (protectedChildToRemove) which guarantees the object to be alive until the function returns control back to the caller:

ALWAYS_INLINE bool ContainerNode::removeNodeWithScriptAssertion(Node& childToRemove, ChildChangeSource source)
{
    Ref<Node> protectedChildToRemove(childToRemove);
    ASSERT_WITH_SECURITY_IMPLICATION(childToRemove.parentNode() == this);
    {
        ScriptDisallowedScope::InMainThread scriptDisallowedScope;
        ChildListMutationScope(*this).willRemoveChild(childToRemove);
    }
    ..

Also see Darin’s RefPtr Basics for further reading.

Weak Pointers in WebKit

In some cases, it’s desirable to express a relationship between two objects without necessarily tying their lifetime. In those cases, WeakPtr is useful. Like std::weak_ptr, this class creates a non-owning reference to an object. There is a lot of legacy code which uses a raw pointer for this purpose, but there is an ongoing effort to always use WeakPtr instead so do that in new code you’re writing.

To create a WeakPtr to an object, we need to make its class inherit from CanMakeWeakPtr as follows:

class A : CanMakeWeakPtr<A> { }

...

function foo(A& a) {
    WeakPtr<A> weakA = makeWeakPtr(a);
}

Dereferencing a WeakPtr will return nullptr when the referenced object is deleted. Because creating a WeakPtr allocates an extra WeakPtrImpl object, you’re still responsible to dispose of WeakPtr at appropriate time.

WeakHashSet

While ordinary HashSet does not support having WeakPtr as its elements, there is a specialized WeakHashSet class, which supports referencing a set of elements weakly. Because WeakHashSet does not get notified when the referenced object is deleted, the users / owners of WeakHashSet are still responsible for deleting the relevant entries from the set. Otherwise, WeakHashSet will hold onto WeakPtrImpl until computeSize is called or rehashing happens.

Understanding Document Object Model

Introduction

Document Object Model (often abbreviated as DOM) is the tree data structured resulted from parsing HTML. It consists of one or more instances of subclasses of Node and represents the document tree structure. Parsing a simple HTML like this:

<!DOCTYPE html>
<html>
<body>hi</body>
</html>

Will generate the following six distinct DOM nodes:

• Document
  • DocumentType
  • HTMLHtmlElement
    • HTMLHeadElement
    • HTMLBodyElement
      • Text with the value of “hi”

Note that HTMLHeadElement (i.e. <head>) is created implicitly by WebKit per the way HTML parser is specified.

Broadly speaking, DOM node divides into the following categories:

• Container nodes such as Document, Element, and DocumentFragment.
• Leaf nodes such as DocumentType, Text, and Attr.

Document node, as the name suggests a single HTML, SVG, MathML, or other XML document, and is the owner of every node in the document. It is the very first node in any document that gets created and the very last node to be destroyed.

Note that a single web page may consist of multiple documents since iframe and object elements may contain a child frame, and form a frame tree. Because JavaScript can open a new window under user gestures and have access back to its opener, multiple web pages across multiple tabs might be able to communicate with one another via JavaScript API such as postMessage.

JavaScript Wrappers

Each DOM node’s behavior is implemented as a C++ class in WebCore. JavaScript API is primarily implemented using Web IDL, an interface description language, from which various JS DOM binding code is auto-generated by a perl script, for example, under WebKitBuild/Debug/DerivedSources/WebCore/ for debug builds. For example, C++ implementation of Node is Node class and its JavaScript interface is implemented by JSNode class, most of which is auto-generated but has some custom bindings code in JSNodeCustom. Similarly, C++ implementation of Range interface is Range class whilst its JavaScript API is implemented by the auto-generated JSRange class. We call instances of the latter JS* classes JS wrappers.

These JS wrappers exist in what we call a DOMWrapperWorld. Each DOMWrapperWorld has its own JS wrapper for each C++ object. As a result, a single C++ object may have multiple JS wrappers in distinct DOMWrapperWorlds. The most important DOMWrapperWorld is the main DOMWrapperWorld which runs the scripts of web pages WebKit loaded while other DOMWrapperWorlds are typically used to run code for browser extensions and other code injected by applications that embed WebKit. JSX.h provides toJS functions which creates a JS wrapper for X in a given global object’s DOMWrapperWorld, and toWrapped function which returns the underlying C++ object. For example, toJS function for Node is defined in Source/WebCore/bindings/js/JSNodeCustom.h.

When there is already a JS wrapper object for a given C++ object, toJS function will find the appropriate JS wrapper in a hash map of the given DOMWrapperWorld. Because a hash map lookup is expensive, some WebCore objects will inherit from ScriptWrappable, which has an inline pointer to the JS wrapper for the main world if one was already created.

JS Wrapper Lifecycle Management

As a general rule, a JS wrapper object keeps its underlying C++ object alive by means of reference counting in JSDOMWrapper temple class from which all JS wrappers in WebCore inherits. However, C++ objects do not keep their corresponding JS wrapper in each world alive by the virtue of them staying alive as such a circular dependency will result in a memory leak.

There are two primary mechanisms to keep JS wrappers alive in WebCore:

• Visit Children - When JavaScriptCore’s garbage collection visits some JS wrapper during the marking phase, visit another JS wrapper or JS object that needs to be kept alive.
• Reachable from Opaque Roots - Tell JavaScriptCore’s garbage collection that a JS wrapper is reachable from an opaque root which was added to the set of opaque roots during marking phase.

FIXME: Explain how to add new IDL files and where derived sources are generated.

Visit Children

FIXME: Explain how visit children works.

Opaque Roots

FIXME: Explain how opaque roots work.

Inserting or Removing DOM Nodes

FIXME: Talk about how a node insertion or removal works.

Understanding Style and Render Tree

FIXME: Describe rendering/layers/compositing

Security Model of Web

For starters, refer to https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy.

FIXME: Write this section.

WebKit2: WebKit’s Multi-Process Architecture

Overview

In order to safeguard the rest of the system and allow the application to remain responsive even if the user had loaded web page that infinite loops or otherwise hangs, the modern incarnation of WebKit uses multi-process architecture. Web pages are loaded in its own WebContent process. Multiple WebContent processes can share a browsing session, which lives in a shared network process. In addition to handling all network accesses, this process is also responsible for managing the disk cache and Web APIs that allow websites to store structured data such as Web Storage API and IndexedDB API: Because a WebContent process can Just-in-Time compile arbitrary JavaScript code loaded from the internet, meaning that it can write to memory that gets executed, this process is tightly sandboxed. It does not have access to any file system unless the user grants an access, and it does not have direct access to the underlying operating system’s clipboard, microphone, or video camera even though there are Web APIs that grant access to those features. Instead, UI process brokers such requests.

FIXME: How is IPC setup

FIXME: How to add / modify an IPC message

Layout Tests: Tests of the Web for the Web

Layout tests are WebKit tests written using Web technology such as HTML, CSS, and JavaScript, and it’s the primary mechanism by which much of WebCore is tested. Relevant layout test should be ran while you’re making code changes to WebCore and before uploading a patch to bugs.webkit.org. While bugs.webkit.org’s Early Warning System will build and run tests on a set of configurations, individual patch authors are ultimately responsible for any test failures that their patches cause.

Test Files and Expected Files

Directory Structure

LayoutTests directory is organized by the category of tests. For example, LayoutTests/accessibility contains accessibility related tests, and LayoutTests/fast/dom/HTMLAnchorElement contains tests for the HTML anchor element.

Any file that ends in .html, .htm, .shtml, .xhtml, .mht, .xht, .xml, .svg, or .php is considered as a test unless it’s preceded with -ref, -notref, -expected, or -expected-mismatch (these are used for ref tests; explained later). It’s accompanied by another file of the same name except it ends in -expected.txt or -expected.png. These are called expected results and constitutes the baseline output of a given test. When layout tests are ran, the test runner generates an output in the form of a plain text file and/or an PNG image, and it is compared against these expected results.

In the case expected results may differ from one platform to another, the expected results for each test is stored in LayoutTests/platform. The expected result of a given test exists in the corresponding directory in each subdirectory of LayoutTests/platform. For example, the expected result of LayoutTests/svg/W3C-SVG-1.1/animate-elem-46-t.svg for macOS Mojave is located at LayoutTests/platform/mac-mojave/svg/W3C-SVG-1.1/animate-elem-46-t-expected.txt.

These platform directories have a fallback order. For example, running tests for WebKit2 on macOS Catalina will use the following fallback path from the most specific to most generic:

• platform/mac-catalina-wk2 - Results for WebKit2 on macOS Catalina.
• platform/mac-catalina - Results for WebKit2 and WebKitLegacy on macOS Catalina.
• platform/mac-wk2 - Results for WebKit2 on all macOS.
• platform/mac - Results for all macOS.
• platform/wk2 - Results for WebKit2 on every operating system.
• generic - Next to the test file.

Imported Tests

Tests under LayoutTests/imported are imported from other repositories. They should not be modified by WebKit patches unless the change is made in respective repositories first.

Most notable is Web Platform Tests, which are imported under LayoutTests/imported/w3c/web-platform-tests. These are cross browser vendor tests developed by W3C. Mozilla, Google, and Apple all contribute many tests to this shared test repository.

HTTP Tests

FIXME: Explain how to start and open tests that require HTTP server.

Test Expectations

FIXME: Explain how test expectations work.

Running Layout Tests

To run layout tests, use Tools/Scripts/run-webkit-tests. It optionally takes file paths to a test file or directory and options on how to run a test. For example, in order to just run LayoutTests/fast/dom/Element/element-traversal.html, do:

Tools/Scripts/run-webkit-tests fast/dom/Element/element-traversal.html

Because there are 50,000+ tests in WebKit, you typically want to run a subset of tests that are relevant to your code change (e.g. LayoutTests/storage/indexeddb/ if you’re working on IndexedDB) while developing the code change, and run all layout tests at the end on your local machine or rely on the Early Warning System on bugs.webkit.org for more thorough testing.

Specify --debug or --release to use either release or debug build. To run tests using iOS simulator, you can specify either --ios-simulator, --iphone-simulator, or --ipad-simulator based on whichever simulator is desired.

By default, run-webkit-tests will run all the tests you specified once in the lexicological order of test paths relative to LayoutTests directory and retry any tests that have failed. If you know the test is going to fail and don’t want retries, specify --no-retry-failures.

Because there are so many tests, run-webkit-tests will runs tests in different directories in parallel (i.e. all tests in a single directory is ran sequentially one after another). You can control the number of parallel test runners using --child-processes option.

run-webkit-tests has many options. Use --help to enumerate all the supported options.

Repeating Layout Tests

When you’re investigating flaky tests or crashes, it might be desirable to adjust this. --iterations X option will specify the number of times the list of tests are ran. For example, if we are running tests A, B, C and --iterations 3 is specified, run-webkit-tests will run: A, B, C, A, B, C, A, B, C. Similarly, --repeat-each option will specify the number of times each test is repeated before moving onto next test. For example, if we’re running tests A, B, C, and --repeat-each 3 is specified, run-webkit-tests will run: A, A, A, B, B, B, C, C, C. --exit-after-n-failures option will specify the total number of test failures before run-webkit-tests will stop. In particular, --exit-after-n-failures=1 is useful when investigating a flaky failure so that run-webkit-tests will stop when the failure actually happens for the first time.

Test Results

Whenever tests do fail, run-webkit-tests will store results in WebKitBuild/Debug/layout-test-results mirroring the same directory structure as LayoutTests. For example, the actual output produced for LayoutTests/editing/inserting/typing-001.html, if failed, will appear in WebKitBuild/Debug/layout-test-results/editing/inserting/typing-001-actual.txt. run-webkit-tests also generates a web page with the summary of results in WebKitBuild/Debug/layout-test-results/results.html and automatically tries to open it in Safari using the local build of WebKit.

If Safari fails to launch, specify --no-show-results and open results.html file manually.

Updating Expected Results

If you’ve updated a test content or test’s output changes with your code change (e.g. more test case passes), then you may have to update -expected.txt file accompanying the test. To do that, first run the test once to make sure the diff and new output makes sense in results.html, and run the test again with --reset-results. This will update the matching -expected.txt file.

You may need to manually copy the new result to other -expected.txt files that exist under LayoutTests for other platforms and configurations. Find other -expected.txt files when you’re doing this.

When a new test is added, run-webkit-tests will automatically generate new -expected.txt file for your test. You can disable this feature by specifying --no-new-test-results e.g. when the test is still under development.

Different Styles of Layout Tests

There are multiple styles of layout tests in WebKit.

Render tree dumps

This is the oldest style of layout tests, and the default mode of layout tests. It’s a text serialization of WebKit’s render tree and its output looks like this:

layer at (0,0) size 800x600
  RenderView at (0,0) size 800x600
layer at (0,0) size 800x600
  RenderBlock {HTML} at (0,0) size 800x600
    RenderBody {BODY} at (8,8) size 784x584
      RenderInline {A} at (0,0) size 238x18 [color=#0000EE]
        RenderInline {B} at (0,0) size 238x18
          RenderText {#text} at (0,0) size 238x18
            text run at (0,0) width 238: "the second copy should not be bold"
      RenderText {#text} at (237,0) size 5x18
        text run at (237,0) width 5: " "
      RenderText {#text} at (241,0) size 227x18
        text run at (241,0) width 227: "the second copy should not be bold"

This style of layout tests is discouraged today because its outputs are highly dependent on each platform, and end up requiring a specific expected result in each platform. But they’re still useful when testing new rendering and layout feature or bugs thereof.

These tests also have accompanying -expected.png files but run-webkit-tests doesn't check the PNG output against the expected result by default. To do this check, pass --pixel. Unfortunately, many pixel tests will fail because we have not been updating the expected PNG results a good chunk of the last decade. However, these pixel results might be useful when diagnosing a new test failure. For this reason, run-webkit-tests will automatically generate PNG results when retrying the test, effectively enabling --pixel option for retries.

dumpAsText test

These are tests that uses the plain text serialization of the test page as the output (as if the entire page’s content is copied as plain text). All these tests call testRunner.dumpAsText to trigger this behavior. The output typically contains a log of text or other informative output scripts in the page produced. For example, LayoutTests/fast/dom/anchor-toString.html is written as follows:

<a href="http://localhost/sometestfile.html" id="anchor">
A link!
</a>
<br>
<br>
<script>
    {
        if (window.testRunner)
            testRunner.dumpAsText();

        var anchor = document.getElementById("anchor");
        document.write("Writing just the anchor object - " + anchor);

        var anchorString = String(anchor);
        document.write("<br><br>Writing the result of the String(anchor) - " + anchorString);

        var anchorToString = anchor.toString();
        document.write("<br><br>Writing the result of the anchor's toString() method - " + anchorToString);
    }
</script>

and generates the following output:

A link! 

Writing just the anchor object - http://localhost/sometestfile.html

Writing the result of the String(anchor) - http://localhost/sometestfile.html

Writing the result of the anchor's toString() method - http://localhost/sometestfile.html

js-test.js and js-test-pre.js tests

These are variants of dumpAsText test which uses WebKit’s assertion library: LayoutTests/resources/js-test.js and LayoutTests/resources/js-test-pre.js. It consists of shouldX function calls which takes two JavaScript code snippet which are then executed and outputs of which are compared. js-test.js is simply a new variant of js-test-pre.js that doesn’t require the inclusion of LayoutTests/resources/js-test-post.js at the end. Use js-test.js in new tests, not js-test-pre.js.

For example, LayoutTests/fast/dom/Comment/remove.html which tests remove() method on Comment node is written as:

<!DOCTYPE html>
<script src="../../../resources/js-test-pre.js"></script>
<div id="test"></div>
<script>

description('This tests the DOM 4 remove method on a Comment.');

var testDiv = document.getElementById('test');
var comment = document.createComment('Comment');
testDiv.appendChild(comment);
shouldBe('testDiv.childNodes.length', '1');
comment.remove();
shouldBe('testDiv.childNodes.length', '0');
comment.remove();
shouldBe('testDiv.childNodes.length', '0');

</script>
<script src="../../../resources/js-test-post.js"></script>

with the following expected result (output):

This tests the DOM 4 remove method on a Comment.

On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".


PASS testDiv.childNodes.length is 1
PASS testDiv.childNodes.length is 0
PASS testDiv.childNodes.length is 0
PASS successfullyParsed is true

TEST COMPLETE

description function specifies the description of this test, and subsequent shouldBe calls takes two strings, both of which are evaluated as JavaScript and then compared.

Some old js-test-pre.js tests may put its test code in a separate JS file but we don’t do that anymore to keep all the test code in one place.

js-test.js and js-test-pre.js provide all kinds of other assertion and helper functions. Here are some examples:

• debug(msg) - Inserts a debug / log string in the output.
• evalAndLog(code) - Similar to debug() but evaluates code as JavaScript.
• shouldNotBe(a, b) - Generates PASS if the results of evaluating a and b differ.
• shouldBeTrue(code) - Shorthand for shouldBe(code, 'true').
• shouldBeFalse(code) - Shorthand for shouldBe(code, 'false').
• shouldBeNaN(code) - Shorthand for shouldBe(code, 'NaN').
• shouldBeNull(code) - Shorthand for shouldBe(code, 'null').
• shouldBeZero(code) - Shorthand for shouldBe(code, '0').
• shouldBeEqualToString(code, string) - Similar to shouldBe but the second argument is not evaluated as string.
• finishJSTest() - When js-test.js style test needs to do some async work, define the global variable named jsTestIsAsync and set it to true. When the test is done, call this function to notify the test runner (don’t call testRunner.notifyDone mentioned later directly). See an example.

It’s important to note that these shouldX functions only add output strings that say PASS or FAIL. If the expected result also contains the same FAIL strings, then run-webkit-tests will consider the whole test file to have passed.

Another way to think about this is that -expected.txt files are baseline outputs, and baseline outputs can contain known failures.

There is a helper script to create a template for a new js-test.js test. The following will create new test named new-test.html in LayoutTests/fast/dom:

Tools/Scripts/make-new-script-test fast/dom/new-test.html

dump-as-markup.js Tests

A dump-as-markup.js test is yet another variant of dumpAsText test, which uses LayoutTests/resources/dump-as-markup.js. This style of test is used when it’s desirable to compare the state of the DOM tree before and after some operations. For example, many tests under LayoutTests/editing use this style of testing to test complex DOM mutation operations such as pasting HTML from the users’ clipboard. dump-as-markup.js adds Markup on the global object and exposes a few helper functions. Like js-test.js tests, a test description can be specified via Markup.description. The test then involves Markup.dump(node, description) to serialize the state of DOM tree as plain text where element is either a DOM node under which the state should be serialized or its id.

For example, LayoutTests/editing/inserting/insert-list-in-table-cell-01.html is written as follows:

<!DOCTYPE html>
<div id="container" contenteditable="true"><table border="1"><tr><td id="element">fsdf</td><td>fsdf</td></tr><tr><td>gghfg</td><td>fsfg</td></tr></table></div>
<script src="../editing.js"></script>
<script src="../../resources/dump-as-markup.js"></script>
<script>
    Markup.description('Insert list items in a single table cell:');

    var e = document.getElementById("element");
    setSelectionCommand(e, 0, e, 1);
    Markup.dump('container', 'Before');

    document.execCommand("insertOrderedList");
    Markup.dump('container', 'After');
</script>

with the following expected result:

Insert list items in a single table cell:

Before:
| <table>
|   border="1"
|   <tbody>
|     <tr>
|       <td>
|         id="element"
|         "<#selection-anchor>fsdf<#selection-focus>"
|       <td>
|         "fsdf"
|     <tr>
|       <td>
|         "gghfg"
|       <td>
|         "fsfg"

After:
| <table>
|   border="1"
|   <tbody>
|     <tr>
|       <td>
|         id="element"
|         <ol>
|           <li>
|             "<#selection-anchor>fsdf<#selection-focus>"
|             <br>
|       <td>
|         "fsdf"
|     <tr>
|       <td>
|         "gghfg"
|       <td>
|         "fsfg"

testharness.js Tests

This is yet another variant of dumpAsText test which uses the test harness of Web Platform Tests, which is W3C’s official tests for the Web. There is an extensive documentation on how this harness works.

As mentioned above, do not modify tests in LayoutTests/imported/w3c/web-platform-tests unless the same test changes are made in Web Platform Tests’ primary repository.

Reference Tests

Reference tests are special in that they don’t have accompanying -expected.txt files. Instead, they have a matching or mismatching expected result file. Both the test file and the accompanying matching or mismatching expected result generate PNG outputs. The test passes if the PNG outputs of the test and the matching expected result are the same; the test fails otherwise. For a test with a mismatching expected result, the test passes if the PNG outputs of the test and the mismatching expected result are not the same, and fails if they are the same.

A matching expected result or a mismatching expected result can be specified in a few ways:

• The file with the same name as the test name except it ends with -expected.* or -ref.* is a matching expected result for the test.
• The file with the same name as the test name except it ends with -expected-mismatch.* or -notref.* is a matching expected result for the test.
• The file specified by a HTML link element in the test file with match relation: <link rel=match href=X> where X is the relative file path is a matching expected result.
• The file specified by a HTML link element in the test file with mismatch relation: <link rel=mismatch href=X> where X is the relative file path is a mismatching expected result.

For example, LayoutTests/imported/w3c/web-platform-tests/2dcontext/line-styles/lineto_a.html specifies lineto_ref.html in the same directory as the matching expected result as follows:

<!DOCTYPE html>
<meta charset=utf-8>
<link rel=match href=lineto_ref.html>
<style>
  html, body {
    margin: 0;
    padding: 0;
  }
</style>
<canvas id="c" width="150" height="150" >
Your browser does not support the HTML5 canvas tag.</canvas>

<script>
var c = document.getElementById("c");
var ctx = c.getContext("2d");

ctx.beginPath();
ctx.moveTo(20, 20);
ctx.lineTo(20, 130);
ctx.lineTo(130, 130);
ctx.lineTo(130, 20);
ctx.closePath();

ctx.fillStyle = '#90EE90';
ctx.fill();
</script>

Test Runners

Most layout tests are designed to be runnable inside a browser but run-webkit-tests uses a special program to run them. Our continuous integration system as well as the Early Warning System uses run-webkit-tests to run layout tests. In WebKit2, this is appropriately named WebKitTestRunner. In WebKit1 or WebKitLegacy, it’s DumpRenderTree, which is named after the very first type of layout tests, which generated the text representation of the render tree.

Extra Interfaces Available in Test Runners

Both WebKitTestRunner and DumpRenderTree expose a few extra interfaces to JavaScript on window (i.e. global object) in order to emulate user inputs, enable or disable a feature, or to improve the reliability of testing.

• GCController
  • GCController.collect() triggers a synchronous full garbage collection. This function is useful for testing crashes or erroneous premature collection of JS wrappers and leaks.
• testRunner
  • TestRunner interface exposes many methods to control the behaviors of WebKitTestRunner and DumpRenderTree. Some the most commonly used methods are as follows:
  • waitUntilDone() / notifyDone() - These functions are useful when writing tests that involve asynchronous tasks which may require the test to continue running beyond when it finished loading. testRunner.waitUntilDone() makes WebKitTestRunner and DumpRenderTree not end the test when a layout test has finished loading. The test continues until testRunner.notifyDone() is called.
  • dumpAsText(boolean dumpPixels) - Makes WebKitTestRunner and DumpRenderTree output the plain text of the loaded page instead of the state of the render tree.
  • overridePreference(DOMString preference, DOMString value) - Overrides WebKit’s preferences. For WebKit2, these preferences are defined in Source/WebKit/Shared/WebPreferences.yaml. For WebKitLegacy, these are defined in Source/WebKitLegacy/mac/WebView/WebPreferences.h for macOS and Source/WebKitLegacy/win/WebPreferences.h for Windows.
• eventSender
  • Exposes methods to emulate mouse, keyboard, and touch actions. Use ui-helpers.js script instead of directly calling methods on this function. This will ensure the test will be most compatible with all the test configurations we have.
• UIScriptController
  • Exposes methods to emulate user inputs like eventSender mostly on iOS WebKit2. Use ui-helpers.js script instead of directly calling methods on this function. This will ensure the test will be most compatible with all the test configurations we have.
• textInputController
  • Exposes methods to test input methods.

Additionally, WebCore/testing exposes a few testing hooks to test its internals:

• internals
  • Exposes various hooks into WebCore that shouldn’t be part of WebKit or WebKitLegacy API.
• internals.settings
  • Exposes various WebCore settings and let tests override them. Note that WebKit layer code depends on Source/WebKit/Shared/WebPreferences.yaml, and will not respect this override. Because of this, it’s preferable to override the equivalent preference via testRunner.overridePreference unless you know for sure WebKit or WebKitLegacy layer of code isn’t affected by the setting you’re overriding.

Enabling or Disabling a Feature in Test Runners

FIXME: Mention test-runner-options

Test Harness Scripts

FIXME: Write about dump-as-markup.js, and ui-helper.js

Investigating Test Failures Observed on Bots

There are multiple tools to investigate test failures happening on our continuous integration system (build.webkit.org). The most notable is flakiness dashboard: results.webkit.org

FIXME: Write how to investigate a test failure.

Debugging Layout Tests in Xcode

The easiest way to debug a layout test is with WebKitTestRunner or DumpRenderTree. In Product > Scheme, select “All Source”.

In Product > Scheme > Edit Scheme, open “Run” tab. Pick WebKitTestRunner or DumpRenderTree, whichever is desired in “Executable”.

Go to Arguments and specify the path to the layout tests being debugged relative to where the build directory is located. e.g. ../../LayoutTests/fast/dom/Element/element-traversal.html if WebKitBuild/Debug is the build directory. You may want to specify OS_ACTIVITY_MODE environmental variable to “disable” in order to suppress all the system logging that happens during the debugging session.

You may also want to specify --no-timeout option to prevent WebKitTestRunner or DumpRenderTree to stop the test after 30 seconds if you’re stepping through code.

Once this is done, you can run WebKitTestRunner or DumpRenderTree by going to Product > Perform Action > Run without Building.

Clicking on “Run” button may be significantly slower due to Xcode re-building every project and framework each time. You can disable this behavior by going to “Build” tab and unchecking boxes for all the frameworks involved for “Run”:

Attaching to WebContent Process

You may find Xcode fails to attach to WebContent or Networking process in the case of WebKitTestRunner. In those cases, attach a breakpoint in UIProcess code such as TestController::runTest in WebKitTestRunner right before TestInvocation::invoke is called. Once breakpoint is hit in the UIProcess, attach to WebContent.Development or Networking.Development process manually in Xcode via Debug > Attach to Process.

Dive into API tests

FIXME: Talk about how to debug API tests.

Logging in WebKit

Setup

Each framework (WebCore, WebKit, WebKitLegacy, WTF) enable their own logging infrastructure independently (though the infrastructure itself is shared). If you want to log a message, #include the relevant framework's Logging.h header. Then, you can use the macros below.

Beware that you can‘t #include multiple framework’s Logging.h headers at the same time - they each define a macro LOG_CHANNEL_PREFIX which will conflict with each other. Only #include the Logging.h header from your specific framework.

If you want to do more advanced operations, like searching through the list of log channels, #include your framework's LogInitialization.h header. These do not conflict across frameworks, so you can do something like

#include "LogInitialization.h"
#include <WebCore/LogInitialization.h>
#include <WTF/LogInitialization.h>

Indeed, WebKit does this to initialize all frameworks' log channels during Web Process startup.

Logging messages

There are a few relevant macros for logging messages:

• LOG(): Log a printf-style message in debug builds. Requires you to name a logging channel to output to.
• LOG_WITH_STREAM() Log an iostream-style message in debug builds. Requires you to name a logging channel to output to.
• RELEASE_LOG(): Just like LOG() but logs in both debug and release builds. Requires you to name a logging channel to output to.
• WTFLogAlways(): Mainly for local debugging, unconditionally output a message. Does not require a logging channel to output to.

Here's an example invocation of LOG():

LOG(MediaQueries, "HTMLMediaElement %p selectNextSourceChild evaluating media queries", this);

That first argument is a log channel. These have 2 purposes:

• Individual channels can be enabled/disabled independently (So you can get all the WebGL logging without getting any Loading logging)
• When multiple channels are enabled, and you're viewing the logs, you can search/filter by the channel

Here's an example invocation of LOG_WITH_STREAM():

LOG_WITH_STREAM(Scrolling, stream << "ScrollingTree::commitTreeState - removing unvisited node " << nodeID);

The macro sets up a local variable named stream which the second argument can direct messages to. The second argument is a collection of statements - not expressions like LOG() and RELEASE_LOG(). So, you can do things like this:

LOG_WITH_STREAM(TheLogChannel,
    for (const auto& something : stuffToLog)
        stream << " " << something;
);

The reason why (most of) these use macros is so the entire thing can be compiled out when logging is disabled. Consider this:

LOG(TheLogChannel, "The result is %d", someSuperComplicatedCalculation());

If these were not macros, you'd have to pay for someSuperComplicatedCalculation() whether logging is enabled or not.

Enabling and disabling log channels

Channels are enabled/disabled at startup by passing a carefully crafted string to initializeLogChannelsIfNecessary(). On the macOS and iOS ports, this string comes from the defaults database. On other UNIX systems and Windows, it comes from environment variables.

You can read the grammar of this string in initializeLogChannelsIfNecessary(). Here is an example:

WebGL -Loading

You can also specify the string all to enable all logging.

On macOS/iOS and Windows, each framework has its own individually supplied string that it uses to enable its own logging channels. On Linux, all frameworks share the same string.

Linux

Set the WEBKIT_DEBUG environment variable.

WEBKIT_DEBUG=Scrolling Tools/Scripts/run-minibrowser --gtk --debug

macOS

On macOS, you can, for example, enable the Language log channel with these terminal commands:

for identifier in com.apple.WebKit.WebContent.Development com.apple.WebKit.WebContent org.webkit.MiniBrowser com.apple.WebKit.WebKitTestRunner org.webkit.DumpRenderTree -g /Users/$USER/Library/Containers/com.apple.Safari/Data/Library/Preferences/com.apple.Safari.plist; do
    for key in WTFLogging WebCoreLogging WebKitLogging WebKit2Logging; do
        defaults write ${identifier} "${key}" "Language"
    done
done

You may also need to specify these strings to com.apple.WebKit.WebContent.Development, the global domain, or the Safari container, depending on what you're running.

You may also pass this key and value as an argument:

Tools/Scripts/run-minibrowser --debug -WebCoreLogging Scrolling

Windows

Set the WebCoreLogging environment variable.

Adding a new log channel

Simply add a line to your framework's Logging.h header. Depending on how the accompanying Logging.cpp file is set up, you may need to add a parallel line there. That should be all you need. It is acceptable to have log channels in different frameworks with the same name - this is what LOG_CHANNEL_PREFIX is for.