commit	71b1fca68a8433e698627dc3b38bf058cfbb8fa9	[log] [tgz]
author	carlosgc@webkit.org <carlosgc@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>	Fri Sep 01 06:36:32 2017 +0000
committer	carlosgc@webkit.org <carlosgc@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>	Fri Sep 01 06:36:32 2017 +0000
tree	c0a9a88cbe5cc5c50750c7427501f897ce0ab219
parent	2b2dbf707668cd79507380311bed6590bf411d7e [diff]

[GTK] Improve the way unit test are run and the results reported
https://bugs.webkit.org/show_bug.cgi?id=176104

Reviewed by Carlos Alberto Lopez Perez.

There are several issues with the way unit tests are run by run-gtk-tests and also with the way results are
reported:

- The results summary only mentions the test binaries, not the actual test cases, so you always have to scroll
up to find the actual test cases failing.
- The number of reported failures is the number of test binaries that failed, so if a new test case fails for
the same binary in a new revision, we won't notice it just looking at the number of failures.
- We show detailed information about skipped test in the results summary, which is just noise.
- In the case of glib tests, when a test case times out, we finish the test suite, instead of continuing with the
rest of the test cases like we do for normal failures or crashes. If a new test case fails after a test case that
timed out we will not notice it until we fix or skip the test cases timing out.
- In the case of glib tests, the timeout is applied to the whole suite, instead of per test case, we have a hack
to make it longer only for that. It has worked so far, but it doesn't scale, and it's an ugly hack.
- It's not currently possible to detect flaky tests, because again, we know the binaries/suites that failed but
not the actual test cases.

This patch fixes all these issues and makes it possible to add support for flaky tests in a follow up patch.

* BuildSlaveSupport/build.webkit.org-config/master.cfg:
(RunGtkAPITests.commandComplete): Update the RunGtkAPITests step to parse the new output.
* Scripts/run-gtk-tests:
(TestRunner._start_timeout): Helper to start the timeout if needed.
(TestRunner._start_timeout._alarm_handler): Raise timeout exception.
(TestRunner._stop_timeout): Helper to stop the timeout if needed.
(TestRunner._waitpid): Merged waitpid and return_code_from_exit_status.
(TestRunner._run_test_glib): Do not double the timeout anymore, we now start/stop the timeout for every test
case. Return a dictionary where keys are test cases and values the results only in case of failures.
(TestRunner._run_test_glib.parse_line): Update the parser to also detect test cases and the results.
(TestRunner._run_test_glib.parse_line.set_test_result): Helper to set the result of a test case.
(TestRunner._run_google_test): Return a dictionary where key is the test case and value is the result only in
case of failure.
(TestRunner._run_google_test_suite): Updated now that _run_google_test returns a dictionary.
(TestRunner.run_tests): Handle the results dictionary and show the results with information about test cases
failing and grouped by test binaries.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@221474 268f45cc-cd09-0410-ab3c-d52691b4dbfc

3 files changed

tree: c0a9a88cbe5cc5c50750c7427501f897ce0ab219

ReadMe.md

WebKit

WebKit is a cross-platform web browser engine. On iOS and macOS, it powers Safari, Mail, iBooks, and many other applications.

Feature Status

Visit WebKit Feature Status page to see which Web API has been implemented, in development, or under consideration.

Trying the Latest

Downloading Safari Technology Preview to test the latest version of WebKit.

Reporting Bugs

Search WebKit Bugzilla to see if there is an existing report for the bug you've encountered.
Create a Bugzilla account to to report bugs (and to comment on them) if you haven't done so already.
File a bug in accordance with our guidelines.

Once your bug is filed, you will receive email when it is updated at each stage in the bug life cycle. After the bug is considered fixed, you may be asked to download the latest nightly and confirm that the fix works for you.

Getting the Code

On Windows, follow the instructions on our website.

Cloning the Git SVN Repository

Run the following command to clone WebKit's Git SVN repository:

git clone git://git.webkit.org/WebKit.git WebKit

If you want to be able to commit changes to the repository, or just want to check out branches that aren’t contained in WebKit.git, you will need track WebKit's Subversion repository. You can run the following command to configure this and other options of the new Git clone for WebKit development.

Tools/Scripts/webkit-patch setup-git-clone

For information about this, and other aspects of using Git with WebKit, read the wiki page.

Checking out the Subversion Repository

Run the following command to check out WebKit's subversion repository:

svn checkout https://svn.webkit.org/repository/webkit/trunk WebKit

Building WebKit

Building Mac Port

Install Xcode and its command line tools if you haven't done so already:

Install Xcode Get Xcode from https://developer.apple.com/downloads. To build WebKit for OS X, Xcode 5.1.1 or later is required. To build WebKit for iOS Simulator, Xcode 7 or later is required.
Install the Xcode Command Line Tools In Terminal, run the command: xcode-select --install

Run the following command to build a debug build with debugging symbols and assertions:

Tools/Scripts/build-webkit --debug

For performance testing, and other purposes, use --release instead.

Using Xcode

You can open WebKit.xcworkspace to build and debug WebKit within WebKit.

If you don't use a custom build location in Xcode preferences, you have to update the workspace settings to use WebKitBuild directory. In menu bar, choose File > Workspace Settings, then click the Advanced button, select “Custom”, “Relative to Workspace”, and enter WebKitBuild for both Products and Intermediates.

Building iOS Port

The first time after you install a new Xcode, you will need to run the following command to enable Xcode to build command line tools for iOS Simulator:

 sudo Tools/Scripts/configure-xcode-for-ios-development

Without this step, you will see the error message: “target specifies product type ‘com.apple.product-type.tool’, but there’s no such product type for the ‘iphonesimulator’ platform.” when building target JSCLLIntOffsetsExtractor of project JavaScriptCore.

Run the following command to build a debug build with debugging symbols and assertions for iOS:

Tools/Scripts/build-webkit --debug --ios-simulator.

Building GTK+ Port

Install the dependencies by running the following command:

Tools/gtk/install-dependencies

Then run the following command to build additional dependencies:

Tools/Scripts/update-webkitgtk-libs

Run the following command to build WebKit with debugging symbols for GTK+ port:

Tools/Scripts/build-webkit --debug --gtk

Note that the procedure for building a release tarball is different. For more information, see the wiki page.

Building Windows Port

For building WebKit on Windows, see the wiki page.

Running WebKit

With Safari and Other macOS Applications

Run the following command to launch Safari with your local build of WebKit:

Tools/Scripts/run-safari --debug

The run-safari script sets the DYLD_FRAMEWORK_PATH environment variable to point to your build products, and then launches /Applications/Safari.app. DYLD_FRAMEWORK_PATH tells the system loader to prefer your build products over the frameworks installed in /System/Library/Frameworks.

To run other applications with your local build of WebKit, run the following command:

Tools/Scripts/run-webkit-app <application-path>

iOS Simulator

Run the following command to launch iOS simulator with your local build of WebKit:

run-safari --debug --ios-simulator

In both cases, if you have built release builds instead, use --release instead of --debug.

Contribute

Congratulations! You’re up and running. Now you can begin coding in WebKit and contribute your fixes and new features to the project. For details on submitting your code to the project, read Contributing Code.