Using The WebPageTest API with Python

The official WebPageTest API wrapper for NodeJS is awesome. Still I’ve found it equally fun and easy to use Python.

In this post, I’ll share with you how I use Python 3 and the WebPageTest API to:

• run a test with default settings
• run tests using any custom settings
• run scripted tests
• get test results
• run batch tests

Getting started

Get a WebPageTest API key.

Then create a directory dir containing at least these files:

dir/
	┝ config.py
	┝ script.py
	└ .gitignore

In config.py you can set your API key.

wpt_api_key = "sOmE-aPi-KeY"

Then add config.py to your .gitignore file to avoid sharing your key with the world on Git.

Run a test with default settings

With this script, you can run a test on a URL with default WebPageTest settings (i.e. “1366x768 desktop browser dimensions, Virginia USA - EC2 - Chrome - Cable”).

#!/usr/bin/env python3
import requests
import config

url = "https://www.example.com"  # The URL you want to run a test on
endpoint = f'https://www.webpagetest.org/runtest.php?k={config.wpt_api_key}&url={url}&f=json'
response = requests.get(endpoint)
print(response.json())

The JSON response containing links to the test results are logged to console.

~ python3 run-default-test.py
{'statusCode': 200, 'statusText': 'Ok', 'data': {'testId': 'SOME_TEST_ID', 'jsonUrl': 'https://www.webpagetest.org/jsonResult.php?test=SOME_TEST_ID', 'xmlUrl': 'https://www.webpagetest.org/xmlResult/SOME_TEST_ID/', 'userUrl': 'https://www.webpagetest.org/result/SOME_TEST_ID/', 'summaryCSV': 'https://www.webpagetest.org/result/SOME_TEST_ID/page_data.csv', 'detailCSV': 'https://www.webpagetest.org/result/SOME_TEST_ID/requests.csv'}}

Run tests using custom settings

To apply custom test settings (e.g. “Frankfurt, Germany - EC2 - Chrome - Emulated Motorola G (gen 4) - 4G - Mobile”), set some options (see API docs) and pass them as params in the request.

#!/usr/bin/env python3
import requests
import config

test_url = "https://www.example.com"  # The URL you want to run a test on

test_options = {
    "label": "My Label",
    "k": config.wpt_api_key,
    "f": "json",
    "runs": 1,
    # Optionally set connectivity after the location. You can also specify a browser using the format 'ec2-eu-central-1:Firefox.4G'
    "location": "ec2-eu-central-1.4G",
    "mobile": 1,
}

endpoint = f'https://www.webpagetest.org/runtest.php?url={test_url}'
response = requests.get(endpoint, params=test_options)

# Do something with the response
print(response.json())

Run scripted tests

To run a scripted test, you can write the tab-delimited commands as a multi-line string literal. By the way, I’ve found that with the Node wrapper, you can also pass the script commands using a string literal instead of using the scriptToString(script) method.

#!/usr/bin/env python3
import requests
import config

test_url = "https://www.example.com"  # The URL you want to run a test on

script = """
overrideHost	*.example.com	some-edge-script.some-user.workers.dev
setHeader	x-bypass-transform: false
navigate	%URL%
"""

test_options = {
    "k": config.wpt_api_key,
    "f": "json",
    "script": script
    # Add additional options here
}

endpoint = f'https://www.webpagetest.org/runtest.php?url={test_url}'
response = requests.get(endpoint, params=test_options)

# Do something with the response
print(response.json())

Get test results

To get the final results of a WebPageTest as JSON, you need to poll the endpoint https://www.webpagetest.org/jsonResult.php periodically using the time module. Note that the parameters &average=0&standard=0&requests=0&lighthouse=0 remove the data on average, standard deviation, requests and lighthouse to keep the size of the response smaller.

#!/usr/bin/env python3
import requests
import config
import time

test_url = "https://www.example.com"  # The URL you want to run a test on

test_options = {
    "k": config.wpt_api_key,
    "f": "json",
}


def run_test_and_return_id(url):
    endpoint = f'https://www.webpagetest.org/runtest.php?url={url}'
    response = requests.get(
        endpoint, params=test_options)
    print("Test Results: ", response.json()['data']['userUrl'])
    return response.json()['data']['testId']


def get_test_results(id):
    response = requests.get(
        f'https://www.webpagetest.org/jsonResult.php?test={id}&average=0&standard=0&requests=0&lighthouse=0').json()
    return response


def poll_for_test_results(id):
    time.sleep(10)
    get_test_results(id)
    status = False
    while not status:
        if (get_test_results(id)['statusCode'] != 200):
            get_test_results(id)
            print(get_test_results(id)['statusText'])
            time.sleep(5)
        else:
            status = True
            return get_test_results(id)


id = run_test_and_return_id(test_url)

# Do something with the test results
print(poll_for_test_results(id))

Run batch tests

To run a series of WebPageTest tests on different URLs using the same settings, simply store the URLs in a list and loop through the list using the methods above for getting test results.

...

test_urls = [
    "https://www.example.com",
    "https://www.example.com/page-1",
    "https://www.example.com/page-2",
    "https://www.example.com/page-3",
]

...

for test_url in test_urls:
    print(f'START test for {test_url}')
    id = run_test_and_return_id(test_url)
    print(poll_for_test_results(id))

Other fun things you can do

Once you’ve gotten the WebPageTest results as JSON, you can do fun things like export metrics to CSV or write them to a database for monitoring them over time (e.g. in Grafana).