At a glance
- Decide what you are measuring and why before you test.
- Record the environment with every result: URL, variant, device profile, network setting, browser version, date and time.
- Run lab tests several times and report the median, not the best run.
- Label every number as field or lab, and say what was not tested.
1. Define the question
Write one sentence: which template, which metric, which audience. For example, "Does the mobile lobby in the UK pass LCP for logged-out visitors?" A precise question decides which data you need and stops a single Lighthouse score from standing in for a whole site.
2. Record the environment
| Field | Example |
|---|---|
| URL and template | /lobby/ (lobby template) |
| Variant | Market, language, logged in or out, consent state |
| Data type | Field (source, period) or lab (tool) |
| Device and network | Lighthouse standard mobile throttling: 150 ms latency, 1.6 Mbps down, 4x CPU |
| Browser and tool version | As shown by the tool |
| Date and time | With time zone |
| Cache state | Cold or warm |
| Runs | Number of runs and the statistic reported (median) |
3. Lab protocol
- Use a fixed profile. Lighthouse's standard mobile throttling is a common reference point.
- Run at least five times and report the median for each metric. Chrome's Lighthouse documentation notes that "a lot of the variability" between runs "is not due to Lighthouse", so single runs mislead.
- Save the trace or report file for each run you cite.
- Change one thing at a time, then repeat with the same settings.
- Call the result a lab result. It shows that a change helped in the lab, not that players will see the same.
4. Field protocol
- Use CrUX through PageSpeed Insights, Search Console or the CrUX API for public pages, noting that PageSpeed Insights reports the previous 28 days at the 75th percentile.
- Remember who CrUX excludes: Chrome on iOS, Android WebView and other Chromium browsers.
- For logged-in pages and iPhone visitors, collect your own field data. Google's open source web-vitals library measures the Core Web Vitals "on real users" and offers an attribution build for diagnosis.
- Segment by template, device type and country where you can.
- After a release, wait for the full 28-day window before comparing.
5. Report with limits
Every report should state, in plain words, what the data covers and what it does not: which variants, which devices, which period, and whether any part is estimated. CasinoPerflyn's guides end with a "What we did not test" note for this reason. If you publish benchmarks, publish the method too.
What CasinoPerflyn does and does not do
CasinoPerflyn has not run measurement campaigns on operators' sites and does not publish scores for named operators. Our guides explain documented mechanisms and methods. If we publish measurements in future, they will follow this protocol and include the raw records.
Questions
Why the median and not the average?
Lab runs sometimes include outliers from network or CPU noise. The median is less affected by one bad or one lucky run.
Can we compare our site with a competitor's?
Use public field data for both and compare like with like: same template type, device and period. Avoid lab comparisons of sites you do not control, because variants and consent states differ.
How long should a before-and-after study run?
For field data, at least the 28-day window after the change. For lab data, as long as it takes to collect enough runs under identical settings.
Sources
- Google for Developers, About PageSpeed Insights, last updated October 21, 2024.
- Chrome for Developers, CrUX methodology, last updated June 20, 2024.
- Chrome for Developers, How Lighthouse calculates your overall Performance score, footer date September 19, 2019.
- GoogleChrome, Lighthouse throttling documentation, no date shown.
- GoogleChrome, web-vitals library, no date shown.