Beacon and page lifecycle stage reliability

Remediating data loss with same-site destination for events and more immediate sending

NoteExecutive summary

In March 2026, the Experiment Platform team conducted two tests of potential improvements to the Test Kitchen JavaScript SDK. The changes were meant to address the problem of event loss, where not all events produced by clients (browsers) would be transmitted. Analysis of the data collected yielded the following recommendations:

• Send all events to the new same-site path (already implemented)
• Disable internal queuing mechanism and send events immediately (T384687)
• Conduct further investigation of page lifecycle reliability

Sending events to the new same-site path yielded up to 23% more events among logged-in users (compared to 7.5% among logged-out) and up to 15% more events among desktop users (compared to 4.5% among mobile visitors). Sidestepping the queuing mechanism also yielded more events, but by a smaller amount.

Background

The Experiment Platform team at the Wikimedia Foundation is building Test Kitchen, a platform which provides a set of tools and services that helps Wikimedia teams make data-informed decisions about product experiences. Test Kitchen may be thought of as an opinionated Event Platform-based system, making web analytics instrumentation more accessible to engineers.

While experimentation is certainly a major focus for us, Test Kitchen SDKs and systems are also used for collecting data about product usage. We seek to make data collection as reliable as we can, not just in the context of experiments, but product analytics in general. To that end, we worked with our colleagues in the Data Engineering team to create a new destination URI to send web beacons to.

Event loss due to blocked destination

There is an external path – https://intake-analytics.wikimedia.org/v1/events – that events produced by Event Platform clients are sent to, whether they are coming from legacy EventLogging-based instruments on the web or the Wikipedia mobile app instruments. This external path is on many blocklists that users of ad and tracker blockers can subscribe to, which has the following negative consequences:

• Limits visibility and biases results
  • Analytics (experiment or otherwise) are not representative of the whole population of users.
  • Experiment results can be biased and not completely generalizable.
• Wastes survey responders’ time
  • Wikimedia Foundation uses an in-house developed tool called QuickSurveys to conduct lightweight on-site surveys.
  • A user might answer a survey but their responses may get blocked by the user’s ad/tracker blocker (cf. T404476).

Our previous research (Ming 2025) has shown that we deal with approximately 20% loss of events, which varies substantially across different segments of users. Logged-in users – who predominately use desktop devices – have the highest event loss.

Notably, this is mainly an issue for experiments with logged-in users and non-experimental data because data collected from experiments on all user traffic already goes through a special path in order for events to be associated with unique cookie-derived, per-experiment subject identifier. It is important for product teams working in the Contributors space – who develop features and interventions for editors and users with extended rights – to have maximum visibility into their users’ behavior to make sure that user experience is not adversely affected and the experimental work does not have any negative impact.

Event loss due to queuing mechanisms

Besides a blocked destination URI, event loss is also affected by queuing mechanisms. When data is sent via sendBeacon, “the data is transmitted asynchronously when the user agent has an opportunity to do so, without delaying unload or the next navigation” – in other words, the data goes into a browser-managed queue. However, before sendBeacon() is called, events produced by Event Platform JS clients on the web actually go into a website-managed queue that was introduced in early 2020 (cf. T225578).

This queuing mechanism exists in:

• Test Kitchen JS SDK’s Instrument#send() and Experiment#send() methods
• mw.eventLog.submit() (used by non-TK client-side instrumentation)

The basic premise of this mechanism was to limit how frequently our analytics instrumentation would force the radio antenna on mobile devices to wake up to transmit data, and we hypothesized that frequent wake-ups drained users’ batteries. By enqueuing multiple events to be sent together – not as a batch with a single network call but as a burst of network calls – we would improve battery usage for our mobile users. This theory was never validated nor were the initially chosen parameters (for time between bursts) optimized.

Crucially, the queuing mechanism reacts when the tab is being closed and tries to flush the queue, converting all the queued events into sendBeacon() calls, which themselves are also queued into the browser’s queue. Therefore, it is possible for some events to be lost simply because they would never get transmitted.

Remediating event loss

We consider two possible changes to the platform:

• Sending events to a new path (same-site destination URI)
• Sending events immediately as they happen via sendImmediately(), rather than queuing them via send()

Rather than just implementing these changes, we decided to conduct small experiments and gather data to inform our decisions. We wanted to see whether implementing these changes would remedy event loss.

Methods

We used Test Kitchen’s experimentation capabilities to sample a small proportion of user traffic to 8 language editions of Wikipedia. We did not attempt to do a comprehensive research/study across the nearly one-thousand-website portfolio, but just enough to obtain a signal. The Wikipedias were chosen to yield some diversity of languages and cultures, but we were not exhaustive in our selection.

We sampled 0.1% of edge uniques on English Wikipedia and 10% of edge uniques on Chinese Wikipedia, French Wikipedia, Indonesian Wikipedia, Japanese Wikipedia, Portuguese Wikipedia, Russian Wikipedia, and Spanish Wikipedia.

Since the sampling employed edge uniques and we used JavaScript (JS) to collect data, we must note that:

• The populations of those 10 Wikipedias are not sampled independently of each other. A visitor who is in the 0.1% sample of English Wikipedia would be in the 10% sample on the other Wikipedias they visit, because the edge unique cookie is shared by all `.wikipedia.org domains.
• The client (browser) must allow cookies. If the client rejects all cookies, it cannot be sampled.¹
• The client (browser) must have JS enabled. There are many reasons why a client would have JS disabled – such as security, performance, and bandwidth considerations – but such clients represent a very small portion of overall traffic. The vast majority of traffic to our sites have JS enabled.

Once a user was enrolled in the experiment, we produced simple page visit events on their visits while they were in the experiment. No personally identifying information (IP address or the URI of the visited page) was collected with these events. User agent strings were collected for the purposes of segmenting traffic by browser to understand how remediation impact differed across popular web browsers.

In one round of data collection, we instrumented as follows:

Path	Send method
Old (external)	navigator.sendBeacon()
New (same-site)	navigator.sendBeacon()
New (same-site)	submitInteraction()

The method submitInteraction() was switched to the new path some weeks prior in order to support bot detection and anti-abuse work, so we aimed for an apples-to-apples comparison by using sendBeacon() method with two different destination URIs.

Note

The full source code for the first round of data collection is available at this commit.

In a follow-up round of data collection we wanted to understand reliability of pagehide browser events, which are fired when the browser starts unloading a page if the user is navigating to another page or if the tab is being closed (Walton 2023). Prior research has shown high variability in reliability of these events across different browsers, and substantially lower reliability among mobile browsers specifically (Jansma 2020).

Since the path that events are sent to was not the question for this round, all events were sent to the new path.

Note

The full source code for the second round of data collection is available at this commit.

Results

New vs old event path reliability

TipRecommendation

All data collected by Test Kitchen’s JS SDK should be sent to the new same-site URI to remedy event loss.

Table 1 shows volume of events by path across different segments of users and the relative difference of the new path compared to the old path. Across the segments we looked at, the new path yielded between 4.5% and 23.3% more events. We saw the largest increases on desktop and among logged-in users (either permanent or temporary users), while the smallest increases were among logged-out users and users browsing the mobile version of the website.

	Event counts by path		Relative difference
	Old path	New path
Overall
-	61.0M	65.7M	+7.6%
By platform
Desktop	16.5M	19.2M	+15.8%
Mobile	44.5M	46.5M	+4.5%
By user authentication status
Logged-out user	60.2M	64.7M	+7.4%
Permanent or temporary user	785.3K	968.7K	+23.3%
By language edition of Wikipedia
Chinese	5.8M	6.2M	+7.3%
English	1.2M	1.4M	+10.7%
French	10.4M	11.5M	+10.0%
Indonesian	908.7K	975.1K	+7.3%
Japanese	17.1M	18.2M	+6.9%
Portuguese	3.0M	3.3M	+8.2%
Russian	12.4M	13.3M	+7.2%
Spanish	10.2M	10.8M	+6.3%

Table 1: Volume of events by path across different segments of users and the relative difference of the new path compared to the old path.

Event submitter queue reliability

TipRecommendation

Stop using an internal queuing mechanism and send analytics events immediately (T384687).

In other words, make the behavior of send() the same as that of sendImmediately(), and then deprecate/remove sendImmediately().

Table 2 shows volume of events by send method, comparing the submitInteraction() method in TK JS SDK which uses a queuing mechanism to navigator.sendBeacon() – to reveal how much the queueing mechanism contributes to event loss. As expected, the queuing mechanism does contribute to event loss, but it is a minor contribution. Most of the analyzed segments showed an increase of 0.5-2%, except Chinese Wikipedia traffic which showed an increase of 5.2%.

	Event counts by send method		Relative difference
	TK JS SDK’s submitInteraction	navigator.sendBeacon
Overall
-	64.6M	65.7M	+1.6%
By platform
Desktop	18.9M	19.2M	+1.1%
Mobile	45.7M	46.5M	+1.9%
By user authentication status
Logged-out user	63.6M	64.7M	+1.7%
Permanent or temporary user	962.3K	968.7K	+0.7%
By language edition of Wikipedia
Chinese	5.9M	6.2M	+5.2%
English	1.3M	1.4M	+1.8%
French	11.3M	11.5M	+1.8%
Indonesian	964.3K	975.1K	+1.1%
Japanese	18.1M	18.2M	+1.0%
Portuguese	3.2M	3.3M	+1.6%
Russian	13.1M	13.3M	+0.9%
Spanish	10.7M	10.8M	+1.6%

Table 2: Volume of events by send method to reveal how much the queueing mechanism contributes to event loss.

Page lifecycle reliability

TipRecommendation

• Stop using an internal queuing mechanism and send analytics events immediately (T384687).
• Conduct a deeper investigation of the pagehide and visibilitychange events.

Table 3 shows volume of events by method and compares their reliability across two page lifecycle events: (1) page load and (2) pagehide. It is surprising to see sendImmediately perform ever-so-slightly worse than send but could potentially be explained by the execution order. When the page starts unloading and a pagehide browser event fires, it triggers flushing of the send queue and a call to sendImmediately. However, behind the scenes, both of those make calls navigator.sendBeacon and those requests go into the browser’s queue to be processed in the background. If Test Kitchen’s queue is flushed – which may include events from other data collection activities such as active experiments – with those events going into the browser’s queue, and then sendImmediately pushes another event to the end of the browser’s queue, then in order for a sendImmediately-sent event to actually make it to our servers, the browser has to process all the requests in its queue. If the browser does not finish processing the whole queue, some events would be lost.

	Page load events by TK JS SDK method			Pagehide events by TK JS SDK method
	send	sendImmediately	Relative difference	send	sendImmediately	Relative difference
Overall
-	67.4M	68.7M	+2.0%	64.8M	64.7M	−0.1%
By platform
Desktop	18.8M	19.5M	+4.0%	20.3M	20.3M	−0.2%
Mobile	48.6M	49.2M	+1.3%	44.5M	44.4M	−0.1%

Table 3: Volume of events by method and their reliability across two page lifecycle events (page load and pagehide).

By browser family

TipRecommendation

Further investigation into client-side page lifecycle reliability, using server-side instrumentation to establish a reference.

Figure 1 compares volume of events across the most popular browsers (as determined by internal pageview data). Curiously, some browsers showed a greater volume of pagehide events (sent when browser hides the current page) compared to page_load events (sent when page loads and the JS runs).

In theory we should always have fewer pagehide events than page_load events, so the fact that we are seeing more in some cases means that page_load events are also not reliable. Given the conflicting directionalities of the results across browsers, possible future work would be to track page visits with server-side instrumentation to establish a trustworthy denominator for calculation of page_load and pagehide client-side events’ arrival/missingness rates and be able to compare them more directly via those rates.

However, we will also need to think about how to deal with the problem that any instrumentation we develop will not exist in a vacuum – that there is other instrumentation active on any given page. If Test Kitchen’s JS SDK was updated to remove the queuing mechanism, if our TK-based instrumentation produces a pagehide or visibilitychange event when the user navigates away from the page, an immediately-sent event will still compete with events coming from a queue flush of non-TK instrumentation.

Figure 1: Events by browser family and send method

Acknowledgements

We would like to thank Dan Andreescu and Thomas Chin on implementing the new event path.

Tools

This report was created with R and Quarto. Data was queried using PySpark and wmfdata Python packages. Data analysis and presentation was done using R packages gt and packages from the tidyverse collection. No AI was used in the production of this report.

References

Jansma, Nic. 2020. “Beaconing In Practice.” In NicJ.net. https://nicj.net/beaconing-in-practice/#beaconing-reliability.

Ming, Clare. 2025. Traffic, Instrumentation, Experiment, and Data Analysis for Event Loss. Wikimedia Foundation.

Walton, Philip. 2023. “Page Lifecycle API.” In Chrome for Developers. https://developer.chrome.com/docs/web-platform/page-lifecycle-api/.

Footnotes

1. See also Edge Uniques technical design document.