The query_explorer plugin can generate a variety of term statistic based features. We did this AB test to try them out and see which are useful to include in training.

This test ran from 02 March 2018 to 15 March 2018 on enwiki. There were 3 test groups: control, classic, explorer. This report includes fulltext searches. Refer to Phabricator ticket T189843 for more details.

Data Clean-up

Fulltext search events: Deleted 5530 duplicated events. Deleted 2133419 unnecessary check-in events and only keep the last one. Deleted 760 events with negative or large load time. Removed 8440 orphan (SERP-less) events. Removed 1 sessions falling into multiple test groups. Removed 725 sessions with more than 50 searches.

Data Summary

Select one of these three tabs:

Test Summary

Days Events Sessions Page IDs SERPs Unique search queries Searches Same-wiki clicks Other clicks
14 1,951,835 501,061 1,371,354 1,041,038 1,113,702 863,041 315,255 0

Select one of these sub-tabs:

Events

Event type identifies the context in which the event was created. Every time a new search is performed a searchResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.

Searches

Searches with n same-wiki results returned

SERPs by offset

Browser & OS

The goal here is to see whether the proportions of operating system (OS) and browser usage are similar between the groups. To aid decision making, Bayes factor is computed for each row. If one group has very different OS/browser share breakdown (Bayes factor > 2), there might be something wrong with the implementation that caused or is causing the sampling to bias in favor of some OSes/browsers. Note that for brevity, we show only the top 10 OSes/browsers, and that we don’t actually expect the numbers to be different so this is included purely as a diagnostic.

Operating systems:

Browsers:

Results of Statistical Analysis

Same-wiki Zero Results Rate

Same-wiki Engagement

classic vs. control explorer vs. control

First Clicked Same-Wiki Result’s Position

Maximum Clicked Position for Same-Wiki Results