This A/B test have one control group and two test groups that show learning-to-rank results.
This test ran from 20 September 2017 to 05 October 2017 on arwiki, dewiki, fawiki, fiwiki, frwiki, hewiki, idwiki, itwiki, jawiki, kowiki, nlwiki, nowiki, plwiki, ptwiki, ruwiki, svwiki, viwiki, zhwiki. There were 3 test groups: control, ltr-1024, ltr-i-1024. This report includes fulltext searches. Refer to Phabricator ticket T175771 for more details.
Fulltext search events: Deleted 60061 duplicated events. Deleted 1383174 unnecessary check-in events and only keep the last one. Deleted 91 events with negative load time. Removed 8265 orphan (SERP-less) events. Removed 0 sessions falling into multiple test groups. Removed 199 sessions with more than 50 searches.
Select one of these three tabs:
Days | Events | Sessions | Page IDs | SERPs | Unique search queries | Searches | Same-wiki clicks | Other clicks |
---|---|---|---|---|---|---|---|---|
16 | 1,624,956 | 455,646 | 1,183,582 | 950,181 | 949,282 | 763,822 | 243,252 | 9,672 |
Select one of these sub-tabs:
Event type identifies the context in which the event was created. Every time a new search is performed a searchResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.
The goal here is to see whether the proportions of operating system (OS) and browser usage are similar between the groups. To aid decision making, Bayes factor is computed for each row. If one group has very different OS/browser share breakdown (Bayes factor > 2), there might be something wrong with the implementation that caused or is causing the sampling to bias in favor of some OSes/browsers. Note that for brevity, we show only the top 10 OSes/browsers, and that we don’t actually expect the numbers to be different so this is included purely as a diagnostic.
Operating systems:
Browsers:
Select one of these sub-tabs:
ltr-1024 vs. control
ltr-1024 vs. control by wiki