This report used data from the second bm25 A/B test and tried to partially reproduced the analysis.
This test ran from 27 October 2016 to 15 November 2016 on zhwiki, jawiki, thwiki. There were 2 test groups: bm25:control, bm25:inclinks_pv. This report includes fulltext searches. Refer to Phabricator ticket T147495 for more details.
Deleted 0 duplicated events. Removed 2878 orphan (SERP-less) events. Removed 0 sessions falling into multiple test groups.
Select one of these three tabs:
Days | Events | Sessions | Page IDs | SERPs | Unique search queries | Searches | Same-wiki clicks | Other clicks |
---|---|---|---|---|---|---|---|---|
20 | 230,478 | 36,121 | 103,433 | 81,281 | 81,199 | 64,486 | 22,993 | 0 |
Select one of these sub-tabs:
Action identifies the context in which the event was created. Every time a new search is performed a searchEngineResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.
Test group | wiki | Search sessions | Searches recorded |
---|---|---|---|
bm25:control | jawiki | 7,579 | 13,815 |
bm25:control | thwiki | 3,889 | 7,001 |
bm25:control | zhwiki | 6,510 | 11,446 |
bm25:inclinks_pv | jawiki | 7,610 | 13,971 |
bm25:inclinks_pv | thwiki | 4,055 | 7,083 |
bm25:inclinks_pv | zhwiki | 6,478 | 11,170 |
Total | All wikis | 36,121 | 64,486 |
Select one of these sub-tabs:
Select one of these sub-tabs:
bm25:inclinks_pv vs. bm25:control
bm25:inclinks_pv vs. bm25:control by wiki
PaulScore is a measure of search results’ relevancy which takes into account the position of the clicked results, and is computed via the following steps:
We can calculate the confidence interval of PaulScore\((F)\) by approximating its distribution via boostrapping.