The query_explorer plugin can generate a variety of term statistic based features. We did this AB test to try them out and see which are useful to include in training.
This test ran from 02 March 2018 to 15 March 2018 on enwiki. There were 3 test groups: control, classic, explorer. This report includes fulltext searches. Refer to Phabricator ticket T189843 for more details.
Fulltext search events: Deleted 5530 duplicated events. Deleted 2133419 unnecessary check-in events and only keep the last one. Deleted 760 events with negative or large load time. Removed 8440 orphan (SERP-less) events. Removed 1 sessions falling into multiple test groups. Removed 725 sessions with more than 50 searches.
Select one of these three tabs:
Days | Events | Sessions | Page IDs | SERPs | Unique search queries | Searches | Same-wiki clicks | Other clicks |
---|---|---|---|---|---|---|---|---|
14 | 1,951,835 | 501,061 | 1,371,354 | 1,041,038 | 1,113,702 | 863,041 | 315,255 | 0 |
Select one of these sub-tabs:
Event type identifies the context in which the event was created. Every time a new search is performed a searchResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.
The goal here is to see whether the proportions of operating system (OS) and browser usage are similar between the groups. To aid decision making, Bayes factor is computed for each row. If one group has very different OS/browser share breakdown (Bayes factor > 2), there might be something wrong with the implementation that caused or is causing the sampling to bias in favor of some OSes/browsers. Note that for brevity, we show only the top 10 OSes/browsers, and that we don’t actually expect the numbers to be different so this is included purely as a diagnostic.
Operating systems:
Browsers:
Select one of these sub-tabs:
classic vs. control explorer vs. control
PaulScore is a measure of search results’ relevancy which takes into account the position of the clicked results, and is computed via the following steps:
We can calculate the confidence interval of PaulScore\((F)\) by approximating its distribution via boostrapping.
Users may click back to the search result page directly after they clickthrough to an article (within 10 mins). We computed two kinds of return rate:
Among users with at least a click in their search, the proportion of searches that return to the same search page
Among users with at least a click in their search session, the proportion of sessions that return to search for different things (different search result page but in the same session)
We use a technique called interleaving to evaluate the user-perceived relevance of search results from the experimental configuration. In it, each user is their own baseline – we perform two searches behind the scenes and then interleave them together into a single set of results using the team draft algorithm described by Chapelle et al. (2012). For all the graphs in this section, A refers to control group or the first group in the interleaved group names, B referes to the name in the interleaved group names or the second group in the interleaved group names. For example, if the interleaved group name is ‘ltr-i-1024’, A is the control group and B is group ‘1024’; if the interleaved group name is ‘ltr-i-20-1024’, A is group ‘20’ and B is group ‘1024’.