Query for full-text events:
SELECT
timestamp,
event_uniqueId AS event_id,
event_mwSessionId,
event_pageViewId AS page_id,
event_searchSessionId AS session_id,
event_subTest AS `group`,
wiki,
MD5(LOWER(TRIM(event_query))) AS query_hash,
event_action AS event,
CASE
WHEN event_position < 0 THEN NULL
ELSE event_position
END AS event_position,
CASE
WHEN event_action = 'searchResultPage' AND event_hitsReturned > 0 THEN 'TRUE'
WHEN event_action = 'searchResultPage' AND event_hitsReturned IS NULL THEN 'FALSE'
ELSE NULL
END AS `some same-wiki results`,
CASE
WHEN event_action = 'searchResultPage' AND event_hitsReturned > -1 THEN event_hitsReturned
WHEN event_action = 'searchResultPage' AND event_hitsReturned IS NULL THEN 0
ELSE NULL
END AS n_results,
event_scroll,
event_checkin,
event_extraParams,
event_msToDisplayResults AS load_time,
event_searchToken AS search_token,
userAgent AS user_agent
FROM TestSearchSatisfaction2_16909631
WHERE LEFT(timestamp, 8) >= '20171018' AND LEFT(timestamp, 8) < '20171026'
AND wiki IN('enwiki')
AND event_subTest IN('control', 'rec_3t_80_66', 'rec_4t_80_66')
AND event_source IN('fulltext')
AND CASE WHEN event_action = 'searchResultPage' THEN event_msToDisplayResults IS NOT NULL
WHEN event_action IN ('click', 'iwclick', 'ssclick') THEN event_position IS NOT NULL AND event_position > -1
WHEN event_action = 'visitPage' THEN event_pageViewId IS NOT NULL
WHEN event_action = 'checkin' THEN event_checkin IS NOT NULL AND event_pageViewId IS NOT NULL
ELSE TRUE
END;
Query for SERP from autocomplete:
SELECT
timestamp,
event_uniqueId AS event_id,
event_mwSessionId,
event_pageViewId AS page_id,
event_searchSessionId AS session_id,
event_subTest AS `group`,
wiki,
event_action AS event,
event_scroll,
event_checkin,
event_searchToken AS search_token,
userAgent AS user_agent
FROM TestSearchSatisfaction2_16909631
WHERE LEFT(timestamp, 8) >= '20171018' AND LEFT(timestamp, 8) < '20171026'
AND wiki IN('enwiki')
AND event_subTest IN('control', 'rec_3t_80_66', 'rec_4t_80_66')
AND event_source = 'autocomplete'
AND event_articleId IS NULL
AND event_action IN('visitPage', 'checkin')
AND CASE WHEN event_action = 'searchResultPage' THEN event_msToDisplayResults IS NOT NULL
WHEN event_action IN ('click', 'iwclick', 'ssclick') THEN event_position IS NOT NULL AND event_position > -1
WHEN event_action = 'visitPage' THEN event_pageViewId IS NOT NULL
WHEN event_action = 'checkin' THEN event_checkin IS NOT NULL AND event_pageViewId IS NOT NULL
ELSE TRUE
END;
We want to test to see if relaxing the default AND of the retrieval query filter with minimum_should_match can help to pull new interesting results.
This test ran from 18 October 2017 to 25 October 2017 on enwiki. There were 3 test groups: control, rec_3t_80_66, rec_4t_80_66. This report includes fulltext searches. Refer to Phabricator ticket T177302 for more details.
Deleted 56 duplicated events. Deleted 0 events with negative load time. Removed 195 orphan (SERP-less) events. Removed 0 sessions falling into multiple test groups. Removed 1 sessions with more than 100 searches.
There are 5863 SERPs from autocomplete.
Select one of these three tabs:
Days | Events | Sessions | Page IDs | SERPs | Unique search queries | Searches | Same-wiki clicks | Other clicks |
---|---|---|---|---|---|---|---|---|
8 | 29,285 | 4,505 | 12,129 | 9,145 | 10,036 | 7,689 | 2,849 | 107 |
Select one of these sub-tabs:
Event type identifies the context in which the event was created. Every time a new search is performed a searchResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.
The goal here is to see whether the proportions of operating system (OS) and browser usage are similar between the groups. To aid decision making, Bayes factor is computed for each row. If one group has very different OS/browser share breakdown (Bayes factor > 2), there might be something wrong with the implementation that caused or is causing the sampling to bias in favor of some OSes/browsers. Note that for brevity, we show only the top 10 OSes/browsers, and that we don’t actually expect the numbers to be different so this is included purely as a diagnostic.
Operating systems:
Browsers:
Select one of these sub-tabs:
rec_3t_80_66 vs. control rec_4t_80_66 vs. control
PaulScore is a measure of search results’ relevancy which takes into account the position of the clicked results, and is computed via the following steps:
We can calculate the confidence interval of PaulScore\((F)\) by approximating its distribution via boostrapping.
Currently we are only able to track the dwell time of search result pages from autocomplete search.
Users may click back to the search result page directly after they clickthrough to an article (within 10 mins). We computed two kinds of return rate:
Among users with at least a click in their search, the proportion of searches that return to the same search page
Among users with at least a click in their search session, the proportion of sessions that return to search for different things (different search result page but in the same session)