We’ll be doing another test to investigate if and how many users click into the language link to see if the article is also written in a language (other than English) and then to see if the user wants to click through to that new article to read. Write up is here https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/explore_similar#Explore_similar_-_for_languages_in_search_results

This test ran from 14 September 2017 to 21 September 2017 on enwiki. There were 2 test groups: explore_similar_control, explore_similar_test. This report includes fulltext searches. Refer to Phabricator ticket T175647 for more details.

Query for full-text events:

 SELECT
  timestamp,
  event_uniqueId AS event_id,
  event_mwSessionId,
  event_pageViewId AS page_id,
  event_searchSessionId AS session_id,
  event_subTest AS `group`,
  wiki,
  MD5(LOWER(TRIM(event_query))) AS query_hash,
  event_action AS event,
  CASE
    WHEN event_position < 0 THEN NULL
    ELSE event_position
    END AS event_position,
  CASE
    WHEN event_action = 'searchResultPage' AND event_hitsReturned > 0 THEN 'TRUE'
    WHEN event_action = 'searchResultPage' AND event_hitsReturned IS NULL THEN 'FALSE'
    ELSE NULL
    END AS `some same-wiki results`,
  CASE
    WHEN event_action = 'searchResultPage' AND event_hitsReturned > -1 THEN event_hitsReturned
    WHEN event_action = 'searchResultPage' AND event_hitsReturned IS NULL THEN 0
    ELSE NULL
    END AS n_results,
  event_scroll,
  event_checkin,
  event_extraParams,
  event_msToDisplayResults AS load_time,
  event_searchToken AS search_token,
  userAgent AS user_agent
FROM TestSearchSatisfaction2_16909631
WHERE LEFT(timestamp, 8) >= '20170914' AND LEFT(timestamp, 8) < '20170922' 
  AND wiki IN('enwiki') 
  AND event_subTest IN('explore_similar_control', 'explore_similar_test') 
  AND event_source IN('fulltext') 
  AND event_searchSessionId <> 'explore_similar_test' 
  AND CASE WHEN event_action = 'searchResultPage' THEN event_msToDisplayResults IS NOT NULL
            WHEN event_action IN ('click', 'iwclick', 'ssclick') THEN event_position IS NOT NULL AND event_position > -1
            WHEN event_action = 'visitPage' THEN event_pageViewId IS NOT NULL
            WHEN event_action = 'checkin' THEN event_checkin IS NOT NULL AND event_pageViewId IS NOT NULL
            ELSE TRUE
       END;

Query for SERP from autocomplete:

 SELECT
  timestamp,
  event_uniqueId AS event_id,
  event_mwSessionId,
  event_pageViewId AS page_id,
  event_searchSessionId AS session_id,
  event_subTest AS `group`,
  wiki,
  event_action AS event,
  event_scroll,
  event_checkin,
  event_searchToken AS search_token,
  userAgent AS user_agent
FROM TestSearchSatisfaction2_16909631
WHERE LEFT(timestamp, 8) >= '20170914' AND LEFT(timestamp, 8) < '20170922' 
  AND wiki IN('enwiki') 
  AND event_subTest IN('explore_similar_control', 'explore_similar_test') 
  AND event_searchSessionId <> 'explore_similar_test' 
 AND event_source = 'autocomplete'
  AND event_articleId IS NULL
  AND event_action IN('visitPage', 'checkin')
  AND CASE WHEN event_action = 'searchResultPage' THEN event_msToDisplayResults IS NOT NULL
            WHEN event_action IN ('click', 'iwclick', 'ssclick') THEN event_position IS NOT NULL AND event_position > -1
            WHEN event_action = 'visitPage' THEN event_pageViewId IS NOT NULL
            WHEN event_action = 'checkin' THEN event_checkin IS NOT NULL AND event_pageViewId IS NOT NULL
            ELSE TRUE
       END;

Data Clean-up

Deleted 200 duplicated events.Deleted 0 events with negative load time. Removed 59 orphan (SERP-less) events. Removed 0 sessions falling into multiple test groups. Removed 3 sessions with more than 100 searches but only have searchResultPage events.

There are 1964 SERPs from autocomplete.

Data Summary

Select one of these three tabs:

Test Summary

Days Events Sessions Page IDs SERPs Unique search queries Searches Same-wiki clicks Other clicks
8 9,892 1,452 4,181 3,239 3,505 2,730 965 22

Select one of these sub-tabs:

Events

Event type identifies the context in which the event was created. Every time a new search is performed a searchResultPage event is created. When the user clicks a link in the results a visitPage event is created. When the user has dwelled for N seconds a checkin event occurs. If the user clicks an interwiki result provided by TextCat language detection, there is a iwclick event. If the user clicks on a sister search result from the sidebar, that’s an ssclick. If the user interacts with a result to explore similar (pages, categories, translations), there are hover-on, hover-off, and esclick events.

Searches

Test group wiki Search sessions Searches recorded
explore_similar_control enwiki 760 1,425
explore_similar_test enwiki 692 1,305
Total All wikis 1,452 2,730

Searches with n same-wiki results returned

SERPs by offset

Browser & OS

The goal here is to see whether the proportions of operating system (OS) and browser usage are similar between the groups. To aid decision making, Bayes factor is computed for each row. If one group has very different OS/browser share breakdown (Bayes factor >= 2), there might be something wrong with the implementation that caused or is causing the sampling to bias in favor of some OSes/browsers. Note that for brevity, we show only the top 10 OSes/browsers, and that we don’t actually expect the numbers to be different so this is included purely as a diagnostic.

wiki os explore_similar_control explore_similar_test Bayes Factor
enwiki Linux 0.9% (7) 1.7% (12) 0.0372
enwiki Mac OS X 10.10 1.4% (11) 1.0% (7) 0.0191
enwiki Mac OS X 10.11 2.9% (22) 3.5% (24) 0.0281
enwiki Mac OS X 10.12 8.6% (65) 7.4% (51) 0.0502
enwiki Other OSes 3.6% (27) 5.6% (39) 0.1659
enwiki Ubuntu 1.2% (9) 0.3% (2) 0.0756
enwiki Windows 10 37.9% (288) 36.0% (249) 0.0843
enwiki Windows 7 33.9% (258) 35.7% (247) 0.0799
enwiki Windows 8 1.1% (8) 1.2% (8) 0.0142
enwiki Windows 8.1 6.6% (50) 5.6% (39) 0.0416
enwiki Windows XP 2.0% (15) 2.0% (14) 0.0186
wiki browser explore_similar_control explore_similar_test Bayes Factor
enwiki Chrome 49 2.0% (15) 2.0% (14) 0.0186
enwiki Chrome 60 39.3% (299) 39.2% (271) 0.0644
enwiki Chrome 61 6.2% (47) 5.5% (38) 0.0361
enwiki Edge 14 2.1% (16) 1.4% (10) 0.0270
enwiki Edge 15 4.2% (32) 4.0% (28) 0.0266
enwiki Firefox 52 1.2% (9) 1.3% (9) 0.0151
enwiki Firefox 55 11.4% (87) 11.7% (81) 0.0426
enwiki IE 11 15.1% (115) 13.3% (92) 0.0756
enwiki Opera 47 0.9% (7) 1.4% (10) 0.0218
enwiki Other browsers 13.4% (102) 14.6% (101) 0.0562
enwiki Safari 10 4.1% (31) 5.5% (38) 0.0620

Explore Similar

Select one of these sub-tabs:

Clicks by section and link’s position on search result page

2nd result Sum
languages 1 1
Sum 1 1

Hover-overs by section and results

0 results 1 result 2 results 3 results 4 results 5+ results Sum
languages 30 7 9 6 9 9 70
Sum 30 7 9 6 9 9 70

Results of Statistical Analysis

Same-wiki Zero Results Rate

Same-wiki Engagement

explore_similar_test vs. explore_similar_control

First Clicked Same-Wiki Result’s Position

Maximum Clicked Position for Same-Wiki Results

PaulScore

PaulScore is a measure of search results’ relevancy which takes into account the position of the clicked results, and is computed via the following steps:

  1. Pick scoring factor \(0 < F < 1\) (larger values of \(F\) increase the weight of clicks on lower-ranked results).
  2. For \(i\)-th search session \(S_i\) \((i = 1, \ldots, n)\) containing \(m\) queries \(Q_1, \ldots, Q_m\) and search result sets \(\mathbf{R}_1, \ldots, \mathbf{R}_m\):
  1. For each \(j\)-th search query \(Q_j\) with result set \(\mathbf{R}_j\), let \(\nu_j\) be the query score: \[\nu_j = \sum_{k~\in~\{\text{0-based positions of clicked results in}~\mathbf{R}_j\}} F^k.\]
  2. Let user’s average query score \(\bar{\nu}_{(i)}\) be \[\bar{\nu}_{(i)} = \frac{1}{m} \sum_{j = 1}^m \nu_j.\]
  1. Then the PaulScore is the average of all users’ average query scores: \[\text{PaulScore}(F)~=~\frac{1}{n} \sum_{i = 1}^n \bar{\nu}_{(i)}.\]

We can calculate the confidence interval of PaulScore\((F)\) by approximating its distribution via boostrapping.

Other Pages of the Search Results

Dwell Time Per Visited Page

Scroll on visited pages

Search Abandon Rate

Scroll on search result page

Dwell time of search result pages

Currently we are only able to track the dwell time of search result pages from autocomplete search.

Return Rate

Users may click back to the search result page directly after they clickthrough to an article (within 10 mins). We computed two kinds of return rate:

  • Among users with at least a click in their search, the proportion of searches that return to the same search page

  • Among users with at least a click in their search session, the proportion of sessions that return to search for different things (different search result page but in the same session)

Load time of search results pages