We fetch webrequest data from Aug 1 to Aug 15, 2017 and count the number of API calls by search type (fulltext, morelike, prefix, geo, language, open), access method (desktop, mobile web, mobile app), agent type (user, spider) and referer class (direct, internal, search engine, external, unknown). See the following pivot table:
For morelike searches, we have known that it is only used by RelatedArticles on mobile (about 94% through mobile web and about 6% through apps). However, we found that there are about 60 million direct morelike API calls per day (44.6% of all morelike traffic) from mobile web. We think they may be misclassified internal traffic which failed to send referer header, or traffic from apps but their user agent string failed to match with “Wikipedia(App|/5.0.)”.
Break down morelike searches by access method and referrer class (Aug 1, 2017):
| referer_class | desktop | mobile app | mobile web |
|---|---|---|---|
| External (but not search engine) | 174 (0%) | NA | 25800 (0.02%) |
| Search engine | 2 (0%) | NA | 191946 (0.14%) |
| Internal | 434 (0%) | NA | 65933308 (49.46%) |
| None (direct) | 20237 (0.02%) | 8197271 (6.15%) | 58928685 (44.21%) |
| Unknown | 1 (0%) | NA | 9 (0%) |
We saw a similar problem in prefix searches as well. Prefix Search isn’t tremendously used by external services, because the cost of making the call to us is too high to provide seamless dropdowns in the fashion we can on Wikipedia. However, we found that there are about 6.8 million direct prefix API calls per day (19.5% of all prefix traffic) from mobile web. We think they may be misclassified internal traffic which failed to send referer header, or traffic from apps but their user agent string failed to match with “Wikipedia(App|/5.0.)”.
Break down prefix searches by access method and referrer class (Aug 1, 2017):
| referer_class | desktop | mobile app | mobile web |
|---|---|---|---|
| External (but not search engine) | 20234 (0.06%) | NA | 3930 (0.01%) |
| Search engine | 3113 (0.01%) | NA | 18 (0%) |
| Internal | 8823393 (25.82%) | NA | 9650187 (28.24%) |
| None (direct) | 235924 (0.69%) | 9044761 (26.47%) | 6385356 (18.69%) |
| Unknown | 20 (0%) | NA | 4 (0%) |
For internal full-text search, we’ve known that desktop and mobile web don’t use API calls to conduct the search, but we still saw about 6.2% from desktop and 1.8% from mobile web. For mobile web, could it be traffic from apps but their user agent string failed to match with “Wikipedia(App|/5.0.)”?
Break down full-text searches by access method and referrer class (Aug 1, 2017):
| referer_class | desktop | mobile app | mobile web |
|---|---|---|---|
| External (but not search engine) | 1773367 (9.9%) | NA | 134 (0%) |
| Search engine | 138278 (0.77%) | NA | 2 (0%) |
| Internal | 1242351 (6.94%) | NA | 370751 (2.07%) |
| None (direct) | 13015133 (72.66%) | 1219485 (6.81%) | 143896 (0.8%) |
| Unknown | 10195 (0.06%) | NA | 1 (0%) |