Baseline iOS Program Metrics for FY18-19
Chelsy Xie
03 October 2018
{ RMarkdown Source | Analysis Codebase }
In FY18-19, the iOS team will develop contribution features of the app (project page).
This report is an initial exploration with two goals:
- Write relevant queries that can be productionized later (made to execute daily with data flowing into Druid/Superset).
- Learn what the program metrics look like right now so we can set our expectations accordingly.
Executive Summary
Total iOS app edit counts: The daily iOS app edit count across all wikis is 773.18 on average. The daily average edits of English, French, Korean, Czech and Hindi Wikipedia are 414.53, 42.58, 20.57, 1.37, 0.4 respectively. For most Wikipedias, content edits contribute to more than 90% of all iOS app edits.
iOS edits per editor: Because of the right skewed distribution of edits per editor and the small number of iOS editors of most wikis, we suggest tracking the proportion of 1 edit editors among all registered editors who edit at least once via iOS app monthly, and tracking the proportion of 5+ edit editors among all registered editors who edit at least once via iOS app monthly.
Across all Wikipedias, the 1 edit/month editor proportion is 45.3%. The 1 edit/month editor proportion of English and French Wikipedia are 44.678% and 43.825% respectively. Because Korean, Czech and Hindi Wikipedia have very small number of iOS app editors, their proportions are unreliable. Across all Wikipedias, the 5+ edit/month editor proportion is 22.5%. The 5+ edit/month editor proportion of English and French Wikipedia are 22.64% and 25.10% respectively. Because Korean, Czech and Hindi Wikipedia have very small number of iOS app editors, their proportions are unreliable.
Revert rate of iOS edits: The revert rate of iOS app edits across all Wikipedias is 14.7%, which is higher than the revert rate across all platforms (6% in August 2018). The iOS edits revert rates of English, French and Korean Wikipedia are 16.5%, 9.7%, 1.3% respectively. Because Czech and Hindi Wikipedia have very small number of iOS app edits, their revert rates are unreliable.
iOS new contributor retention rate: For new iOS app editors of English and French Wikipedia, the second month retentions are 12.48% and 10.42% respectively. Since Korean, Czech and Hindi Wikipedia don’t have enough new editors, their retention rates are unreliable.
iOS contributor retention rate: For all iOS app editors of English and French Wikipedia, the second month retentions are 28.377% and 28.696% respectively. Since Korean, Czech and Hindi Wikipedia don’t have enough editors, their retention rates are unreliable.
Metrics
In the annual plan, we defined the following primary metrics:
Total iOS app edit counts: Number of non-bot edits which are tagged as made through the iOS app interface. We hope to increase the total number of edits made via the iOS app.
iOS edits per editor: For any given time period, of the non-bot registered users that completed at least 1 edit via the iOS app, how many edits did they complete on average (arithmetic mean/median). Edits on other platform won’t affect this metric. This metric will measure the efficiency of editing via the iOS app for existing editors, and we hope to increase it.
Since the distribution of edits per editor is very right skewed with lots of 1 edit editors, the median edits/editor/month is 1 or 2 for most wikis with at least 10 editors, which means it would be very hard to see a change on this metric. And since we have so few iOS editors for most wikis, the average edits/editor/month is highly influenced by some very active editors who make a lot more edits than others.
Instead, we suggest tracking the proportion of 1 edit editors among all registered editors who edit at least once via iOS app monthly. As we hope to make it much more efficient and enjoyable for these casual editors to do more via the app, the smaller this number is the better. Also, since we consider editors with at least 5 edits per month as active editors historically, we want to track the proportion of 5+ edit editors among all registered editors who edit at least once via iOS app monthly, although this threshold may be too high for mobile editing. We hope to increase this proportion on the iOS app.
We also defined these health monitoring metrics:
Revert rate of iOS edits: Proportion of iOS app edits which have been reverted. We will closely monitor revert rates for edits from the app to ensure that we are not increasing the burden on existing communities or reducing the quality of the projects. Although we expect some noise and fluctuation in this metric, it will need to remain roughly stable as we make changes.
iOS contributor retention rate: Number of editors who have made additional edits through the iOS app in the N days following their first iOS edit in the given time span (does not include anonymous editors and editors who have only made additional edits through other platforms following their first iOS edit).
iOS new contributor retention rate: Number of new editors who have made additional edits through the iOS app in the N days following their first ever iOS edit (does not include anonymous editors and editors who have only made additional edits through other platforms following their first iOS edit). Platform-specific edit data was not available until June 29 via T194424. Prior to the deployment of the patch, all edits made in iOS and Android apps were lumped together under a single change tag “mobile app edit”. Since multi-platform usage and platform switching are not common occurrences, any editor who has made a “mobile app edit”-tagged contribution previously is ineligible to be considered a new iOS editor.
We believe that by making editing more efficient, enjoyable and understandable we will naturally see editors returning to the app and editing on a repeated basis. We are not building features specifically to boost this number, but if we are seeing more edits, but unable to keep those editors engaged that will be a negative signal that we will have to address.
In this report, we will compute the baseline for all languages of Wikipedia, with a focus on our target Wikipedia in the annual plan: English, French, Korean, Hindi and Czech.
Query:
SELECT
ios_app_edit_counts.local_user_id AS local_user_id,
IFNULL(previous_app_editors.new_ios_editor, 'TRUE') AS new_ios_editor,
edit_date,
edits,
content_edits
FROM (
SELECT
local_user_id,
edit_date,
COUNT(*) AS edits,
SUM(namespace = 0) AS content_edits
FROM (
-- Edits made with iOS app on visible pages:
SELECT
rev_user AS local_user_id,
DATE(LEFT(rev_timestamp, 8)) AS edit_date,
page.page_namespace AS namespace
FROM change_tag
INNER JOIN revision ON (
revision.rev_id = change_tag.ct_rev_id
AND revision.rev_timestamp >= '{start_date}'
AND revision.rev_timestamp < '{end_date}'
AND change_tag.ct_tag = 'ios app edit'
)
LEFT JOIN page ON revision.rev_page = page.page_id
UNION ALL
-- Edits made with iOS app on deleted pages:
SELECT
ar_user AS local_user_id,
DATE(LEFT(ar_timestamp, 8)) AS edit_date,
ar_namespace AS namespace
FROM change_tag
INNER JOIN archive ON (
archive.ar_rev_id = change_tag.ct_rev_id
AND archive.ar_timestamp >= '{start_date}'
AND archive.ar_timestamp < '{end_date}'
AND change_tag.ct_tag = 'ios app edit'
)
) AS ios_app_edits
GROUP BY local_user_id, edit_date
) AS ios_app_edit_counts
LEFT JOIN (
-- Editors who have used a mobile app (Android/iOS)
-- to edit Wikipedia before the start_date:
SELECT DISTINCT local_user_id,
'FALSE' as new_ios_editor
FROM (
-- Editors who have previously used a mobile app to edit visible pages:
SELECT rev_user AS local_user_id
FROM change_tag
INNER JOIN revision ON (
revision.rev_id = change_tag.ct_rev_id
AND revision.rev_timestamp < '{start_date}'
AND change_tag.ct_tag = 'mobile app edit'
)
UNION ALL
-- Editors who have previously used a mobile app to edit deleted pages:
SELECT ar_user AS local_user_id
FROM change_tag
INNER JOIN archive ON (
archive.ar_rev_id = change_tag.ct_rev_id
AND archive.ar_timestamp < '{start_date}'
AND change_tag.ct_tag = 'mobile app edit'
)
) AS combined_revisions
) AS previous_app_editors
ON previous_app_editors.local_user_id = ios_app_edit_counts.local_user_idTotal iOS app edit counts
Total iOS app edit counts is the number of non-bot edits which are tagged as made through the iOS app interface. Since it’s very unlikely that a bot would use the app to edit, bot filter was not included in the query. We hope to increase the total number of edits made via the iOS app.
All Edits
This following graph shows the daily number of iOS app edits by all active Wikipedias, from 2018-07-01 to 2018-09-30.
The table below shows the mean, median and total iOS app edits by all active Wikipedias, from 2018-07-01 to 2018-09-30.
Content Edits
Content edit counts is the number of iOS app edits made on the main namespace (namespace 0) of each Wikipedia. This following graph shows the daily number of iOS app content edits by all active Wikipedias, from 2018-07-01 to 2018-09-30.
The table below shows the mean, median, total iOS app content edits and their percentage of all edits, by all active Wikipedias from 2018-07-01 to 2018-09-30.
Summary
- The daily iOS app edit count across all wikis is 773.18 on average. The daily average edits of English, French, Korean, Czech and Hindi Wikipedia are 414.53, 42.58, 20.57, 1.37, 0.4 respectively.
- The daily iOS app content edit count across all wikis is 746.54 on average. The daily average content edits of English, French, Korean, Czech and Hindi Wikipedia are 406.39, 41.9, 19.37, 1.37, 0.28 respectively.
- For most Wikipedias, content edits contribute to more than 90% of all iOS app edits. Among wikis with more than 1 edit daily on average, only Indonesian and Latvian Wikipedia’s content edits are less than 90% – only 36.0% edits on Indonesian and 48.4% edits on Latvian are content edits.
iOS edits by editor
For any given time period, of the non-bot registered users that completed at least 1 edit via the iOS app, we’re interested in knowing how many edits did they complete on average (arithmetic mean/median). This metric will measure the efficiency of editing via the iOS app for existing editors, and we hope to increase it. Edits on other platform won’t affect this metric. Since it’s very unlikely that a bot would use the app to edit, bot filter was not included in the query.
Here we started by counting the number of edits per editor in July, August and September 2018 without doing any aggregation, since we want to see the distribution before determine whether we should use mean or median, or other aggregation metrics. The following graph shows the distribution of edits per editor per month in July, August and September 2018 by target Wikipedias. We can see that most editors only complete one edit per month.
Since the distribution of edits per editor is very right skewed with lots of 1 edit editors, from the table below, we can see the median edits/editor/month is 1 or 2 for most wikis with at least 10 editors, which means it would be very hard to see a change on this metric. And since we have so few iOS editors for most wikis, the average edits/editor/month is highly influenced by some very active editors who make a lot more edits than others.
Instead, we suggest tracking the proportion of 1 edit editors among all registered editors who edit at least once via iOS app monthly. As we hope to make it much more efficient and enjoyable for these casual editors to do more via the app, the smaller this number is the better. Also, since we consider editors with at least 5 edits per month as active editors historically, we want to track the proportion of 5+ edit editors among all registered editors who edit at least once via iOS app monthly, although this threshold may be too high for mobile editing. We hope to increase this proportion on the iOS app.
The graph and table below show the 1 edit/month editors proportion and 5+ edit/month editors proportion. Please note that for Wikipedias with small number of editors, these metrics are unreliable.
Summary
- Since the distribution of edits per editor is very right skewed with lots of 1 edit editors, the median edits/editor/month is 1 or 2 for most wikis with at least 10 editors, which means it would be very hard to see a change on this metric. And since we have so few iOS editors for most wikis, the average edits/editor/month is highly influenced by some very active editors who make a lot more edits than others. Instead, we suggest tracking the proportion of 1 edit editors among all registered editors who edit at least once via iOS app monthly, and tracking the proportion of 5+ edit editors among all registered editors who edit at least once via iOS app monthly.
- Across all Wikipedias, the 1 edit/month editor proportion is 45.3%. The 1 edit/month editor proportion of English, French, Korean, Czech and Hindi Wikipedia are 44.7%, 43.8%, 34.3%, 58.3%, 0.0% respectively. Because Korean, Czech and Hindi Wikipedia have very small number of iOS app editors, their proportions are unreliable.
- Across all Wikipedias, the 5+ edit/month editor proportion is 22.5%. The 5+ edit/month editor proportion of English, French, Korean, Czech and Hindi Wikipedia are 22.6%, 25.1%, 28.6%, 0.0%, 16.7% respectively. Because Korean, Czech and Hindi Wikipedia have very small number of iOS app editors, their proportions are unreliable.
Revert rate of iOS edits
Proportion of iOS app edits which have been reverted. We will closely monitor revert rates for edits from the app to ensure that we are not increasing the burden on existing communities or reducing the quality of the projects. Although we expect some noise and fluctuation in this metric, it will need to remain roughly stable as we make changes.
The following table shows the revert rate of iOS app edits by all active Wikipedias. Please note that for wikis with small number of edits, the revert rate is unreliable.
Note: This total edits is different from what we see in the total iOS app edit counts section since we can’t get revert status of some revisions due to bugs in mwreverts.Summary
- The revert rate of iOS app edits across all wikis is 14.7%, which is higher than the revert rate across all platforms (6% in August 2018).
- The iOS edits revert rates of English, French, Korean, Czech and Hindi Wikipedia are 16.5%, 9.7%, 1.3%, 32.5%, 21.6% respectively. Because Czech and Hindi Wikipedia have small number of iOS app edits, their revert rates are unreliable.
Retention
Number of editors who have made additional edits through the iOS app in the N days following their first iOS edit in the given time span (does not include anonymous editors and editors who have only made additional edits through other platforms following their first iOS edit). If the target editor group is new iOS editors, the “first iOS edit” is the editor’s first ever iOS edit, and the editors have not previously edited Wikipedia with either of the mobile apps (since edits from iOS and Android were not tagged separately until July 2018); if the target editor group is all iOS editors, the “first iOS edit” is the editor’s first edit with the iOS app in a given time span.
Traditionally, N is 30 days following 30 days’ trial period, but we’d like to explore different trial and survival periods in this report and see how they impact the editor retention. However, the longer the trial + survival period is, the less eligible editors we have (we are NOT counting any editors whose trial periods haven’t ended, OR any editors whose survival periods haven’t ended and who haven’t make any edit yet.), which means the retention we’re computing is less reliable.
We believe that by making editing more efficient, enjoyable and understandable we will naturally see editors returning to the app and editing on a repeated basis. We are not building features specifically to boost this number, but if we are seeing more edits, but unable to keep those editors engaged that will be a negative signal that we will have to address.
New iOS editor retention
A new iOS editors is any account that has made at least one edit with the iOS app since – but not any edits with either app prior to – July 1st 2018. This includes users who have previously edited on desktop and/or mobile web.
The graph below shows the weekly retention of new editors since July 1st 2018 by target Wikipedias and other Wikipedias with more than 30 new editors since July 1st 2018. From the error bands, we can see that the retention rates are unreliable for wikis with small number of new editors.
The following table shows various retention rates of new iOS app editors by all active Wikipedias since July 1st 2018. The second column shows the number of new editors we have so far, but not all of them are counted in the retention computation – we are NOT counting any editors who haven’t reached their survival periods, OR any editors whose survival periods haven’t ended and who haven’t make any edit yet. Also, please note that for wikis with small number of editors, the retention rate is unreliable.
iOS editor retention
An iOS editors is any account that has made at least one edit with the iOS app since July 1st 2018, because edits from iOS and Android were not tagged separately until July 2018.
The graph below shows the weekly retention of all editors since July 1st 2018 by target Wikipedias and other Wikipedias with more than 50 editors since July 1st 2018. From the error bands, we can see that the retention rates are unreliable for wikis with small number of editors.
The following table shows various retention rates of all iOS app editors by all active Wikipedias since July 1st 2018. The second column shows the number of editors we have so far, but not all of them are counted in the retention computation – we are NOT counting any editors who haven’t reached their survival periods, OR any editors whose survival periods haven’t ended and who haven’t make any edit yet. Also, please note that for wikis with small number of editors, the retention rate is unreliable.
Summary
- For new iOS app editors across all wikis, their first week retention is 14.6% and it drops to 6.47% in the second week, then stabilizes around 4.22%. We saw a similar pattern in English and French Wikipedia as well. For English Wikipedia, their new iOS editors weekly retentions are 14.1%, 6.4%, 4.3%, 4.8%, 4.4%, 4.0% from week 1 to week 6. For French Wikipedia, their new iOS editors weekly retentions are 12.5%, 6.7%, 5.3%, 3.4%, 2.5%, 2.8% from week 1 to week 6. Since Korean, Czech and Hindi Wikipedia don’t have enough new editors, their retention rates are unreliable.
- For new iOS app editors of English and French Wikipedia, the 31st-60th day retentions are 12.48% and 10.42% respectively, which is higher than the new editor retention across all wikis and platforms (5.1% in August 2018). But please note that the birth dates of users are different for these metrics – for retention across all wikis and platforms, the birth date is user’s registration date; while for iOS new editor retention, the birth date is the date when users make their first ever edit with the iOS app.
- For all iOS app editors across all wikis, their first week retention is 19.3% and it drops to 12.8% in the second week, then stabilizes around 10.7%. We saw a similar pattern in English Wikipedia as well. For English Wikipedia, their iOS editors weekly retentions are 19.46%, 13.25%, 11.48%, 10.91%, 10.90%, 10.91% from week 1 to week 6. For French Wikipedia, their iOS editors weekly retentions wobbles a bit – 17.65%, 13.37%, 14.69%, 9.70%, 10.90%, 11.72% from week 1 to week 6. Since Korean, Czech and Hindi Wikipedia don’t have enough editors, their retention rates are unreliable.
- For all iOS app editors of English and French Wikipedia, the 31st-60th day retentions are 28.377% and 28.696% respectively.
Appendix
iOS vs Android on target wikis
