Code
# Define pre- and post-intervention periods for each group
pre_period_1 <- as.Date(c("2023-07-11", "2024-07-10"))
post_period_1 <- as.Date(c("2024-07-11", "2024-12-11"))Evaluating the Impact of the South American Data Center on Unique Devices
SDS 1.1.2, FY24-25
Introduction
Expanding its network of servers to new regions is one of the key investments the Wikimedia movement can make to enhance accessibility and user experience with its available resources. A faster, more reliable connection to Wikipedia and other Wikimedia projects is essential for improving engagement, particularly in regions where access has historically been slower. In pursuit of this goal, the Wikimedia Foundation launched the São Paulo data center (Magru DC) —its first South American data center, in May 2024.
The new data center has already resulted in notable improvements in access speed for users in the region, reducing load times and enhancing overall usability. Previous research on the Singapore data center confirmed significant and sustained increases in unique devices among countries that transitioned to using it.
The introduction of the Magru DC provided an excellent opportunity to conduct a natural experiment during the implementation process. This experiment aimed to deepen our understanding of data centers as a lever for improving “relevance,” as measured by unique devices. It also offered the chance to build on prior research by leveraging higher-quality data and observing immediate changes in unique devices following the data center switch.
Experiment
Countries Routed to Magru
Countries in South America are routed to the Magru data center when real-world measurements indicate lower latency compared to other data centers (previously the Dallas data center). For countries in the northern part of the continent—such as Ecuador, Colombia, Venezuela, and Guyana—the routing decision remains undecided, as their closer proximity to Dallas requires further analysis.
Based on latency analysis, six South American countries were routed to Magru. They are Argentina, Brazil, Chile, Paraguay, Uruguay where the new data center clearly improved latency, and Bolivia where latency improved for some but not all users. These six countries currently account for 67% of South America’s traffic
Experiment Design
We initially planned to conduct a controlled experiment with the Magru DC switch. Specifically, a portion of the traffic in South American countries would be routed to Magru as the treatment group, while the remaining traffic would continue to flow to the previous data center as the control group. By comparing the observed uplift in the treatment group with that in the control group, we could estimate both the size of any increase attributable to the intervention and our confidence in that attribution.
One of our primary considerations was how to route the traffic. In Neil Shah-Quinn’s analysis of the Singapore data center, he noted that a key improvement for future studies would be a randomized controlled trial in which the units of analysis would be individual users or subnational regions (e.g., cities or provinces), rather than entire countries. Under such a design, these units would be randomly assigned to connect to a particular data center. However, due to technical constraints in GeoDNS routing precision, we were required to conducted the experiment at the country level. Large DNS resolvers often mask users’ true IP addresses, making subnational or user-level routing unreliable: traffic intended for a specific city or province could be misclassified and routed to incorrect data centers, invalidating assignments. Conducting the experiment at the country level mitigates this risk, as geographic categorization at this scale is more robust to resolver-induced inaccuracies, despite sacrificing granularity.
Given this constraint, we examined three quasi-experiment approaches outlined in the South American Data Center Experiment Proposal. Option 1 routes traffic exclusively at the country level, while Options 2 and 3 route traffic based on a combination of countries and the Wikimedia projects users visit. Although Options 2 and 3 would have accounted for differences across various projects and generated more granular, higher-quality data, implementing these approaches would have required substantial additional engineering work, significantly delaying the launch of the Magru data center, as detailed in Examining the Feasibility of the South American Data Center Experiment Proposal.
With all these constraints and the lack of randomization, we decided to proceed with Option 1. The chosen six South American countries were divided into three cohorts and routed to the new data center according to the following schedule:
| switch date | country names | |
|---|---|---|
| cohort 1 | 2024-06-11 | Bolivia, Paraguay |
| cohort 2 | 2024-07-11 | Brazil |
| cohort 3 | 2024-12-11 | Argentina, Chile, Uruguay |
During the six-month study period, the countries in Cohorts 1 and 2 (Bolivia, Brazil, and Paraguay) were routed to the new data center, constituting the treatment group. The countries in Cohort 3 (Argentina, Chile, and Uruguay) remained routed to the original Dallas data center, serving as the control group. The experiment ran for six months for Cohort 1 and five months for Cohort 2. At the end of the period, Cohort 3 countries transitioned to the new data center, resulting in all six countries being fully routed to the new data center.
Methods
Metrics Selection
Unique device data for the Wikipedia project family is by far the most common choice for an overall measure of users of Wikimedia projects. There are two project-family scope unique devices datasets differing in their time grains: daily and monthly. In this analysis, we are focusing on the daily dataset in the project family scope.
Although monthly unique device data is almost always the choice for high-level metrics, it’s inherently less sensitive to short-term changes. By aggregating data over an entire month, any immediate fluctuations become smoothed out, making it harder to detect the early impact of the reduced latency from switching to the new data center, especially over our relatively short, five- to six-month experiment.
In contrast, the daily granularity of unique-device data offers a more sensitive and granular view of user behavior. For instance, if faster load times encourage users to return more frequently—even transitioning from visiting every other month to visiting monthly or weekly—those changes would be visible in daily data. Thus, while monthly data remains an important metric for long-term analysis, focusing on the daily data helps confirm whether the switch to the Magru DC is indeed driving sustained engagement and providing valuable early insights.
In addition, all unique device metrics are subdivided into the “underestimate” (devices making a first request in the time with at least some Wikimedia cookies) and the “offset” (devices making a request with no Wikimedia cookies, which might be a human first request or a bot request which was not filtered out by one of several different heuristics). The overall estimate composed of the sum of these two components is almost always used. We use only the underestimate component for this analysis since it excludes most bots and produces less noisy data. Although we are ultimately interested in the impact on total unique devices, it is reasonable to assume the relative impact of the Magru DC is the same on both the underestimate and the offset.
Data Preparation
We assigned three countries—Argentina, Chile, and Uruguay—to a control group, and three countries—Bolivia, Brazil, and Paraguay—to a treatment group. Each treatment country was matched with a control country based on similarities in historical daily unique device data and initial latency measurements before migrating to the Magru data center. This pairing ensures comparable baseline trends for measuring the data center’s impact.
For each country pair, we compiled a country-specific list of holidays to account for events affecting daily unique device activity. To address anomalies unrelated to these holidays, we applied tsclean() function. This method decomposes the time series using Seasonal-Trend decomposition via Loess, identifies outliers in the residual component that exceed Tukey’s far-out threshold (3×IQR), and replaces them via linear interpolation. This process preserves the underlying trend and seasonality while removing extreme deviations not attributable to holidays or baseline patterns.”
After replacing the outliers, we standardized all columns using the mean and standard deviation from the pre-intervention period. This step ensured that the final results were invariant to linear transformations, allowing a more consistent comparison across countries.
BSTS Modeling
For each pair, we constructed a bayesian structural time series (BSTS) model (via the bsts package) to predict daily unique devices in the treatment country. The model incorporates the paired control country’s historical device data as a predictor, along with covariates such as trends, seasonality, data length, and holiday effects. This integration ensures the counterfactual prediction is informed by the control group’s trajectory, helping isolate the treatment effect from external factors common to both countries.
To ensure robustness and pick the most fit model, we performed a rolling window cross-validation procedure. Models were iteratively trained on six- to twelve-month windows of pre-intervention data, then tested in the subsequent periods. We repeated this rolling process across six windows, evaluating model performance using mean absolute error (MAE) and root mean square deviation (RMSE). We selected the model that demonstrated both a low average MAE/RMSE and consistent performance (low variance) across all folds. This approach ensures the chosen model generalizes well to unseen data while preserving the temporal structure of the series.
Causal Impact Analysis
After training on the pre-intervention data, the BSTS model generated a counterfactual forecast of daily unique device counts for the treatment country, effectively serving as a synthetic control. This model incorporates post-intervention data from the paired control country to estimate what would have occurred if the treatment country had not switched to the new data center.
We then applied the CausalImpact package to compare the actual observed post-intervention data to these counterfactual predictions. The difference between actual and predicted outcomes represents the estimated causal effect of the intervention. If this difference is sufficiently large, it can be confidently attributed to the Magru data center switch. Additionally, CausalImpact provides a posterior probability, quantifying the likelihood that the observed effect genuinely resulted from the intervention and helping to clarify the statistical certainty of the results.
Assumptions
This method has two key assumptions:
No Spillover Effects: control countries’ traffic is unaffected by the Magru data center. We believe this holds because we explicitly control how traffic is routed at the country level. While IP geolocation is not perfect, it is still highly reliable for most users, and we use the same geolocation database both to route traffic and to record unique devices. This setup helps ensure that a country’s device data consistently corresponds to the users receiving that country’s routing—even if a small fraction of users are physically located elsewhere.
Stable Pre-Intervention Relationships: the correlations between unique devices in the treatment countries and those in the control countries do not change for reasons unrelated to Magru. While there is good reason to believe these correlations remain consistent during our analysis period, we cannot be entirely certain. Potential deviations are discussed further in the “Limitations and Future Work” section.
Results
Daily Unique Devices
Brazil
The average daily effect in Brazil during the 5 months after the switch is a 6.9% increase (SD 4 pp). In absolute terms, this is a daily increase of 124K unique devices (SD 70K), though note this is calculated based on the underestimate.
The posterior probability that the new data center increased unique devices is 96.1%, indicating strong confidence in a positive effect.
Paraguay
The average daily effect in Paraguay during 6 months after the switch is a 4.9% increase (SD 1.7 pp). In absolute terms, this is an increase of 3,011 unique devices (SD 1K), though note this is calculated based on the underestimate.
The posterior probability that the new data center increased unique devices is 99.0%, indicating very strong confidence in a positive effect.
Bolivia
Over the six months following the switch, Bolivia’s average daily effect shows a 6.4% decrease (SD 84 pp). In absolute terms, this corresponds to a drop of roughly 6,150 unique devices (SD 80K), under the conservative (“underestimate”) model. However, the posterior probability that the new data center truly reduces unique devices is only 53%, indicating relatively weak evidence of a negative impact.
Combined Effect
Combining the average effects from Brazil and Paraguay—both showing strong evidence of increased daily unique devices—the overall impact is 6.7% (approximately 127K devices). This combined figure is very close to Brazil’s effect, largely because Brazil’s daily unique device volume is about 28 times larger than Paraguay’s. Consequently, the combined impact is predominantly driven by the results from Brazil.
Monthly Unique Devices
We also examined monthly unique devices, although this was not the primary focus due to the relatively short experiment duration (5–6 months). Similar to the daily data, the monthly data shows clear positive growth in Brazil. While a possible increase in Paraguay, and a possible decrease in Bolivia. For detailed results of the monthly unique devices analysis, please refer to this notebook.
Conclusion
In summary, this analysis provides strong evidence that switching to the Magru data center has notably increased daily unique devices in Brazil and Paraguay. Although the precise magnitude of this increase remains uncertain, consistently positive outcomes across multiple cohorts support the hypothesis that improved site performance enhances user engagement.
In contrast, Bolivia did not exhibit a significant increase, with the observed decrease lacking statistical confidence. This aligns with initial latency measurements described earlier, which indicated that Bolivia’s latency improvements were smaller relative to those in other countries. The limited improvement in Bolivia likely constrained a broader positive impact.
While these findings primarily reflect short-term daily metrics, they suggest potential long-term benefits as well. If better performance leads users to visit more frequently, such increases in engagement would eventually be reflected in monthly unique devices counts and sustained metrics. Overall, these results highlight the tangible value of infrastructure improvements, reinforcing the strategic importance of investing in data center expansion where latency gains are robust and consistent.
Limitations and Future Work
Granular Randomization
Our current approach relies on the assumption that the relationship between unique devices in treatment and control countries remains stable before and after the intervention. Ideally, a fully randomized controlled trial—at the individual or sub-national level—would eliminate potential confounding effects and provide greater certainty. However, due to practical constraints, our randomization was still limited to the country level in this study.
Examining Long-term Effects with High-level Metrics
Our analysis primarily uses daily unique device metrics, capturing short-term changes in user engagement. Future analyses could incorporate monthly unique device counts or alternative metrics, such as pageviews, evaluated over longer periods (e.g., 12 months or more). Analyzing these higher-level metrics would help determine whether the immediate improvements observed in daily data translate into sustained user retention and long-term engagement following the switch to the Magru data center.
Acknowledgements
Many thanks to Jennifer Wang and Mikhail Popov for their support throughout various stages of experiment design, analysis and evaluation. To Neil Shah-Quinn for reviewing the draft of this report and providing insightful feedback. Additionally, appreciations to Sukhbir Singh and SRE Traffic team for their critical support in evaluating and implementing the data center experiment. To Omari Sefu’s support on pushing the backfill of unique devices data issue.
References
Brodersen, Kay H., Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L. Scott. 2015. “Inferring Causal Impact Using Bayesian Structural Time-Series Models.” Annals of Applied Statistics. https://doi.org/10.1214/14-AOAS788.
Nurkholis, Zaki. 2023. “A Practical Guide to Causal Impact Package with Custom Model and Prior.” Blog post. https://medium.com/@mouse3mic3/a-practical-guide-to-causal-impact-package-with-custom-model-and-prior-31b5cb714a41.
SCOTT, STEVEN. 2017. “Fitting Bayesian Structural Time Series with the Bsts r Package.” Blog post. https://www.unofficialgoogledatascience.com/2017/07/fitting-bayesian-structural-time-series.html.
Scott, Steven L., and Hal R. Varian. 2013. “Predicting the Present with Bayesian Structural Time Series.” https://doi.org/10.2139/ssrn.2304426.
Shah-Quinn, Neil. 2024. “Effect of the Singapore Data Center on Unique Devices Accessing Wikipedia.” https://people.wikimedia.org/~nshahquinn-wmf/new-data-center-impact.
Xie, Xiaoxi Chelsy, Isaac Johnson, and Anne Gomez. 2019. “Detecting and Gauging Impact on Wikipedia Page Views.” https://doi.org/10.48550/arXiv.1903.10670.
Appendix
Causal impact analysis and models
Group 1
Code
#Remove non-holiday related outliers and standardize a all columns of the data using moments estimated from the pre-intervention period before fitting the model.
group1_train <- prep_country_data(
data = wikipedia_ud_data,
y_country = "Brazil",
x_country = "Argentina",
pre_period = pre_period_1,
post_period = post_period_1
)Code
#Create zoo timeseries to fit the bsts model. And the model have to fit on data where the response variable was set to NA during the post-treatment period.
group1_train_wide <- group1_train %>%
select(dt, country, normalized_device) %>%
pivot_wider(names_from = country, values_from = normalized_device) %>%
arrange(dt)
group1_zoo <- group1_train_wide
group1_zoo$Brazil[group1_train_wide$dt >= post_period_1[1]] <- NA
group1_zoo <- to_zoo(group1_zoo)Based on the model seletion, for group 1, we are using one year of unique devices daily data, because the daily data does not show a yearly (monthly annual) seasonality. It’s built on local level trend with weekly seasonality and holiday effect.
Cross validation result for this model:
| fold | rmse | mae |
|---|---|---|
| 1 | 0.4405401 | 0.3458638 |
| 2 | 0.3761925 | 0.2861960 |
| 3 | 0.3761925 | 0.2861960 |
| 4 | 0.2756485 | 0.2168940 |
| 5 | 0.3839335 | 0.2839962 |
| 6 | 0.3604388 | 0.2448254 |
Average RMSE: 0.37, Average MAE = 0.28
Code
# create holiday list for BSTS model
br_holiday <- createHolidayList(holiday_data, "Brazil")
model_br <- fit_bsts_model(
data = group1_zoo,
response_col = "Brazil",
predictor_col = "Argentina",
trend_types = "local_level",
seasonality_types = c("weekly"),
holiday_list = br_holiday,
niter = 15000
)Here a fitted bsts model is provided to the CausalImpact analysis. The actual observed data during this period must then be passed to the function in post.period.response.
Code
br_post_response <- group1_train_wide %>%
filter(dt >= post_period_1[1], dt <= post_period_1[2]) %>%
pull(Brazil)
# CausalImpact analysis
ci_br <- CausalImpact(
bsts.model = model_br, # Pre-trained model with predictors
post.period.response = br_post_response, # Observed post-intervention values
model.args = list(standardize.data = FALSE)
)Model evaluation:
From the residual plot, the residuals appear to center around zero without any obvious trends or patterns, suggesting the model captures key dynamics and that variance remains relatively consistent over time.
Code
eval_causalimpact_metrics(ci_br)RMSE: 0.4082295
MAE: 0.2457039
R-squared: 0.8447696 With an average prediction error around 0.4 standard deviations and absolute errors near 0.25 standard deviations, this model’s estimates stay quite close to observed values. Explaining roughly 84% of the series’ variance points to a robust fit. Overall, these measures indicate that the BSTS model is a reasonably strong model and delivers reliable forecasts.
Unstandardize the result and print out the causal impact analysis summary and plots.
Code
print_impact_summary(ci_br, group1_train, "Brazil", length(br_post_response))The CausalImpact analysis estimates that, after the intervention, Brazil observed around 1.9 × 10^6 daily unique devices on average—about 120,000 more than the model’s predicted 1.8 × 10^6. This represents roughly a 7% increase (with a 95% CI from −1% to +16%). The low posterior tail‐area probability (p ≈ 0.039) and a posterior probability of a causal effect near 96% both suggest moderate to strong evidence that the intervention caused this uptick in daily unique devices.
Group 2
Code
pre_period_2 <- as.Date(c("2022-06-11", "2024-06-10"))
post_period_2 <- as.Date(c("2024-06-11", "2024-12-11"))Code
group2_train <- prep_country_data(
data = wikipedia_ud_data,
y_country = "Paraguay",
x_country = "Chile",
pre_period = pre_period_2,
post_period = post_period_2
)Code
group2_train_wide <- group2_train %>%
select(dt, country, normalized_device) %>%
pivot_wider(names_from = country, values_from = normalized_device) %>%
arrange(dt)
group2_zoo <- group2_train_wide
group2_zoo$Paraguay[group2_train_wide$dt >= post_period_2[1]] <- NA
group2_zoo <- to_zoo(group2_zoo)Based on the model seletion, for group 2, we are using two years of unique devices daily data, since the daily data shows a yearly (monthly annual) seasonality. It’s built on local level trend with yearly seasonality and holiday effect.
Cross validation result for this model:
| fold | rmse | mae |
|---|---|---|
| 1 | 0.3950305 | 0.3155283 |
| 2 | 0.3766124 | 0.2928474 |
| 3 | 0.3669890 | 0.2217063 |
| 4 | 0.3409642 | 0.2181441 |
| 5 | 0.2635942 | 0.1920462 |
| 6 | 0.2273070 | 0.1816038 |
Average RMSE: 0.32, Average MAE = 0.23
Code
# create holiday list for BSTS model
pa_holiday <- createHolidayList(holiday_data, "Paraguay")
model_pa <- fit_bsts_model(
data = group2_zoo,
response_col = "Paraguay",
predictor_col = "Chile",
trend_types = "local_level",
seasonality_types = c("monthly_annual"),
holiday_list = pa_holiday,
niter = 15000
)Code
pa_post_response <- group2_train_wide %>%
filter(dt >= post_period_2[1], dt <= post_period_2[2]) %>%
pull(Paraguay)
ci_pa <- CausalImpact(
bsts.model = model_pa, # Pre-trained model with predictors
post.period.response = pa_post_response, # Observed post-intervention values
model.args = list(standardize.data = FALSE)
)Model evaluation:
Overall, the residuals fluctuate near zero with no obvious long-term upward or downward trend. The model seems to be capturing the general pattern reasonably well. And the early spike (the outlier) is explainable by a holiday.
Code
eval_causalimpact_metrics(ci_pa)RMSE: 0.1606927
MAE: 0.1007452
R-squared: 0.9681976 With an average prediction error around 0.16 standard deviations and absolute errors near 0.10 standard deviations, this model’s forecasts closely match the observed data. Explaining roughly 97% of the variance points to a highly robust fit. Overall, these metrics indicate that the model performs well, delivering reliable forecasts.
Code
print_impact_summary(ci_pa, group2_train, "Paraguay", length(pa_post_response))The actual daily average of 64,000 unique devices exceeded the predicted counterfactual of 61,000 (±1,267), implying an additional 3,136 devices per day. This equates to a 5.2% average relative increase (±2.1%), with the 95% credible interval spanning 0.16% to 9%. Furthermore, the posterior probability of a positive causal effect is 97.7%, indicating strong evidence that the intervention produced a meaningful uplift in daily unique devices in Paraguay.
Group 3
Code
pre_period_3 <- as.Date(c("2022-06-11", "2024-06-10"))
post_period_3 <- as.Date(c("2024-06-11", "2024-12-11"))Code
group3_train <- prep_country_data(
data = wikipedia_ud_data,
y_country = "Bolivia",
x_country = "Uruguay",
pre_period = pre_period_3,
post_period = post_period_3
)Code
group3_train_wide <- group3_train %>%
select(dt, country, normalized_device) %>%
pivot_wider(names_from = country, values_from = normalized_device) %>%
arrange(dt)
group3_zoo <- group3_train_wide
group3_zoo$Bolivia[group3_train_wide$dt >= post_period_3[1]] <- NA
group3_zoo <- to_zoo(group3_zoo)Based on the model seletion, for group 3, we are using two years of unique devices daily data, since the daily data shows a yearly (monthly annual) seasonality. It’s built on local level trend with weekly and yearly seasonality with no holiday effect.
Cross validation result for this model:
| fold | rmse | mae |
|---|---|---|
| 1 | 0.4282244 | 0.3357268 |
| 2 | 0.3505878 | 0.2547625 |
| 3 | 0.4253283 | 0.3097628 |
| 4 | 0.4021777 | 0.2998559 |
| 5 | 0.4691432 | 0.3089045 |
| 6 | 0.2913825 | 0.2501399 |
Average RMSE: 0.39, Average MAE = 0.29
Code
model_bo <- fit_bsts_model(
data = group3_zoo,
response_col = "Bolivia",
predictor_col = "Uruguay",
trend_types = "local_level",
seasonality_types = c("weekly", "monthly_annual"),
niter = 15000
)Code
bo_post_response <- group3_train_wide %>%
filter(dt >= post_period_3[1], dt <= post_period_3[2]) %>%
pull(Bolivia)
ci_bo <- CausalImpact(
bsts.model = model_bo, # Pre-trained model with predictors
post.period.response = bo_post_response, # Observed post-intervention values
model.args = list(standardize.data = FALSE)
)Model evaluation:
Overall, the residuals appear to center around zero without any obvious trends or patterns, suggesting the model captures key dynamics and that variance remains relatively consistent over time.
Code
eval_causalimpact_metrics(ci_bo)RMSE: 0.2729294
MAE: 0.1445569
R-squared: 0.9176845 The model’s average prediction error is about 0.27 standard deviations, and absolute errors are 0.14 standard deviations, showing low deviation from actual values. The model explains about 92% of the variation in the time series, indicating a strong overall fit. In summary, these metrics suggest that the BSTS model captures the majority of the variability in the normalized daily data and provides reasonably accurate forecasts.
Code
print_impact_summary(ci_bo, group3_train, "Bolivia", length(bo_post_response))Code
plot_impact(ci_bo, group3_train, "Bolivia")
The result indicates a slight reduction of about 8.4% in daily unique devices in Bolivia compared to the counterfactual prediction, but the confidence intervals are wide and include no change. The causal impact analysis suggests only a 54% probability of a real effect, so there isn’t strong evidence to conclude the intervention (or event) definitively changed the daily unique device count.