Paradox of the big data

Author: Yen-Chi Chen (University of Washington)
Date: 08/03/2026

This note is based on Prof. Xiao-Li Meng's famous paper:

Meng, X. L. (2018). Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. The Annals of Applied Statistics, 12(2), 685-726.

Short conclusion

Even if you have a massive sample that is just slighly biased, the sample's bias could be much larger than a tiny simple random sample of a much smaller size.

Motivation: a large convenient sample versus a small random sample

Suppose our population consists of $N=2,000,000$ people. We have two samples.

[Large sample] A massive sample of $n=1,000,000$ people (half of the population!) but this sample is slightly biased in the sense that the correlation between participant being included and their response is $\rho_{Y,R} = 0.05$ .
[Simple random sample] A small random sample of $n=1,000$ people that every one has equal probability to be included.

Our goal is to estimate the population's proportion of response to a binary question.

Question:

In this case, which sample should we trust? The large sample that covers almost half of the population but has a tiny bias (correlation is $0.05$ ) or the small random sample?

Answer:

We will show that the small sample has a smaller statistical error! In fact, the half-population sample only has an effective sample size of $n=400$ as a simple random sample!!!

The bias formula

Let $y_i\in\{0,1\}$ be the response of individual $i$ in the population for any $i=1,\cdots, N$ .

The true population mean of $Y$ is
$\bar y_N = \frac{1}{N} \sum_{i=1}^N y_i .$
In our sample of size $n$ , the sample mean can be written as
$\bar y_n = \frac{\sum_{i=1}^N r_i y_i}{ \sum_{j=1}^N r_j},$
where $r_i \in\{0,1\}$ is the binary indicator of whether individual $i$ is collected in our sample.
Suppose our sample contains $n$ responses, we have $\sum_{j=1}^N r_j =n$ .

How much is the bias between $\bar y_n - \bar y_N$ ?
Fortunately, [Meng 2018] offers a simple formula for this problem:

The bias formula

$\text{Bias} = \text{Data Quality} \times \text{Data Quantity} \times \text{Problem Difficulty}$
Specifically,
$\bar{y}_n - \bar{y}_N = \rho_{R, Y} \times \sqrt{\frac{N-n}{n}} \times \sigma_Y,$
where

$\rho_{R,Y} = {\sf Cor}(R,Y)$ is the correlation between $R$ and $Y$ ,
$\sigma_Y = \sqrt{\bar y_N (1-\bar y_N)}$ .
The random variables $(Y,R)$ has equal probability $\frac{1}{N}$ being any pair of $(y_1,r_1),\cdots, (y_N, r_N)$ .

The derivation is in this section.

Analysis of the two-samples motivating example

For the large sample, $\rho_{R,Y} = 0.05$ , $N=2,000,000$ , $n=1,000,000$ , the formula gives a bias of
$\bar y_n - \bar y_N = 0.05 \times 1\times \sigma_Y = 0.05\sigma_Y.$
Thus, the mean-square error (MSE)
${\sf MSE}_{\sf Large} \approx 0.0025\sigma^2_Y$
We ignore the variance in MSE since this massive sample size will make the standard error very small.

For the small sample, there is no bias but the standard error from sampling is
$\sigma_{\sf small} = \sqrt{\frac{N - n_{\sf small}}{N-1} \frac{1}{n_{\sf small}}} \sigma_Y\approx 0.032 \sigma_Y.$
So its MSE is
${\sf MSE}_{\sf Small} \approx \frac{1}{n_{\sf small}}\sigma_Y^2 = 0.001\sigma^2_Y< 0.0025\sigma^2_Y \approx {\sf MSE}_{\sf Large}.$
Therefore, the small sample of size $n=1,000$ actually has a smaller error than the massive sample of size $n=1,000,000$ that is 1000 times larger!

Moreover, we can invert the above analysis into an effective sample size of the large sample.
The MSE of the large sample is
${\sf MSE}_{\sf Large} \approx 0.05^2 \sigma_Y^2 = \frac{1}{400}\sigma^2_Y.$
The MSE of a small sample is roughly
${\sf MSE}_{\sf Small} \approx \frac{1}{n_{\sf small}} \sigma_Y^2.$
Therefore, we conclude that the large sample only has an effective sample size of $n=400$ random sample!

Conclusion

The 1000-size random sample a MSE has only 0.4 times of the MSE of the half-population, slightly biased large sample!
The half-population, slightly biased large sample only has an effective sample size of 400 individual to a random sample!

What if we have both samples

In many modern scenarios, we may have access to both samples (a massive but bias sample and a clean but tiny sample). In this case, we can use a weighted approach of the two samples that we give the small sample a higher weight while the large sample a smaller weight. By properly weight the two sample, we are essentially tradeoff between bias and variance and may acheive a better performance.

A simple convex combination

Let $\bar y_{L}$ be the sample mean of the large sample and $\bar y_{S}$ be the sample mean of the small sample.
In our previous analysis, we know
${\sf MSE}(\bar y_L) \approx 0.0025\sigma^2_Y, \qquad {\sf MSE}(\bar y_S) \approx 0.001\sigma^2_Y$
Consider a simple convex combination of the two estimators:
$\bar y_{\alpha} = \alpha \bar y_L + (1-\alpha) \bar y_S.$
The MSE of $\bar y_{\alpha}$ is
${\sf MSE}(\bar y_{\alpha}) \approx \alpha^2 0.0025\sigma^2_Y + (1-\alpha)^2 0.001 \sigma^2_Y.$
Thus, minimizing the above with respect to $\alpha$ leads to the optimal weight
$\alpha^* = \frac{0.001}{0.0025+0.001} = \frac{2}{7}.$
Under this choice, we have the optimal MSE
${\sf MSE}(\bar y_{\alpha^*}) \approx 0.000714 \sigma^2_Y \approx \frac{1}{1400}\sigma^2_Y.$
Thus, under this optimal choice $\alpha^*$ , the estimator $\bar y_{\alpha^*}$ has a MSE effective to a random sample of size $n^* = 1400 = 1000 + 400$ , which is exactly the effective sample size of the large sample plus the small random sample.

Assumption: independence of the two samples

Note that the above analysis assumes that the two samples are independent. If they are correlated, the effective sample will not be simply addition of the two.

The weights per individual

If we consider the weights per individual, our optimal estimator is
$\bar y_{\alpha^*} = \frac{2}{7} \bar y_L + \frac{5}{7} \bar y_S = \sum_{i=1}^{n_{\sf large}} \frac{2}{7n_{\sf large}} Y^*_i + \sum_{j=1}^{n_{\sf small}} \frac{5}{7 n_{\sf small}} Y^\dagger_j,$
where $Y_i^*$ is an observed outcome of large sample and $Y^\dagger_j$ is an observed outcome of the small sample.
So each observation in the large sample has a weight of $\frac{2}{7n_{\sf large}}\approx 2.857\times 10^{-7}$ while each observation in the small sample has a weight of $\frac{5}{n_{\sf small}}\approx 7.143\times 10^{-4}$ . Every observation in the small random sample has a weight of $2500$ times of the weight of an observation from the large convenience sample!

What if our large sample is not so large?

Having a half-population conveninent sample is generally not practical. Now we consider a practical scenario where the large sample has only a size of $n_{\sf large} = 100,000$ . What is its effective sample size?

Using the bias formula, the difference in our calculation is the sample size factor
$\sqrt{\frac{N-n_{\sf large}}{n_{\large}}} = \sqrt{19}\approx 4.36.$
In terms of MSE, this sample's average is 19 times more than the half-population sample ( $n=1,000,000$ ).

Thus, its effective sample size is only $400/19\approx$ 21.

A simple random sample of 21 people has (almost) the same MSE as this $n_{\sf large}=100,000$ , slightly biased convenient sample!

Some remarks

Insights into the training of large language models (LLMs). While the training of large langugage models is not the same as simply taking a sample mean, it does exhibits some similar phenomena. People have noticed that LLMs trained on a small but clean data could outperform LLMs trained on a massive and low quality data. Famous examples are LIMA: Less Is More for Alignment and Textbooks Are All You Need

Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., ... & Levy, O. (2023). Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 55006-55021.

Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., ... & Li, Y. (2023). Textbooks are all you need. arXiv preprint arXiv:2306.11644.

Weights are the key. As in our analysis on having two samples, we have seen that when multiple datasets with varying qualities are avaiable, a proper combination of them could improve the accuracy. This principle has been used a lot in modern training of LLMs in the data curation. Researchers use over/under-sampling to create a high-quality data (data curation), which is in spirit a similar idea as the weighting. Some examples are:

Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., ... & Irving, G. (2021). Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.

Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., ... & Launay, J. (2023). The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Information Processing Systems, 36, 79155-79172.

Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., ... & Lo, K. (2024, August). Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 15725-15788).

Derivation of the bias formula

While the bias formula may look abstract, its derivation is very elementary.

Consider random variables $(Y,R)$ that has equal probability of being any pair of $(y_1,r_1),\cdots, (y_N, r_N)$ .
Clearly,
$\bar y_n =\frac{\sum_{i=1}^N r_i y_i}{ \sum_{j=1}^N r_j} = \frac{\mathbb{E}(YR)}{\mathbb{E}(R)}$
and
$\bar y_N = \frac{1}{N} \sum_{i=1}^N y_i = \mathbb{E}(Y).$
Thus,
$\begin{align*} \bar y_n - \bar y_N &= \frac{\mathbb{E}(YR)}{\mathbb{E}(R)} - \mathbb{E}(Y)\\ & = \frac{\mathbb{E}(YR) - \mathbb{E}(Y) \mathbb{E}(R)}{\mathbb{E}(R)} \\ & = \frac{{\sf Cov}(Y,R)}{\mathbb{E}(R)}\\ & = {\sf Cor}(Y,R) \frac{\sqrt{{\sf Var}(Y) {\sf Var}(R)}}{\mathbb{E}(R)}\\ & = {\sf Cor}(Y,R) \sqrt{\frac{{\sf Var}(R)}{\mathbb{E}^2(R)}} \sqrt{{\sf Var}(Y)}\\ & = \rho_{Y,R} \sqrt{\frac{f(1-f)}{f^2}} \sigma_Y\qquad (f = \frac{n}{N})\\ & = \rho_{Y,R} \sqrt{\frac{(1-f)}{f}} \sigma_Y\\ & = \rho_{Y,R} \sqrt{\frac{N-n}{n}} \sigma_Y, \end{align*}$
which is the bias formula we used.