Abstract

Mean absolute percentage error (MAPE), the measure most often used for evaluating subnational demographic estimates, is not always valid. We describe guidelines for determining when MAPE is valid. Applying them to case study data, we find that MAPE understates accuracy because it is unduly influenced by outliers. To overcome this problem, we calculate a transformed MAPE (MAPET) using a modified Box-Cox method. Because MAPE-T is not in the same scale as the untransformed absolute percentage errors, we provide a procedure for calculating MAPE-R, a measure in the same scale as the original observations. We argue that MAPE-R is a more appropriate summary measure of average absolute percentage error when the guidelines indicate that MAPE is not valid.

The text of this article is only available as a PDF.

References

Barnett, V., & Lewis, T. (
1984
).
Outliers in Statistical Data
. 2nd ed.
New York
:
Wiley
.
Box, G., & Cox, D. (
1964
).
An Analysis of Transformations
.
Journal of the Royal Statistical Society, Series B
,
26
,
211
52
.
Bryan, T. 1999. “U.S. Census Bureau Population Estimates and Evaluation With Loss Functions.” Presented at the 52nd Congress of the International Statistical Institute, Helsinki, Finland, August 17.
Carmines, E., & Zeller, R. (
1979
).
Reliability and Validity Assessment
.
Beverly Hills
:
Sage
.
D’Agostino, R., Belanger, A., & D’Agostino, R. (
1990
).
A Suggestion for Using Powerful and Informative Tests of Normality
.
The American Statistician
,
44
,
316
21
. 10.2307/2684359
Davis, S. (
1994
).
Evaluation of Postcensal County Estimates for the 1980s
.
Washington, DC
:
Population Division, U.S. Bureau of the Census
.
Draper, N., & Smith, H. (
1981
).
Applied Regression Analysis
.
New York
:
Wiley
.
Emerson, J., & Stoto, M. (
1983
).
Transforming Data
. In D. Hoaglin, F. Mosteller, & J. Tukey (Eds.),
Understanding Robust and Exploratory Data Analysis
(pp.
97
128
).
New York
:
Wiley
.
Emerson, J., & Strenio, J. (
1983
).
Boxplots and Batch Comparison
. In D. Hoaglin, F. Mosteller, & J. Tukey (Eds.),
Understanding Robust and Exploratory Data Analysis
(pp.
58
96
).
New York
:
Wiley
.
Fonseca, L., & Tayman, J. (
1989
).
Postcensal Estimates of Household Income Distributions
.
Demography
,
26
,
149
59
. 10.2307/2061501
Goldberg, D., Rao, V.R., & Namboodiri, K. (
1964
).
A Test of the Accuracy of Ratio-Correlation Population Estimates
.
Land Economics
,
40
,
100
102
. 10.2307/3144467
Goodall, C. (
1983
).
M-Estimators of Location: An Outline of the Theory
. In D. Hoaglin, F. Mosteller, & J. Tukey (Eds.),
Understanding Robust and Exploratory Data Analysis
(pp.
339
403
).
New York
:
Wiley
.
Hamburg, M. (
1985
).
Basic Statistics: A Modern Approach
. 3rd ed.
San Diego
:
Harcourt Brace Jovanovich
.
Hintze, J. (
1996
).
NCSS 6.0 User’s Guide
.
Kaysville, UT
:
NCSS
.
Kitagawa, E. (
1980
).
Estimating Population and Income of Small Areas
.
Washington, DC
:
National Academy Press
.
Lin, P. 1999. “Colorado Methods of Population Estimates.” Presented at the Population Estimates Methods Conference, Population Estimates Branch, Population Division, U.S. Bureau of the Census, Suitland, MD, June 8.
Martindale, M. 1999. “Overview of California’s County Population Estimating Methods.” Presented at the Population Estimates Methods Conference, Population Estimates Branch, Population Division, U.S. Bureau of the Census, Suitland, MD, June 8.
Morrison, P. (
1971
).
Demographic Information for Cities: A Manual for Estimating and Projecting Local Population Characteristics
.
Santa Monica
:
RAND
.
Murdock, S., & Hoque, N. (
1995
).
The Effect of Undercount on the Accuracy of Small-Area Population Estimates: Implications for the Use of Administrative Data for Improving Population Enumeration
.
Population Research and Policy Review
,
14
,
251
71
. 10.1007/BF01074461
Norusis, M. (
1986
).
The SPSS Guide to Data Analysis
.
Chicago
:
SPSS
.
Rosenberger, J., & Gasko, M. (
1983
).
Comparing Location Estimators: Trimmed Means, Medians, and Trimean
. In D. Hoaglin, F. Mosteller, & J. Tukey (Eds.),
Understanding Robust and Exploratory Data Analysis
(pp.
297
338
).
New York
:
Wiley
.
Shahidullah, M. and M. Flotow. 1999. “Review of Administrative Records and Ratio-Correlation Methods for Producing Post 2000 County Population Estimates in Illinois.” Presented at the Population Estimates Methods Conference, Population Estimates Branch, Population Division, U.S. Bureau of the Census, Suitland, MD, June 8.
Sink, L. (
1997
).
Race and Ethnicity Classification Consistency Between the Census Bureau and the National Center for Health Statistics
.
Washington, DC
:
Population Division, U.S. Bureau of the Census
.
Smith, S. and S. Cody. 1999. “Evaluating the Housing Unit Method: A Case Study of 1990 Population Estimates in Florida.” Presented at the Population Estimates Methods Conference, Population Estimates Branch, Population Division, U.S. Bureau of the Census, Suitland, MD, June 8.
Smith, S., & Sincich, T. (
1988
).
Stability Over Time in the Distribution of Forecast Errors
.
Demography
,
25
,
461
74
. 10.2307/2061544
Snedecor, G., & Cochran, W. (
1980
).
Statistical Methods
. 7th ed.
Ames
:
Iowa State Press
.
Snow, E.C. (
1911
).
The Application of the Method of Multiple Correlation to the Estimation of Post-Censal Populations
.
Journal of the Royal Statistical Society
,
74
,
575
620
. 10.2307/2340076
Stigler, S. (
1986
).
The History of Statistics: The Measurement of Uncertainty Before 1900
.
Cambridge, MA
:
Belknap Press
.
Swanson, D. (
1980
).
Improving Accuracy in Multiple Regression Estimates of Population Using Principles From Causal Modeling
.
Demography
,
17
,
413
28
. 10.2307/2061154
Tayman, J., Schafer, E., & Carter, L. (
1998
).
The Role of Population Size in the Determination and Prediction of Population Forecast Errors: An Evaluation Using Confidence Intervals for Subcounty Areas
.
Population Research and Policy Review
,
17
,
1
20
. 10.1023/A:1005766424443
Tayman, J., & Swanson, D. (
1996
).
On the Utility of Population Forecasts
.
Demography
,
33
,
523
28
. 10.2307/2061785
Tayman, J., & Swanson, D. (
1999
).
On the Validity of MAPE as a Measure of Population Forecast Accuracy
.
Population Research and Policy Review
,
18
,
299
322
. 10.1023/A:1006166418051
Tayman, J., D. Swanson, and C. Barr. Forthcoming. “In Search of the Ideal Measure of Accuracy for Subnational Demographic Forecasts.” Population Research and Policy Review.
Tedrow, L.M., & Swanson, D. (
1990
).
Population Estimation Bibliography
State and Local Agencies Preparing Population and Housing Estimates, Current Population Reports
(pp.
41
52
).
Washington, DC
:
U.S. Bureau of the Census
.
Zitter, M. and F. Cavanaugh. 1980. “Postcensal Estimates of Population.” Presented at the annual meeting of the American Association for the Advancement of Science, San Francisco, January 5