Statistical Note - Measuring Uncertainty 2015-16

Date: 23 March 2017
Coverage: United Kingdom
Theme: Economy

Measuring uncertainty

This Statistical Notice advises users of the change in methodology for measuring uncertainty in three National Statistics publications from Analytical Services Unit FRS team - the Poverty Bulletin, Households Below Average Income (HBAI) and Pensioners’ Incomes (PI) Series publications. This change will take effect for the 2015/16 publications.

The new methodology provides an improved measure of uncertainty around key HBAI and related PI estimates by:

  • Creating resamples of the HBAI dataset by simulating stratified, cluster sampling for Great Britain (GB) and stratified sampling for Northern Ireland (NI).
  • Creating a unique set of grossing factors for each resample using the HBAI grossing process to gross the GB and NI resamples to the UK population.

The old methodology used a simpler technique of creating simple random resamples of the UK HBAI dataset and re-using the original HBAI grossing factors.

Further detail on the new methodology will be presented in a Quality & Methodology Report later in the year along with associated publications.

If you have any queries on these changes please send them to: PSU@communities-ni.gov.uk

Background

1. As the FRS is a sample from the UK population, any statistics derived from it are only estimates of the true numbers for the overall population. Bootstrapping techniques are used to measure how different a HBAI or PI estimate might have looked if different FRS samples had been drawn.

2. The previous bootstrapping methodology applied the original HBAI grossing factors to simple random resamples of the HBAI dataset to calculate confidence intervals for HBAI estimates.

3. The Institute for Fiscal Studies (IFS) were commissioned to develop the DWP methodology further to account as fully as possible for the specific features of the FRS sampling design for Great Britain (GB) and Northern Ireland (NI) and HBAI grossing process.

4. The new methodology produces:

  • GB resamples simulating the FRS stratified, cluster sampling of GB households.
  • NI resamples simulating the FRS stratified sampling of NI households.
  • A unique set of grossing factors for each GB and NI resample, replicating the original HBAI grossing process, to produce lower and upper confidence intervals.

Impact on HBAI and PI Uncertainty Estimates

5. Accounting for:

  • At a UK level, cluster sampling widens confidence intervals for most estimates, reflecting that this feature makes survey estimates less precise.
  • Post-sample grossing to population totals narrows confidence intervals for estimates sensitive to incomes towards the very top of the income distribution, as specific control totals are set for high income individuals.

6. Further details on methodological work undertaken by IFS, together with illustrative details of the impact of different aspects of the new methodology on key HBAI estimates for 2013/14, are available in a published IFS report .

The Way Forward

7. Published uncertainty estimates in NI Poverty Bulletin, HBAI and PI will be calculated using the new methodology from the 2015/16 publication onwards.

8. A UK back-series of resamples datasets to 2002/03 will be made available on the UK Data Archive later in the year to allow users to produce consistent uncertainty estimates using the new methodology.

Back to top