Coverage Estimation for Health Programs using Administrative Data (viz, HMIS/DHIS2)

In my previous **blog**, I shared my thoughts on applying the Bayesian paradigm in Public Health. In this blog, I will throw an idea of estimating any public health indicators by combining the administrative data (HMIS) and a sample survey, with a small sample size. If we can scale up this idea, I believe that there will be a substantial reduction in the cost of any sample survey.

Source: https://qafya.co.ke/

This blog seeks to promote an easily implementable, cost-effective, and scientifically grounded methodology for estimating the coverage of various public health programs, including immunization coverage and maternal, neonatal, and child health (MNCH) program coverage, etc. This approach is facilitated by leveraging the advantages afforded by the Bayesian Paradigm.

Before going into unpacking the terms such as coverage estimation and Bayesian paradigm, let’s start with the problem that we want to address.

The problem statement

How do we use the aggregated administrative data to provide robust population-level estimates of health program coverage?

Health coverage data is collected from facility-level records and subsequently entered into the Health Management Information System (HMIS). For instance, information from delivery room registers across all facilities is compiled and integrated into HMIS (or DHIS2 in some countries). This process ultimately produces the total number of women who delivered in a health facility within an administrative unit, district, state/province, or country.

However, several factors impede the reliability of this data source for estimating coverage in low- and middle-income countries (LMICs). Firstly, this data source exclusively represents individuals who visit public sector health facilities; the quality of many indicators is suboptimal, with substantial grounds for both over- and underreporting, as these data are utilized for gauging the performance of health officials (indirectly). Furthermore, challenges persist in accurately estimating the denominators, which are often crudely approximated by combining survey and census data.

As a result, there is a marked reliance on periodic sample survey data, which is costly, occurs at an interval of 3 to 5 years (such as the Demographic and Health Surveys — DHS), and is only accessible for larger administrative areas. Consequently, estimates for smaller administrative units are typically either unavailable or unreliable.

Public health experts have proposed certain methodologies, such as adjusting the numerators and denominators by employing a correction factor (Isabella Maina, 2017) and the statistical annealing technique (AT) that combines survey data and HMIS data to generate reliable estimates along with the corresponding standard error, which serves as the frequentist measure of uncertainty (Caroline Jeffery, 2022).

However, these methodologies are not extensively applied, primarily due to the necessity for accurate data quality estimations and the infrequency of available survey data at regular intervals.

Propose methodology

I would like to propose an easy and robust method that we can implement as a concurrent method, which is low-cost in comparison to other methods, and provides a robust and globally accepted measure of coverage and uncertainty. At the same time provides a real-time measure of the data quality of HMIS/DHIS2 reports.

This means we need to provide

a. The estimated level of coverage (for example, the proportion of women delivered at the institution)

b. The level of uncertainty associated with the coverage indicator (Uncertainty interval)

I am proposing a simple Bayesian method and explaining it step by step.

In this approach, we start with an initial belief or assumption about the likely values of an indicator — this is called the ‘prior.’ Then, we collect new data — this is called the ‘evidence’ or ‘likelihood.’ By combining our initial belief with the new data, we update our understanding and get a final result, called the ‘posterior’, which gives us the most likely estimate based on both our prior belief and the observed data

The example we are using here is estimating institutional delivery in a small area of 10,000–20,000 population, and one facility, which can be easily scaled without any change, following the same steps.

The steps are as follows, and the calculation can be automated without much effort

1. Let’s start with a prior belief about the institutional delivery (ID) rate in the assessment areas (a catchment area of a facility or intervention, or an administrative area). The prior belief about the institutional rates and uncertainty can be arrived at using HMIS/DHIS2 aggregated data. Where the rate (ID) can be calculated by dividing the numerator, the Number of deliveries at the health facility, and the Denominator: Preferably, an HMIS indicator with a very high level of coverage (>90%) validated through any large-scale survey such as DHS (a WHO method1) OR use the crude measure of estimating the denominator using Crude Birth Rate (CBR).
2. Next, capture the belief about uncertainty in consultation with the local implementor or experts and assign an uncertainty. The uncertainty statement would be like this: “We think from HMIS that the institutional delivery is 67%, and we are almost sure that it is within the range of 60% or 75%; there is a small chance that it will go outside the range of 60–70%”.
3. Points 1 and 2 can be used for drawing a distribution of prior, believe- in this case, it will be a Beta distribution with location (a) and scale parameters (b). A Beta (a, b) distribution. The values of a’ andb’ can be chosen to reflect the prior belief about the institutional delivery rate, which is 67%, with an uncertainty that the value can vary between 60% and 70%. In this example, the beta function is Beta (36, 18), which is easy to compute.
4. The next step is to draw the sample from the population (Generate likelihood function). The sample size does not matter in this paradigm. However, randomness is an essential component, like the conventional frequentist approach. Let us say that we decide to collect a random sample of 5 from the population, who have delivered during the reporting month, and capture the outcome (institutional or home delivery) if 3 (=y)of the 5 =n) samples had an institutional delivery, then we get the likelihood function: where p is the prior rate of institutional delivery, we guessed from HMIS.

5. We can now get the final estimates with uncertainty measures. The posterior distribution = (Prior X Likelihood)/ (Normalising constant), which in this case is also a Beta distribution by property, Beta (a+y, b+n-y), where n=5 and y=3 in our example.

6. Therefore, if the prior is Beta (a=36, b=18), the posterior becomes Beta (39, 20).

7. We can now take a sample from the Beta (39,20) distribution (Figure 1) to compute several summary statistics about the institutional delivery (ID) rate in the catchment area, including mean, median, quantiles, and credible interval CI (a measure of uncertainty).

8. Here, we can easily compute the data quality measure without additional assessment and prioritize the catchment areas where data quality is poor (or needs improvement). In places where the posterior is close prior (belief based on HMIS), the data quality is good. However, where the posterior is close to the likelihood, the data quality of HMIS is relatively poor. There is no need for a separate data quality assessment. Figure 1 is an example of good data quality. In this case, the posterior distribution (in red) is close to the prior distribution based on HMIS (blue). The inference did not change substantially by accruing new evidence.

Figure 1: Prior, likelihood, and Posterior

9. Now, if we implement this method across several hyperlocal or sub-geographies (PHC areas, Health Post, Sub-Centre areas), we can answer various types of implementation and planning-related questions. For example, if there are 10 catchment areas, with 80–90% institutional delivery in each area. It is difficult to prioritize catchment areas that need more support. However, in this method, we can ask questions such as, which are the areas where there is a> 90% chance that institutional delivery is less than 70% (say).

The following snippet shows the Python code for the above example:

# The code
import os
os.chdir('PATH')

# Import libraries 

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta, binom

# Prior parameters
alpha_prior, beta_prior = 36, 18
alpha_post, beta_post = 39, 20
n, x = 5, 3  # Survey data: 5 trials, 3 successes

# Define range of probabilities
p = np.linspace(0, 1, 1000)

# Prior distribution
prior_pdf = beta.pdf(p, alpha_prior, beta_prior)

# Likelihood function
likelihood = binom.pmf(x, n, p)

# Posterior distribution
posterior_pdf = beta.pdf(p, alpha_post, beta_post)

# Plotting
plt.figure(figsize=(12, 8))
plt.plot(p, prior_pdf, label=f'Prior: Beta({alpha_prior}, {beta_prior})', color='blue', linewidth=2)
plt.plot(p, likelihood * 10, label=f'Likelihood: Binomial({n}, {x}) (scaled)', color='green', linestyle='--', linewidth=2)
plt.plot(p, posterior_pdf, label=f'Posterior: Beta({alpha_post}, {beta_post})', color='red', linewidth=2)

# Labels and legend
plt.xlabel('p (Institutional Delivery Rate)')
plt.ylabel('Density / Scaled Probability')
plt.title('Prior, Likelihood, and Posterior Distributions')
plt.legend()
plt.grid(True)
plt.tight_layout()

plt.show()Reference:
1. Maina I, Wanjala P, Soti D, Kipruto H, Droti B, Boerma T. 
Using health-facility data to assess subnational coverage of maternal and 
child health indicators, Kenya. Bull World Health Organ.
2017 Oct 1;95(10):683-694. doi: 810.2471/BLT.17.194399. 
Epub 2017 Aug 28. PMID: 29147041; PMCID: PMC5689197.

2. Jeffery C, Pagano M, Devkota B, et alInnovative approach to improve 
information accuracin a two-district cross-sectional study in Bihar, 
IndiaBMJ Open 2022;12:e051427. doi: 10.1136/bmjopen-2021-05

I will be happy to know your thoughts on the same.

Reference:
1. Maina I, Wanjala P, Soti D, Kipruto H, Droti B, Boerma T. 
Using health-facility data to assess subnational coverage of maternal and 
child health indicators, Kenya. Bull World Health Organ. 
2017 Oct 1;95(10):683-694. doi: 810.2471/BLT.17.194399. 
Epub 2017 Aug 28. PMID: 29147041; PMCID: PMC5689197.

2. Jeffery C, Pagano M, Devkota B, et alInnovative approach to 
improve information accuracy in a two-district cross-sectional 
study in Bihar, IndiaBMJ Open 2022;12:e051427. 
doi: 10.1136/bmjopen-2021-051427