Uncertainty, Confidence Intervals, and Error Bars

Every number comes with uncertainty. Confidence intervals and error bars make that uncertainty visible so readers can judge whether differences are meaningful or just statistical wobble.

What We Track

Intervals around rates: Proportions such as “share with a new petition within 12 months” should include uncertainty bands, especially for small cohorts.
Intervals around counts: Event counts (petitions, referrals) can use Poisson or quasi-Poisson intervals when modeled as rates per exposure (e.g., per person-year).
Comparisons over time or groups: Intervals help distinguish signal from noise across years, counties, or programs.

Choosing an Interval Method

Wilson / Agresti–Coull (binomial proportions): Recommended for rates with modest sample sizes; more stable than the simple “p ± 1.96·SE” approximation.
Exact (Clopper–Pearson): Conservative intervals for very small denominators or rare events.
Poisson intervals: For event counts per exposure time; use when modeling incident events.
Bootstrap: Non-parametric option when distributional assumptions are unclear; document resample size.

How to Read Overlap

Overlapping bars ≠ no difference: Visual overlap of 95% intervals does not strictly test equality; formal tests or modeled contrasts are needed for inference.
Consistent separation: Non-overlapping intervals across multiple years or groups suggest persistent differences beyond sampling noise.
Multiple comparisons: Many side-by-sides inflate false positives; note when adjustments are applied.

Small-N Effects and Pooling

With tiny cohorts, intervals become wide or asymmetric. Year pooling (e.g., 2–3 years) reduces variance and stabilizes interpretation, but pooled spans must be labeled directly on the chart. Where counts remain too small, suppress values and annotate the reason.

Display Conventions

Label the interval type (e.g., “95% Wilson CI”) in tooltips or footnotes.
Use consistent confidence levels across a dashboard (typically 95%).
Distinguish raw series from smoothed/pooled series with line styles and legends.

Data & Methods

The research source emphasizes that administrative rates often lack visible uncertainty, encouraging overconfident interpretation. Interval choice should match the data-generating process (binomial for person-based rates, Poisson for event counts per exposure). Document the method, level (e.g., 90% vs. 95%), any pooling, and whether intervals reflect design weights or clustering.

Transparency note: Charts should disclose the interval method, confidence level, and any pooling. Where uncertainty dominates the signal, say so—silence is a bias.

EDORA • Learn