This page documents every step taken to produce the data shown in the dashboard — from raw government CSV files to cleaned datasets, joined tables, AQI calculations, and machine learning models. Nothing is hidden. Every number on the dashboard is traceable to a specific output file.
What this notebook does: Before any cleaning, Phase 0 mapped out
exactly what raw data existed. It loaded the station metadata file
(stations_info.csv), counted all CSV files (one per city),
and catalogued the 97 distinct column names used across 453 stations —
since different stations reported different pollutants with different naming conventions.
Phase 0.5 explored the disease dataset separately — confirming that all five source files shared the same column structure (GBD standard format), that only metric_id 1 (Number) and 3 (Rate) were relevant, and that exactly 21 unique disease cause names appeared across the files. Row counts were verified: 6,720 + 6,720 + 9,408 + 4,032 + 1,344 = 28,224 total rows.
What this notebook does: This is the most complex phase. It loads every city CSV, standardises column names, converts units where measurements were recorded in non-standard formats, drops unusable columns, and aggregates hourly readings into monthly means.
The unit conversion problem: Different stations labelled the
same pollutant in different units. For example, a column named NOx (ppm)
at one station should produce the same values as NOx (ppb) at another —
but 1 ppm = 1,000 ppb. Without correction, those stations would appear to have
1,000× less pollution than reality. Ten conversion rules were applied based on the
original column label, using atmospheric chemistry constants at 25°C, 1 atm.
Data quality tiering: Each city was assigned a quality tier (Tier 1 / 2 / 3) based on how many years of data it had and how complete the readings were. Tier 1 cities had the most complete long-term records and were prioritised for the ML model.
What this notebook does: Loads all five GBD disease files, combines them into a single DataFrame, and filters to keep only Number and Rate metrics (dropping Percent). The national aggregate rows (location = "India") are separated from state-level rows and saved independently.
Evidence strength labelling: Every disease was assigned one of three labels based on the scientific literature — direct (WHO/IARC confirmed causal link, e.g. lung cancer, asthma), strong (mechanism well understood, e.g. tuberculosis, upper respiratory infections), or indirect (pollution increases systemic vulnerability, e.g. neonatal birth outcomes, congenital defects). These labels drive the colour coding in the correlation chart.
The fundamental mismatch: Pollution data is at city × month level. Disease data is at state × year level. They cannot be joined directly.
Phase 3 solves this by collapsing pollution data from city-month rows up to state-year rows — computing the mean PM2.5 (and other pollutants) per state per year, weighted by data coverage. The result is then joined to the disease data on the keys state name + year.
State name standardisation was critical: CPCB uses different naming conventions from GBD (e.g. "Jammu and Kashmir" vs "Jammu & Kashmir and Ladakh"). A manual mapping was applied to ensure clean joins rather than silent NaN mismatches.
join_quality == 'full'
were used in the machine learning analysis — meaning both pollution AND disease data
were present for that state-year combination. Partial joins were excluded to avoid
biased model training.
What this notebook does: Applies the official CPCB AQI breakpoint tables (from the Ministry of Earth Sciences Standard Operating Procedure, citing CPCB National Air Quality Index 2014) to compute a sub-index per pollutant for each city-month. The final AQI is the maximum sub-index across all available pollutants.
Pollutants used: PM2.5, PM10, NO₂, SO₂, CO, NH₃, and Ozone. Lead (Pb) is excluded — it is not measured in the CPCB station network dataset. The AQI formula uses linear interpolation between official breakpoint pairs.
Two separate predictive tasks were defined:
Task B (done first — simpler): Does state-level PM2.5 exposure predict disease death rates? Ordinary Least Squares (OLS) linear regression was applied for each of the 21 disease categories separately. Death rate (per 100,000 population) was used as the target — not raw death count — to control for the large population differences between states. The model was trained on 96 state-year observations.
Task A (time-series ML): Predict next month's city-level PM2.5 from current pollution and weather. The model progression rule was: start with Linear Regression. If R² ≥ 0.6, stop. Since Linear Regression achieved R² = 0.47 on the test set, a Random Forest was trained next. Random Forest achieved R² = 0.594 on the test set.
train_test_split() was explicitly not used — it shuffles rows
randomly, which would allow the model to train on 2022 data while testing on 2019
data. That would produce artificially inflated accuracy. Time-based splitting
is the only valid approach for time-series forecasting.