Where the EA Said No
A browser-native map and scraping pipeline that pulls flood-risk planning decisions out of hundreds of separate council portals and shows, case by case, where homes were approved on floodplains against the Environment Agency's own objection.
- Period
- 2026
- Role
- Developer and data engineer
- Scale
- England and Wales planning system
- Context
- Civic, Research
The Environment Agency objects to some planning applications because the site sits in a floodplain, yet councils sometimes grant permission anyway. That pattern matters to anyone who might one day live in, buy, or insure those homes, but it is scattered across roughly 426 separate council planning portals and buried inside unstructured application PDFs, so nobody can see it whole. Flooded People UK held a gov.uk list of EA flood-risk objections and their outcomes, but it was only reference numbers and metadata; the task was to turn that thin index into a searchable, mappable evidence base without a single central planning database to draw on.
My contribution
Working from the Flooded People UK objections list, I built the full pipeline end to end. I fingerprinted all 426 authority portals by the software running them (Idox is dominant, with 13,674 applications queued against that scraper alone) so one general scraper could serve many councils instead of one per site. I wrote the Idox scraper to pull application detail pages and their document sets, then built a PDF extraction pipeline that sends each planning form natively to the Gemini API and returns structured JSON: applicant, agent, agent company, site address and proposal. I geocoded sites, joined them to the EA Flood Map, and collated 23,372 applications across the 426 authorities into a single Parquet and PMTiles store. The front end is a Next.js 15 and MapLibre map that runs its analytics client-side in DuckDB-WASM over the Parquet file, with the flood-zone overlay and an LPA choropleth, served as static files from Cloudflare R2 on zero-egress so it costs almost nothing to keep live.
Decisions and constraints
Decisions I made
- Fingerprint portals by platform first, then write one scraper per platform, so targeting the software rather than each of 426 councils turned an intractable per-site problem into a small prioritised set of general scrapers.
- Send each PDF to the model visually rather than parse its text layer, because the council name is only in the rendered header; native visual extraction lifted field accuracy from 80.9% to 99.7% on the scored set.
- Run analytics client-side in DuckDB-WASM over Parquet so the map is a static-file deployment with no backend to run, secure or pay for.
- Keep provenance visible by separating advice-followed, granted-against-advice and unknown outcomes rather than smoothing them, so the reader can see how much is still unresolved.
Operating constraint
No central planning database exists in England: each of the 426 authorities publishes to its own portal, on differing software, with the council's own identity absent from the machine-readable text layer of the PDFs. The dataset also grows as more portals are scraped, so the whole thing had to be built to keep coverage honest rather than pretend completeness.
Claim boundary: Work in progress: only a subset of the objected applications is fully scraped and geocoded so far, so the map shows a slice, not the whole. The source objections list was obtained by Flooded People UK from public gov.uk records; I built the pipeline and map on top of it. The 99.7% figure is measured on a small scored test set, not the full corpus. The map surfaces patterns for investigation and does not, on its own, prove wrongdoing by any council or developer.
Evidence
- 23,372 applications collated across 426 local planning authorities, of which 636 are recorded as permission granted against Environment Agency flood-risk advice, mapped and rankable by authority.
- A PDF extraction pipeline scoring 99.7% field accuracy on the scored test set at 2.4s per document sequential and about 0.25s per document across 10 workers, with a costed path to the full corpus of roughly 370,000 PDFs.
- Platform-general council-portal scrapers (Idox implemented, 13,674 applications queued) plus applicant and agent parsing that surfaces repeat developers.
- A browser-native map (Next.js 15, MapLibre, PMTiles, DuckDB-WASM over Parquet) with an EA flood-zone overlay and LPA choropleth, served from object storage on zero egress.