Data Foundations

Wed, 25 Feb 2026 12:00:00 +0100

Before building anything, we need a principled way to manage scientific data — datasets that are large, heterogeneous, version-sensitive, and expensive to recompute. The data foundations layer establishes the abstractions that every subsequent lesson builds on.

Lessons Covered

Lesson 00 — Foundations

Datasets, lazy evaluation, and the shape of scientific data management. Introduces the @evaluate_datasets decorator pattern: scientific functions declare what data they need, and the framework resolves, caches, and validates dependencies automatically.

DEM Processing: Reading the Earth's Surface

Wed, 25 Feb 2026 12:00:00 +0100

Everything starts with elevation. The SRTM (Shuttle Radar Topography Mission) provides 1-arc-second (~30 m) digital elevation models covering the entire Parvati Valley and surrounding ranges. These are the raw material from which terrain meshes, hillshades, and satellite-textured landscapes are built.

Data Source

SRTM tiles from AWS Mapzen elevation tiles (public, no authentication):

N31E077 and N32E077 — two tiles covering the Parvati Valley extent
3601 × 3601 pixels per tile, signed 16-bit big-endian
~30 m horizontal resolution, ~1 m vertical accuracy

Processing Pipeline

Download — fetch .hgt.gz tiles from S3
Parse — load as NumPy float32, handle voids (NaN)
Stitch — combine adjacent tiles into continuous elevation grids
Subsample — reduce resolution for mesh generation (step=3 to step=8)
Hillshade — compute synthetic illumination for 2D visualisation
Satellite texture — fetch Sentinel-2 Cloudless composite from EOX WMS