The static site for old.datahub.io: the original DataHub catalog (11,383 public
datasets). Scaffolded with /portaljs-new-portal and wired to CKAN with
/portaljs-connect-ckan, adapted for a large catalog.
There is no backend. The DataHub v1 CKAN (api.old.datahub.io) was retired on
2026-11-17. The catalog is a pinned snapshot in this repo, uploaded files are served from
R2 by the Worker, and the old API answers 410 Gone. The CKAN database dump is in
gs://archived-dx-projects/database-backups/datahub/datahub-prod-hetzner-2026-11-17.dump
(pg_dump -Fc).
| Step | What happens |
|---|---|
prebuild → npm run load:catalog |
scripts/load-catalog.ts unpacks the pinned snapshot data/catalog.json.gz into .cache/catalog.json (read by lib/providers/ckan-provider.ts), worker/redirect-map.json and worker/download-ids.json. |
next build |
Static export to out/: one page per dataset at /@<org>/<dataset>/. |
postbuild → pagefind --site out |
Builds the chunked static search index in out/pagefind/. /search queries it in the browser. Facets are publisher, format, tag and license; only tags on 5+ datasets become filters. |
data/catalog.json.gz is the last snapshot of the CKAN, taken with
scripts/fetch-catalog.ts (package_search, 1000 per request), which also rendered the
Markdown notes and rewrote upload links to old.datahub.io. That script now only works
against another CKAN: DMS=https://other-ckan npm run fetch:catalog.
npm install
npm run build # full catalog
MAX_DATASETS=300 npm run build # quick dev build
npx serve out # or any static serverThe site is the Cloudflare Worker in worker/. It serves out/ as Workers Static Assets
and handles old CKAN URLs before any asset is served:
- Old CKAN URLs get a 301 to the new pages:
/dataset/<name|id>,/dataset/<x>/resource/<id>(to the#<id>row),/organization/<org>,/dataset?q=…&tags=…&res_format=…. - File downloads (
/dataset/<x>/resource/<id>/download/<file>) are served from R2: bucketdx-ckan-enterprise,datahub-prod/resources/<resource id>/<file>, where CKAN stored the uploads. Only the 1,801 uploads on public datasets are served (worker/download-ids.json): the prefix also holds files of private and deleted datasets. Files are found by resource id, since the name in a link doesn't always match the stored one. /api/*answers 410 Gone in CKAN's error shape;/uploads/*is a 404.api.old.datahub.io, the retired CKAN's host, is served by the same Worker: downloads and the 410 there too, every other path a 301 to old.datahub.io.- Dataset pages are looked up with
@percent-encoded, so/@<org>/<dataset>/stays a clean 200.
npm run build # portal + redirect map
npx tsx worker/redirects.test.ts
cd worker && npx wrangler deploy # → https://old.datahub.ioOnly changed files are re-uploaded on later deploys. PortalJS Arc (/portaljs-deploy) is
the intended host, but its upload API currently fails on this 34k-file export (Cloudflare
1102, a resource limit). It also has no custom domains, so the Worker would be needed in
front of it anyway.
Live on old.datahub.io, a Workers custom domain since 2026-10-13, and on
api.old.datahub.io since 2026-11-17. routes in worker/wrangler.toml keeps both domains
attached across deploys.
PostHog project DataHub v1 (EU cloud, project 303946), set up in lib/analytics.ts:
- Privacy: cookieless (
cookieless_mode: 'always', stateful server hashing) and anonymized IPs, so no consent banner is needed. - What's recorded: pageviews, plus
dataset_search(query, filters, result count) andresource_download(dataset, publisher, format, resource id). No autocapture, replay, surveys or flags. - Where it runs: the SDK loads lazily and only on old.datahub.io. posthog-js drops headless browsers and bots, so test from a real browser.
- Crawl entry points:
public/robots.txtandpublic/sitemap.xml(generated byscripts/generate-sitemap.ts). - Crawlable links: publisher pages
/@<org>/and/publishers/give every dataset a plain inbound link. - Dataset markup: each dataset page has schema.org
DatasetJSON-LD (Google Dataset Search), plus canonical and OG tags (components/Seo.tsx). - After a deploy:
npm run indexnownotifies Bing, Yandex and others. Google gets the sitemap through Search Console.
Not included on purpose:
- DCAT feeds. At this size they would be ~100 MB of single files; use
/portaljs-add-dcatwith pagination if needed. - The datastore. It wasn't restored.
- Chart and map views. Those skills target the static-manifest showcase.
A template variant for portals with many datasets. Instead of one page file per
dataset, the catalog is driven by a single manifest (datasets.json) and rendered by a
dynamic route (pages/[owner]/[slug].tsx + getStaticPaths). Adding a dataset is one
JSON entry plus a data file — no new page.
| Surface | Route | File | What it is |
|---|---|---|---|
| Home | / |
pages/index.tsx |
Landing page: hero + search CTA + suggested-query chips |
| Catalog / search | /search |
pages/search.tsx |
The dataset list with client-side full-text filtering |
| Dataset showcase | /@<namespace>/<slug> |
pages/[owner]/[slug].tsx |
One dataset: metadata, data preview, download/API, views |
The home page's search box and chips navigate to /search?q=…; each search result links
to its showcase at /@<namespace>/<slug>.
portaljs-template |
portaljs-catalog (this) |
|
|---|---|---|
| Dataset pages | one .tsx file per dataset |
one dynamic [owner]/[slug].tsx for all |
| Registration | hardcoded array in index.tsx |
datasets.json manifest |
| Best for | a handful of datasets | dozens to hundreds |
Both ship the same lightweight components/Table.tsx, Tailwind setup, and Next 14 config.
cd examples/portaljs-catalog
npm install
npm run dev- Drop the file in
public/data/(e.g.public/data/my-data.csv). - Append an entry to
datasets.json:
{
"slug": "my-data",
"namespace": "reference",
"name": "My Data",
"description": "One-line description.",
"file": "my-data.csv",
"format": "csv"
}That's it. getStaticPaths picks up the new (namespace, slug) pair at build time and
/@reference/my-data renders automatically. CSV and TSV files are previewed in an
interactive <Table />; other formats (json, geojson) show a download link.
The bundled sample CSVs under public/data/ are small reference data that ships
inline so the basic table demos run offline with zero credentials. The two binary
samples — region-metrics (Parquet) and world-boundaries (PMTiles) — are not
committed; they are hosted on R2 via Giftless and referenced by their
data.portaljs.com URLs in datasets.json, so the showcase demonstrates the real
serverless range-read path rather than shipping bytes in git. For large data, see below.
Committing big files to public/data/ doesn't scale — git bloats and the static export
ships every byte. So the template wires large data through Git LFS → Cloudflare R2
(epic po-g9y):
-
.gitattributesroutes added data through Git LFS per file — it ships minimal and/portaljs-add-datasetappends a path-specific entry for each file it sends to R2 (format-agnostic, any binary). The bundledpublic/data/sample is a fenced exception — it stays inline so the portal runs offline with zero credentials. -
.lfsconfigpoints LFS at the Giftless endpoint, which streams the bytes to R2 and leaves a tiny pointer in git. It carries no credentials — auth is local-only (seegiftless/README.md). -
Serving: the browser fetches the bytes straight from R2. A dataset whose data lives in R2 sets its
resource.path(orfile) to the absolute R2 URL —resourceUrl()inlib/datasets.tspasses absolute URLs through unchanged, so no bytes are copied into the repo or the static export. R2 CORS + range headers are configured and verified (giftless/r2-cors.json), which also unlocks the DuckDB-Wasm range-query tier. -
Deploy:
/portaljs-deployserves large data from R2 too — it never pulls LFS bytes into the static export and gates the upload onnpm run check-export(scripts/check-export.mjs), which fails on LFS pointer leaks or oversized files inout/. The export stays lean; the bytes stay in R2.
/portaljs-add-dataset automates this routing: local files → R2 via LFS by default;
remote URLs → recorded as-is (passthrough) by default, or adopted into R2 on request.
Inline storage is a fenced exception for bundled sample data.
Once data is on R2, the showcase can query it in place instead of downloading the whole file. Convert CSV/TSV to Parquet and the showcase reads it with DuckDB-Wasm over HTTP range requests — projection + predicate pushdown mean a browser fetches only the row groups and columns a query touches, so a phone can query a multi-GB file by pulling a few MB:
scripts/csv-to-parquet.sh public/data/orders.csv orders.parquet
# version with Git LFS (→ R2), then point the resource at the R2 URL with format: "parquet"Every tabular resource renders the in-browser SQL explorer by default
(DATA_QUERY = 'duckdb' in lib/datasets.ts); set it to 'flat' for a
preview-only portal that never needs querying. The engine runs entirely client-side
(no server, no catalog DB) and loads only when a query view mounts. For data too
big for the browser, the same DataQuery seam can route to an edge Worker or
MotherDuck — see lib/query/README.md for the contract and
device-tier / fallback guidance.
Big GeoJSON can't be previewed by loading the whole file. Tile it into a single
PMTiles archive and the showcase renders it with MapLibre GL over HTTP range
requests — the map fetches only the tiles in view, so a multi-GB dataset pans and
zooms with no tile server (served from R2 here, or any static host / /public/data).
Make PMTiles from GeoJSON/Shapefile with tippecanoe
(brew install tippecanoe):
tippecanoe -zg --drop-densest-as-needed -o boundaries.pmtiles boundaries.geojson
# small reference layers can cap the zoom instead: tippecanoe -z5 -o out.pmtiles in.geojsonThen add the resource with format: "pmtiles" (see the R2-hosted
@reference/world-boundaries sample) — the showcase renders the interactive map
preview with click-to-inspect feature properties. The /portaljs-add-geo skill
automates this: it shells out to tippecanoe (and duckdb) on your machine to
derive PMTiles + GeoParquet from one geo upload, pushes them to R2, and registers
the dual-tier dataset for you — the manual tippecanoe recipe below is the
underlying step it runs.
Where PMTiles renders geometry, GeoParquet lets you query it. Add a
resource with format: "geoparquet" and the showcase opens components/GeoQuery.tsx:
DuckDB-Wasm (with the spatial extension) reads the remote GeoParquet in place
over HTTP range requests and runs spatial SQL — the geospatial analog of the
Parquet SQL editor above. Results render as a live MapLibre overlay, a table, and
a bar chart; clicking a feature does a DuckDB point-in-polygon lookup for its full
attributes. A dataset can ship both a pmtiles and a geoparquet resource so
render + query compose on one page (see @reference/world-boundaries).
The queries use a bbox-first pattern — a cheap covering-bbox filter prunes
Parquet row groups, then ST_Intersects refines to the exact predicate:
SELECT name, ST_AsGeoJSON(geometry) AS geojson -- `geojson` column drives the overlay
FROM data
WHERE bbox.xmin <= 40 AND bbox.xmax >= -25 -- prune row groups (Parquet stats)
AND bbox.ymin <= 72 AND bbox.ymax >= 34
AND ST_Intersects(geometry, ST_MakeEnvelope(-25, 34, 40, 72)) -- exact predicateBuild a GeoParquet twin from the same source with duckdb — Hilbert-sorted, with a
GeoParquet 1.1 covering bbox column so row-group pruning works:
duckdb -c "LOAD spatial;
COPY (
SELECT * EXCLUDE (geom),
{'xmin': ST_XMin(geom), 'ymin': ST_YMin(geom),
'xmax': ST_XMax(geom), 'ymax': ST_YMax(geom)} AS bbox,
geom AS geometry
FROM ST_Read('boundaries.geojson')
ORDER BY ST_Hilbert(geom, {'min_x':-180,'min_y':-90,'max_x':180,'max_y':90}::BOX_2D)
) TO 'boundaries.parquet' (FORMAT PARQUET, ROW_GROUP_SIZE 25000);"Keep result sets small (ST_AsGeoJSON is the slow path — decode, not query); for
very large sets project WKB/GeoArrow instead of GeoJSON. @duckdb/duckdb-wasm
loads on demand in the browser only. /portaljs-add-geo runs this exact duckdb
recipe for you as part of its auto-ingest (GeoParquet query tier alongside the
PMTiles render tier); the manual step above is what it automates.
Dataset showcase URLs are namespaced under @ (/@<owner-or-theme>/<dataset>) so they
never collide with regular content/static pages (which never start with @). The
dataset route is a 2-segment dynamic route (pages/[owner]/[slug].tsx) resolved entirely
by static generation from the manifest — so content/static pages should not use
2-segment non-@ paths, or they would clash with the dataset route's matcher.
A portal uses exactly one namespace mode, set via NAMESPACE_TYPE in
lib/datasets.ts:
'theme'— a single-publisher portal whose datasets are grouped by subject (e.g.@reference/country-codes). The showcase labels the namespace "Theme".'owner'— a multi-publisher portal whose datasets are grouped by who published them (e.g.@worldbank/country-codes). The showcase labels the namespace "Owner".
Picking one mode keeps every (@namespace, slug) pair unique. The URL shape is
/@<namespace>/<slug> regardless of which mode is chosen — NAMESPACE_TYPE only changes
the metadata label on the showcase.
/portaljs-new-portal replaces these at scaffold time:
| Token | Replaced with |
|---|---|
Old DataHub |
Human-readable portal name |
old-datahub |
URL-safe slug |
The original DataHub: 11,000+ open datasets published since 2010. |
One-sentence portal description |
The template ships with the PortalJS mark as a clearly-swappable placeholder so a fresh
portal looks intentional before you customize it: a favicon, a navbar logo (with a tasteful
spin on hover that respects prefers-reduced-motion), and social/PWA icons.
To make it your own, replace the icon files in public/ with your own brand marks — the
links in pages/_document.tsx and the logo in components/Navbar.tsx then need no changes:
| File | Used for |
|---|---|
public/icon.svg |
navbar logo + modern-browser favicon (scalable) |
public/favicon.ico |
classic browser-tab favicon (16/32/48) |
public/apple-touch-icon.png |
iOS home-screen icon (180×180) |
public/icon-512.png |
PWA / social card (512×512) |
The navbar's brand text uses the same Old DataHub token, so it is already set to your
portal name after scaffolding.
datasets.json — manifest: the single source of truth for the catalog
lib/datasets.ts — typed loader (getDatasets / getDataset / datasetHref / NAMESPACE_TYPE)
pages/index.tsx — landing page: hero + search CTA + suggested chips
pages/search.tsx — searchable dataset list, reads manifest via getStaticProps
pages/[owner]/[slug].tsx — dynamic dataset showcase (/@<namespace>/<slug>)
pages/_app.tsx — renders the Navbar on every page
pages/_document.tsx — favicon / icon links + default meta description
public/data/ — bundled sample CSVs (inline); Parquet/PMTiles samples live on R2 (data.portaljs.com)
public/{icon.svg,favicon.ico,apple-touch-icon.png,icon-512.png} — branding (placeholder)
components/Navbar.tsx — site navbar: logo (hover-spin) + name + link to /search
components/Table.tsx — interactive table (search, sort, paginate)
.gitattributes — routes large data formats through Git LFS (→ R2)
.lfsconfig — Git LFS endpoint (Giftless → R2); no credentials