Skip to content

About

PortalJS based Data Portal for DataHub

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

DataHub v1

The static site for old.datahub.io: the original DataHub catalog (11,383 public datasets). Scaffolded with /portaljs-new-portal and wired to CKAN with /portaljs-connect-ckan, adapted for a large catalog.

There is no backend. The DataHub v1 CKAN (api.old.datahub.io) was retired on 2026-11-17. The catalog is a pinned snapshot in this repo, uploaded files are served from R2 by the Worker, and the old API answers 410 Gone. The CKAN database dump is in gs://archived-dx-projects/database-backups/datahub/datahub-prod-hetzner-2026-11-17.dump (pg_dump -Fc).

How it builds

Step What happens
prebuild → npm run load:catalog scripts/load-catalog.ts unpacks the pinned snapshot data/catalog.json.gz into .cache/catalog.json (read by lib/providers/ckan-provider.ts), worker/redirect-map.json and worker/download-ids.json.
next build Static export to out/: one page per dataset at /@<org>/<dataset>/.
postbuild → pagefind --site out Builds the chunked static search index in out/pagefind/. /search queries it in the browser. Facets are publisher, format, tag and license; only tags on 5+ datasets become filters.

data/catalog.json.gz is the last snapshot of the CKAN, taken with scripts/fetch-catalog.ts (package_search, 1000 per request), which also rendered the Markdown notes and rewrote upload links to old.datahub.io. That script now only works against another CKAN: DMS=https://other-ckan npm run fetch:catalog.

npm install
npm run build                    # full catalog
MAX_DATASETS=300 npm run build   # quick dev build
npx serve out                    # or any static server

Deploy

The site is the Cloudflare Worker in worker/. It serves out/ as Workers Static Assets and handles old CKAN URLs before any asset is served:

  • Old CKAN URLs get a 301 to the new pages: /dataset/<name|id>, /dataset/<x>/resource/<id> (to the #<id> row), /organization/<org>, /dataset?q=…&tags=…&res_format=….
  • File downloads (/dataset/<x>/resource/<id>/download/<file>) are served from R2: bucket dx-ckan-enterprise, datahub-prod/resources/<resource id>/<file>, where CKAN stored the uploads. Only the 1,801 uploads on public datasets are served (worker/download-ids.json): the prefix also holds files of private and deleted datasets. Files are found by resource id, since the name in a link doesn't always match the stored one.
  • /api/* answers 410 Gone in CKAN's error shape; /uploads/* is a 404.
  • api.old.datahub.io, the retired CKAN's host, is served by the same Worker: downloads and the 410 there too, every other path a 301 to old.datahub.io.
  • Dataset pages are looked up with @ percent-encoded, so /@<org>/<dataset>/ stays a clean 200.
npm run build                                   # portal + redirect map
npx tsx worker/redirects.test.ts
cd worker && npx wrangler deploy                # → https://old.datahub.io

Only changed files are re-uploaded on later deploys. PortalJS Arc (/portaljs-deploy) is the intended host, but its upload API currently fails on this 34k-file export (Cloudflare 1102, a resource limit). It also has no custom domains, so the Worker would be needed in front of it anyway.

Live on old.datahub.io, a Workers custom domain since 2026-10-13, and on api.old.datahub.io since 2026-11-17. routes in worker/wrangler.toml keeps both domains attached across deploys.

Analytics

PostHog project DataHub v1 (EU cloud, project 303946), set up in lib/analytics.ts:

  • Privacy: cookieless (cookieless_mode: 'always', stateful server hashing) and anonymized IPs, so no consent banner is needed.
  • What's recorded: pageviews, plus dataset_search (query, filters, result count) and resource_download (dataset, publisher, format, resource id). No autocapture, replay, surveys or flags.
  • Where it runs: the SDK loads lazily and only on old.datahub.io. posthog-js drops headless browsers and bots, so test from a real browser.

SEO

  • Crawl entry points: public/robots.txt and public/sitemap.xml (generated by scripts/generate-sitemap.ts).
  • Crawlable links: publisher pages /@<org>/ and /publishers/ give every dataset a plain inbound link.
  • Dataset markup: each dataset page has schema.org Dataset JSON-LD (Google Dataset Search), plus canonical and OG tags (components/Seo.tsx).
  • After a deploy: npm run indexnow notifies Bing, Yandex and others. Google gets the sitemap through Search Console.

Not included on purpose:

  • DCAT feeds. At this size they would be ~100 MB of single files; use /portaljs-add-dcat with pagination if needed.
  • The datastore. It wasn't restored.
  • Chart and map views. Those skills target the static-manifest showcase.

PortalJS Catalog Template (dynamic routes)

A template variant for portals with many datasets. Instead of one page file per dataset, the catalog is driven by a single manifest (datasets.json) and rendered by a dynamic route (pages/[owner]/[slug].tsx + getStaticPaths). Adding a dataset is one JSON entry plus a data file — no new page.

The three surfaces

Surface Route File What it is
Home / pages/index.tsx Landing page: hero + search CTA + suggested-query chips
Catalog / search /search pages/search.tsx The dataset list with client-side full-text filtering
Dataset showcase /@<namespace>/<slug> pages/[owner]/[slug].tsx One dataset: metadata, data preview, download/API, views

The home page's search box and chips navigate to /search?q=…; each search result links to its showcase at /@<namespace>/<slug>.

When to use this vs portaljs-template

portaljs-template portaljs-catalog (this)
Dataset pages one .tsx file per dataset one dynamic [owner]/[slug].tsx for all
Registration hardcoded array in index.tsx datasets.json manifest
Best for a handful of datasets dozens to hundreds

Both ship the same lightweight components/Table.tsx, Tailwind setup, and Next 14 config.

Running

cd examples/portaljs-catalog
npm install
npm run dev

Adding a dataset

  1. Drop the file in public/data/ (e.g. public/data/my-data.csv).
  2. Append an entry to datasets.json:
{
  "slug": "my-data",
  "namespace": "reference",
  "name": "My Data",
  "description": "One-line description.",
  "file": "my-data.csv",
  "format": "csv"
}

That's it. getStaticPaths picks up the new (namespace, slug) pair at build time and /@reference/my-data renders automatically. CSV and TSV files are previewed in an interactive <Table />; other formats (json, geojson) show a download link.

The bundled sample CSVs under public/data/ are small reference data that ships inline so the basic table demos run offline with zero credentials. The two binary samples — region-metrics (Parquet) and world-boundaries (PMTiles) — are not committed; they are hosted on R2 via Giftless and referenced by their data.portaljs.com URLs in datasets.json, so the showcase demonstrates the real serverless range-read path rather than shipping bytes in git. For large data, see below.

Large data (Git LFS → R2)

Committing big files to public/data/ doesn't scale — git bloats and the static export ships every byte. So the template wires large data through Git LFS → Cloudflare R2 (epic po-g9y):

  • .gitattributes routes added data through Git LFS per file — it ships minimal and /portaljs-add-dataset appends a path-specific entry for each file it sends to R2 (format-agnostic, any binary). The bundled public/data/ sample is a fenced exception — it stays inline so the portal runs offline with zero credentials.

  • .lfsconfig points LFS at the Giftless endpoint, which streams the bytes to R2 and leaves a tiny pointer in git. It carries no credentials — auth is local-only (see giftless/README.md).

  • Serving: the browser fetches the bytes straight from R2. A dataset whose data lives in R2 sets its resource.path (or file) to the absolute R2 URL — resourceUrl() in lib/datasets.ts passes absolute URLs through unchanged, so no bytes are copied into the repo or the static export. R2 CORS + range headers are configured and verified (giftless/r2-cors.json), which also unlocks the DuckDB-Wasm range-query tier.

  • Deploy: /portaljs-deploy serves large data from R2 too — it never pulls LFS bytes into the static export and gates the upload on npm run check-export (scripts/check-export.mjs), which fails on LFS pointer leaks or oversized files in out/. The export stays lean; the bytes stay in R2.

/portaljs-add-dataset automates this routing: local files → R2 via LFS by default; remote URLs → recorded as-is (passthrough) by default, or adopted into R2 on request. Inline storage is a fenced exception for bundled sample data.

Query tier — query large data without downloading it

Once data is on R2, the showcase can query it in place instead of downloading the whole file. Convert CSV/TSV to Parquet and the showcase reads it with DuckDB-Wasm over HTTP range requests — projection + predicate pushdown mean a browser fetches only the row groups and columns a query touches, so a phone can query a multi-GB file by pulling a few MB:

scripts/csv-to-parquet.sh public/data/orders.csv orders.parquet
# version with Git LFS (→ R2), then point the resource at the R2 URL with format: "parquet"

Every tabular resource renders the in-browser SQL explorer by default (DATA_QUERY = 'duckdb' in lib/datasets.ts); set it to 'flat' for a preview-only portal that never needs querying. The engine runs entirely client-side (no server, no catalog DB) and loads only when a query view mounts. For data too big for the browser, the same DataQuery seam can route to an edge Worker or MotherDuck — see lib/query/README.md for the contract and device-tier / fallback guidance.

Map tier — preview any-size geo data (PMTiles)

Big GeoJSON can't be previewed by loading the whole file. Tile it into a single PMTiles archive and the showcase renders it with MapLibre GL over HTTP range requests — the map fetches only the tiles in view, so a multi-GB dataset pans and zooms with no tile server (served from R2 here, or any static host / /public/data).

Make PMTiles from GeoJSON/Shapefile with tippecanoe (brew install tippecanoe):

tippecanoe -zg --drop-densest-as-needed -o boundaries.pmtiles boundaries.geojson
# small reference layers can cap the zoom instead: tippecanoe -z5 -o out.pmtiles in.geojson

Then add the resource with format: "pmtiles" (see the R2-hosted @reference/world-boundaries sample) — the showcase renders the interactive map preview with click-to-inspect feature properties. The /portaljs-add-geo skill automates this: it shells out to tippecanoe (and duckdb) on your machine to derive PMTiles + GeoParquet from one geo upload, pushes them to R2, and registers the dual-tier dataset for you — the manual tippecanoe recipe below is the underlying step it runs.

Geo query tier — spatial SQL over GeoParquet (no download)

Where PMTiles renders geometry, GeoParquet lets you query it. Add a resource with format: "geoparquet" and the showcase opens components/GeoQuery.tsx: DuckDB-Wasm (with the spatial extension) reads the remote GeoParquet in place over HTTP range requests and runs spatial SQL — the geospatial analog of the Parquet SQL editor above. Results render as a live MapLibre overlay, a table, and a bar chart; clicking a feature does a DuckDB point-in-polygon lookup for its full attributes. A dataset can ship both a pmtiles and a geoparquet resource so render + query compose on one page (see @reference/world-boundaries).

The queries use a bbox-first pattern — a cheap covering-bbox filter prunes Parquet row groups, then ST_Intersects refines to the exact predicate:

SELECT name, ST_AsGeoJSON(geometry) AS geojson   -- `geojson` column drives the overlay
FROM data
WHERE bbox.xmin <= 40 AND bbox.xmax >= -25        -- prune row groups (Parquet stats)
  AND bbox.ymin <= 72 AND bbox.ymax >= 34
  AND ST_Intersects(geometry, ST_MakeEnvelope(-25, 34, 40, 72))  -- exact predicate

Build a GeoParquet twin from the same source with duckdb — Hilbert-sorted, with a GeoParquet 1.1 covering bbox column so row-group pruning works:

duckdb -c "LOAD spatial;
  COPY (
    SELECT * EXCLUDE (geom),
      {'xmin': ST_XMin(geom), 'ymin': ST_YMin(geom),
       'xmax': ST_XMax(geom), 'ymax': ST_YMax(geom)} AS bbox,
      geom AS geometry
    FROM ST_Read('boundaries.geojson')
    ORDER BY ST_Hilbert(geom, {'min_x':-180,'min_y':-90,'max_x':180,'max_y':90}::BOX_2D)
  ) TO 'boundaries.parquet' (FORMAT PARQUET, ROW_GROUP_SIZE 25000);"

Keep result sets small (ST_AsGeoJSON is the slow path — decode, not query); for very large sets project WKB/GeoArrow instead of GeoJSON. @duckdb/duckdb-wasm loads on demand in the browser only. /portaljs-add-geo runs this exact duckdb recipe for you as part of its auto-ingest (GeoParquet query tier alongside the PMTiles render tier); the manual step above is what it automates.

Why dataset URLs start with @

Dataset showcase URLs are namespaced under @ (/@<owner-or-theme>/<dataset>) so they never collide with regular content/static pages (which never start with @). The dataset route is a 2-segment dynamic route (pages/[owner]/[slug].tsx) resolved entirely by static generation from the manifest — so content/static pages should not use 2-segment non-@ paths, or they would clash with the dataset route's matcher.

A portal uses exactly one namespace mode, set via NAMESPACE_TYPE in lib/datasets.ts:

  • 'theme' — a single-publisher portal whose datasets are grouped by subject (e.g. @reference/country-codes). The showcase labels the namespace "Theme".
  • 'owner' — a multi-publisher portal whose datasets are grouped by who published them (e.g. @worldbank/country-codes). The showcase labels the namespace "Owner".

Picking one mode keeps every (@namespace, slug) pair unique. The URL shape is /@<namespace>/<slug> regardless of which mode is chosen — NAMESPACE_TYPE only changes the metadata label on the showcase.

Placeholder tokens

/portaljs-new-portal replaces these at scaffold time:

Token Replaced with
Old DataHub Human-readable portal name
old-datahub URL-safe slug
The original DataHub: 11,000+ open datasets published since 2010. One-sentence portal description

Branding (placeholder — swap it)

The template ships with the PortalJS mark as a clearly-swappable placeholder so a fresh portal looks intentional before you customize it: a favicon, a navbar logo (with a tasteful spin on hover that respects prefers-reduced-motion), and social/PWA icons.

To make it your own, replace the icon files in public/ with your own brand marks — the links in pages/_document.tsx and the logo in components/Navbar.tsx then need no changes:

File Used for
public/icon.svg navbar logo + modern-browser favicon (scalable)
public/favicon.ico classic browser-tab favicon (16/32/48)
public/apple-touch-icon.png iOS home-screen icon (180×180)
public/icon-512.png PWA / social card (512×512)

The navbar's brand text uses the same Old DataHub token, so it is already set to your portal name after scaffolding.

Structure

datasets.json              — manifest: the single source of truth for the catalog
lib/datasets.ts            — typed loader (getDatasets / getDataset / datasetHref / NAMESPACE_TYPE)
pages/index.tsx            — landing page: hero + search CTA + suggested chips
pages/search.tsx           — searchable dataset list, reads manifest via getStaticProps
pages/[owner]/[slug].tsx   — dynamic dataset showcase (/@<namespace>/<slug>)
pages/_app.tsx             — renders the Navbar on every page
pages/_document.tsx        — favicon / icon links + default meta description
public/data/               — bundled sample CSVs (inline); Parquet/PMTiles samples live on R2 (data.portaljs.com)
public/{icon.svg,favicon.ico,apple-touch-icon.png,icon-512.png} — branding (placeholder)
components/Navbar.tsx       — site navbar: logo (hover-spin) + name + link to /search
components/Table.tsx       — interactive table (search, sort, paginate)
.gitattributes             — routes large data formats through Git LFS (→ R2)
.lfsconfig                 — Git LFS endpoint (Giftless → R2); no credentials

About

PortalJS based Data Portal for DataHub

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages