Skip to content

Guide

Quickstart

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 1) Install (editable dev + docs tooling)
python -m pip install -U pip
pip install -e .[docs,notebook,parquet]

# 2) Configure credentials (Kaggle preferred)
set KAGGLE_USERNAME=...   # or export on Linux/macOS
set KAGGLE_KEY=...
# Optional data root (default = ./cleantech_data)
set CLEANTECH_DATA_DIR=...

# 3) Fetch Bronze data
cleantech-fetch --openalex-pages 3 --openalex-per-page 200 --openalex-mailto you@example.com

# 4) Build Silver artifacts
ctclean media
ctclean patents
ctclean openalex
# or all three:
ctclean all

# 5) Validate & unify latest Silver buckets
ctunify run

Data layout

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
cleantech_data/
  bronze/
    kaggle/
      cleantech-media-dataset/YYYY-MM-DD/{ original.zip, raw.jsonl.gz, raw_manifest.jsonl, extracted/ }
      cleantech-google-patent-dataset/YYYY-MM-DD/{ ... }
    openalex/
      YYYY-MM-DD/{ raw.jsonl.gz, raw_manifest.jsonl }
      topics/YYYY-MM-DD/{ topics.jsonl.gz, raw_manifest.jsonl, extracted/topics.jsonl? }
  silver/
    media/<DATE>/{ media_canonical.*, media_dupe_links.*, media_excluded_*.* }
    patents/<DATE>/{ patent_canonical.*, patent_dupe_links.*, patents_normalized.* }
    openalex/<DATE>/{ topics_canonical.*, topic_*.*, *_ref.* }
    unified/unified_docs.parquet

Repro tips

  • Kaggle mirror: Bronze Kaggle downloads keep the original ZIP and optionally mirror tabular/JSON into a single raw.jsonl.gz (one object per line). See --kaggle-no-mirror to disable.
  • OpenAlex: Works/Topics are fetched with exponential backoff and cursor paging. Use --openalex-pages 0 for “until exhaustion.”
  • Silver gates: Media applies a 200‑char content gate; patents require abstract length ≥ 40; both perform canonicalization with conservative rules.
  • Parquet writing: ctclean.io.safe_write coerces tricky object columns; falls back to CSV.gz if Arrow rejects the schema.
  • CI notebooks: GitHub Actions exports .ipynb → docs/notebooks/*.md (cached). Edit notebooks under /notebooks; CI overwrites the cached pages.