Tutorial 10 of 13
Public datasets, snapshots and comparisons over time
Intermediate 8 min
Fetch electricity, spending, package-download or web-archive data into a dataset, refresh it with Run all and compare every refresh with the previous version.
The Public data steps of Data & charts call site tools that read open data sources and write the result into a dataset. Every later step — charts, time series, statistics — works on that dataset like on your own data.
The sources
Open Data → Fetch public data or the Fetch public data tile on the start page:
| Step | Source | Dataset it creates |
|---|---|---|
| Electricity data | Energy-Charts (Fraunhofer ISE) | Generation by source: GWh, share, peak MW, renewable |
| Public spending | USAspending.gov or EU tenders (TED) | One row per award: buyer, recipient, amount, dates |
| Package downloads | npm, PyPI or crates.io statistics | Weekly downloads |
| Web archive changes | Internet Archive Wayback Machine | One row per capture, with new versions marked |
| Satellite images | NASA GIBS | No table: dates and links to the before/after images |
Fetch a year of package downloads
Open the example Open-source package adoption. The first step, Fetch npm downloads, asks the Software Adoption Explorer for chart.js. Change Package name to your own package if you like, then press Run all above the list of steps.
Run all works in a fixed order: public-data steps first (they produce the data), then unit conversions, then every other tool step. It shows a pause countdown when the server asks RavingDesk to slow down — heavy tools are limited to a few calls per minute — and continues on its own. You can keep editing meanwhile; Stop ends the run after the current call.
After the run, the dataset Weekly downloads of chart.js is filled, the local chart Weekly downloads draws it and Download trend has its findings, for example the growth of the last four weeks and a peak week.
Refresh and compare
Press Run all again next week. Before the fetch replaces the data, RavingDesk stores the old table as a snapshot (Before the refresh of …). Open the Snapshots desk (Snapshots and comparisons):
- Dataset, Older version and Newer version pick what to compare.
- Match rows by chooses the key column (RavingDesk suggests the first column whose values are unique, here Week start).
- The cards count Added, Removed, Changed and Unchanged rows; Column totals shows how each numeric column’s sum moved; Changed values lists individual cells.
Download comparison (CSV) exports the whole list. Restore puts an old version back (the current one becomes a snapshot itself), and Take a snapshot now keeps a version manually — for example before you clean data.
Remind yourself
Open the Reminders desk and press Add a reminder: RavingDesk suggests a title and the repeat interval from the dataset’s Expected update interval. Download calendar (.ics) creates repeating events for Google Calendar, Outlook or Apple Calendar. RavingDesk itself does not run in the background; the reminder brings you back to press Run all.
Honest limits
- Public sources can be slow or temporarily unavailable; the step then says so in a sentence and keeps its previous result.
- Spending searches list at most 100 awards; when more exist, the findings say that the sums cover only the listed awards.
- Satellite images open from NASA’s servers in a new tab — the tool compares dates and areas, not the pixels.
Ready to try it?