Repository navigation
[Feature] Automate Flickr Data Fetching #164
Description
Caution
Because it requires a paid subscription (Flickr Pro), Flickr is not currently a viable data source.
Problem
I've been exploring the codebase and noticed that Flickr data collection is still in pre-automation/flickr/ while other sources like Google Custom Search and GitHub have been automated. The current Flickr scripts appear to be manual and haven't been integrated with the automated pipeline yet.
Description
Looking at pre-automation/flickr/photos.py, it currently:
- Searches for photos with different Creative Commons licenses (IDs: 1,2,3,4,5,6,9,10)
- Gets 500 photos per license
- Saves everything to a JSON file
I noticed some differences compared to the automated scripts:
- Output: Flickr uses JSON, but GCS/GitHub use CSV
- Arguments: The automated scripts have
--enable-save,--enable-gitetc. - Setup: Automated scripts use
shared.setup()instead of the oldquantify.setup() - Integration: They integrate with git automation and quarterly folders
The Pipfile already mentions flickrapi and has a planned script path: flickr_fetched = "./scripts/1-fetch/flickr_fetched.py"
Implementation Plan
Following the guidance to plan each phase before implementation, and after studying the 2025Q3 report structure:
Research Findings
The GCS report is way more detailed than I expected, it's not just counting licenses, but showing breakdowns by different countries, languages, and grouping licenses into categories. All the charts look the same with bars on the left and pie charts on the right, and they use the same colors throughout.
Looking at this made me understand why planning carefully is important, with such a large Flickr dataset, I could easily collect tons of data that doesn't actually help with making useful reports.
Fetch Phase
Build new automation (using pre-automation as reference only):
- Create
scripts/1-fetch/flickr_fetch.pyfollowing the same pattern asgcs_fetch.py - Use the same setup method, save to CSV files, same command options
- Focus on collecting data that will actually be useful for the reporting phase
- Get license types (1,2,3,4,5,6,9,10) plus any metadata that supports analysis
Process Phase
Transform data to support reporting:
- Map Flickr license IDs to actual CC license names and versions using the Flickr API reference
- Group licenses the same way as GCS (Latest/Prior/Retired categories)
- Process any geographic or language data to match the report structure
- Make sure everything aligns with what will actually be shown in charts
Report Phase
Create visualizations that add value:
- License distribution analysis (similar to GCS "Products totals")
- Status breakdown (Latest/Prior/Retired)
- Geographic or language analysis if the data supports it
- Focus on analyses that make sense for visual content
Next Steps
I'd like to start by understanding the license mapping and what kinds of reports would be most useful. Then build the fetch script to collect only the data that will actually be used in reporting.
This way I can avoid doing a lot of fetch work that won't end up being helpful for the final reports.
Alternatives
- Build Flickr automation following a different pattern than GCS/GitHub
- Focus on other data sources first and return to Flickr later
- Start with a smaller subset of licenses to test the approach
Additional Context
- I'm still learning the codebase patterns, but I can see the structure used by other automated sources
- The scripts use Creative Commons license IDs that Flickr recognizes
- Currently limited to 500 photos per license and outputs to JSON format
- Could potentially be modernized to follow the pattern used by other automated sources in
scripts/1-fetch/ - Pre-automation effort:
Activity
oree-xx commentedon Oct 10, 2025
Hi @SaurabhCodesAI I just like to share some insight. So contribute to the project now, you would follow the structure of the three phase process of fetching, processing and reporting the data which in under the script directory. So from the pre automation code you have gotten some insights on how flickr api works and all, you just need to adapt it to the structure of gcs_fetch.py or github_fetch.py which has the best practices to implement the flickr data source.
TimidRobot commentedon Oct 10, 2025
@SaurabhCodesAI Yes, adding Flickr would be helpful.
Please note that the nothing in the
pre-automation/directory is currently used (is is there as a reference, not a manual approach).TimidRobot commentedon Oct 10, 2025
@SaurabhCodesAI Also, the Flickr dataset is quite large. I recommend documenting what will be done in each phase (fetch, process, report) prior to implementation. It would be easy to do a lot of fetch work that wouldn't be merged because it isn't helpful for reporting.
SaurabhCodesAI commentedon Oct 10, 2025
Thank you @TimidRobot! Really excited to work on this.
@oree-xx thanks for the tip about following the three phase structure!
Got it about the pre automation directory being reference only, that makes much more sense now. And your point about planning before implementation is really important , I don't want to fetch a bunch of data that won't be useful for reporting.
I'll put together a planning document covering what to do in each phase. Should I post that here or create a separate issue for it?
Also, would it help if I look at the existing GCS/GitHub reports first to understand what kind of output we're aiming for?
Thanks for the guidance!
TimidRobot commentedon Oct 10, 2025
@SaurabhCodesAI I think here in this issue is a good place to post your plan.
The most recent report is here: https://github.com/creativecommons/quantifying/tree/main/data/2025Q3
SaurabhCodesAI commentedon Oct 10, 2025
Thanks for the report link @TimidRobot , I'm going through the 2025Q3 analysis to understand the structure and will put together the planning document. Should have it ready soon.
SaurabhCodesAI commentedon Oct 13, 2025
TimidRobot commentedon Oct 14, 2025
Yes. See https://www.flickr.com/services/api/flickr.photos.licenses.getInfo.html
TimidRobot commentedon Oct 14, 2025
@SaurabhCodesAI Also, I prefer that you edit the description instead of replacing it with a comment.
SaurabhCodesAI commentedon Oct 14, 2025
Thanks for the API link @TimidRobot! I've updated the issue description as you suggested.
I studied the license mapping and I'm ready to start building the fetch script following the
gcs_fetch.pypattern.Could you assign this issue to me? I'd like to get started on the implementation and should have some initial progress to show soon.
Let me know if there are any specific priorities I should focus on first.
SaurabhCodesAI commentedon Oct 14, 2025
Hi @TimidRobot,
I've been working on this implementation over the past few days following your guidance. I'm ready to submit a PR shortly.
I noticed PR #194 was also submitted for this issue. While reviewing both implementations, I wanted to clarify the scope to ensure we're aligned with the project requirements.
Scope Question:
The pre automation reference (pre-automation/flickr/photos.py line 39) shows 8 license IDs:
{1, 2, 3, 4, 5, 6, 9, 10}. Should the automation include all 8 of these licenses?License Names:
I cross referenced the official Flickr API documentation (https://www.flickr.com/services/api/flickr.photos.licenses.getInfo.html) for the exact license name format. Just wanted to confirm this is the preferred source for naming consistency.
My Implementation:
I've completed an implementation that:
Note: I wasn't able to test with live Flickr API as they now require a PRO subscription for API access. The code follows established patterns and should work once credentials are available.
I'll submit my PR for review. Happy to collaborate with @chrisjindu if needed to ensure we get the best solution merged!
Thanks!
Add Flickr data fetching automation
Add Flickr fetch script and workflow integration
Add Flickr fetch script with workflow integration
TimidRobot commentedon Oct 15, 2025
Unfortunately, the Flickr API documentation is out of date:
Their Help Center has more current information:
Flickr Pro is a paid subscription
Because it requires a paid subscription (Flickr Pro), Flickr is not currently a viable data source.