Parse pip cache dir

[pombredanne]

It can happen that we find a pip cache directory in a codebase. It contains both code archives and JSON that would be useful to parse and collect.
For instance:

pombreda@computer4:~$ ll .cache/pip/wheels/eb/60/37/ee40403cbd895ccdb57eb28b03b0afabeb449d5df9ce776a0d/
total 80
drwxrwxr-x 2 pombreda pombreda  4096 Mar 11 17:12 ./
drwxrwxr-x 3 pombreda pombreda  4096 Mar 11 17:12 ../
-rw-rw-r-- 1 pombreda pombreda   332 Mar 11 17:12 origin.json
-rw-rw-r-- 1 pombreda pombreda 69427 Mar 11 17:12 pefile-2022.5.30-py3-none-any.whl

$ ll .cache/pip/wheels/eb/60/37/ee40403cbd895ccdb57eb28b03b0afabeb449d5df9ce776a0d/
total 80
drwxrwxr-x 2 foo bar  4096 Mar 11 17:12 ./
drwxrwxr-x 3 foo bar  4096 Mar 11 17:12 ../
-rw-rw-r-- 1 foo bar   332 Mar 11 17:12 origin.json
-rw-rw-r-- 1 foo bar 69427 Mar 11 17:12 pefile-2022.5.30-py3-none-any.whl




$ less .cache/pip/wheels/eb/60/37/ee40403cbd895ccdb57eb28b03b0afabeb449d5df9ce776a0d/origin.json 

{"archive_info": {"hash": "sha256=a5488a3dd1fd021ce33f969780b88fe0f7eebb76eb20996d7318f307612a045b", "hashes": {"sha256": "a5488a3dd1fd021ce33f969780b88fe0f7eebb76eb20996d7318f307612a045b"}}, "url": "https://files.pythonhosted.org/packages/48/30/4559d06bad5bb627733dac1ef28c34f5e35f1461247ba63e5f6366901277/pefile-2022.5.30.tar.gz"}

[Spurgeon008]

can i fix this ?

[kalpshah18]

@pombredanne Is this issue open?

[vamshi567bh]

@pombredanne please assign me

[levi42x]

Hi @pombredanne
I’ve started working on this issue and had a couple of doubts:

1. Is it necessary that both files .whl and origin.json must be present inside the pip/cache directory? (Typically, the cache contains both, but I’d like to confirm for clarity.)
   what should the handler do if one of them is missing?

2. Should the handler only detect origin.json inside .cache/pip/wheels/, or do we also need to support other pip/cache subdirectories (like .cache/pip/http/)?

[Jayant-kernel]

Hi @pombredanne,

I'd like to work on this issue. Before I start implementation, I have a few clarification questions:

1. Scope: Should this parse only the origin.json files in pip's HTTP cache (~/.cache/pip/http/), or also handle the wheel cache (~/.cache/pip/wheels/)?

2. Output format: Should this create Package objects similar to existing pip handlers, or is there a specific format for cache-derived packages?

3. Cache location: Should we support custom cache directories (via PIP_CACHE_DIR), or only the default location?
   Looking forward to your guidance!

[Jayant-kernel]

Hi @pombredanne,
I started looking into this and ran into this,
My pip cache (c:\users\jayan\appdata\local\pip\cache) has:

Wheel cache: Totally empty
HTTP cache: Has files but they're in serial - looks like msgpack format from CacheControl
origin.json files: Couldn't find any

So I'm a bit stuck on which direction to take. The issue description shows origin.json files in the wheel cache, but I don't have any to work with. The HTTP cache files are there but they're in binary format.
Quick questions:

Should I tackle the binary HTTP cache files? (Would need msgpack to serealized)
Or focus on the wheel cache origin.json parsing? (Just need to find/generate test data first)
Do you have a pip cache directory with origin.json I could test against?

Just want to make sure I'm solving the right problem before moving forward.

[TheSangamX]

Hi @pombredanne,
I’m new to ScanCode Toolkit and exploring this issue as a first contribution.
Before starting implementation, I wanted to confirm the intended scope:

• Should we focus first on parsing origin.json from pip’s HTTP cache?
• Or is the expectation to also support the wheel cache?
  I’m happy to start with a minimal, incremental implementation if that works.
  Thanks!