how to share large packages between projects/environments?

[fancyerii]

I have many projects with shared large packages such as pytorch(4G+). Every time I create a new project I need to install it. It wastes time and disk space. Is there any way to share these large packages like maven(java) or docker.

Java's maven is project based-- it manage each projects indenpendently. But all the dependents(jars) are stored centrally in .m2. So If project A and project B share exactly the same version of a lib, There is only one copy of this jar file.

I tried and searched many python package manager tools: pip,conda,pipenv,poetry,PDM. No of them has this features. A similar issue can be found pypa/packaging-problems#328. But I can't find solution there.

[matteius]

@oz123 might be able to way in on this as he has published docker image in the past. I have used AWS ECR and currently use Digital Ocean container registry to publish artifacts from CI builds, so I imagine its something of that nature you are looking for, but I am not sure how pipenv can be extended to help support that use case.

FWIW, I had done some work a while back end of last year to help enable builder of pytorch to have package hashes so there is fewer packages to scrape when locking, but other than I hear what you are saying.

[goyalyashpal]

yep, i asked same in pipx lol:

• space efficient non-isolated caches? or use deps solver like `micromamba`? pipx#1167