Collect all GenAI licenses for correct detection

[pombredanne]

We need to ensure that LLMs have a clear license. This is messy out there with a lot of new, weird, proprietary licenses.

There is something that would be super useful wrt. AI: find all these licenses!

The process would be:

• Collect the metadata files from all Hugging Face GenAi repos
• Scan GenAi metadata files with scancode
• Review GenAi metadata file scan results for correctness
• Create new licenses, and new license rules used @ HuggingFace

When cloning all the repos @ Hugging Face, these should shallow clones, without Git LFS and without xet large files, and possibly partial clone to focus only on a few files: model cards and licenses, and related metadata files. See for ideas:

• Investigate git partial clone/sparse checkout federatedcode#82

Running ScanCode license detection on that and review the results to find all the funky new AI-licenses and notices may be best done in ScanCode.io for efficiency, but the licenses are ultimately created in ScanCode toolkit

[Godzilaa]

please assign this to me i would wanna work on this

[armijnhemel]

please assign this to me i would wanna work on this

You actually don't need permission to work on it.

[DennisClark]

thanks for the comment @armijnhemel and yes @Godzilaa you are quite welcome to collect GenAI licenses that are not currently defined in https://scancode-licensedb.aboutcode.org/index.html and report them in a text file that provides the URL to each license page. That would be extremely helpful, thanks!

[Tarun-goswamii]

@DennisClark sir , if they issue still not solved can i work on it?