Inconsistent URL detection

[stefan6419846]

Description

Scanning for URLs does not seem to be consistent and I sometimes see expected and sometimes unexpected results.

Let's take the following files as an example:

• https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/trig.py does not report http://www.w3.org/TR/trig/
• https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/trix.py reports http://www.w3.org/2004/03/trix/trix-1/, but not http://www.w3.org/XML/1998/namespace
• https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/turtle.py reports http://www.w3.org/TeamSubmission/turtle/ correctly.

How To Reproduce

Run scancode.api.get_urls(path) on the corresponding files.

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Linux
• What version of scancode-toolkit was used to generate the scan file? 32.3.3
• What installation method was used to install/run scancode? pip

[pombredanne]

Thank you for the report. We do some filtering in URLs at:

• https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/cluecode/finder_data.py#L61

Do you think we should relax this? make it more aggressive? Add an option to not filter?

[stefan6419846]

I think an option to disable filtering would be the best approach here. Currently, I have to do a second run (or manual review) of all URLs which were possibly excluded, especially since for using them in Python files some further analysis might be desired, while for cases with XML files just referencing the namespace it might be fine to actually filter them.