Fix intermittent package_versions.py error when retrieving pkg:deb/debian packages

[johnmhoran]

As described in detail in purldb issue 247, the server for pkg:deb/debian/* queries intermittently fails to respond properly and instead returns a status code error (often 503) and a traceback that interrupts the flow of the purlcli command (versions). I've seen this behavior off and on in my recent weeks of testing involving deb/debian packages.

One reply comment has suggested that the solution may be returning an empty list.

[johnmhoran]

@keshav-space Here's a fresh example of the problem from a few seconds ago:

(venv) Tue Mar 05, 2024 08:05 AM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$ python -m purldb_toolkit.purlcli versions --purl pkg:deb/debian/2ping --output -
Error while fetching 'https://sources.debian.org/api/src/2ping': 503
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 989, in <module>
    purlcli()
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 829, in get_versions
    purl_versions = get_versions_details(purls, output, file, unique, command_name)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 873, in get_versions_details
    for package_version in list(versions(purl)):
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/fetchcode/package_versions.py", line 205, in get_deb_versions_from_purl
    for release in response["versions"]:
TypeError: 'NoneType' object is not subscriptable

(venv) Tue Mar 05, 2024 08:06 AM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$

[johnmhoran]

And here's an example of the 503 error from Google Chrome at the same time my versions command is returning an error.

[johnmhoran]

This problem also is occasionally reflected in purlcli versions by a problem with the PurlDB validate endpoint, which I catch with a try/except, e.g.,

    logger = logging.getLogger(__name__)

    api_query = "https://public.purldb.io/api/validate/"
    request_body = {"purl": purl, "check_existence": True}

    try:
        response = requests.get(api_query, params=request_body).json()
    except json.decoder.JSONDecodeError as e:

        print(f"validate_purl(): json.decoder.JSONDecodeError for '{purl}': {e}")

        logging.basicConfig(
            filename="purldb-toolkit/src/purldb_toolkit/app.log",
            level=logging.ERROR,
            format="%(levelname)s - %(message)s",
            filemode="w",
        )

        logger.error(f"validate_purl(): json.decoder.JSONDecodeError for '{purl}': {e}")

    except Exception as e:
        print(f"'validate' endpoint error for '{purl}': {e}")
    else:
        if response is None:
            print(f"'{purl}' -- response.status_code for None = {response.status_code}")
        return response

Reported in the console like this just now:

(venv) Tue Mar 05, 2024 09:06 AM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$ python -m purldb_toolkit.purlcli versions --purl pkg:deb/debian/2ping --output -
validate_purl(): json.decoder.JSONDecodeError for 'pkg:deb/debian/2ping': Expecting value: line 2 column 1 (char 1)
'pkg:deb/debian/2ping' encountered a validation error
{
    "headers": [
        {
            "tool_name": "purlcli",
            "tool_version": "0.2.0",
            "options": {
                "command": "versions",
                "--purl": [
                    "pkg:deb/debian/2ping"
                ],
                "--file": null,
                "--output": "<stdout>"
            },
            "purls": [
                "pkg:deb/debian/2ping"
            ],
            "errors": [
                "ERROR - validate_purl(): json.decoder.JSONDecodeError for 'pkg:deb/debian/2ping': Expecting value: line 2 column 1 (char 1)\n"
            ],
            "warnings": [
                "'pkg:deb/debian/2ping' encountered a validation error"
            ]
        }
    ],
    "packages": []
}
(venv) Tue Mar 05, 2024 09:07 AM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$