Version 0.14.0 misses several source packages

[sschuberth]

When upgrading ORT from python-inspector 0.10.0 to 0.14.0, our tests start to fail as for some packages the information about source packages is not missing. See e.g.

    {
      "type": "pypi",
      "namespace": null,
      "name": "ply",
      "version": "3.11",
      "qualifiers": {},
      "subpath": null,
      "primary_language": "Python",
      "description": "Python Lex & Yacc\nPLY is yet another implementation of lex and yacc for Python. Some notable\nfeatures include the fact that its implemented entirely in Python and it\nuses LALR(1) parsing which is efficient and well suited for larger grammars.\n\nPLY provides most of the standard lex/yacc features including support for empty \nproductions, precedence rules, error recovery, and support for ambiguous grammars. \n\nPLY is extremely easy to use and provides very extensive error checking. \nIt is compatible with both Python 2 and Python 3.",
      "release_date": "2018-02-15T19:01:27",
      "parties": [
        {
          "type": "person",
          "role": "author",
          "name": "David Beazley",
          "email": "dave@dabeaz.com",
          "url": null
        }
      ],
      "keywords": [
        "Programming Language :: Python :: 2",
        "Programming Language :: Python :: 3"
      ],
      "homepage_url": "http://www.dabeaz.com/ply/",
      "download_url": "https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl",
      "size": 49567,
      "sha1": null,
      "md5": "62b6ad5affddc9926ab5571f390cc840",
      "sha256": "096f9b8350b65ebd2fd1346b12452efe5b9607f7482813ffca50c22722a807ce",
      "sha512": null,
      "bug_tracking_url": null,
      "code_view_url": null,
      "vcs_url": null,
      "copyright": null,
      "license_expression": null,
      "declared_license": {
        "license": "BSD"
      },
      "notice_text": null,
      "source_packages": [],
      "file_references": [],
      "extra_data": {},
      "dependencies": [],
      "repository_homepage_url": null,
      "repository_download_url": null,
      "api_data_url": "https://pypi.org/pypi/ply/3.11/json",
      "datasource_id": null,
      "purl": "pkg:pypi/ply@3.11"
    },

which previously contained

https://files.pythonhosted.org/packages/e5/69/882ee5c9d017149285cab114ebeab373308ef0f874fcdac9beb90e0ac4da/ply-3.11.tar.gz

which is missing now.

[sschuberth]

Ok, interesting: With --prefer-source I can get the source package again, but then the binary package URL is missing. Is there a way to restore the original behavior to get the binary and source package URLs in a single run?

[sschuberth]

So, the way it seems to have worked in python-inspector 0.10.0 is that whole separate packages were created for binary and source packages, essentially doubling the number of packages. With python-inspector 0.14.0 now only single packages get created for a single dependency, which is good, but in exchange download_url only points to either the binary or the source package.

I wonder why the source package does not simply get filled into source_packages. However, that would lack metadata like the file's hash. But that should be easy to extend.

[pombredanne]

@sschuberth can you give a simple command and example you have used for 0.10.0?

[sschuberth]

We use these options:

    "options": [
      "--analyze-setup-py-insecurely",
      "--index-url https://pypi.org/simple",
      "--json-pdt /tmp/ort-PythonInspector5393608600190755772/python-inspector8932738510953399825.json",
      "--operating-system linux",
      "--python-version 27",
      "--setup-py /home/sebastian/Development/GitHub/oss-review-toolkit/ort/plugins/package-managers/python/src/funTest/assets/projects/external/spdx-tools-python/setup.py",
      "--verbose"
    ],

On this file:

https://github.com/spdx/tools-python/blob/a48022e65a8897d0e4f2e93d8e53695d2c13ea23/setup.py

[sschuberth]

Is there a way to restore the original behavior to get the binary and source package URLs in a single run?

Ping on this. I'm just curious whether it was a deliberate decision not to do so.