Add test protocol to validate resolution #6

docs/source/dependencies-design.rst

@@ -1,6 +1,6 @@
-=========================================
-  Resolving Python package dependencies
-=========================================
+====================================================
+  Python package dependencies resolver design
+====================================================
 
 
 This is a design to create a new command line tool to resolve Python

docs/source/index.rst

@@ -1,11 +1,12 @@
-Welcome to nexb-skeleton's documentation!
-=========================================
+Welcome to  python-inspector's documentation!
+================================================
 
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
    dependencies-design
+   test-protocol
 
 Indices and tables
 ==================

docs/source/test-protocol.rst

@@ -0,0 +1,308 @@
+==============================================================
+  Validation tests for Python package dependency resolution
+==============================================================
+
+In order to validate that the python-inspector dependency resolution is correct
+here is a procedure.
+
+
+Requirements
+***************
+
+You need to:
+
+- have access to some Python codebase to use as a test bed.
+- have a working installation of python-inspector
+- have a working installation of scancode-toolkit v31.x (inlucing RCs) or higher
+
+
+Test protocol
+***************
+
+Step 1: Collect development code details
+--------------------------------------------
+
+In this step, you will collect the list of requirements files that exist
+in the codebase. You can use scancode::
+
+    scancode --package --info <path/to/development/codebase/dir> \
+      --json-pp <path/to/development-scan-output.json> --processes 4
+
+You then need to review the scanc results to assemble a list of pip
+requirement files used in development.
+
+Or a simpler approach is to use the ``find`` command::
+
+    find <path/to/development/codebase/dir> -name "requirement*.txt" \
+      > <path/to/requirements-file-paths.txt>
+
+The output is a list of pip requirement files used in the development codebase.
+
+
+
+Step 2: Build your code
+----------------------------
+
+In this step, you need to run the build of your codebase and obtain the set
+of deployed binaries. Building requires both dependency resolution and
+installation of the resolved packages. The set of installed packages is used to
+establish the ground truth and the expected results.
+
+As an output, keep the Python version that is used and the directory(ies) where
+the deployed code is built named further down as <path/to/deployed/codebase/dir>.
+
+
+Step 3: Collect built code details
+---------------------------------------
+
+In this step, you will extract the deployed code and run a scan to
+collect existing packages as resolved during the original build.
+
+For this we will use extractcode and scancode with these commands::
+
+    extractcode --shallow <path/to/deployed/codebase/dir>
+    scancode --package --info <path/to/deployed/codebase/dir> \
+      --json-pp <path/to/deployed-scan-output.json> --processes 4
+
+You can adjust the number of processes up or down based on available CPU cores.
+
+The output is the file <path/to/scan-output.json> that contains all the detected
+packages that were installed during the build.
+
+
+Step 4: Resolve dependencies using development requirement files
+--------------------------------------------------------------------
+
+In this step, you will resolve the dependencies using python-inspector dad
+command for each of the requirements files identified in Step 1 using the
+Python version identified in Step 2. Run this command for each requirements
+file, using each time a different output file name. We assume here Python
+version 3.8 (note the absence of dot when passed as a command line option::
+
+    dad --python-version 38 --requirement <path/to/requirements.txt> \
+      --json <path/to/resolved-requirements.txt.json> \
+      --netrc <path/to/.netrc>
+
+The output is a list of JSON files with resolved packages for each of the
+input requirements files.
+
+
+Step 5: Collect expected resolved packages
+----------------------------------------------
+
+In this step, you need to collect the list of built packages "purl" found in
+the top level "packages" attribute of the JSON scan output from Step 3.
+
+The output is a list of expected purls with a version.
+
+
+Step 6: Collect actual resolved packages
+----------------------------------------------
+
+In this step, you need to collect the list of purls "package" attribute resolved
+found in the top level "resolved_dependencies" attribute of the python-inspector
+resolution output from Step 4.
+
+The output is a list of actual purls with a version.
+
+
+Step 7: Compare expected with actual resolved packages
+---------------------------------------------------------
+
+In this step, you compare the output from Step 5 and Step 6 and validate the
+correctness of python-inpector resolution.
+
+The output is a list of differences:
+
+- list 1: expected purls not present in the resolved purls.
+- list 2: resolved purls not present in the expected purls.
+
+
+(and if needed you can also collate the list of similar purls for reference).
+
+
+If the build, build scan and resolution worked as expected, the list 1 and list 2
+should be empty. The differences need to be investigated.
+
+The possible causes could be:
+
+- extra actual resolved packages when some of the requirement files from Step 3
+  also contains files for requirements that are not deployed (e.g. test, tools.)
+
+- missing actual resolved packages when the reference codebase from Step 1 uses
+  other build manifests beyond requirements.txt files. If this is the case, you
+  should carefully identify which other build manifests are used  by reviewing
+  the development scan from Step 3. and report each manifest format as enhancement
+  request in the python-inspector issue tracker.
+
+- finally it can be a bug in python-inspector proper. Please report an issue
+  in the python-inspector issue tracker. Attach the list or JSON output(s)
+  from Step 1, Step 3 and Step 4. And the results of the review of Step 5, 6
+  and 7. Alternatively you can share these files with python-inspector
+  maintainers if these are private.
+
+
+End-to-end example
+**********************
+
+Setup
+------
+
+
+We use this repo https://github.com/tjcsl/ion as a sample codebase.
+And the reference Python version is 3.8::
+
+    mkdir -p ~/tmp/pyinsp-example/
+    cd ~/tmp/pyinsp-example/
+    git clone https://github.com/tjcsl/ion
+
+
+Another example could be https://github.com/digitalocean/sample-django
+
+We use the latest main branch from python-inspector and scancode-toolkit 31.0.0rc2
+installed on Linux with Python 3.8 using the release tarball from:
+https://github.com/nexB/scancode-toolkit/releases/tag/v31.0.0rc2
+
+ScanCode setup::
+
+    mkdir -p ~/tmp/pyinsp-example/tools
+    cd ~/tmp/pyinsp-example/tools
+    wget https://github.com/nexB/scancode-toolkit/releases/download/v31.0.0rc2/scancode-toolkit-31.0.0rc2_py38-linux.tar.xz
+    tar -xf scancode-toolkit-31.0.0rc2_py38-linux.tar.xz
+    cd scancode-toolkit-31.0.0rc2/
+    ./scancode --help
+
+python-inspector setup::
+
+    cd ~/tmp/pyinsp-example/tools
+    git clone https://github.com/nexB/python-inspector
+    python3.8 -m venv venv
+    source venv/bin/activate
+    pip install --upgrade pip setuptools wheel
+    cd python-inspector
+    ./configure
+
+We will store all outputs in this directory::
+
+    mkdir -p ~/tmp/pyinsp-example/output
+
+
+Step 1: Collect development code details
+--------------------------------------------
+
+We run a simple find::
+
+    find ~/tmp/pyinsp-example/ion \
+      -name "requirement*.txt" > ~/tmp/pyinsp-example/output/requirements-file-paths.txt
+
+We find these two requirement files in ~/tmp/pyinsp-example/output/requirements-file-paths.txt::
+
+    ~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt
+    ~/tmp/pyinsp-example/ion/requirements.txt
+
+
+Step 2: Build your code
+----------------------------
+
+We perform a simple "editable" build in place::
+
+    cd ~/tmp/pyinsp-example/codebase/ion
+    python3.8 -m venv venv
+    source venv/bin/activate
+    pip install --upgrade pip setuptools wheel
+    pip install --editable .
+    deactivate
+
+
+Step 3: Collect built code details
+---------------------------------------
+
+We extract in place::
+
+    cd ~/tmp/pyinsp-example/tools/scancode-toolkit-31.0.0rc2/
+    ./extractcode --shallow ~/tmp/pyinsp-example/codebase/ion
+
+And collect built details::
+
+    ./scancode --package --info ~/tmp/pyinsp-example/codebase/ion \
+      --json-pp ~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json --processes 4
+
+The output files is::
+
+    ~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json
+
+
+Step 4: Resolve dependencies using development requirement files
+--------------------------------------------------------------------
+
+    cd ~/tmp/pyinsp-example/tools/python-inspector
+    source venv/bin/activate
+
+    dad --requirement ~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt \
+      --json ~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json
+
+    dad --requirement ~/tmp/pyinsp-example/ion/requirements.txt \
+      --json ~/tmp/pyinsp-example/output/resolved-requirements.txt.json
+
+    deactivate
+
+The output files are::
+
+    ~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json
+    ~/tmp/pyinsp-example/output/resolved-requirements.txt.json
+
+
+Step 5: Collect expected resolved packages
+----------------------------------------------
+
+Run this python script to generate text file with expected purls
+
+::
+
+  import json
+  with open("~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json") as f:
+      scancode_data = json.load(f)
+  scancode_purls = []
+  for package in scancode_data["packages"]:
+      if package["purl"] not in scancode_purls:
+          scancode_purls.append(package["purl"])
+  scancode_purls = sorted(scancode_purls)
+  with open("~/tmp/pyinsp-example/codebase/output/scan.txt", "w") as f:
+      f.write("\n".join(scancode_purls))
+
+
+The output is a list of expected purls with a version.
+
+
+Step 6: Collect actual resolved packages
+----------------------------------------------
+
+Run this python script to generate text file with actual purls
+
+::
+
+  import json
+  py_insp_purls = []
+  for json_file in [
+      "~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json",
+      "~/tmp/pyinsp-example/output/resolved-requirements.txt.json",
+  ]:
+      with open(json_file) as f:
+          py_insp_data = json.load(f)
+      for package in py_insp_data["resolved_dependencies"]:
+          if package["package"] not in py_insp_purls:
+              py_insp_purls.append(package["package"])
+  py_insp_purls = sorted(py_insp_purls)
+  with open("~/tmp/pyinsp-example/codebase/output/py-insp.txt", "w") as f:
+      f.write("\n".join(py_insp_purls))
+
+
+The output is a list of actual purls with a version.
+
+
+Step 7: Compare expected with actual resolved packages
+---------------------------------------------------------
+
+We run a sdiff command::
+
+    sdiff ~/tmp/pyinsp-example/codebase/output/py-insp.txt ~/tmp/pyinsp-example/codebase/output/scan.txt

tests/test_resolution.py

@@ -82,11 +82,11 @@ def test_get_resolved_dependencies_with_flask_and_python_36():
     assert as_list == [
         "pkg:pypi/click@8.1.3",
         "pkg:pypi/flask@2.1.2",
-        "pkg:pypi/importlib-metadata@4.11.4",
+        "pkg:pypi/importlib-metadata@4.12.0",
         "pkg:pypi/itsdangerous@2.1.2",
         "pkg:pypi/jinja2@3.1.2",
         "pkg:pypi/markupsafe@2.0.1",
-        "pkg:pypi/typing-extensions@4.2.0",
+        "pkg:pypi/typing-extensions@4.3.0",
         "pkg:pypi/werkzeug@2.1.2",
         "pkg:pypi/zipp@3.8.0",
     ]
@@ -107,7 +107,7 @@ def test_get_resolved_dependencies_with_tilde_requirement_using_json_api():
     assert as_list == [
         "pkg:pypi/click@8.1.3",
         "pkg:pypi/flask@2.1.2",
-        "pkg:pypi/importlib-metadata@4.11.4",
+        "pkg:pypi/importlib-metadata@4.12.0",
         "pkg:pypi/itsdangerous@2.1.2",
         "pkg:pypi/jinja2@3.1.2",
         "pkg:pypi/markupsafe@2.1.1",