Skip to content

Commit db336bd

Browse files
TG1999TG1999
authored
Merge pull request #29 from nexB/test-protocol
Add test protocol to validate resolution #6
2 parents 6a40806 + f6a84b2 commit db336bd

4 files changed

Lines changed: 315 additions & 6 deletions

File tree

‎docs/source/dependencies-design.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
=========================================
2-
Resolving Python package dependencies
3-
=========================================
1+
====================================================
2+
Python package dependencies resolver design
3+
====================================================
44

55

66
This is a design to create a new command line tool to resolve Python

‎docs/source/index.rst‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
1-
Welcome to nexb-skeleton's documentation!
2-
=========================================
1+
Welcome to python-inspector's documentation!
2+
================================================
33

44
.. toctree::
55
:maxdepth: 2
66
:caption: Contents:
77

88
dependencies-design
9+
test-protocol
910

1011
Indices and tables
1112
==================

‎docs/source/test-protocol.rst‎

Lines changed: 308 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,308 @@
1+
==============================================================
2+
Validation tests for Python package dependency resolution
3+
==============================================================
4+
5+
In order to validate that the python-inspector dependency resolution is correct
6+
here is a procedure.
7+
8+
9+
Requirements
10+
***************
11+
12+
You need to:
13+
14+
- have access to some Python codebase to use as a test bed.
15+
- have a working installation of python-inspector
16+
- have a working installation of scancode-toolkit v31.x (inlucing RCs) or higher
17+
18+
19+
Test protocol
20+
***************
21+
22+
Step 1: Collect development code details
23+
--------------------------------------------
24+
25+
In this step, you will collect the list of requirements files that exist
26+
in the codebase. You can use scancode::
27+
28+
scancode --package --info <path/to/development/codebase/dir> \
29+
--json-pp <path/to/development-scan-output.json> --processes 4
30+
31+
You then need to review the scanc results to assemble a list of pip
32+
requirement files used in development.
33+
34+
Or a simpler approach is to use the ``find`` command::
35+
36+
find <path/to/development/codebase/dir> -name "requirement*.txt" \
37+
> <path/to/requirements-file-paths.txt>
38+
39+
The output is a list of pip requirement files used in the development codebase.
40+
41+
42+
43+
Step 2: Build your code
44+
----------------------------
45+
46+
In this step, you need to run the build of your codebase and obtain the set
47+
of deployed binaries. Building requires both dependency resolution and
48+
installation of the resolved packages. The set of installed packages is used to
49+
establish the ground truth and the expected results.
50+
51+
As an output, keep the Python version that is used and the directory(ies) where
52+
the deployed code is built named further down as <path/to/deployed/codebase/dir>.
53+
54+
55+
Step 3: Collect built code details
56+
---------------------------------------
57+
58+
In this step, you will extract the deployed code and run a scan to
59+
collect existing packages as resolved during the original build.
60+
61+
For this we will use extractcode and scancode with these commands::
62+
63+
extractcode --shallow <path/to/deployed/codebase/dir>
64+
scancode --package --info <path/to/deployed/codebase/dir> \
65+
--json-pp <path/to/deployed-scan-output.json> --processes 4
66+
67+
You can adjust the number of processes up or down based on available CPU cores.
68+
69+
The output is the file <path/to/scan-output.json> that contains all the detected
70+
packages that were installed during the build.
71+
72+
73+
Step 4: Resolve dependencies using development requirement files
74+
--------------------------------------------------------------------
75+
76+
In this step, you will resolve the dependencies using python-inspector dad
77+
command for each of the requirements files identified in Step 1 using the
78+
Python version identified in Step 2. Run this command for each requirements
79+
file, using each time a different output file name. We assume here Python
80+
version 3.8 (note the absence of dot when passed as a command line option::
81+
82+
dad --python-version 38 --requirement <path/to/requirements.txt> \
83+
--json <path/to/resolved-requirements.txt.json> \
84+
--netrc <path/to/.netrc>
85+
86+
The output is a list of JSON files with resolved packages for each of the
87+
input requirements files.
88+
89+
90+
Step 5: Collect expected resolved packages
91+
----------------------------------------------
92+
93+
In this step, you need to collect the list of built packages "purl" found in
94+
the top level "packages" attribute of the JSON scan output from Step 3.
95+
96+
The output is a list of expected purls with a version.
97+
98+
99+
Step 6: Collect actual resolved packages
100+
----------------------------------------------
101+
102+
In this step, you need to collect the list of purls "package" attribute resolved
103+
found in the top level "resolved_dependencies" attribute of the python-inspector
104+
resolution output from Step 4.
105+
106+
The output is a list of actual purls with a version.
107+
108+
109+
Step 7: Compare expected with actual resolved packages
110+
---------------------------------------------------------
111+
112+
In this step, you compare the output from Step 5 and Step 6 and validate the
113+
correctness of python-inpector resolution.
114+
115+
The output is a list of differences:
116+
117+
- list 1: expected purls not present in the resolved purls.
118+
- list 2: resolved purls not present in the expected purls.
119+
120+
121+
(and if needed you can also collate the list of similar purls for reference).
122+
123+
124+
If the build, build scan and resolution worked as expected, the list 1 and list 2
125+
should be empty. The differences need to be investigated.
126+
127+
The possible causes could be:
128+
129+
- extra actual resolved packages when some of the requirement files from Step 3
130+
also contains files for requirements that are not deployed (e.g. test, tools.)
131+
132+
- missing actual resolved packages when the reference codebase from Step 1 uses
133+
other build manifests beyond requirements.txt files. If this is the case, you
134+
should carefully identify which other build manifests are used by reviewing
135+
the development scan from Step 3. and report each manifest format as enhancement
136+
request in the python-inspector issue tracker.
137+
138+
- finally it can be a bug in python-inspector proper. Please report an issue
139+
in the python-inspector issue tracker. Attach the list or JSON output(s)
140+
from Step 1, Step 3 and Step 4. And the results of the review of Step 5, 6
141+
and 7. Alternatively you can share these files with python-inspector
142+
maintainers if these are private.
143+
144+
145+
End-to-end example
146+
**********************
147+
148+
Setup
149+
------
150+
151+
152+
We use this repo https://github.com/tjcsl/ion as a sample codebase.
153+
And the reference Python version is 3.8::
154+
155+
mkdir -p ~/tmp/pyinsp-example/
156+
cd ~/tmp/pyinsp-example/
157+
git clone https://github.com/tjcsl/ion
158+
159+
160+
Another example could be https://github.com/digitalocean/sample-django
161+
162+
We use the latest main branch from python-inspector and scancode-toolkit 31.0.0rc2
163+
installed on Linux with Python 3.8 using the release tarball from:
164+
https://github.com/nexB/scancode-toolkit/releases/tag/v31.0.0rc2
165+
166+
ScanCode setup::
167+
168+
mkdir -p ~/tmp/pyinsp-example/tools
169+
cd ~/tmp/pyinsp-example/tools
170+
wget https://github.com/nexB/scancode-toolkit/releases/download/v31.0.0rc2/scancode-toolkit-31.0.0rc2_py38-linux.tar.xz
171+
tar -xf scancode-toolkit-31.0.0rc2_py38-linux.tar.xz
172+
cd scancode-toolkit-31.0.0rc2/
173+
./scancode --help
174+
175+
python-inspector setup::
176+
177+
cd ~/tmp/pyinsp-example/tools
178+
git clone https://github.com/nexB/python-inspector
179+
python3.8 -m venv venv
180+
source venv/bin/activate
181+
pip install --upgrade pip setuptools wheel
182+
cd python-inspector
183+
./configure
184+
185+
We will store all outputs in this directory::
186+
187+
mkdir -p ~/tmp/pyinsp-example/output
188+
189+
190+
Step 1: Collect development code details
191+
--------------------------------------------
192+
193+
We run a simple find::
194+
195+
find ~/tmp/pyinsp-example/ion \
196+
-name "requirement*.txt" > ~/tmp/pyinsp-example/output/requirements-file-paths.txt
197+
198+
We find these two requirement files in ~/tmp/pyinsp-example/output/requirements-file-paths.txt::
199+
200+
~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt
201+
~/tmp/pyinsp-example/ion/requirements.txt
202+
203+
204+
Step 2: Build your code
205+
----------------------------
206+
207+
We perform a simple "editable" build in place::
208+
209+
cd ~/tmp/pyinsp-example/codebase/ion
210+
python3.8 -m venv venv
211+
source venv/bin/activate
212+
pip install --upgrade pip setuptools wheel
213+
pip install --editable .
214+
deactivate
215+
216+
217+
Step 3: Collect built code details
218+
---------------------------------------
219+
220+
We extract in place::
221+
222+
cd ~/tmp/pyinsp-example/tools/scancode-toolkit-31.0.0rc2/
223+
./extractcode --shallow ~/tmp/pyinsp-example/codebase/ion
224+
225+
And collect built details::
226+
227+
./scancode --package --info ~/tmp/pyinsp-example/codebase/ion \
228+
--json-pp ~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json --processes 4
229+
230+
The output files is::
231+
232+
~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json
233+
234+
235+
Step 4: Resolve dependencies using development requirement files
236+
--------------------------------------------------------------------
237+
238+
cd ~/tmp/pyinsp-example/tools/python-inspector
239+
source venv/bin/activate
240+
241+
dad --requirement ~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt \
242+
--json ~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json
243+
244+
dad --requirement ~/tmp/pyinsp-example/ion/requirements.txt \
245+
--json ~/tmp/pyinsp-example/output/resolved-requirements.txt.json
246+
247+
deactivate
248+
249+
The output files are::
250+
251+
~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json
252+
~/tmp/pyinsp-example/output/resolved-requirements.txt.json
253+
254+
255+
Step 5: Collect expected resolved packages
256+
----------------------------------------------
257+
258+
Run this python script to generate text file with expected purls
259+
260+
::
261+
262+
import json
263+
with open("~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json") as f:
264+
scancode_data = json.load(f)
265+
scancode_purls = []
266+
for package in scancode_data["packages"]:
267+
if package["purl"] not in scancode_purls:
268+
scancode_purls.append(package["purl"])
269+
scancode_purls = sorted(scancode_purls)
270+
with open("~/tmp/pyinsp-example/codebase/output/scan.txt", "w") as f:
271+
f.write("\n".join(scancode_purls))
272+
273+
274+
The output is a list of expected purls with a version.
275+
276+
277+
Step 6: Collect actual resolved packages
278+
----------------------------------------------
279+
280+
Run this python script to generate text file with actual purls
281+
282+
::
283+
284+
import json
285+
py_insp_purls = []
286+
for json_file in [
287+
"~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json",
288+
"~/tmp/pyinsp-example/output/resolved-requirements.txt.json",
289+
]:
290+
with open(json_file) as f:
291+
py_insp_data = json.load(f)
292+
for package in py_insp_data["resolved_dependencies"]:
293+
if package["package"] not in py_insp_purls:
294+
py_insp_purls.append(package["package"])
295+
py_insp_purls = sorted(py_insp_purls)
296+
with open("~/tmp/pyinsp-example/codebase/output/py-insp.txt", "w") as f:
297+
f.write("\n".join(py_insp_purls))
298+
299+
300+
The output is a list of actual purls with a version.
301+
302+
303+
Step 7: Compare expected with actual resolved packages
304+
---------------------------------------------------------
305+
306+
We run a sdiff command::
307+
308+
sdiff ~/tmp/pyinsp-example/codebase/output/py-insp.txt ~/tmp/pyinsp-example/codebase/output/scan.txt

‎tests/test_resolution.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ def test_get_resolved_dependencies_with_flask_and_python_36():
8686
"pkg:pypi/itsdangerous@2.1.2",
8787
"pkg:pypi/jinja2@3.1.2",
8888
"pkg:pypi/markupsafe@2.0.1",
89-
"pkg:pypi/typing-extensions@4.2.0",
89+
"pkg:pypi/typing-extensions@4.3.0",
9090
"pkg:pypi/werkzeug@2.1.2",
9191
"pkg:pypi/zipp@3.8.0",
9292
]

0 commit comments

Comments
 (0)