Refactor scraper logic and datastructure #16 - #17
Conversation
- All data is scraped in 1 loop - Remove the need for regex - Return a dict datastructure - Enhanced the tests using input as collected on the source Signed-off-by: Thomas Druez <tdruez@nexb.com>
|
@tdruez Looks great! 👍 |
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
| fields_names = ['status', 'urgency', 'fixed_version'] | ||
|
|
||
| return cve_id, package_name, vulnerability_status | ||
| return [{name: version_detail.get(name) for name in fields_names} |
There was a problem hiding this comment.
@pombredanne @tdruez This will get the three mentioned fields, but obviously not Package Name & Vulnerability. I can't seem to return data in {Dicitonary[list{dicitonary}]} which is {package_name [cve-ids{cve_id_data}]} format.
There was a problem hiding this comment.
Do not ever use complicated and complex comprehensions with more than a single level.
Use regular loops instead.
There was a problem hiding this comment.
@kartiksibal FWIW, @tdruez recommendation to use a dict comprehension was for when you have a more simple, single-level iteration... not multiple level nested loops
| def extract_cves_from_tracker(html): | ||
| def debian_extract_data(): | ||
| """ | ||
| Return all CVEs extracted from the given `html` input. |
There was a problem hiding this comment.
The docstring is no longer up to date
| # if package name is empty, use the previous package name | ||
| if href == '/tracker/source-package/': | ||
| package_name.append(pkg) | ||
| test_input = urlopen("https://security-tracker.debian.org/tracker/data/json").read() |
There was a problem hiding this comment.
you need to load this with the json module...
e.g. once you have read the content, call debian_vulnerabilities = json.loads(test_input)
| fields_names = ['status', 'urgency', 'fixed_version'] | ||
|
|
||
| return cve_id, package_name, vulnerability_status | ||
| return [{name: version_detail.get(name) for name in fields_names} |
There was a problem hiding this comment.
Do not ever use complicated and complex comprehensions with more than a single level.
Use regular loops instead.
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
| for distro, version_detail in details.get('releases', {}).items(): | ||
| if distro == 'jessie': | ||
| final_data.append({ | ||
| "package_name": package_name, |
There was a problem hiding this comment.
Please use single quotes except for doc strings and use all lower case keys.
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
| 'vulnerability_id': 'TEMP-0807341-84E914' | ||
| }, | ||
|
|
||
| { |
There was a problem hiding this comment.
Collecting this have no value.
If the 'jessie' data is not available we should not create an entry in the output.
I'm talking about the sysvinit entry.
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
…rablecode into scraper_datastructure
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
* organize and sort imports * simplify reading test data from JSON * format expectation Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* use global URL as a default for a json_data() url arg * use base_release as an arg for extract_data() with default to jessie * do not use fragile chained get in extract_data() and check for possible empty values ahead of use * use more descriptive variable names * improve docstrings * organize imports and improve code formatting Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
3 participants
Signed-off-by: Thomas Druez tdruez@nexb.com