Feat: add lwn scraper - #160
Conversation
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
|
@EslamHiko thanks you ... this is looking very good at first glance! @sbs2001 @haikoschol your review is welcomed. |
| dists = getDistributors() | ||
| packagesVulns = {} | ||
| for dist in dists: | ||
| distUrl = base_url + "Alerts/" + dist + "?n=100" |
There was a problem hiding this comment.
"?n=100" how about not hardcoding this. Consider iterating every page(n=0 to n=last page)
There was a problem hiding this comment.
@sbs2001 n isn't the page number it's how many packages to show in the table after the offset, the default value is 20 & the max is 100.
ex : https://lwn.net/Alerts/Ubuntu/?n=20 & https://lwn.net/Alerts/Ubuntu/?n=100 & https://lwn.net/Alerts/Ubuntu/?n=200
| @@ -0,0 +1,118 @@ | |||
| # Author: Navonil Das (@NavonilDas) | |||
There was a problem hiding this comment.
Consider removing this, or put your name instead :))
| text = articleSoup.get_text() | ||
| total = int(text[text.find("(") + 1:text.find(")")].split()[0]) | ||
| curr_offset = 0 | ||
| while curr_offset < total: |
There was a problem hiding this comment.
This could use a for curr_offset in range(0,total,100)
| base_url = "https://lwn.net/" | ||
|
|
||
|
|
||
| def extractPackageData(advisoryLink, dist, advisoryId): |
There was a problem hiding this comment.
This misses the key component, package's version, please extract the package version and refactor that in data dump.
There was a problem hiding this comment.
I can't find a way to scrape the versions of the packages, but we can get the CVE if there's a source we can use it to get the versions I'll work on it. any ideas for it?
There was a problem hiding this comment.
I haven't seen all the 'articles', so I may be wrong :) but it seems Gentoo,Fedora,Oracle distributions are providing patched or vulnerable versions of the package and they do follow a pattern. I think those could be extracted. Vulnerabilities without vulnerable packages are kind of useless. BTW we already have a scraper for Debian,Archlinux,Ubuntu so those could be safely skipped in this scraper.
Signed-off-by: Islam ElHakmi <eslam.elhakmey3@gmail.com>
| 'cve_ids': cves, | ||
| 'references': references, | ||
| 'summary': summary, | ||
| 'advisory_id': advisoryId, |
There was a problem hiding this comment.
I'm not sure how this is handled for Gentoo but by looking at the sample data given
......
"summary": "[gentoo-announce] [ GLSA 201911-07 ] Mozilla Firefox: Multiple vulnerabilities",
"advisory_id": "201911-07",
......
I think it will be more correct if it looked something like this "advisory_id":" GLSA 201911-07" . Ofcourse that would mean crafting separate logic for gentoo(mailing lists are painful)
It does? I thought we wanted to get rid of the little web scraping we have, not add more. Why do we even have a ticket about LWN? What exactly is the information they publish? After looking around a bit it seems like it's just security advisories from various Linux distributions. I think we should get that data from the distributions directly, in a machine-readable format. |
I think this came up earlier(about giving a second thought to every data collection ticket). We need to flag duplicates(in this case lwn debian is duplicate of debian json) and eliminate human-readable advisories as much as possible. |
Pretty much all tickets that are not bugs or part of any milestones should be considered a brain dump; reminders to look into potential data sources and decide whether we want to write code for that. It is possible that there are duplicates or overlapping scope in those bazillion tickets but that's a secondary concern, after the initial research about a potential data source has been done. @EslamHiko Of course there was no way for you to know that. I'm sorry that you put in so much effort here. This project is just woefully unprepared to accept contributions. Apart from the need to categorize and flesh out the tickets, there are some fundamentals that need to be straightened out before it makes sense to write dozens of importers.
To me this is a decision on our side and personally I would decide not to bother with web scraping. We could use a ticket to discuss this questions. That's probably a more appropriate place than the chat or comments on pull requests. |
|
@haikoschol no problems at all I'll take a look at the milestones. |
4 participants
This is a PR to solve: #77
This a sample data file shows the data I could collect (10 from items from each distributor) : sample-data.zip
small example :