Skip to content

Scrapes data from scrapers and dump in the database - #25

Merged
tdruez merged 31 commits into
developfrom
data_dump
Aug 8, 2017
Merged

Scrapes data from scrapers and dump in the database#25
tdruez merged 31 commits into
developfrom
data_dump

Conversation

@kartiksibal

Copy link
Copy Markdown
Contributor
  • Also adds description in Debian's scraper logic.

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Comment thread data_dump.py Outdated

vulnerability.save()
vulnerability_reference.save()
package.save()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the save() need to be in the loop...
Otherwise you are only saving the latest item. Add some tests that show that

Comment thread data_dump.py Outdated


def ubuntu_dump():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some docstring

Comment thread data_dump.py Outdated


def debian_dump():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add docstring

Comment thread data_dump.py Outdated
json_data = debian.json_data()
extracted_data = debian.extract_data(json_data)

for i, v in enumerate(extract_data):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you enumerate here at all? this code makes no sense at all.

@pombredanne pombredanne Jul 31, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still need to process the comment I made above ;)

@kartiksibal kartiksibal Jul 31, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne I have removed enumerate?

* moved .save() in the loop
* remove enumerate

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
* uses debian' test data to scrape
* dumps data in the test DB and verifies it

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
* Changed test cases accordingly

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to start by testing things with more assertions and ensure that all the records you expect are there. You will then see that your code is not inserting all the records

Comment thread app/vulncode_app/data_dump.py Outdated

def debian_dump(extract_data):
"""
Dump data scraped from Debian' security tracker.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Save might be a better name than dump ;)

Comment thread app/vulncode_app/data_dump.py Outdated
vulnerability_reference.save()
package.save()

return vulnerability, vulnerability_reference, package

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to return anything? Things are saved in the DB.
Also if you return here.... this will return at the first iteration. skipping all the other records.

Comment thread app/vulncode_app/test_data_dump.py Outdated
extract_data = debian.extract_data(test_data)
data_dump = debian_dump(extract_data)

self.assertEqual(data_dump[0].summary, extract_data[0].get('description'))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your tests should instead do a query in the DB. In particular you should start by asserting that the number of records you think you inserted based on the contents of tests/test_data/debian.json is correct.
Then test the values of all records

Comment thread app/vulncode_app/test_models.py Outdated
summary_create = Vulnerability.objects.create(summary="Affected package xyz")
summary_get = Vulnerability.objects.get(pk=summary_create.pk)

self.assertEqual(str(summary_create), "Affected package xyz")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not convert to a string. Instead test the attributes values here and in your other tests

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne Do you think we should be keeping test_models. Won't, test_data_dump suffice?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, keep these tests

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and do not convert to str

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne Will update. 👍

* put expected value before test value
* retrieve attr values instead of converting to strings
* remove return statemetn in

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Comment thread app/vulncode_app/data_dump.py Outdated
"""
Dump data scraped from Ubuntu's security tracker.
"""
for data in extract_data:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You did not run the tests ;) where could extract_data come from?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot see a parameter in https://github.com/nexB/vulnerablecode/pull/25/files#diff-102b650ccd1cc2cb42f5359f5101e8e1R66
def ubuntu_dump(): .... there is no args: hence I can only conclude that the code has never be run or tested ;)

Comment thread app/vulncode_app/test_data_dump.py Outdated
extract_data = debian.extract_data(test_data)
data_dump = debian_dump(extract_data)

for i in range(3):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why limit yourself to 3?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But why doing three iterations?

@kartiksibal kartiksibal Aug 2, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne I was iterating over extract_data and incrementing pk. To match the expected data with the data we are getting. And since we have three entries, in the test data. Hence, 3

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a robust approach for testing. Instead use a query, this is an iterable that you can then handle. Never expect that the pk have a specific values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne Got it. 👍

Comment thread app/vulncode_app/test_data_dump.py Outdated
self.assertEqual(extract_data[i].get('vulnerability_id'),
VulnerabilityReference.objects.get(pk=i+1).reference_id)
self.assertEqual(extract_data[i].get('package_name'),
Package.objects.get(pk=i+1).name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should do a query, not hope that the PK will be incremented linearly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating. 👍

Comment thread app/vulncode_app/test_data_dump.py Outdated
data_dump = debian_dump(extract_data)

for i in range(3):
self.assertEqual(3, len(Vulnerability.objects.all()))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the query in the range loop?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also you should not assert on all() but instead with a filter that select what you expect there.

@kartiksibal kartiksibal Aug 2, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne The query shouldn't be in the loop.
I'll update that. I'll factor in the second point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne Since, we are testing the number of entries. Isn't .all() a requirement here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main point was not to make that test three times in a range loop.
As far all is concerned, that is fine to count, but then use count() not all()

@kartiksibal kartiksibal Aug 3, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha 👍
I have addressed your first point.

Well there is no loop anymore. So, the root is gone.

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that you did not run the code nor the tests.
You need to document how to run those new tests in the README and make sure those are part of the continuous integration test suite (Travis CI)

Comment thread app/vulncode_app/data_dump.py Outdated
from scraper import debian, ubuntu


def debian_data():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in the scraper/debian.py module, for consistency with https://github.com/nexB/vulnerablecode/blob/develop/scraper/ubuntu.py#L50

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez Valid point. 👍

"""
Save data scraped from Debian' security tracker.
"""
for data in extract_data:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use ModelClass.objects.create() instead of save() for simplicity.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I doubt the following objects creation works, the required fields are not properly set.

@kartiksibal kartiksibal Aug 2, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez Could you elaborate this? Are you talking in context to models.py?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example: Vulnerability.objects.create(summary="Affected package xyz")
-> django.db.utils.IntegrityError: NOT NULL constraint failed: vulncode_app_vulnerability.cvss
Since the cvss is not a nullable field, and since you do not set a value for it, your save() cannot work

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez Yes, I get your point. I'll add the updated models along the next commit.

Comment thread app/vulncode_app/data_dump.py Outdated
package.save()


def ubuntu_data():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is un-needed, just call ubuntu.scrape_cves() in the ubuntu_dump() function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez gotcha 👍

Comment thread app/vulncode_app/test_models.py Outdated
class TestVulnerabilityReference(TestCase):
def test_vulnerability_reference(self):
ref_id_create = VulnerabilityReference.objects.create(reference_id="CVE-2017-8564")
ref_id_get = VulnerabilityReference.objects.get(pk=ref_id_create.pk)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un-used variable.

Comment thread app/vulncode_app/test_models.py Outdated
class TestPackage(TestCase):
def test_package(self):
package_name_create = Package.objects.create(name="Firefox")
package_name_get = Package.objects.get(pk=package_name_create.pk)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un-used variable.

Comment thread app/vulncode_app/test_models.py Outdated
class TestPackageReference(TestCase):
def test_package_reference(self):
platform_create = PackageReference.objects.create(platform="Maven")
platform_get = PackageReference.objects.get(pk=platform_create.pk)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un-used variable.

Comment thread app/vulncode_app/test_models.py Outdated
class TestVulnerability(TestCase):
def test_vulnerability(self):
summary_create = Vulnerability.objects.create(summary="Affected package xyz")
summary_get = Vulnerability.objects.get(pk=summary_create.pk)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un-used variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez test_models.py was just a trial method to test models, initially. I haven't updated it. I was under the impression that test_data_dump will suffice.

Will update this now. 👍

* removed loop, added individual asserts
* removed str()
* added test cases for ubuntu
* updated code formatting and test cases in test_models.py

Signed-off-by: kartik sibal <kartiksibal@gmail.com>
* Moved tests and scraper code in app/

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
* added null=true in fields which can be nullable

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
* used Class.objects.create() instead of .save()
* remove un-necessary ubuntu_data method

Signed-off-by: Kartik Sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use null=True on CharField but blank=True instead, see https://docs.djangoproject.com/en/1.11/ref/models/fields/#null

Comment thread app/vulncode_app/models.py Outdated
class Vulnerability(models.Model):
summary = models.TextField(max_length=50, help_text="Summary of the vulnerability")
cvss = models.FloatField(max_length=50, help_text="CVSS Score")
summary = models.TextField(max_length=50, help_text="Summary of the vulnerability", null=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the use case for creating Vulnerability without a summary?

Comment thread app/vulncode_app/models.py Outdated
source = models.CharField(max_length=50, help_text="Source's name eg:NVD")
reference_id = models.CharField(max_length=50, help_text="Reference ID, eg:CVE-ID")
url = models.URLField(max_length=1024, help_text="URL of Vulnerability data")
vulnerability = models.ForeignKey('Vulnerability', null=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the use case for creating VulnerabilityReference without a Vulnerability reference?

Comment thread app/vulncode_app/models.py Outdated
name = models.CharField(max_length=50, help_text="Package name")
version = models.CharField(max_length=50, help_text="Pacakge version")
platform = models.CharField(max_length=50, help_text="Package platform eg:maven", null=True)
name = models.CharField(max_length=50, help_text="Package name", null=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the use case for creating Package without a name?

Comment thread app/vulncode_app/models.py Outdated

class PackageReference(models.Model):
package = models.ForeignKey('Package')
package = models.ForeignKey('Package', null=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the use case for creating PackageReference without a Package reference

* changed test cases accordingly
* added blank=true
* cleaned test_models.py

Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
* an empty string will be returned instead of None

Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
* added `cd app/` in before_script

Signed-off-by: kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
Comment thread app/app/settings.py Outdated
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this blank line?

Comment thread app/scraper/debian.py Outdated
import json
from urllib.request import urlopen

import pprint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un-used import.

# VulnerableCode is a free software code scanning tool from nexB Inc. and others.
# Visit https://github.com/nexB/vulnerablecode/ for support and download.

from vulncode_app.models import Vulnerability

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove un-used imports

Comment thread app/vulncode_app/data_dump.py Outdated
"""
for data in extract_data:
vulnerability = Vulnerability.objects.create(summary=data.get('description', ''))
vulnerability_ref = VulnerabilityReference.objects.create(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to create a variable if you do not need to use the object.

Comment thread app/vulncode_app/models.py Outdated
url = models.URLField(max_length=1024, help_text="URL of Vulnerability data")
source = models.CharField(max_length=50, help_text="Source's name eg:NVD", blank=True)
reference_id = models.CharField(max_length=50, help_text="Reference ID, eg:CVE-ID", blank=True)
url = models.URLField(max_length=1024, help_text="URL of Vulnerability data", null=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

URLField is a subclass of CharField, use blank instead.

Comment thread app/vulncode_app/models.py Outdated
version = models.CharField(max_length=50, help_text="Pacakge version")
platform = models.CharField(max_length=50, help_text="Package platform eg:maven", blank=True)
name = models.CharField(max_length=50, help_text="Package name", blank=True)
version = models.CharField(max_length=50, help_text="Pacakge version", blank=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in "Pacakge"

Comment thread app/vulncode_app/models.py Outdated
class Vulnerability(models.Model):
summary = models.TextField(max_length=50, help_text="Summary of the vulnerability")
cvss = models.FloatField(max_length=50, help_text="CVSS Score")
summary = models.TextField(max_length=50, help_text="Summary of the vulnerability", blank=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using a TextField and not a CharField here?


from django.test import TestCase

from vulncode_app.models import Vulnerability

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, un-used import and un-used variable in that file.

* removed un-wanted creation of vars
* added blank=true in urlfield
* removed un-used imports

Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
* added doc-strings
* added unique together

Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
tdruez added 2 commits August 8, 2017 11:45
- Add missing initial migration file
- Code style and naming consistency
- Refine the README content
- Remove un-used module and section of code

Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made some comments, most are minor and easy to address. And then we can merge this

Comment thread .travis.yml Outdated
before_script:
- pycodestyle --exclude=migrations,settings.py,lib,tests --max-line-length=100 .
- cd app/
- python3 manage.py makemigrations

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this. Migrations should be committed

Comment thread .travis.yml

script:
- python3.6 -m pytest -v tests/
- python3.6 manage.py test

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use one or the other for running tests.... do not run tests twice

Comment thread README.md
@@ -28,18 +29,28 @@ pycodestyle --exclude=migrations,settings.py,lib,tests --max-line-length=100 .
cd app/
python3.6 -m pytest -v tests/

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here too, we should only use one way to run tests. Not two

Comment thread app/vulncode_app/models.py Outdated
summary = models.TextField(max_length=1024,
help_text="Summary of the vulnerability")
cvss = models.FloatField(help_text="CVSS Score")
summary = models.CharField(max_length=50, help_text="Summary of the vulnerability", blank=True)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use single quotes for all strings (unless this is a docstring or it contains a single quote itself)
This applies here and below

Comment thread app/vulncode_app/test_data_dump.py Outdated

self.assertTrue(VulnerabilityReference.objects.get(reference_id="CVE-2009-2458"))

self.assertTrue(VulnerabilityReference.objects.get(reference_id="CVE-2009-2459"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single quotes for strings throughout.

Comment thread app/vulncode_app/urls.py Outdated

urlpatterns = [
url(r'(?P<name>[a-z]+)/(?P<version>[0-9]+)', views.package_version, name="package_version"),
url(r'^(?P<name>[a-z]+)', views.package, name="package"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single quotes

Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
Signed-off-by: Kartik sibal <kartiksibal@gmail.com>
@tdruez
tdruez merged commit 95f725f into develop Aug 8, 2017
@tdruez
tdruez deleted the data_dump branch August 8, 2017 11:55
pombredanne added a commit that referenced this pull request Apr 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants