Store summary in `_insert_vulnerabilities_and_references()` by haikoschol · Pull Request #195 · aboutcode-org/vulnerablecode

haikoschol · 2020-06-08T11:25:17Z

Bugfix: For advisories with a CVE ID that don't exist in the database yet, the summary was not stored.

Bugfix: For advisories with a CVE ID that don't exist in the database yet, the summary was not stored. Signed-off-by: Haiko Schol <hs@haikoschol.com>

sbs2001 · 2020-06-09T01:05:58Z

I am getting this


Importing data from rust
Successfully imported data from rust
Importing data from alpine
Successfully imported data from alpine
Importing data from archlinux
Successfully imported data from archlinux
Importing data from debian
Traceback (most recent call last):
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get
    raise self.model.DoesNotExist(
vulnerabilities.models.DoesNotExist: Package matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(50)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/shivam/coding/opensource/oriv/with_importer/vulnerablecode/vulnerablecode/vulnerabilities/management/commands/import.py", line 66, in handle
    self._import_data(Importer.objects.all(), options['cutoff_date'])
  File "/home/shivam/coding/opensource/oriv/with_importer/vulnerablecode/vulnerablecode/vulnerabilities/management/commands/import.py", line 104, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/home/shivam/coding/opensource/oriv/with_importer/vulnerablecode/vulnerablecode/vulnerabilities/import_runner.py", line 78, in run
    _process_updated_advisories(data_source)
  File "/home/shivam/coding/opensource/oriv/with_importer/vulnerablecode/vulnerablecode/vulnerabilities/import_runner.py", line 132, in _process_updated_advisories
    pkg, created = _get_or_create_package(rpkg_url)
  File "/home/shivam/coding/opensource/oriv/with_importer/vulnerablecode/vulnerablecode/vulnerabilities/import_runner.py", line 180, in _get_or_create_package
    return models.Package.objects.get_or_create(**query_kwargs)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 562, in get_or_create
    return self._create_object_from_params(kwargs, params)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 596, in _create_object_from_params
    obj = self.create(**params)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/base.py", line 782, in save_base
    updated = self._save_table(
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/base.py", line 924, in _do_insert
    return manager._insert(
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
    cursor.execute(sql, params)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(50)

sbs2001 · 2020-06-09T02:10:20Z

Seems unrelated to your PR

haikoschol · 2020-06-09T15:31:29Z

I just reproduced the issue by running ./manage.py import --all. As you mentioned in gitter, the initial migration that creates the Package model still has max_length=50 for version. I forgot to change that when switching to the latest version of packageurl, which has max_length=100. I'll file a separate PR.

sbs2001 · 2020-06-09T15:42:58Z

Which specific Importer caused it ?

If it is was the debian one, I managed to fix that by writing versions to db without normalizing the version string, as mentioned in gitter.

I also tried setting max_length = 100 manually in the db and not changing anything else, that also fixed the issue.

haikoschol · 2020-06-09T15:53:21Z

Which specific Importer caused it ?

It was debian, specifically this query: INSERT INTO "vulnerabilities_package" ("type", "namespace", "name", "version", "qualifiers", "subpath") VALUES ('deb', 'debian', 'novnc', '1:0.4%2Bdfsg%2B1%2B20131010%2Bgitf68af8af3d-4%2Bdeb8u1', 'distro=jessie', NULL) RETURNING "vulnerabilities_package"."id"

Regardless of any other changes, the migration needs to be fixed as well.

haikoschol requested review from sbs2001 and singh1114 June 8, 2020 11:25

Store summary in _insert_vulnerabilities_and_references()

8c4876d

Bugfix: For advisories with a CVE ID that don't exist in the database yet, the summary was not stored. Signed-off-by: Haiko Schol <hs@haikoschol.com>

haikoschol force-pushed the fix_store_summary branch from 592473e to 8c4876d Compare June 8, 2020 13:08

sbs2001 approved these changes Jun 9, 2020

View reviewed changes

haikoschol merged commit 826de10 into aboutcode-org:develop Jun 9, 2020

haikoschol deleted the fix_store_summary branch June 9, 2020 16:24

Navigation Menu

Uh oh!

Uh oh!

Store summary in `_insert_vulnerabilities_and_references()` - #195

Store summary in `_insert_vulnerabilities_and_references()`#195
haikoschol merged 1 commit into
aboutcode-org:developfrom
haikoschol:fix_store_summary

haikoschol commented Jun 8, 2020

Uh oh!

sbs2001 commented Jun 9, 2020

Uh oh!

sbs2001 commented Jun 9, 2020

Uh oh!

haikoschol commented Jun 9, 2020

Uh oh!

sbs2001 commented Jun 9, 2020 •

edited

Loading

Uh oh!

haikoschol commented Jun 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

haikoschol commented Jun 8, 2020

Uh oh!

sbs2001 commented Jun 9, 2020

Uh oh!

sbs2001 commented Jun 9, 2020

Uh oh!

haikoschol commented Jun 9, 2020

Uh oh!

sbs2001 commented Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haikoschol commented Jun 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sbs2001 commented Jun 9, 2020 •

edited

Loading