diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 96eb3bacb..fa7f34795 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -2,6 +2,36 @@ Release notes ============= + +Version v30.2.0 +---------------- + +This is a critical bug fix release including features updates. + +- We fixed critical performance issues that made the web UI unusable. This include + removing some less interesting redundant details displayed in the web UI for + vulnerabilities. +- We made minor documentation updates. +- We re-enabled support for Arch linux, Debian, and Ubuntu security advisories importers +- We added a new improver for Oval data sources +- We improved Alpine linux and Gitlab security advisories importers + +The summary of performance improvements include these fixes: + +- Cascade queries from exact to approximate searches to avoid full table scans + in all cases. This is a band-aid for now. The proper solution will likely + require using full text search instead. +- Avoid iceberg queries with "prefetch related" to limit the number of queries + that are needed in the UI +- Do not recreate querysets from scratch but instead allow these to be chained + for simpler and correct code. +- Remove extra details from the vulnerability pacge: each package was further + listing its related vulnerabilities creating an iceberg query. +- Enable the django-debug-toolbar with a setting to easily profile queries on demand + by setting both VULNERABLECODE_DEBUG and VULNERABLECODE_DEBUG_TOOLBAR enviroment + variables. + + Version v30.1.1 ---------------- diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index daf7552e4..96d557f25 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -5,7 +5,7 @@ Contributing to VulnerableCode Thank you so much for being so interested in contributing to VulnerableCode. We are always on the lookout for enthusiastic contributors like you who can make -our project better, and we're willing to lend a helping hand if you have any +our project better, and we are willing to lend a helping hand if you have any questions or need guidance along the way. That being said, here are a few resources to help you get started. @@ -33,13 +33,13 @@ join our community. Below are some examples to get involved: First Timers ^^^^^^^^^^^^ -You are here to help, but you're a new contributor! No worries, we always +You are here to help, but you are a new contributor! No worries, we always welcome newcomer contributors. We maintain some `good first issues `_ and encourage new contributors to work on those issues for a smooth start. .. tip:: - If you're an open-source newbie, make sure to check the extra resources at + If you are an open-source newbie, make sure to check the extra resources at the bottom of this page to get the hang of the contribution process! Code Contributions @@ -75,7 +75,7 @@ Other Ways ^^^^^^^^^^ You want to contribute to other aspects of the VulnerableCode project, and you -can't find what you're looking for! You can always discuss new topics, ask +cannot find what you are looking for! You can always discuss new topics, ask questions, and interact with us and other community members on `AboutCode Gitter `_ and `VulnerableCode Gitter `_ diff --git a/docs/source/index.rst b/docs/source/index.rst index dc84c96ec..7a49c83b3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,9 +1,18 @@ -VulnerableCode documentation +Welcome to VulnerableCode! ============================= -Welcome to VulnerableCode! In this documentation you’ll find information on: +*VulnerableCode* provides an open database of software packages that are affected +by known security vulnerabilities aka. *"vulnerable packages"*. -- An overview of VulnerableCode +VulnerableCode is also a free and open source software (FOSS) project that +provides the tools to build this open database. The tools handle collecting, +aggregating and correlating these vulnerabilities and relating them to a correct +package version. Our project also supports a public cloud instance of this +database - VulnerableCode.io. + +In this documentation you will find information on: + +- An overview of VulnerableCode and what you can do with it - Installation instructions - How to make technical contributions to the project and the community @@ -33,6 +42,7 @@ Welcome to VulnerableCode! In this documentation you’ll find information on: reference_improver_overview reference_framework_overview command-line-interface + importers_link .. toctree:: :maxdepth: 1 diff --git a/docs/source/introduction.rst b/docs/source/introduction.rst index b4f10fb60..20fec6bc1 100644 --- a/docs/source/introduction.rst +++ b/docs/source/introduction.rst @@ -3,18 +3,61 @@ VulnerableCode Overview ======================== -VulnerableCode is a FOSS project that provides tools to build a database -of software vulnerabilities and the packages they impact. The tools -handle collecting, aggregating and correlating these vulnerabilities. -Our project also supports a public Cloud instance of this database – -VulnerableCode.io. +*VulnerableCode* provides an open database of software packages that are affected +by known security vulnerabilities aka. *"vulnerable packages"*. + +VulnerableCode is also a free and open source software (FOSS) project that +provides the tools to build this open database. The tools handle collecting, +aggregating and correlating these vulnerabilities and relating them to a correct +package version. Our project also supports a public cloud instance of this +database - VulnerableCode.io. + + +What can I do with VulnerableCode? +------------------------------------ + +**For security researchers and software developers, VulnerableCode offers a web +UI and a JSON API to efficient find if the FOSS packages and dependencies that +you use may be affected by known vulnerabilities and which version of a package +you should upgrade to to fix this issue.** + + +- With the web UI, you can search by package using Package URLs or search by + vulnerability like by CVE. From there you can navigate to the package + vulnerabilities and to the vulnerable packages. + +- With the JSON API, you can perform package queries using Package URLs or query + by vulnerability id. You can also query by CPEs and vulnerability aliases. + The API provides paginated index and detail endpoints and includes indexes + of vulnerable CPEs and vulnerable Package URLs (purl). + +You can also install VulnerableCode locally or use the provided publicly hosted instance, +or host your own installation. You can also contact the VulnerableCode authors and team +for special needs including commercial support. + Why VulnerableCode? ------------------- -Existing vulnerability database solutions are primarily commercial or -proprietary which does not make sense because the data is about FOSS -(Free and Open Source Software). +VulnerableCode provides open correlated data and eventually does provide curated +data. Our approach is to privilege upstream data sources and to merge multiple +vulnerability data sources after comparison and correlation. The vulnerability +data is keyed by Package URL ensuring quick and accurate lookup with minimal +friction. We further continuously validate and refine the collected data for +quality, accuracy and consistency using "improver" jobs. +An example of such improver can validate that a package version reported as +vulnerable effectively exists (several do not exist); Or an improver can +re-evaluate a vulnerable version ranges based on the latest releases of a +package. + +The benefits of our approach is that we will eventually provide better, more +accurate vulnerability data, more efficiently related to actual packages scanned +or reported in an SBOM. This should contribute to more efficient vulnerability +management with less noise from false positives. + +Furthermore, existing vulnerability database solutions are primarily commercial +or proprietary which does not make sense because the bulk of the vulnerability +data is about FOSS. The National Vulnerability Database, which is a primary centralized data source for known vulnerabilities, is not particularly well suited to diff --git a/docs/source/tutorial_add_new_importer.rst b/docs/source/tutorial_add_new_importer.rst index 137317985..4ee5879f6 100644 --- a/docs/source/tutorial_add_new_importer.rst +++ b/docs/source/tutorial_add_new_importer.rst @@ -15,7 +15,8 @@ TL;DR #. Create a new importer subclass inheriting from the ``Importer`` superclass defined in ``vulnerabilites.importer``. It is conventional to end an importer name with *Importer*. #. Specify the importer license. -#. Implement the ``advisory_data`` method to process the data source you're writing an importer for. +#. Implement the ``advisory_data`` method to process the data source you are + writing an importer for. #. Add the newly created importer to the importers registry at ``vulnerabilites/importers/__init__.py`` @@ -136,7 +137,9 @@ version management from `univers `_. .. note:: - It is possible that the versioning scheme you are targetting has not yet been implemented in the `univers `_ library. If this is the case, you'll need to head over there and implement one. + It is possible that the versioning scheme you are targetting has not yet been + implemented in the `univers `_ library. + If this is the case, you will need to head over there and implement one. .. code-block:: python @@ -235,12 +238,12 @@ Finally, register your importer in the importer registry at IMPORTERS_REGISTRY = {x.qualified_name: x for x in IMPORTERS_REGISTRY} -Congratulations! You've written your first importer. +Congratulations! You have written your first importer. Run Your First Importer ^^^^^^^^^^^^^^^^^^^^^^^^^^ -If everything went well, you'll see your importer in the list of available importers. +If everything went well, you will see your importer in the list of available importers. .. code-block:: console :emphasize-lines: 5 @@ -284,7 +287,7 @@ For more visibility, turn on debug logs in :file:`vulnerablecode/settings.py`. }, } -Invoke the import command now and you'll see (in a fresh database): +Invoke the import command now and you will see (in a fresh database): .. code-block:: console diff --git a/docs/source/tutorial_add_new_improver.rst b/docs/source/tutorial_add_new_improver.rst index 5d56885f7..16fc7beab 100644 --- a/docs/source/tutorial_add_new_improver.rst +++ b/docs/source/tutorial_add_new_improver.rst @@ -76,7 +76,7 @@ Explore Package Managers (Optional) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your Improver depends on the discrete versions of a package, the package managers' VersionAPI -located at :file:`vulnerabilites/package_managers.py` could come in handy. You'll need to +located at :file:`vulnerabilites/package_managers.py` could come in handy. You will need to instantiate the relevant ``VersionAPI`` in the improver's constructor and use it later in the implemented methods. See an already implemented improver (NginxBasicImprover) for an example usage. @@ -184,12 +184,12 @@ Finally, register your improver in the improver registry at IMPROVERS_REGISTRY = {x.qualified_name: x for x in IMPROVERS_REGISTRY} -Congratulations! You've written your first improver. +Congratulations! You have written your first improver. Run Your First Improver ^^^^^^^^^^^^^^^^^^^^^^^^^^ -If everything went well, you'll see your improver in the list of available improvers. +If everything went well, you will see your improver in the list of available improvers. .. code-block:: console :emphasize-lines: 6 @@ -244,7 +244,7 @@ For more visibility, turn on debug logs in :file:`vulnerablecode/settings.py`. }, } -Invoke the improve command now and you'll see (in a fresh database, after importing): +Invoke the improve command now and you will see (in a fresh database, after importing): .. code-block:: console diff --git a/pyproject.toml b/pyproject.toml index f9b73e098..0eef62af2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -46,23 +46,16 @@ addopts = [ "--doctest-modules", # Ignore the following doctests until these files are migrated to # import-improve structure - "--ignore=vulnerabilities/importers/alpine_linux.py", "--ignore=vulnerabilities/importers/apache_httpd.py", "--ignore=vulnerabilities/importers/apache_kafka.py", "--ignore=vulnerabilities/importers/apache_tomcat.py", - "--ignore=vulnerabilities/importers/archlinux.py", - "--ignore=vulnerabilities/importers/debian.py", "--ignore=vulnerabilities/importers/elixir_security.py", "--ignore=vulnerabilities/importers/gentoo.py", - "--ignore=vulnerabilities/importers/github.py", "--ignore=vulnerabilities/importers/istio.py", "--ignore=vulnerabilities/importers/kaybee.py", "--ignore=vulnerabilities/importers/npm.py", - "--ignore=vulnerabilities/importers/nvd.py", - "--ignore=vulnerabilities/importers/openssl.py", "--ignore=vulnerabilities/importers/postgresql.py", "--ignore=vulnerabilities/importers/project_kb_msr2019.py", - "--ignore=vulnerabilities/importers/redhat.py", "--ignore=vulnerabilities/importers/retiredotnet.py", "--ignore=vulnerabilities/importers/ruby.py", "--ignore=vulnerabilities/importers/rust.py", diff --git a/setup.cfg b/setup.cfg index 3d91eb983..192974522 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = vulnerablecode -version = 30.1.1 +version = 30.2.0 license = Apache-2.0 AND CC-BY-SA-4.0 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390 diff --git a/vulnerabilities/importers/alpine_linux.py b/vulnerabilities/importers/alpine_linux.py index b1fb861ac..b4a05e8b7 100644 --- a/vulnerabilities/importers/alpine_linux.py +++ b/vulnerabilities/importers/alpine_linux.py @@ -15,19 +15,13 @@ from typing import Mapping from urllib.parse import urljoin -import requests from bs4 import BeautifulSoup -from django.db.models.query import QuerySet from packageurl import PackageURL from univers.versions import AlpineLinuxVersion from vulnerabilities.importer import AdvisoryData from vulnerabilities.importer import AffectedPackage from vulnerabilities.importer import Importer -from vulnerabilities.improver import MAX_CONFIDENCE -from vulnerabilities.improver import Improver -from vulnerabilities.improver import Inference -from vulnerabilities.models import Advisory from vulnerabilities.references import WireSharkReference from vulnerabilities.references import XsaReference from vulnerabilities.references import ZbxReference diff --git a/vulnerabilities/importers/archlinux.py b/vulnerabilities/importers/archlinux.py index a059ab9e3..c188a3e5b 100644 --- a/vulnerabilities/importers/archlinux.py +++ b/vulnerabilities/importers/archlinux.py @@ -10,7 +10,6 @@ from typing import Iterable from typing import List from typing import Mapping -from urllib.request import urlopen from packageurl import PackageURL from univers.version_range import ArchLinuxVersionRange diff --git a/vulnerabilities/importers/github.py b/vulnerabilities/importers/github.py index a643c12d7..1532dae6c 100644 --- a/vulnerabilities/importers/github.py +++ b/vulnerabilities/importers/github.py @@ -184,12 +184,16 @@ def advisory_data(self) -> Iterable[AdvisoryData]: def get_purl(pkg_type: str, github_name: str) -> Optional[PackageURL]: """ - Return a PackageURL by splitting the `github_name` using the `pkg_type` convention. - Return None and log an error if we can not split or it is an unknown package type. - >>> get_purl("maven", "org.apache.commons:commons-lang3") - PackageURL(type="maven", namespace="org.apache.commons", name="commons-lang3") - >>> get_purl("composer", "foo/bar") - PackageURL(type="composer", namespace="foo", name="bar") + Return a PackageURL by splitting the `github_name` using the `pkg_type` + convention. Return None and log an error if we can not split or it is an + unknown package type. + + For example:: + >>> expected = PackageURL(type='maven', namespace='org.apache.commons', name='commons-lang3') + >>> assert get_purl("maven", "org.apache.commons:commons-lang3") == expected + + >>> expected = PackageURL(type="composer", namespace="foo", name="bar") + >>> assert get_purl("composer", "foo/bar") == expected """ if pkg_type == "maven": if ":" not in github_name: diff --git a/vulnerabilities/importers/gitlab.py b/vulnerabilities/importers/gitlab.py index b085e02b5..afdb3f865 100644 --- a/vulnerabilities/importers/gitlab.py +++ b/vulnerabilities/importers/gitlab.py @@ -20,7 +20,6 @@ import saneyaml from dateutil import parser as dateparser from django.db.models.query import QuerySet -from fetchcode.vcs import fetch_via_vcs from packageurl import PackageURL from univers.version_range import RANGE_CLASS_BY_SCHEMES from univers.version_range import VersionRange diff --git a/vulnerabilities/tests/example_importer_improver.py b/vulnerabilities/tests/example_importer_improver.py index 1d315fd57..8717d607d 100644 --- a/vulnerabilities/tests/example_importer_improver.py +++ b/vulnerabilities/tests/example_importer_improver.py @@ -11,7 +11,6 @@ from datetime import timezone from typing import Iterable -import requests from django.db.models.query import QuerySet from packageurl import PackageURL from univers.version_range import NginxVersionRange diff --git a/vulnerabilities/tests/test_api.py b/vulnerabilities/tests/test_api.py index cd4b9c406..487819978 100644 --- a/vulnerabilities/tests/test_api.py +++ b/vulnerabilities/tests/test_api.py @@ -10,21 +10,17 @@ import os -from collections import OrderedDict from random import choices -from unittest.mock import MagicMock from urllib.parse import quote from django.test import TestCase from django.test.client import RequestFactory -from rest_framework.test import APIClient -from rest_framework.test import APIRequestFactory from vulnerabilities.api import PackageSerializer from vulnerabilities.models import Package BASE_DIR = os.path.dirname(os.path.abspath(__file__)) -TEST_DATA = os.path.join(BASE_DIR, "test_data/") +TEST_DATA = os.path.join(BASE_DIR, "test_data") def cleaned_response(response): @@ -75,7 +71,7 @@ def cleaned_response(response): class TestDebianResponse(TestCase): - fixtures = ["debian.json"] + fixtures = [os.path.join(TEST_DATA, "api_fixtures/debian.json")] @classmethod def setUpTestData(cls): @@ -164,7 +160,7 @@ def test_query_by_package_url_without_namespace(self): class APIResponseRelations(TestCase): - fixtures = ["openssl.json"] + fixtures = [os.path.join(TEST_DATA, "api_fixtures/openssl.json")] def test_vulnerability_package_relations(self): @@ -207,7 +203,7 @@ def test_vulnerability_package_relations(self): class TestSerializers(TestCase): - fixtures = ["debian.json"] + fixtures = [os.path.join(TEST_DATA, "api_fixtures/debian.json")] def test_package_serializer(self): pk = Package.objects.filter(name="mimetex") @@ -228,7 +224,7 @@ def test_package_serializer(self): class TestBulkAPIResponse(TestCase): - fixtures = ["github.json"] + fixtures = [os.path.join(TEST_DATA, "api_fixtures/github.json")] def test_bulk_packages_api(self): request_body = { diff --git a/vulnerabilities/tests/test_archlinux.py b/vulnerabilities/tests/test_archlinux.py index 3b1908e73..f5ee38b75 100644 --- a/vulnerabilities/tests/test_archlinux.py +++ b/vulnerabilities/tests/test_archlinux.py @@ -12,10 +12,6 @@ import os from unittest.mock import patch -from django.test import TestCase - -from vulnerabilities import models -from vulnerabilities.import_runner import ImportRunner from vulnerabilities.importers import archlinux from vulnerabilities.tests import util_tests diff --git a/vulnerabilities/fixtures/debian.json b/vulnerabilities/tests/test_data/api_fixtures/debian.json similarity index 100% rename from vulnerabilities/fixtures/debian.json rename to vulnerabilities/tests/test_data/api_fixtures/debian.json diff --git a/vulnerabilities/fixtures/github.json b/vulnerabilities/tests/test_data/api_fixtures/github.json similarity index 100% rename from vulnerabilities/fixtures/github.json rename to vulnerabilities/tests/test_data/api_fixtures/github.json diff --git a/vulnerabilities/fixtures/openssl.json b/vulnerabilities/tests/test_data/api_fixtures/openssl.json similarity index 100% rename from vulnerabilities/fixtures/openssl.json rename to vulnerabilities/tests/test_data/api_fixtures/openssl.json diff --git a/vulnerabilities/tests/test_import_command.py b/vulnerabilities/tests/test_import_command.py index 4807f5a2d..f5094113f 100644 --- a/vulnerabilities/tests/test_import_command.py +++ b/vulnerabilities/tests/test_import_command.py @@ -15,7 +15,6 @@ from django.core.management.base import CommandError from django.test import TestCase -from vulnerabilities.importer import AdvisoryData from vulnerabilities.importer import Importer diff --git a/vulnerabilities/tests/test_openssl.py b/vulnerabilities/tests/test_openssl.py index 090a36b7c..328f025ea 100644 --- a/vulnerabilities/tests/test_openssl.py +++ b/vulnerabilities/tests/test_openssl.py @@ -9,13 +9,11 @@ import datetime import json -import os from pathlib import Path import defusedxml.ElementTree as DET import pytest from commoncode import testcase -from django.db.models.query import QuerySet from packageurl import PackageURL from univers.version_constraint import VersionConstraint from univers.version_range import OpensslVersionRange diff --git a/vulnerabilities/tests/test_performance.py b/vulnerabilities/tests/test_performance.py index 1dd376aee..05f043aed 100644 --- a/vulnerabilities/tests/test_performance.py +++ b/vulnerabilities/tests/test_performance.py @@ -9,8 +9,8 @@ # import pytest -# this import are used in the script -from vulnerabilities.importers import redhat +# this import is used in the script +from vulnerabilities.importers import redhat # NOQA script = """for i, data in enumerate(redhat.RedhatImporter().advisory_data()): if 1 == 100: diff --git a/vulnerabilities/tests/test_ubuntu.py b/vulnerabilities/tests/test_ubuntu.py index b0531ee2b..cc6c6630a 100644 --- a/vulnerabilities/tests/test_ubuntu.py +++ b/vulnerabilities/tests/test_ubuntu.py @@ -7,7 +7,6 @@ # See https://aboutcode.org for more information about nexB OSS projects. # -import asyncio import os import xml.etree.ElementTree as ET diff --git a/vulnerablecode/__init__.py b/vulnerablecode/__init__.py index 85560a2b1..b50d54a13 100644 --- a/vulnerablecode/__init__.py +++ b/vulnerablecode/__init__.py @@ -12,7 +12,7 @@ import warnings from pathlib import Path -__version__ = "30.1.1" +__version__ = "30.2.0" def command_line(): diff --git a/vulnerablecode/urls.py b/vulnerablecode/urls.py index 8fb18017d..8d09fa72e 100644 --- a/vulnerablecode/urls.py +++ b/vulnerablecode/urls.py @@ -7,7 +7,6 @@ # See https://aboutcode.org for more information about nexB OSS projects. # -from django.contrib import admin from django.urls import include from django.urls import path from django.urls import re_path