Skip to content

Improve docs #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbs2001 merged 6 commits into from
Feb 13, 2021
Merged

Improve docs #316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ea199c3
Removing link to external https://makeapullrequest.com/
pombredanne Jan 27, 2021
0cd35e4
Improve README and bundle PDF
pombredanne Jan 27, 2021
bd4167c
Merge remote latest main branch
pombredanne Jan 27, 2021
8ed231f
Improve readability of Nix docs #275
pombredanne Jan 27, 2021
8f77147
Update README.rst
sbs2001 Feb 13, 2021
029f259
Merge branch 'main' into improve-docs
sbs2001 Feb 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
286 changes: 153 additions & 133 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,95 +1,115 @@
VulnerableCode
==============

|Build Status| |License| |Python 3.8| |stability-wip| |Gitter chat| |PRs
Welcome|
|Build Status| |License| |Python 3.8| |stability-wip| |Gitter chat|

.. image:: README.gif

.. |Build Status| image:: https://travis-ci.org/nexB/vulnerablecode.svg?branch=develop
:target: https://travis-ci.org/nexB/vulnerablecode
.. |License| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:target: https://opensource.org/licenses/Apache-2.0
.. |Python 3.8| image:: https://img.shields.io/badge/python-3.8-blue.svg
:target: https://www.python.org/downloads/release/python-380/
.. |stability-wip| image:: https://img.shields.io/badge/stability-work_in_progress-lightgrey.svg
.. |Gitter chat| image:: https://badges.gitter.im/gitterHQ/gitter.png
:target: https://gitter.im/aboutcode-org/vulnerablecode

The What
--------

VulnerableCode is a FOSS database of vulnerabilities and the FOSS
packages they impact. It is made by the FOSS community to improve and
secure the open source software ecosystem.
VulnerableCode is a free and open database of FOSS software package
vulnerabilities and the tools to create and keep the data current.

The Why
-------
It is made by the FOSS community to improve and secure the open source software
ecosystem.

The existing solutions are commercial proprietary vulnerability
databases, which in itself does not make sense because the data is about
FOSS.
.. image:: README.gif

National Vulnerability Database which is the primary data source for all
things security, is not particularly catered to address FOSS security
issues, because:
Why?
----

The existing solutions are commercial proprietary vulnerability databases, which
in itself does not make sense because the data is about FOSS (Free and Open
Source Software).

The National Vulnerability Database which is a primary centralized data source
for known vulnerabilities is not particularly well suited to address FOSS
security issues because:

1. It predates the explosion of FOSS software usage
2. It's data format reflects a commercial vendor-centric point of view,
this is due to the usage of
`CPE <https://nvd.nist.gov/products/cpe>`__ to map vulnerabilities
and the packages.
3. CPEs are just not designed to map FOSS to vulnerabilities owing to
their vendor-product centric semantics. This makes it really hard to
answer the fundamental question "Is package foo vulnerable to
vulnerability bar?"


The How
-------

VulnerableCode independently aggregates many software vulnerability data
sources that can easily be recreated in a decentralized fashion. These
data sources (see complete list `here <./SOURCES.rst>`_) include security
advisories published by distros, package managers, etc. Due to this, the
data obtained is not generalized to apply for other ecosystems. This
increases the accuracy as the same version of a package across different distros
may or may not be vulnerable to some vulnerability.

The packages are identified using
`PURL <https://github.com/package-url/purl-spec>`__ rather than CPEs.
This makes it really easy to answer questions like "Is package foo
vulnerable to vulnerability bar ? ".

The web interface enables community curation of data by enabling
the addition of new packages, vulnerabilities, and modifying the
relationships between them as shown in GIF. Along with the web interface
the API allows seamless consumption of the data.
2. It's data format reflects a commercial vendor-centric point of view in part
due to the usage of `CPE <https://nvd.nist.gov/products/cpe>`__ to map
vulnerabilities to existing packages.
3. CPEs are just not designed to map FOSS to vulnerabilities owing to their
vendor-product centric semantics. This makes it really hard to answer the
fundamental questions "Is package foo vulnerable" and "Is package foo
vulnerable to vulnerability bar?"

How
---

VulnerableCode independently aggregates many software vulnerability data sources
and supports data re-creation in a decentralized fashion. These data sources
(see complete list `here <./SOURCES.rst>`_) include security advisories
published by Linux and BSD distributions, application software package managers
and package repositories, FOSS projects, GitHub and more. Thanks to this
approach, the data is focused on specific ecosystems yet aggregated in a single
database that enables querying a richer graph of relations between multiple
incarnations of a package. Being specific increases the accuracy and validity
of the data as the same version of an upstream package across different
ecosystems may or may not be vulnerable to the same vulnerability.

The packages are identified using Package URL `PURL
<https://github.com/package-url/purl-spec>`__ as primary identifiers rather than
CPEs. This makes answers to questions such as "Is package foo vulnerable
to vulnerability bar?" much more accurate and easy to interpret.


The primary access to the data is through a REST API.

In addition, an emerging web interface goal is to support vulnerabilities data
browsing and search and progressively to enable community curation of the data
with the addition of new packages and vulnerabilities, and reviewing and
updating their relationships.

We also plan to mine for vulnerabilities which didn't receive any
exposure due to various reasons like but not limited to the complicated
procedure to receive CVE ID or not able to classify a bug as a security
compromise.

Check VulnerableCode at `Open Source Summit 2020
<https://ossna2020.sched.com/event/c46p/why-is-there-no-free-software-vulnerability-database-philippe-ombredanne-aboutcodeorg-and-nexb-inc-michael-herzog-nexb-inc>`__
Recent presentations:

- `Open Source Summit 2020 <docs/Why-Is-There-No-Free-Software-Vulnerability-Database-v1.0.pdf>`__



Setting up VulnerableCode
-------------------------

Clone the source code:

::
First clone the source code::

git clone https://github.com/nexB/vulnerablecode.git
cd vulnerablecode




Using Docker Compose
~~~~~~~~~~~~~~~~~~~~

An easy way to set up VulnerableCode is with docker containers and
docker compose. For this you need to have the following installed.
An easy way to set up VulnerableCode is with docker containers and docker
compose. For this you need to have the following installed.

- Docker Engine. Find instructions to install it `here <https://docs.docker.com/get-docker/>`__
- Docker Compose. Find instructions to install it `here <https://docs.docker.com/compose/install/#install-compose>`__
- Docker Engine. Find instructions to install it
`here <https://docs.docker.com/get-docker/>`__
- Docker Compose. Find instructions to install it
`here <https://docs.docker.com/compose/install/#install-compose>`__

Use ``sudo docker-compose up`` to start VulnerableCode. Access
VulnerableCode at http://localhost:8000/ or at http://127.0.0.1:8000/ .
Use ``sudo docker-compose up`` to start VulnerableCode. Then access
VulnerableCode at http://localhost:8000/ or at http://127.0.0.1:8000/

Use ``sudo docker-compose exec web bash`` to access the VulnerableCode
container. From here you can access ``manage.py`` and run management
commands to import data as specified below.
container. From here you can access ``manage.py`` and run management commands
to import data as specified below.


Without Docker Compose
~~~~~~~~~~~~~~~~~~~~~~
Expand All @@ -100,50 +120,57 @@ Without Docker Compose
- PostgreSQL 9+
- Compiler toolchain and development files for Python and PostgreSQL

On Debian-based distros, these can be installed with
``sudo apt install python3-venv python3-dev postgresql libpq-dev build-essential``.
On Debian-based distros, these can be installed with::

**Database configuration** - Create a user named ``vulnerablecode``. Use
``vulnerablecode`` as password when prompted:
``sudo -u postgres createuser --no-createrole --no-superuser --login --inherit --createdb --pwprompt vulnerablecode``
sudo apt-get install python3-venv python3-dev postgresql libpq-dev build-essential

- Create a databased named ``vulnerablecode``:
``createdb --encoding=utf-8 --owner=vulnerablecode --user=vulnerablecode --password --host=localhost --port=5432 vulnerablecode``

**Application dependencies**
**Database configuration**

- Create a user named ``vulnerablecode``. Use ``vulnerablecode`` as password
when prompted::

Create a virtualenv, install dependencies, and run the database
migrations:
sudo -u postgres createuser --no-createrole --no-superuser --login \
--inherit --createdb --pwprompt vulnerablecode``

::
- Create a databased named ``vulnerablecode``::

createdb --encoding=utf-8 --owner=vulnerablecode --user=vulnerablecode \
--password --host=localhost --port=5432 vulnerablecode


**Application dependencies**

Create a virtualenv, install dependencies, and run the database migrations::

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
DJANGO_DEV=1 python manage.py migrate

The environment variable ``DJANGO_DEV`` is used to load settings
suitable for development, defined in ``vulnerablecode/dev.py``. If you
The environment variable ``DJANGO_DEV`` is used to load settings suitable for
development, defined in ``vulnerablecode/dev.py``. If you
don't want to type it every time use ``export DJANGO_DEV=1`` instead.
Do not use `DJANGO_DEV` in a production environment.


For a production mode, an environment variable named ``SECRET_KEY`` needs to be
set. The recommended way to generate this key is to use the code Django includes
for this purpose::

SECRET_KEY=$(python -c "from django.core.management import utils; print(utils.get_random_secret_key())")

When not running in development mode, an environment variable named
``SECRET_KEY`` needs to be set. The recommended way to generate this key
is to use the code Django includes for this purpose:
``SECRET_KEY=$(python -c "from django.core.management import utils; print(utils.get_random_secret_key())")``.

Using Nix
~~~~~~~~~

You can install VulnerableCode with `Nix <https://nixos.org/download.html>`__ (`Flake <https://nixos.wiki/wiki/Flakes>`__ support is needed).

::
You can install VulnerableCode with `Nix <https://nixos.org/download.html>`__
(`Flake <https://nixos.wiki/wiki/Flakes>`__ support is needed)::

cd etc/nix
nix --print-build-logs flake check # build & run tests

There are several options to use the Nix version

::
There are several options to use the Nix version::

# Enter an interactive environment with all dependencies set up.
cd etc/nix
Expand All @@ -159,53 +186,74 @@ There are several options to use the Nix version

**Keeping the Nix setup in sync**

The Nix installation uses `mach-nix <https://github.com/DavHau/mach-nix>`__ to handle Python dependencies because some dependencies are currently not available as Nix packages.
All Python dependencies are automatically fetched from ``./requirements.txt``.
If the ``mach-nix``-based installation fails, you might need to update ``mach-nix`` itself and the `pypi-deps-db <https://github.com/DavHau/pypi-deps-db>`_ version in use (see ``etc/nix/flake.nix:inputs.machnix`` and ``machnixFor.pypiDataRev``).
The Nix installation uses `mach-nix <https://github.com/DavHau/mach-nix>`__ to
handle Python dependencies because some dependencies are currently not available
as Nix packages. All Python dependencies are automatically fetched from
``./requirements.txt``. If the ``mach-nix``-based installation fails, you might
need to update ``mach-nix`` itself and the `pypi-deps-db
<https://github.com/DavHau/pypi-deps-db>`_ version in use (see
``etc/nix/flake.nix:inputs.machnix`` and ``machnixFor.pypiDataRev``).

Non-Python dependencies are curated in::

etc/nix/flake.nix:vulnerablecode.propagatedBuildInputs

Non-Python dependencies are curated in ``etc/nix/flake.nix:vulnerablecode.propagatedBuildInputs``.


Tests
-----
Run Tests
---------

::
Use these commands to run code style checks and the test suite::

pycodestyle --exclude=migrations,settings.py,venv,lib_oval.py,test_ubuntu.py,test_suse.py,test_data_source.py --max-line-length=100 .
DJANGO_DEV=1 pytest


Data import
-----------

Many data importers use GitHub APIs. For this, first set up value of the ``GH_TOKEN`` environment variable by running :

::
Some data importers use the GitHub APIs. For this, export the ``GH_TOKEN``
environment variable with::

export GH_TOKEN=yourgithubtoken


See `GitHub docs <https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token>`_ for instructions on how to obtain your GitHub token.
See `GitHub docs
<https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token>`_
for instructions on how to obtain your GitHub token.


To run all data importers use :
::
To run all data importers use::

DJANGO_DEV=1 python manage.py import --all

To list available importers use :
::
To list available importers use::

DJANGO_DEV=1 python manage.py import --list

To run specific importers :
::
To run specific importers::

DJANGO_DEV=1 python manage.py import rust npm


If you want to run the import periodically, you can use a systemd timer:
REST API access
---------------

Start the webserver::

DJANGO_DEV=1 python manage.py runserver


For full documentation about API endpoints use this URL::

http://127.0.0.1:8000/api/docs


::

Continuous periodic Data import
-------------------------------


If you want to run the import periodically, you can use a systemd timer::

$ cat ~/.config/systemd/user/vulnerablecode.service

Expand All @@ -228,37 +276,9 @@ If you want to run the import periodically, you can use a systemd timer:
[Install]
WantedBy=multi-user.target

Start it with

::

systemctl --user daemon-reload && systemctl --user start vulnerablecode.timer
Start this "timer" with::

API
---

Start the webserver

::

DJANGO_DEV=1 python manage.py runserver

In your browser access:

::

http://127.0.0.1:8000/api/docs
systemctl --user daemon-reload
systemctl --user start vulnerablecode.timer

For full documentation about API endpoints.

.. |Build Status| image:: https://travis-ci.org/nexB/vulnerablecode.svg?branch=develop
:target: https://travis-ci.org/nexB/vulnerablecode
.. |License| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:target: https://opensource.org/licenses/Apache-2.0
.. |Python 3.8| image:: https://img.shields.io/badge/python-3.8-blue.svg
:target: https://www.python.org/downloads/release/python-360/
.. |stability-wip| image:: https://img.shields.io/badge/stability-work_in_progress-lightgrey.svg
.. |Gitter chat| image:: https://badges.gitter.im/gitterHQ/gitter.png
:target: https://gitter.im/aboutcode-org/vulnerablecode
.. |PRs Welcome| image:: https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square
:target: http://makeapullrequest.com
Binary file added docs/Why-Is-There-No-Free-Software-Vulnerability-Database-v1.0.pdf
View file
Open in desktop
Binary file not shown.