Skip to content

Commit 58a253f

Browse files
tushar912tushar912
committed
Merge new changes and use load_yaml from helpers
used load_yaml from helpers Signed-off-by: Tushar912 <tushar.912u@gmail.com> fix typo in importer Signed-off-by: Tushar912 <tushar.912u@gmail.com>
2 parents ee6dd62 + bdf7c85 commit 58a253f

47 files changed

Lines changed: 1821 additions & 354 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/main.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: CI
2+
on:
3+
pull_request:
4+
types: [opened, synchronize]
5+
jobs:
6+
unit_tests:
7+
runs-on: ubuntu-latest
8+
9+
services:
10+
# Label used to access the service container
11+
postgres:
12+
image: postgres
13+
env:
14+
POSTGRES_PASSWORD: vulnerablecode
15+
POSTGRES_DB: vulnerablecode
16+
# Set health checks to wait until postgres has started
17+
options: >-
18+
--health-cmd pg_isready
19+
--health-interval 10s
20+
--health-timeout 5s
21+
--health-retries 5
22+
ports:
23+
# Maps tcp port 5432 on service container to the host
24+
- 5432:5432
25+
steps:
26+
- name: Check out repository code
27+
uses: actions/checkout@v2
28+
29+
- name: Set up Python 3.8
30+
uses: actions/setup-python@v2
31+
with:
32+
python-version: 3.8
33+
34+
- name: Install dependencies
35+
run: |
36+
sudo apt install python3-dev postgresql libpq-dev build-essential libxml2-dev libxslt1-dev
37+
python -m pip install --upgrade pip
38+
pip install -r requirements.txt
39+
40+
- name: Run tests
41+
run: pytest
42+
env:
43+
# The hostname, username used to communicate with the PostgreSQL service container
44+
POSTGRES_HOST: localhost
45+
VC_DB_USER: postgres
46+
POSTGRES_PORT: 5432
47+
DJANGO_DEV: 1
48+
GH_TOKEN: 1

‎.travis.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ notifications:
2828
- https://webhooks.gitter.im/e/b119fa557626081e1f36
2929
on_success: change # options: [always|never|change] default: always
3030
on_failure: always # options: [always|never|change] default: always
31-
on_start: never # options: [always|never|change] default: always
31+
on_start: never # options: [always|never|change] default: always

‎README.rst‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,11 @@ The How
4141

4242
VulnerableCode independently aggregates many software vulnerability data
4343
sources that can easily be recreated in a decentralized fashion. These
44-
data sources include security advisories published by distros, package
45-
managers, etc. Due to this, the data obtained is not generalized to apply
46-
for other ecosystems. This increases the accuracy as the same version of
47-
a package across different distros may or may not be vulnerable to some
48-
vulnerability.
44+
data sources (see complete list `here <./SOURCES.rst>`_) include security
45+
advisories published by distros, package managers, etc. Due to this, the
46+
data obtained is not generalized to apply for other ecosystems. This
47+
increases the accuracy as the same version of a package across different distros
48+
may or may not be vulnerable to some vulnerability.
4949

5050
The packages are identified using
5151
`PURL <https://github.com/package-url/purl-spec>`__ rather than CPEs.
@@ -143,14 +143,31 @@ Tests
143143
Data import
144144
-----------
145145

146+
Many data importers use GitHub APIs. For this, first set up value of the ``GH_TOKEN`` environment variable by running :
147+
148+
::
149+
150+
export GH_TOKEN=yourgithubtoken
151+
152+
153+
See `GitHub docs <https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token>`_ for instructions on how to obtain your GitHub token.
154+
155+
156+
To run all data importers use :
146157
::
147158

148159
DJANGO_DEV=1 python manage.py import --all
149160

150-
It is not required to run all the importers . To quickly get started with some data run :
151-
::
161+
To list available importers use :
162+
::
163+
164+
DJANGO_DEV=1 python manage.py import --list
165+
166+
To run specific importers :
167+
::
168+
169+
DJANGO_DEV=1 python manage.py import rust npm
152170

153-
DJANGO_DEV=1 python manage.py import rust ruby
154171

155172
If you want to run the import periodically, you can use a systemd timer:
156173

@@ -210,4 +227,4 @@ For full documentation about API endpoints.
210227
.. |Gitter chat| image:: https://badges.gitter.im/gitterHQ/gitter.png
211228
:target: https://gitter.im/aboutcode-org/vulnerablecode
212229
.. |PRs Welcome| image:: https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square
213-
:target: http://makeapullrequest.com
230+
:target: http://makeapullrequest.com

‎SOURCES.rst‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
2+
|Importer Name: | Data Source |Ecosystems Covered |
3+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
4+
|rust | https://github.com/RustSec/advisory-db |rust crates |
5+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
6+
|alpine | https://gitlab.alpinelinux.org/alpine/infra/alpine-secdb |alpine packages |
7+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
8+
|archlinux | https://security.archlinux.org/json |arch packages |
9+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
10+
|debian | https://security-tracker.debian.org/tracker/data/json |debian packages |
11+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
12+
|npm | https://github.com/nodejs/security-wg.git |npm packages |
13+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
14+
|ruby | https://github.com/rubysec/ruby-advisory-db.git |ruby gems |
15+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
16+
|ubuntu | https://people.canonical.com/~ubuntu-security/oval/ |ubuntu packages |
17+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
18+
|retiredotnet | https://github.com/RetireNet/Packages.git |.NET packages |
19+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
20+
|suse_backports | http://ftp.suse.com/pub/projects/security/yaml/ |SUSE packages |
21+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
22+
|debian_oval | https://www.debian.org/security/oval/ |debian packages |
23+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
24+
|redhat | https://access.redhat.com/hydra/rest/securitydata/cve.json |rpm packages |
25+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
26+
|nvd | https://nvd.nist.gov/vuln/data-feeds#JSON_FEED |none |
27+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
28+
|gentoo | https://anongit.gentoo.org/git/data/glsa.git |gentoo packages |
29+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
30+
|openssl | https://www.openssl.org/news/vulnerabilities.xml |openssl |
31+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
32+
|ubuntu_usn | https://usn.ubuntu.com/usn-db/database-all.json.bz2 |ubuntu packages |
33+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
34+
|github | https://api.github.com/graphql |maven, .NET, php-composer, pypi packages. ruby gems |
35+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
36+
|msr2019 | https://raw.githubusercontent.com/SAP/project-kb/master/MSR2019/dataset/vulas_db_msr2019_release.csv |maven packages |
37+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
38+
|apache_httpd | https://httpd.apache.org/security/vulnerabilities-httpd.xml |apache-httpd |
39+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
40+
|kaybee | https://github.com/SAP/project-kb.git |maven packages |
41+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
42+
|nginx | http://nginx.org/en/security_advisories.html |nginx |
43+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+
44+
|postgresql | https://www.postgresql.org/support/security/ |postgresql |
45+
+----------------+------------------------------------------------------------------------------------------------------+----------------------------------------------------+

‎docs/how-to-guides/add_new_importer.rst‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,13 @@ The Building Blocks A.K.A Prerequisites
3636
summary: str
3737
impacted_package_urls: Iterable[PackageURL]
3838
resolved_package_urls: Iterable[PackageURL] = dataclasses.field(default_factory=list)
39-
reference_urls: Sequence[str] = dataclasses.field(default_factory=list)
40-
reference_ids: Sequence[str] = dataclasses.field(default_factory=list)
39+
vuln_references: List[Reference] = dataclasses.field(default_factory=list)
4140
cve_id: Optional[str] = None
4241
42+
class Reference:
43+
44+
url: str = ''
45+
reference_id: str = ''
4346
4447
Steps to build an Importer
4548
--------------------------

‎requirements.txt‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,7 @@ pytest-dependency==0.4.0
4242
pytest-django==3.7.0
4343
pytest-mock==1.13.0
4444
python-dateutil==2.8.1
45-
pytoml==0.1.21
4645
pytz==2019.3
47-
PyYAML==5.3.1
48-
saneyaml==0.4
4946
schema==0.7.1
5047
six==1.13.0
5148
soupsieve==1.9.5
@@ -56,3 +53,5 @@ wcwidth==0.1.7
5653
whitenoise==5.0.1
5754
zipp==0.6.0
5855
requests==2.23.0
56+
toml==0.10.2
57+
PyYAML==5.3.1

‎vulnerabilities/helpers.py‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Copyright (c) nexB Inc. and others. All rights reserved.
2+
# http://nexb.com and https://github.com/nexB/vulnerablecode/
3+
# The VulnerableCode software is licensed under the Apache License version 2.0.
4+
# Data generated with VulnerableCode require an acknowledgment.
5+
#
6+
# You may not use this software except in compliance with the License.
7+
# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
8+
# Unless required by applicable law or agreed to in writing, software distributed
9+
# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
10+
# CONDITIONS OF ANY KIND, either express or implied. See the License for the
11+
# specific language governing permissions and limitations under the License.
12+
#
13+
# When you publish or redistribute any data created with VulnerableCode or any VulnerableCode
14+
# derivative work, you must accompany this data with the following acknowledgment:
15+
#
16+
# Generated with VulnerableCode and provided on an "AS IS" BASIS, WITHOUT WARRANTIES
17+
# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
18+
# VulnerableCode should be considered or used as legal advice. Consult an Attorney
19+
# for any legal advice.
20+
# VulnerableCode is a free software from nexB Inc. and others.
21+
# Visit https://github.com/nexB/vulnerablecode/ for support and download.
22+
23+
import json
24+
import re
25+
26+
import yaml
27+
import requests
28+
import toml
29+
30+
31+
def load_yaml(path):
32+
with open(path) as f:
33+
return yaml.safe_load(f)
34+
35+
36+
def load_json(path):
37+
with open(path) as f:
38+
return json.load(f)
39+
40+
41+
def load_toml(path):
42+
with open(path) as f:
43+
return toml.load(f)
44+
45+
46+
def create_etag(data_src, url, etag_key):
47+
"""
48+
Etags are like hashes of web responses. For a data source `data_src`,
49+
we maintain (url, etag) mappings in the DB. `create_etag` creates
50+
(`url`, etag) pair. If a (`url`, etag) already exists then the code
51+
skips processing the response further to avoid duplicate work.
52+
53+
`etag_key` is the name of header which contains the etag for the url.
54+
"""
55+
etag = requests.head(url).headers.get(etag_key)
56+
if not etag:
57+
return True
58+
59+
elif url in data_src.config.etags:
60+
if data_src.config.etags[url] == etag:
61+
return False
62+
63+
data_src.config.etags[url] = etag
64+
return True
65+
66+
67+
is_cve = re.compile(r"CVE-\d+-\d+", re.IGNORECASE).match

‎vulnerabilities/import_runner.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ class ImportRunner:
9090
Efficiency:
9191
- Bulk inserts should be used whenever possible.
9292
- Checking whether a record already exists should be kept to a minimum
93-
(the data source should know this instead).
93+
(the data source should know this instead).
9494
- All update and select operations must use indexed columns.
9595
"""
9696

‎vulnerabilities/importer_yielder.py‎

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@
179179
'data_source': 'GitHubAPIDataSource',
180180
'data_source_cfg': {
181181
'endpoint': 'https://api.github.com/graphql',
182-
'ecosystems': ['MAVEN', 'NUGET', 'COMPOSER']
182+
'ecosystems': ['MAVEN', 'NUGET', 'COMPOSER', 'PIP', 'RUBYGEMS']
183183
}
184184
},
185185
{
@@ -188,7 +188,7 @@
188188
'last_run': None,
189189
'data_source': 'ProjectKBMSRDataSource',
190190
'data_source_cfg': {
191-
'etag': {}
191+
'etags': {}
192192
}
193193
},
194194
{
@@ -216,7 +216,7 @@
216216
'last_run': None,
217217
'data_source': 'NginxDataSource',
218218
'data_source_cfg': {
219-
'etag': {}
219+
'etags': {}
220220
},
221221
},
222222
{
@@ -227,13 +227,22 @@
227227
'data_source_cfg': {},
228228
},
229229
{
230-
'name': 'elixir_security',
231-
'license': '',
232-
'last_run': None,
233-
'data_source': 'ElixirSecurityDataSource',
234-
'data_source_cfg': {
235-
'repository_url': 'https://github.com/dependabot/elixir-security-advisories'
230+
'name': 'elixir_security',
231+
'license': '',
232+
'last_run': None,
233+
'data_source': 'ElixirSecurityDataSource',
234+
'data_source_cfg': {
235+
'repository_url': 'https://github.com/dependabot/elixir-security-advisories'
236+
},
236237
},
238+
{
239+
'name': 'apache_tomcat',
240+
'license': '',
241+
'last_run': None,
242+
'data_source': 'ApacheTomcatDataSource',
243+
'data_source_cfg': {
244+
"etags": {}
245+
},
237246
},
238247

239248
]

‎vulnerabilities/importers/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,4 @@
4444
from vulnerabilities.importers.suse_backports import SUSEBackportsDataSource
4545
from vulnerabilities.importers.ubuntu import UbuntuDataSource
4646
from vulnerabilities.importers.ubuntu_usn import UbuntuUSNDataSource
47+
from vulnerabilities.importers.apache_tomcat import ApacheTomcatDataSource

0 commit comments

Comments
 (0)