Skip to content

Commit cc26ed4

Browse files
pombredannepombredanne
committed
Merge remote add-nix-support branch #275
From https://github.com/ngi-nix/vulnerablecode/ Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
2 parents 3b903ae + 3f732cd commit cc26ed4

8 files changed

Lines changed: 367 additions & 0 deletions

File tree

‎.github/workflows/test-import-using-nix.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
on:
2+
workflow_dispatch: # allow manual execution
3+
push:
4+
pull_request:
5+
schedule:
6+
# run on the 3rd each month at 10:00am
7+
- cron: '0 10 3 * *'
8+
9+
jobs:
10+
nix-check-and-import:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v2
15+
with:
16+
fetch-depth: 0
17+
- uses: cachix/install-nix-action@v11
18+
env:
19+
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
20+
with:
21+
nix_path: nixpkgs=https://github.com/NixOS/nixpkgs/archive/20.09.tar.gz
22+
install_url: https://github.com/numtide/nix-flakes-installer/releases/download/nix-3.0pre20201007_5257a25/install
23+
extra_nix_config: |
24+
experimental-features = nix-command flakes
25+
- name: run checks & test import
26+
run: |
27+
cd etc/nix
28+
nix --print-build-logs flake check
29+
./test-import-using-nix.sh alpine

‎README.rst‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,41 @@ When not running in development mode, an environment variable named
131131
is to use the code Django includes for this purpose:
132132
``SECRET_KEY=$(python -c "from django.core.management import utils; print(utils.get_random_secret_key())")``.
133133

134+
Using Nix
135+
~~~~~~~~~
136+
137+
You can install VulnerableCode with `Nix <https://nixos.org/download.html>`__ (`Flake <https://nixos.wiki/wiki/Flakes>`__ support is needed).
138+
139+
::
140+
141+
cd etc/nix
142+
nix --print-build-logs flake check # build & run tests
143+
144+
There are several options to use the Nix version
145+
146+
::
147+
148+
# Enter an interactive environment with all dependencies set up.
149+
cd etc/nix
150+
nix develop
151+
> ../../manage.py ... # invoke the local checkout
152+
> vulnerablecode-manage.py ... # invoke manage.py as installed in the nix store
153+
154+
# Test the import prodecure using the Nix version.
155+
etc/nix/test-import-using-nix.sh --all # import everything
156+
# Test the import using the local checkout.
157+
INSTALL_DIR=. etc/nix/test-import-using-nix.sh ruby # import ruby only
158+
159+
160+
**Keeping the Nix setup in sync**
161+
162+
The Nix installation uses `mach-nix <https://github.com/DavHau/mach-nix>`__ to handle Python dependencies because some dependencies are currently not available as Nix packages.
163+
All Python dependencies are automatically fetched from ``./requirements.txt``.
164+
If the ``mach-nix``-based installation fails, you might need to update ``mach-nix`` itself and the `pypi-deps-db <https://github.com/DavHau/pypi-deps-db>`_ version in use (see ``etc/nix/flake.nix:inputs.machnix`` and ``machnixFor.pypiDataRev``).
165+
166+
Non-Python dependencies are curated in ``etc/nix/flake.nix:vulnerablecode.propagatedBuildInputs``.
167+
168+
134169
Tests
135170
-----
136171

‎etc/nix/default.nix‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
(import (fetchTarball
2+
"https://github.com/edolstra/flake-compat/archive/master.tar.gz") {
3+
src = ./.;
4+
}).defaultNix

‎etc/nix/flake.lock‎

Lines changed: 96 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎etc/nix/flake.nix‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
{
2+
description =
3+
"Vulnerablecode - A free and open vulnerabilities database and the packages they impact.";
4+
5+
inputs.nixpkgs = {
6+
type = "github";
7+
owner = "NixOS";
8+
repo = "nixpkgs";
9+
ref = "20.09";
10+
};
11+
12+
inputs.machnix = {
13+
type = "github";
14+
owner = "DavHau";
15+
repo = "mach-nix";
16+
ref = "3.1.1";
17+
};
18+
19+
outputs = { self, nixpkgs, machnix }:
20+
let
21+
22+
vulnerablecode-src = ./../..;
23+
24+
# Extract version from setup.py.
25+
version = builtins.head (builtins.match ''.*version=["']?([^"',]+).*''
26+
(builtins.readFile (vulnerablecode-src + "/setup.py")));
27+
28+
# Common shell code.
29+
libSh = ./lib.sh;
30+
31+
# System types to support.
32+
supportedSystems = [ "x86_64-linux" ];
33+
34+
# Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
35+
forAllSystems = f:
36+
nixpkgs.lib.genAttrs supportedSystems (system: f system);
37+
38+
# Nixpkgs instantiated for supported system types.
39+
nixpkgsFor = forAllSystems (system:
40+
import nixpkgs {
41+
inherit system;
42+
overlays = [ self.overlay ];
43+
});
44+
45+
# mach-nix instantiated for supported system types.
46+
machnixFor = forAllSystems (system:
47+
import machnix {
48+
pkgs = (nixpkgsFor.${system}).pkgs;
49+
python = "python38";
50+
51+
# Pin pypi repo to a specific commit which includes all necessary
52+
# Python deps. The default version is updated with every mach-nix
53+
# release might be be sufficient for newer releases.
54+
# The corresponding sha256 hash can be obtained with:
55+
# $ nix-prefetch-url --unpack https://github.com/DavHau/pypi-deps-db/tarball/<pypiDataRev>
56+
pypiDataRev = "c86b4490a7d838bd54a2d82730455e96c6e4eb14";
57+
pypiDataSha256 =
58+
"0al490gi0qda1nkb9289z2msgpc633rv5hn3w5qihkl1rh88dmjd";
59+
});
60+
61+
in {
62+
63+
# A Nixpkgs overlay.
64+
overlay = final: prev:
65+
with final.pkgs; {
66+
67+
pythonEnv = machnixFor.${system}.mkPython {
68+
requirements =
69+
builtins.readFile (vulnerablecode-src + "/requirements.txt");
70+
};
71+
72+
vulnerablecode = stdenv.mkDerivation {
73+
inherit version;
74+
name = "vulnerablecode-${version}";
75+
src = vulnerablecode-src;
76+
dontConfigure = true; # do not use ./configure
77+
propagatedBuildInputs = [ pythonEnv postgresql ];
78+
79+
postPatch = ''
80+
# Make sure the pycodestyle binary in $PATH is used.
81+
substituteInPlace vulnerabilities/tests/test_basics.py \
82+
--replace 'join(bin_dir, "pycodestyle")' '"pycodestyle"'
83+
'';
84+
85+
installPhase = ''
86+
cp -r . $out
87+
'';
88+
};
89+
90+
};
91+
92+
# Provide a nix-shell env to work with vulnerablecode.
93+
devShell = forAllSystems (system:
94+
with nixpkgsFor.${system};
95+
mkShell {
96+
# will be available as env var in `nix develop` / `nix-shell`.
97+
VULNERABLECODE_INSTALL_DIR = vulnerablecode;
98+
buildInputs = [ vulnerablecode ];
99+
shellHook = ''
100+
alias vulnerablecode-manage.py=${vulnerablecode}/manage.py
101+
'';
102+
});
103+
104+
# Provide some packages for selected system types.
105+
packages = forAllSystems
106+
(system: { inherit (nixpkgsFor.${system}) vulnerablecode; });
107+
108+
# The default package for 'nix build'.
109+
defaultPackage =
110+
forAllSystems (system: self.packages.${system}.vulnerablecode);
111+
112+
# Tests run by 'nix flake check' and by Hydra.
113+
checks = forAllSystems (system: {
114+
inherit (self.packages.${system}) vulnerablecode;
115+
116+
vulnerablecode-test = with nixpkgsFor.${system};
117+
stdenv.mkDerivation {
118+
name = "${vulnerablecode.name}-test";
119+
120+
buildInputs = [ wget vulnerablecode ];
121+
122+
# Used by pygit2.
123+
# See https://github.com/NixOS/nixpkgs/pull/72544#issuecomment-582674047.
124+
SSL_CERT_FILE = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
125+
126+
unpackPhase = "true";
127+
128+
buildPhase = ''
129+
source ${libSh}
130+
initPostgres $(pwd)
131+
export DJANGO_DEV=1
132+
${vulnerablecode}/manage.py migrate
133+
'';
134+
135+
doCheck = true;
136+
checkPhase = ''
137+
# Run pytest on the installed version. A running postgres
138+
# database server is needed.
139+
(cd ${vulnerablecode} && pytest)
140+
141+
# Launch the webserver and call the API.
142+
${vulnerablecode}/manage.py runserver &
143+
sleep 2
144+
wget http://127.0.0.1:8000/api/
145+
kill %1 # kill background task (i.e. webserver)
146+
'';
147+
148+
installPhase =
149+
"mkdir -p $out"; # make this derivation return success
150+
};
151+
});
152+
};
153+
}

‎etc/nix/lib.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
3+
# Setup postgres; see the README for the latest instructions.
4+
#
5+
# $RUNDIR is used to prevent postgres from accessings its default run dir at
6+
# /run/postgresql. See
7+
# https://github.com/NixOS/nixpkgs/issues/83770#issuecomment-607992517
8+
function initPostgres() {
9+
ROOTDIR=$1
10+
DATADIR=$ROOTDIR/pgdata
11+
RUNDIR=$ROOTDIR/run
12+
ENCODING="UTF-8"
13+
mkdir -p "$RUNDIR"
14+
initdb -D "$DATADIR" -E $ENCODING
15+
pg_ctl -D "$DATADIR" -o "-k $RUNDIR" -l "$DATADIR/logfile" start
16+
createuser --host "$RUNDIR" --no-createrole --no-superuser --login --inherit --createdb vulnerablecode
17+
createdb --host "$RUNDIR" -E $ENCODING --owner=vulnerablecode --user=vulnerablecode --port=5432 vulnerablecode
18+
}

‎etc/nix/shell.nix‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
(import (fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) {
2+
src = ./.;
3+
}).shellNix

‎etc/nix/test-import-using-nix.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/usr/bin/env nix-shell
2+
#!nix-shell -i bash
3+
4+
# Populate a test database using either the Nix installation or the local
5+
# checkout.
6+
7+
set -e
8+
9+
THIS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
10+
DEFAULT_INSTALL_DIR=$VULNERABLECODE_INSTALL_DIR # in the Nix store, see flake.nix
11+
INSTALL_DIR=${INSTALL_DIR:-$DEFAULT_INSTALL_DIR}
12+
ARGS=$(if [ $# -eq 0 ]; then echo "--all"; else echo "$@"; fi)
13+
export DJANGO_DEV=${DJANGO_DEV:-1}
14+
TEMPDIR=$(mktemp -d -p "$THIS_DIR")
15+
export TEMPDIR
16+
17+
source "$THIS_DIR/lib.sh"
18+
19+
cleanup() {
20+
pg_ctl -D "$DATADIR" stop
21+
rm -rf "$TEMPDIR"
22+
}
23+
24+
trap cleanup EXIT
25+
26+
initPostgres "$TEMPDIR"
27+
28+
"$INSTALL_DIR/manage.py" migrate
29+
"$INSTALL_DIR/manage.py" import $ARGS

0 commit comments

Comments
 (0)