Skip to content

Commit 0ffa256

Browse files
authored
Create a ScanCode.io pipeline to integrate Grimoire (#2202)
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
1 parent a76e0cc commit 0ffa256

13 files changed

Lines changed: 612 additions & 0 deletions

docker-compose.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ services:
4747
- /etc/scancodeio/:/etc/scancodeio/
4848
- workspace:/var/scancodeio/workspace/
4949
- static:/var/scancodeio/static/
50+
- healthycode_venv:/opt/healthycode
5051
restart: always
5152
depends_on:
5253
db:
@@ -67,6 +68,7 @@ services:
6768
- .env:/opt/scancodeio/.env
6869
- /etc/scancodeio/:/etc/scancodeio/
6970
- workspace:/var/scancodeio/workspace/
71+
- healthycode_venv:/opt/healthycode
7072
restart: always
7173
depends_on:
7274
db:
@@ -97,10 +99,19 @@ services:
9799
- workspace:/var/scancodeio/workspace/
98100
restart: always
99101

102+
healthycode:
103+
profiles:
104+
- healthycode
105+
image: ghcr.io/aboutcode-org/healthycode:v1.0.0
106+
volumes:
107+
- healthycode_venv:/opt/healthycode
108+
restart: "no"
109+
100110
volumes:
101111
db_data:
102112
redis_data:
103113
clamav_data:
104114
static:
105115
workspace:
106116
webroot:
117+
healthycode_venv:

docker.env

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@ SCANCODEIO_DB_HOST=db
77
SCANCODEIO_RQ_REDIS_HOST=redis
88
SCANCODEIO_ASYNC=True
99
SCANCODEIO_WORKSPACE_LOCATION=/var/scancodeio/workspace/
10+
GRIMOIRELAB_METRICS_EXECUTABLE=/opt/healthycode/.venv/bin/grimoirelab-metrics

docs/application-settings.rst

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,55 @@ Also provide the name and email that will be used to sign off on commits to Git
419419

420420
FEDERATEDCODE_GIT_SERVICE_EMAIL=insert_email_here
421421

422+
.. _scancodeio_settings_grimoirelab:
423+
424+
GrimoireLab
425+
^^^^^^^^^^^^^
426+
427+
GrimoireLab is a platform for software development analytics and insights.
428+
429+
Before launching the GrimoireLab service, ensure that your GrimoireLab
430+
instance is version 2.x and that OpenSearch is running.
431+
432+
See https://github.com/chaoss/grimoirelab#getting-started for more details
433+
on the installation.
434+
435+
To configure your local environment, set the following in your ``.env`` file::
436+
437+
GRIMOIRELAB_METRICS_EXECUTABLE="./healthycode/venv/bin/grimoirelab-metrics"
438+
GRIMOIRELAB_URL="http://example.com:8000"
439+
GRIMOIRELAB_USERNAME="username"
440+
GRIMOIRELAB_PASSWORD="password"
441+
GRIMOIRELAB_OPENSEARCH_URL="https://example.com:9200"
442+
GRIMOIRELAB_OPENSEARCH_INDEX="events"
443+
GRIMOIRELAB_OPENSEARCH_USERNAME="opensearch_username"
444+
GRIMOIRELAB_OPENSEARCH_PASSWORD="opensearch_password"
445+
GRIMOIRELAB_ECOSYSTEM="npmlargerecosystem"
446+
GRIMOIRELAB_PROJECT="NPMMostUsed"
447+
448+
The following variables configure the connection to the GrimoireLab instance:
449+
450+
- ``GRIMOIRELAB_METRICS_EXECUTABLE``: Path to the grimoirelab-metrics executable, See
451+
here for more details on how to set up
452+
the grimoirelab-metrics executable: https://github.com/aboutcode-org/healthycode#installation
453+
454+
- ``GRIMOIRELAB_URL``: URL of the GrimoireLab instance.
455+
- ``GRIMOIRELAB_USERNAME``: Username used to authenticate with GrimoireLab.
456+
- ``GRIMOIRELAB_PASSWORD``: Password used to authenticate with GrimoireLab.
457+
458+
The following variables configure the connection to OpenSearch:
459+
460+
- ``GRIMOIRELAB_OPENSEARCH_URL``: URL of the OpenSearch instance.
461+
- ``GRIMOIRELAB_OPENSEARCH_INDEX``: OpenSearch index containing the GrimoireLab events.
462+
- ``GRIMOIRELAB_OPENSEARCH_USERNAME``: Username used to authenticate with OpenSearch.
463+
- ``GRIMOIRELAB_OPENSEARCH_PASSWORD``: Password used to authenticate with OpenSearch.
464+
465+
The following variables specify the GrimoireLab project where the project
466+
data is stored:
467+
468+
- ``GRIMOIRELAB_ECOSYSTEM``: Ecosystem containing the project data.
469+
- ``GRIMOIRELAB_PROJECT``: Name of the GrimoireLab project containing the project data.
470+
422471

423472
.. _scancodeio_settings_fetch_authentication:
424473

124 KB
Loading
176 KB
Loading

docs/index.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ Learn via practical step-by-step guides.
4848
- :ref:`tutorial_web_ui_symbol_and_string_collection`
4949
- :ref:`tutorial_cli_end_to_end_scanning_to_dejacode`
5050
- :ref:`tutorial_analyze_symbols_reachability`
51+
- :ref:`tutorial_web_ui_scan_repo_health`
5152

5253
.. rst-class:: column column2 bottom-left
5354

@@ -118,6 +119,7 @@ Indices and tables
118119
tutorial_web_ui_symbol_and_string_collection
119120
tutorial_cli_end_to_end_scanning_to_dejacode
120121
tutorial_analyze_symbols_reachability
122+
tutorial_web_ui_scan_repo_health
121123
scanpipe-concepts
122124
built-in-pipelines
123125
custom-pipelines
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
.. _tutorial_web_ui_scan_repo_health:
2+
3+
Scan repo health (Web UI)
4+
=========================
5+
6+
This tutorial guides you through scanning the health
7+
of a git repository using the ScanCode.io Web UI.
8+
9+
.. note::
10+
This tutorial assumes you have a recent version of ScanCode.io installed
11+
locally on your machine and **running with Docker**.
12+
If you do not have it installed, see our :ref:`installation` guide for instructions.
13+
14+
Requirements
15+
------------
16+
17+
Before you follow the instructions in this tutorial, you need to:
18+
19+
- Install **ScanCode.io** locally.
20+
- Access to the web application from your preferred browser on http://localhost/ or
21+
http://localhost:8001/ if you run on a local development setup.
22+
- Make sure the following GrimoireLab
23+
service is correctly configured: :ref:`scancodeio_settings_grimoirelab`.
24+
25+
Instructions
26+
------------
27+
28+
1. From the homepage, click the ``New Project`` button.
29+
2. Enter ``npm-example-repo-health`` as the project **Name** for example.
30+
3. Add a valid git repository url into the **Download URL** field:
31+
https://github.com/aboutcode-org/npm-project-example
32+
4. Select ``scan_repo_health`` from the **Pipeline** dropdown.
33+
5. Check the **Execute pipeline now** box to run the pipeline immediately upon creation.
34+
6. Click **Create**.
35+
36+
.. image:: images/tutorial-web-ui-project-form-repo-health.png
37+
:alt: Create project form for repo health scan
38+
39+
7. Once the pipeline execution is complete, you can download the generated output result in
40+
**JSON** format (for example, ``metrics-2026-09-10-12-02-38.json``)
41+
or directly view the data from the project's result view.
42+
43+
.. image:: images/tutorial-web-ui-project-form-repo-health-result.png
44+
:alt: Scan repo health output pipeline

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,7 @@ scan_codebase = "scanpipe.pipelines.scan_codebase:ScanCodebase"
176176
scan_for_virus = "scanpipe.pipelines.scan_for_virus:ScanForVirus"
177177
scan_maven_package = "scanpipe.pipelines.scan_maven_package:ScanMavenPackage"
178178
scan_single_package = "scanpipe.pipelines.scan_single_package:ScanSinglePackage"
179+
scan_repo_health = "scanpipe.pipelines.scan_repo_health:ScanRepoHealth"
179180

180181
[tool.setuptools.packages.find]
181182
where = ["."]
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# http://nexb.com and https://github.com/aboutcode-org/scancode.io
4+
# The ScanCode.io software is licensed under the Apache License version 2.0.
5+
# Data generated with ScanCode.io is provided as-is without warranties.
6+
# ScanCode is a trademark of nexB Inc.
7+
#
8+
# You may not use this software except in compliance with the License.
9+
# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
10+
# Unless required by applicable law or agreed to in writing, software distributed
11+
# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
12+
# CONDITIONS OF ANY KIND, either express or implied. See the License for the
13+
# specific language governing permissions and limitations under the License.
14+
#
15+
# Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES
16+
# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
17+
# ScanCode.io should be considered or used as legal advice. Consult an Attorney
18+
# for any legal advice.
19+
#
20+
# ScanCode.io is a free software code scanning tool from nexB Inc. and others.
21+
# Visit https://github.com/aboutcode-org/scancode.io for support and download.
22+
23+
from scanpipe.pipelines import Pipeline
24+
from scanpipe.pipes import repo_health
25+
from scanpipe.pipes.repo_health import GRIMOIRELAB_ECOSYSTEM
26+
from scanpipe.pipes.repo_health import GRIMOIRELAB_METRICS_EXECUTABLE
27+
from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_INDEX
28+
from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_PASSWORD
29+
from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_URL
30+
from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_USERNAME
31+
from scanpipe.pipes.repo_health import GRIMOIRELAB_PASSWORD
32+
from scanpipe.pipes.repo_health import GRIMOIRELAB_PROJECT
33+
from scanpipe.pipes.repo_health import GRIMOIRELAB_URL
34+
from scanpipe.pipes.repo_health import GRIMOIRELAB_USERNAME
35+
36+
37+
class ScanRepoHealth(Pipeline):
38+
"""Run a Repo Health scan to extract repository metrics and health score."""
39+
40+
results_url = "/project/{slug}/resources/?extra_data=grimoire_data"
41+
download_inputs = False
42+
43+
@classmethod
44+
def steps(cls):
45+
return (
46+
cls.get_repo_url_input,
47+
cls.collect_and_store_grimoire_metric,
48+
cls.format_metrics_output,
49+
)
50+
51+
@classmethod
52+
def get_availability(cls):
53+
if not (
54+
GRIMOIRELAB_METRICS_EXECUTABLE
55+
and GRIMOIRELAB_OPENSEARCH_INDEX
56+
and GRIMOIRELAB_OPENSEARCH_PASSWORD
57+
and GRIMOIRELAB_OPENSEARCH_URL
58+
and GRIMOIRELAB_OPENSEARCH_USERNAME
59+
and GRIMOIRELAB_PASSWORD
60+
and GRIMOIRELAB_URL
61+
and GRIMOIRELAB_USERNAME
62+
and GRIMOIRELAB_ECOSYSTEM
63+
and GRIMOIRELAB_PROJECT
64+
):
65+
return "Grimoirelab is not configured."
66+
67+
def get_repo_url_input(self):
68+
"""Validate and extract the repository URL from the project's input sources"""
69+
self.repo_url = repo_health.get_repo_url_input(project=self.project)
70+
71+
def collect_and_store_grimoire_metric(self):
72+
"""
73+
Run the grimoirelab-metrics command against the input source.
74+
Save the generated metrics JSON to the project output directory.
75+
"""
76+
self.metrics_output_path = repo_health.collect_and_store_grimoire_metric(
77+
project=self.project, repo_url=self.repo_url, logger=self.log
78+
)
79+
80+
def format_metrics_output(self):
81+
"""
82+
Format the GrimoireLab metrics output by extracting the repository URL,
83+
score, and metrics from the generated JSON and overwriting it with a
84+
simplified structure, and updating the project's extra data.
85+
"""
86+
repo_health.format_metrics_output(
87+
project=self.project, metrics_output_path=self.metrics_output_path
88+
)

0 commit comments

Comments
 (0)