Create a ScanCode.io pipeline to integrate Grimoire

docker-compose.yml

@@ -47,6 +47,7 @@ services:
       - /etc/scancodeio/:/etc/scancodeio/
       - workspace:/var/scancodeio/workspace/
       - static:/var/scancodeio/static/
+      - healthycode_venv:/opt/healthycode
     restart: always
     depends_on:
       db:
@@ -67,6 +68,7 @@ services:
       - .env:/opt/scancodeio/.env
       - /etc/scancodeio/:/etc/scancodeio/
       - workspace:/var/scancodeio/workspace/
+      - healthycode_venv:/opt/healthycode
     restart: always
     depends_on:
       db:
@@ -97,10 +99,19 @@ services:
       - workspace:/var/scancodeio/workspace/
     restart: always
 
+  healthycode:
+    profiles:
+      - healthycode
+    image: ghcr.io/aboutcode-org/healthycode:v1.0.0
+    volumes:
+      - healthycode_venv:/opt/healthycode
+    restart: "no"
+
 volumes:
   db_data:
   redis_data:
   clamav_data:
   static:
   workspace:
   webroot:
+  healthycode_venv:

docker.env

@@ -7,3 +7,4 @@ SCANCODEIO_DB_HOST=db
 SCANCODEIO_RQ_REDIS_HOST=redis
 SCANCODEIO_ASYNC=True
 SCANCODEIO_WORKSPACE_LOCATION=/var/scancodeio/workspace/
+GRIMOIRELAB_METRICS_EXECUTABLE=/opt/healthycode/.venv/bin/grimoirelab-metrics

docs/application-settings.rst

@@ -419,6 +419,55 @@ Also provide the name and email that will be used to sign off on commits to Git
 
     FEDERATEDCODE_GIT_SERVICE_EMAIL=insert_email_here
 
+.. _scancodeio_settings_grimoirelab:
+
+GrimoireLab
+^^^^^^^^^^^^^
+
+GrimoireLab is a platform for software development analytics and insights.
+
+Before launching the GrimoireLab service, ensure that your GrimoireLab
+instance is version 2.x and that OpenSearch is running.
+
+See https://github.com/chaoss/grimoirelab#getting-started for more details
+on the installation.
+
+To configure your local environment, set the following in your ``.env`` file::
+
+    GRIMOIRELAB_METRICS_EXECUTABLE="./healthycode/venv/bin/grimoirelab-metrics"
+    GRIMOIRELAB_URL="http://example.com:8000"
+    GRIMOIRELAB_USERNAME="username"
+    GRIMOIRELAB_PASSWORD="password"
+    GRIMOIRELAB_OPENSEARCH_URL="https://example.com:9200"
+    GRIMOIRELAB_OPENSEARCH_INDEX="events"
+    GRIMOIRELAB_OPENSEARCH_USERNAME="opensearch_username"
+    GRIMOIRELAB_OPENSEARCH_PASSWORD="opensearch_password"
+    GRIMOIRELAB_ECOSYSTEM="npmlargerecosystem"
+    GRIMOIRELAB_PROJECT="NPMMostUsed"
+
+The following variables configure the connection to the GrimoireLab instance:
+
+- ``GRIMOIRELAB_METRICS_EXECUTABLE``: Path to the grimoirelab-metrics executable, See
+  here for more details on how to set up
+  the grimoirelab-metrics executable: https://github.com/aboutcode-org/healthycode#installation
+
+- ``GRIMOIRELAB_URL``: URL of the GrimoireLab instance.
+- ``GRIMOIRELAB_USERNAME``: Username used to authenticate with GrimoireLab.
+- ``GRIMOIRELAB_PASSWORD``: Password used to authenticate with GrimoireLab.
+
+The following variables configure the connection to OpenSearch:
+
+- ``GRIMOIRELAB_OPENSEARCH_URL``: URL of the OpenSearch instance.
+- ``GRIMOIRELAB_OPENSEARCH_INDEX``: OpenSearch index containing the GrimoireLab events.
+- ``GRIMOIRELAB_OPENSEARCH_USERNAME``: Username used to authenticate with OpenSearch.
+- ``GRIMOIRELAB_OPENSEARCH_PASSWORD``: Password used to authenticate with OpenSearch.
+
+The following variables specify the GrimoireLab project where the project
+data is stored:
+
+- ``GRIMOIRELAB_ECOSYSTEM``: Ecosystem containing the project data.
+- ``GRIMOIRELAB_PROJECT``: Name of the GrimoireLab project containing the project data.
+
 
 .. _scancodeio_settings_fetch_authentication:
 

docs/index.rst

@@ -48,6 +48,7 @@ Learn via practical step-by-step guides.
 - :ref:`tutorial_web_ui_symbol_and_string_collection`
 - :ref:`tutorial_cli_end_to_end_scanning_to_dejacode`
 - :ref:`tutorial_analyze_symbols_reachability`
+- :ref:`tutorial_web_ui_scan_repo_health`
 
 .. rst-class:: column column2 bottom-left
 
@@ -118,6 +119,7 @@ Indices and tables
     tutorial_web_ui_symbol_and_string_collection
     tutorial_cli_end_to_end_scanning_to_dejacode
     tutorial_analyze_symbols_reachability
+    tutorial_web_ui_scan_repo_health
     scanpipe-concepts
     built-in-pipelines
     custom-pipelines

docs/tutorial_web_ui_scan_repo_health.rst

@@ -0,0 +1,44 @@
+.. _tutorial_web_ui_scan_repo_health:
+
+Scan repo health (Web UI)
+=========================
+
+This tutorial guides you through scanning the health
+of a git repository using the ScanCode.io Web UI.
+
+.. note::
+    This tutorial assumes you have a recent version of ScanCode.io installed
+    locally on your machine and **running with Docker**.
+    If you do not have it installed, see our :ref:`installation` guide for instructions.
+
+Requirements
+------------
+
+Before you follow the instructions in this tutorial, you need to:
+
+- Install **ScanCode.io** locally.
+- Access to the web application from your preferred browser on http://localhost/ or
+  http://localhost:8001/ if you run on a local development setup.
+- Make sure the following GrimoireLab
+  service is correctly configured: :ref:`scancodeio_settings_grimoirelab`.
+
+Instructions
+------------
+
+1. From the homepage, click the ``New Project`` button.
+2. Enter ``npm-example-repo-health`` as the project **Name** for example.
+3. Add a valid git repository url into the **Download URL** field:
+   https://github.com/aboutcode-org/npm-project-example
+4. Select ``scan_repo_health`` from the **Pipeline** dropdown.
+5. Check the **Execute pipeline now** box to run the pipeline immediately upon creation.
+6. Click **Create**.
+
+.. image:: images/tutorial-web-ui-project-form-repo-health.png
+   :alt: Create project form for repo health scan
+
+7. Once the pipeline execution is complete, you can download the generated output result in
+   **JSON** format (for example, ``metrics-2026-09-10-12-02-38.json``)
+   or directly view the data from the project's result view.
+
+.. image:: images/tutorial-web-ui-project-form-repo-health-result.png
+   :alt: Scan repo health output pipeline

pyproject.toml

@@ -176,6 +176,7 @@ scan_codebase = "scanpipe.pipelines.scan_codebase:ScanCodebase"
 scan_for_virus = "scanpipe.pipelines.scan_for_virus:ScanForVirus"
 scan_maven_package = "scanpipe.pipelines.scan_maven_package:ScanMavenPackage"
 scan_single_package = "scanpipe.pipelines.scan_single_package:ScanSinglePackage"
+scan_repo_health = "scanpipe.pipelines.scan_repo_health:ScanRepoHealth"
 
 [tool.setuptools.packages.find]
 where = ["."]

scanpipe/pipelines/scan_repo_health.py

@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# http://nexb.com and https://github.com/aboutcode-org/scancode.io
+# The ScanCode.io software is licensed under the Apache License version 2.0.
+# Data generated with ScanCode.io is provided as-is without warranties.
+# ScanCode is a trademark of nexB Inc.
+#
+# You may not use this software except in compliance with the License.
+# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software distributed
+# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+# CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+#
+# Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES
+# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
+# ScanCode.io should be considered or used as legal advice. Consult an Attorney
+# for any legal advice.
+#
+# ScanCode.io is a free software code scanning tool from nexB Inc. and others.
+# Visit https://github.com/aboutcode-org/scancode.io for support and download.
+
+from scanpipe.pipelines import Pipeline
+from scanpipe.pipes import repo_health
+from scanpipe.pipes.repo_health import GRIMOIRELAB_ECOSYSTEM
+from scanpipe.pipes.repo_health import GRIMOIRELAB_METRICS_EXECUTABLE
+from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_INDEX
+from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_PASSWORD
+from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_URL
+from scanpipe.pipes.repo_health import GRIMOIRELAB_OPENSEARCH_USERNAME
+from scanpipe.pipes.repo_health import GRIMOIRELAB_PASSWORD
+from scanpipe.pipes.repo_health import GRIMOIRELAB_PROJECT
+from scanpipe.pipes.repo_health import GRIMOIRELAB_URL
+from scanpipe.pipes.repo_health import GRIMOIRELAB_USERNAME
+
+
+class ScanRepoHealth(Pipeline):
+    """Run a Repo Health scan to extract repository metrics and health score."""
+
+    results_url = "/project/{slug}/resources/?extra_data=grimoire_data"
+    download_inputs = False
+
+    @classmethod
+    def steps(cls):
+        return (
+            cls.get_repo_url_input,
+            cls.collect_and_store_grimoire_metric,
+            cls.format_metrics_output,
+        )
+
+    @classmethod
+    def get_availability(cls):
+        if not (
+            GRIMOIRELAB_METRICS_EXECUTABLE
+            and GRIMOIRELAB_OPENSEARCH_INDEX
+            and GRIMOIRELAB_OPENSEARCH_PASSWORD
+            and GRIMOIRELAB_OPENSEARCH_URL
+            and GRIMOIRELAB_OPENSEARCH_USERNAME
+            and GRIMOIRELAB_PASSWORD
+            and GRIMOIRELAB_URL
+            and GRIMOIRELAB_USERNAME
+            and GRIMOIRELAB_ECOSYSTEM
+            and GRIMOIRELAB_PROJECT
+        ):
+            return "Grimoirelab is not configured."
+
+    def get_repo_url_input(self):
+        """Validate and extract the repository URL from the project's input sources"""
+        self.repo_url = repo_health.get_repo_url_input(project=self.project)
+
+    def collect_and_store_grimoire_metric(self):
+        """
+        Run the grimoirelab-metrics command against the input source.
+        Save the generated metrics JSON to the project output directory.
+        """
+        self.metrics_output_path = repo_health.collect_and_store_grimoire_metric(
+            project=self.project, repo_url=self.repo_url, logger=self.log
+        )
+
+    def format_metrics_output(self):
+        """
+        Format the GrimoireLab metrics output by extracting the repository URL,
+        score, and metrics from the generated JSON and overwriting it with a
+        simplified structure, and updating the project's extra data.
+        """
+        repo_health.format_metrics_output(
+            project=self.project, metrics_output_path=self.metrics_output_path
+        )