CRAVEX-integration: Integration with SCA Tools: ORT

[pombredanne]

Integrate and import and export ORT results.

There are a couple things to consider:

1. Formats to use and parse out of ORT for import into ScanCode
2. Formats to create and generate from ScanCode for export into ORT

For 2. there are various input: SBOM, analyzer and package list:

@tsteenbe told me:

This is the ORT package list format for importing things ORT doesn't support see https://github.com/oss-review-toolkit/ort/blob/main/cli-helper/src/funTest/resources/package-list.yml

[heliocastro]

The reference is SADL base image to start:
https://hub.docker.com/u/osadl

[tsteenbe]

Summarizing below in-person conversations, calls and Slack messages from the past weeks.

As @pombredanne We have two objectives in this issue

1. How can we use ORT results as input for ScanCode.io Referred to as ORT => ScanCode.io below.
2. How can we use ScanCode.io results as input for ORT? Referred to as ScanCode.io => ORT below.

1. ORT => ScanCode.io

ScanCode.io supports as input an SBOM via its built-in Load SBOM pipeline. ORT support generating SBOMs via its reporter tools. A logical integration for using ORT results in ScanCode.io would to generate various CycloneDx v1.5/v1.6 and SPDX v2.2/v2.3 with ORT and then "load" these into ScanCode.io

Tasks

• Copy ORT reporter test assets related to SBOM generation into ScanCode.io code base and write a GitHub action workflow within ScanCode.io code repository that on weekly basis tests importing generated SBOM into the latest version of ScanCode.io via assert fails/succeeds on number of packages, licenses and vulnerabilities ScanCode.io found versus known numbers outputted by ORT. Related SBOM test assets in ORT: CycloneDX reporter | SPDX reporter
• Create GitHub action workflow within ScanCode.io code repository that uses GitHub Action for ORT to convert ORTHW-shell example ORT scan-result JSON file for mime types 2.1.26 into CycloneDX v1.5, v1.6 and SPDX v2.2, v2.3 SBOMs, then import them into into the latest version of ScanCode.io via assert fails/succeeds on number of packages, licenses and vulnerabilities ScanCode.io found versus known numbers outputted by ORT
• Create GitHub action workflow that on a weekly basis imports ORT's synthetic tests for various package managers then use GitHub Action for ORT to generate CycloneDX v1.5, v1.6 and SPDX v2.2, v2.3 SBOMs, importing generated SBOMs into the latest version of ScanCode.io and then asserts (fails/succeeds) on number of packages, licenses and vulnerabilities found by ScanCode.io versus known numbers outputted by ORT.

2. ScanCode.io => ORT

ScanCode.io inputs are either
A) File uploads
B) Download URLs e.g. https://domain.com/archive.zip, docker://postgres:13 or a package UR such as pkg:npm/foobar@12.3.1 or pkg:github/package-url/purl-spec@244fd47e07d1004f0aed9c

ORT was designed for package manager / build tools analyzes and expects as import a version controlled source code repository either Git, Mercurial or Subversion. We believe ScanCode.io to ORT conversion for Package URLs such as pkg:github/package-url/purl-spec@244fd47e07d1004f0aed9c would be straightforward as this is basically code repository coordinate (URL + code revision). What do for pkg:npm/foobar@12.3.1 - make a temp repository and then add this package as a dependency?

However, ORT was never designed compared to ScanCode.io to handle as input file archives or docker images. ORT core function is to resolve every found package back to its source code which is not something one can easily do for Docker images due to lack of missing meta/build data. This is however the use case we know several ORT community members would like to use ScanCode.io for.

Although we believe the best approach maybe to integrate ScanCode.io as ORT Scanner plugin this would require a significant amount of code changes within ORT which is beyond the scope of this issue. We therefore choice to make a first step - data from any package manager not supported by ORT can by feed in it via either package-list.yml format using create-analyzer-result-from-package-list ort helper command or by putting a specially crafted SPDX package.spdx.yml or project.spdx.yml within the to be scanned code repository.

After several discussions we choose to generate the simpler package-list.yml format as it's newer and known to have lesser issues than SPDX file input. Downside is that package-list.yml requires a VCS repository to be specified which given Docker image as input does not exist (point to discuss further within ORT community).

We thought of the following mapping from ScanCode.io to ORT's package-list.yml format:

---
projectName: "Alpine Docker Base Image"
projectVcs: <=  Will be all empty for when docker URL is provided to ScanCode.io but should be set if package URL of code repository is provided
  type: "Git"
  url: ""
  revision: ""
  path: ""
  sourceArtifact: <= Should we add support for defining a sourceArtifact within ORT's package-list.yml format?
    url: "https://api.scancode.io/fetch?alpine-docker-base-image:v3.22-latest => ideally the tar url within the Docker regisitry
dependencies:
  - id: "docker::alpine/alpine-base:3.22.0-r0"
    purl: "pkg:alpine/alpine-base@3.22.0-r0?arch=aarch64&uuid=71ba8d88-c2f2-4975-a9b6-6f09759adac2"
    vcs: <= mapped from ScanCode.io `vcs_url` e.g. git+https://git.alpinelinux.org/aports/commit/?id=76769759d295378b24d0ed9a9d252dfe525cf1c0
      type: "Git"
      url: "https://git.alpinelinux.org/aports/commit/?id=76769759d295378b24d0ed9a9d252dfe525cf1c0" => ORT VCS curation can fix this to https://gitlab.alpinelinux.org/alpine/aports.git
      revision: "76769759d295378b24d0ed9a9d252dfe525cf1c0"
    sourceArtifact: <= Only set if ScanCode.io does not detect
      url: "??" <= Mapped from ScanCode.io's `download_url`
    declaredLicenses:
      - "MIT" <= Mapped from ScanCode.io's `declared_license_expression_spdx` field or if empty `declared_license_expression`
    description:  <= Mapped from ScanCode.io description field, needs to added to as new property to https://github.com/oss-review-toolkit/ort/blob/main/cli-helper/src/main/kotlin/commands/CreateAnalyzerResultFromPackageListCommand.kt#L149
    homepageUrl:  <= Mapped from ScanCode.io homepage_url field, needs to added to as new property to https://github.com/oss-review-toolkit/ort/blob/main/cli-helper/src/main/kotlin/commands/CreateAnalyzerResultFromPackageListCommand.kt#L149

Tasks

• Expand CreateAnalyzerResultFromPackageListCommand in ORT to support description and homepageURL in package-list file
• Implement a new ScanCode.io output the ORT package list format as shown in https://github.com/oss-review-toolkit/ort/blob/main/cli-helper/src/funTest/resources/package-list.yml. A package list file can converted to ORT analyzer result output file using ort/cli-helper/build/install/orth/bin/orthw create-analyzer-result-from-package-list -i package-list.yml -o analyzer-result.yml.
• Add ORT package-list.yml output format #1852
• Create a GitHub action workflow similar to other SCA integration in .github/workflow have ScanCode.io scan python:3.13.0-slim, export to package-list.yml, convert to ORT analyzer using create-analyzer-result-from-package-list and then do assert to verity exported package-list.yml matched known package-list.yml.
• Add a workflow for the ScanCode.io to ORT integration #1866