Add model training, export, and read-only inference

CHANGELOG.rst

@@ -9,3 +9,4 @@ v0.0.0
 
 - Add required phrase dataset extraction.
 - Add composite rule required phrase updates.
+- Add required phrase model training and ONNX export.

MANIFEST.in

@@ -12,6 +12,7 @@ include *.rst
 include *.png
 include setup.*
 include configure*
+include conftest.py
 include requirements*
 include .dockerignore
 include .gitignore

README.rst

@@ -35,6 +35,37 @@ The command uses existing required phrases from single-key rules to update
 composite rules. A rule is updated only when every relevant license key has a
 non-overlapping match. It operates on the installed ScanCode rules directory.
 
+Train and export a model
+========================
+
+Install the optional training dependencies and train from a generated dataset:
+
+.. code-block:: console
+
+    python -m pip install ".[training,training-8bit]"
+    train-required-phrase-model --data-dir dataset-output --output-dir model-output \
+        --model-revision 64a8c8eab3e352a784c658aef62be1662607476f \
+        --optimizer adamw-8bit
+
+Test-set evaluation is opt-in and should only be used for a final selected run.
+The completed model is written to ``model-output/final-model`` only after local
+reload and validation succeed. See ``docs/source/training.rst`` for details.
+
+Run read-only prediction
+========================
+
+Load a validated final model and return candidate required phrases without
+changing a ScanCode rule or file:
+
+.. code-block:: python
+
+    from scancode_required_phrases.inference import RequiredPhrasePredictor
+
+    predictor = RequiredPhrasePredictor.from_model_dir("model-output/final-model")
+    result = predictor.predict("Permission is hereby granted ...")
+
+Predictions require human review before they are added to license rules.
+
 Development
 ===========
 

azure-pipelines.yml

@@ -7,6 +7,30 @@
 
 jobs:
 
+    - job: ubuntu24_ml_tests
+      pool:
+          vmImage: ubuntu-24.04
+      steps:
+          - checkout: self
+            fetchDepth: 10
+          - task: UsePythonVersion@0
+            inputs:
+                versionSpec: '3.12'
+                architecture: x64
+            displayName: Install Python 3.12
+          - script: |
+                ./configure --clean
+                ./configure --dev
+                venv/bin/pip install -e ".[training]"
+            displayName: Configure training test environment
+          - script: |
+                USE_TF=0 venv/bin/pytest -q \
+                    tests/test_training.py \
+                    tests/test_model.py \
+                    tests/test_export.py \
+                    tests/test_inference.py
+            displayName: Run training unit tests
+
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: run_code_checks

conftest.py

@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) nexB Inc. and others. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Pytest collection settings for optional training dependencies."""
+
+import importlib.util
+
+
+training_dependencies = ("torch", "torchcrf", "transformers")
+if any(importlib.util.find_spec(name) is None for name in training_dependencies):
+    collect_ignore = ["src/scancode_required_phrases/model.py"]
+else:
+    collect_ignore = []

docs/source/index.rst

@@ -10,6 +10,7 @@ license rules.
 
    dataset
    composite_rules
+   training
    contribute/contrib_doc
 
 Indices and tables

docs/source/training.rst

@@ -0,0 +1,99 @@
+Train and export a required phrase model
+=========================================
+
+The training command fine-tunes the reviewed DeBERTa word-input classifier with
+BIOES labels and an optional constrained word-level CRF. It consumes the hybrid
+JSONL splits produced by ``build-required-phrases-dataset`` without moving,
+deduplicating, retokenizing, relabeling, or repairing records.
+
+Installation
+------------
+
+Install training support with ``python -m pip install ".[training]"``. Add the
+``training-8bit`` extra only when using ``adamw-8bit``. ONNX support remains
+optional and is installed separately with ``.[onnx]``. The training extra pins
+Transformers 4.57.3 and Hugging Face Hub 0.36.2, the versions validated by the
+single-T4 Kaggle smoke run.
+
+Training contract
+-----------------
+
+A run requires a full immutable model-host commit in ``--model-revision`` and a
+new or empty ``--output-dir``. For the current DeBERTa base model, run:
+
+.. code-block:: console
+
+    train-required-phrase-model --data-dir dataset-output --output-dir model-output \
+        --model-revision 64a8c8eab3e352a784c658aef62be1662607476f \
+        --optimizer adamw-8bit
+
+Resume is unsupported. Every non-empty line in
+all three splits is validated before tokenizer loading or ``--limit`` selection.
+Records must use exact field types, contain a positive valid BIOES sequence, and
+have unique identifiers across all splits. Exact token and label duplicates are
+reported but never removed.
+
+Alignment uses the dataset's existing words. Full untruncated fast-tokenizer
+coverage proves the complete retained word prefix. Coverage gaps, zero-subword
+words, partial words, and truncation that omits any non-``O`` label reject the
+record. ``--limit`` is applied only after complete validation and alignment.
+
+The run records versioned raw-byte (H0), validated-record (H1), and effective
+example (H2) hashes. ``run_manifest.json`` is atomically replaced through
+pre-run, failure, or success state and contains source/runtime provenance from
+runtime APIs, never an environment dump.
+
+Final model
+-----------
+
+Selection remains based only on best validation strict span F1. The selected
+checkpoint, selected in-memory model, and staged model must have exactly equal
+states. ``final-model.tmp`` is then reloaded using only its local configuration,
+tokenizer, and full weights; ordered validation predictions, labels, invalid
+path count, and metrics must match exactly. The stage is atomically promoted to
+``final-model`` and ``SUCCESS.json`` is written last. A model is publishable
+only while that marker and every recorded file hash validate.
+
+Strict metrics reject malformed gold paths and malformed CRF predictions.
+Malformed non-CRF predictions are not repaired: they produce no predicted spans,
+count all gold spans as false negatives, and increment ``invalid_paths``.
+
+ISR retains its existing name but measures predicted-phrase locatability only.
+It does not exercise injection gates or rule mutation and is not evidence of
+injection success.
+
+Read-only prediction
+--------------------
+
+``RequiredPhrasePredictor`` loads only a final model that passes the publication
+checks. Its ``predict()`` method uses the same ScanCode tokenization as the
+training dataset and returns phrase text, word offsets, confidence, and whether
+the input was truncated. It does not change rules or write files.
+
+.. code-block:: python
+
+    from scancode_required_phrases.inference import RequiredPhrasePredictor
+
+    predictor = RequiredPhrasePredictor.from_model_dir("model-output/final-model")
+    result = predictor.predict("Permission is hereby granted ...")
+
+Treat every prediction as a candidate requiring human review.
+
+Export
+------
+
+``export-required-phrase-model`` validates a publishable local final model and
+exports constrained CRF matrices by default:
+
+.. code-block:: console
+
+    export-required-phrase-model --model-dir model-output/final-model \
+        --output-dir model-export
+
+Run a separate export with ``--operation onnx`` and a different new output
+directory for optional ONNX emissions; missing ONNX packages cannot disable CRF
+export, training, finalization, or offline reload.
+
+Generated datasets, reports, manifests, checkpoints, final models, matrix
+files, ONNX files, smoke outputs, and caches must remain outside commits and the
+pull request.

pyproject.toml

@@ -103,7 +103,7 @@ force-single-line = true
 lines-after-imports = 1
 default-section = "first-party"
 known-first-party = ["scancode_required_phrases", "tests", "etc/scripts/**/*.py"]
-known-third-party = ["click", "pytest"]
+known-third-party = ["click", "numpy", "pytest", "safetensors", "torch", "torchcrf", "transformers"]
 
 sections = { django = ["django"] }
 section-order = [

setup.cfg

@@ -56,9 +56,26 @@ where = src
 console_scripts =
     add-composite-required-phrases = scancode_required_phrases.composite_rules:add_composite_required_phrases
     build-required-phrases-dataset = scancode_required_phrases.dataset:main
+    export-required-phrase-model = scancode_required_phrases.export:main
+    train-required-phrase-model = scancode_required_phrases.training:main
 
 
 [options.extras_require]
+training =
+    accelerate >= 0.33
+    huggingface-hub == 0.36.2
+    protobuf >= 3.20
+    pytorch-crf == 0.7.2
+    safetensors >= 0.4
+    sentencepiece >= 0.2
+    torch >= 2.0
+    transformers == 4.57.3
+training-8bit =
+    bitsandbytes >= 0.43
+onnx =
+    numpy
+    onnx >= 1.16
+    onnxruntime >= 1.18
 dev =
     pytest >= 7.0.1
     pytest-xdist >= 2