Add human review workflow for model predictions

.github/workflows/tests.yml

@@ -57,4 +57,6 @@ jobs:
             tests/test_model.py \
             tests/test_export.py \
             tests/test_inference.py \
-            tests/test_model_rules.py
+            tests/test_model_rules.py \
+            tests/test_review.py \
+            tests/test_model_cli.py

CHANGELOG.rst

@@ -11,3 +11,4 @@ v0.0.0
 - Add composite rule required phrase updates.
 - Add required phrase model training and ONNX export.
 - Add model prediction and rule integration.
+- Add resumable review for model-predicted required phrases.

README.rst

@@ -70,14 +70,34 @@ return candidate required phrases without changing a ScanCode rule or file:
 
 Predictions require human review before they are added to license rules.
 
-Prepare predicted phrases for review
-====================================
-
-The ``model_rules`` module loads eligible rules, validates model predictions,
-and prepares complete rule updates without changing the original rules. It also
-provides an atomic writer that requires the exact rule path and its current file
-hash. User-facing review and application are added by the stacked review
-workflow.
+Review model-predicted phrases
+==============================
+
+Install the inference dependencies and run the review command. It uses the
+pinned public model by default:
+
+.. code-block:: console
+
+    python -m pip install ".[inference]"
+    add-model-required-phrases --rule path/to/example.RULE
+
+The default mode shows each phrase, model score, text context, and exact rule
+diff before asking for approval. Decisions are saved in a resumable session.
+Use ``--rules-dir`` for top-level rule files in a directory or ``--all`` for
+eligible installed ScanCode rules.
+
+Read-only prediction never creates a session or changes a rule:
+
+.. code-block:: console
+
+    add-model-required-phrases --rule path/to/example.RULE --predict-only
+
+A wrong required phrase can cause a false negative. Batch processing therefore
+requires explicit score thresholds and ``--yes`` before any write. Rules with
+pending phrases are deferred unchanged while fully decided rules can be applied.
+``--dry-run`` always writes zero rules. Run ``scancode-reindex-licenses`` after
+changing installed ScanCode rules. Use ``--model`` for a custom local or remote
+model.
 
 Development
 ===========

azure-pipelines.yml

@@ -29,7 +29,9 @@ jobs:
                     tests/test_model.py \
                     tests/test_export.py \
                     tests/test_inference.py \
-                    tests/test_model_rules.py
+                    tests/test_model_rules.py \
+                    tests/test_review.py \
+                    tests/test_model_cli.py
             displayName: Run training unit tests
 
     - template: etc/ci/azure-posix.yml

docs/source/model_rules.rst

@@ -1,19 +1,89 @@
-Model prediction and rule preparation
-=====================================
+Review model-predicted required phrases
+=======================================
 
-Install the ``inference`` extra before loading a final model. A local model must
-pass the final-model publication checks. A remote Hugging Face model also
-requires its full commit hash.
+Install the inference dependencies before using the command:
 
-The ``scancode_required_phrases.model_rules`` module provides reusable functions
-to:
+.. code-block:: console
 
-- load eligible rules from one file, a directory, or installed ScanCode data;
-- return model predictions with their ScanCode validation result;
-- prepare a complete rule update without mutating the original rule;
-- serialize and atomically write a rule to its exact source path.
+    python -m pip install ".[inference]"
 
-Phrase text found more than once is rejected because ScanCode's mutation helper
-would mark every occurrence. A complete phrase set is prepared before any file
-is written. The stacked review workflow provides the user-facing command and
-human approval process.
+The public model is available without a Hugging Face token:
+
+.. code-block:: text
+
+    Kaushik-Kumar-CEG/scancode-required-phrases-deberta-bioes-crf-hardened
+    11215925b0f9b64cfcfbbb5492b52d6aeb5a572b
+
+A wrong required phrase can prevent a true license match. Review predictions
+carefully and use disposable rule copies before changing installed data.
+
+Interactive review
+------------------
+
+Review one rule:
+
+.. code-block:: console
+
+    add-model-required-phrases --rule path/to/example.RULE
+
+Use ``--rules-dir`` to review sorted top-level ``.RULE`` files in a directory.
+Use ``--all`` to review eligible rules installed with ScanCode Toolkit. The
+command shows the rule, expression, predicted phrase, model score, context, and
+exact diff. Approve, reject, edit, skip, or save and quit at each prompt.
+
+Sessions are created automatically and retained for audit. The command prints an
+exact resume command when review remains unfinished:
+
+.. code-block:: console
+
+    add-model-required-phrases --resume path/to/session.jsonl
+
+Review and application are resumable. An interrupted prediction run starts
+prediction again.
+
+Read-only prediction
+--------------------
+
+Print validated and rejected model output without creating decisions or writing
+rules:
+
+.. code-block:: console
+
+    add-model-required-phrases --rule path/to/example.RULE --predict-only
+
+Use ``--json predictions.json`` for machine-readable output or ``--json -`` for
+JSON on standard output.
+
+Batch classification
+--------------------
+
+Batch mode has no default thresholds. Both values must be chosen explicitly:
+
+.. code-block:: console
+
+    add-model-required-phrases --all --batch \
+        --auto-score 0.90 --review-score 0.70 --dry-run
+
+Scores at or above ``--auto-score`` are staged for automatic approval only after
+ScanCode validation. Scores from ``--review-score`` up to ``--auto-score`` stay
+pending. Lower scores are ignored. Truncated rules always require review. A rule
+with any pending phrase is deferred unchanged. Other fully decided rules may be
+applied after complete preflight.
+
+``--yes`` permits non-interactive writes for ready rules. ``--dry-run`` always
+takes precedence and writes zero rules. Use ``--model`` and ``--model-revision``
+to override the pinned public model.
+
+Safe application
+----------------
+
+Review never mutates rule files. Before application, the command verifies every
+path and file hash, prepares all approved updates, and displays a final summary.
+Each changed rule is then written once with an atomic replacement at its exact
+source path. Rejected and below-threshold phrases are not written.
+
+After changing installed ScanCode rules, run:
+
+.. code-block:: console
+
+    scancode-reindex-licenses

setup.cfg

@@ -55,6 +55,7 @@ where = src
 [options.entry_points]
 console_scripts =
     add-composite-required-phrases = scancode_required_phrases.composite_rules:add_composite_required_phrases
+    add-model-required-phrases = scancode_required_phrases.model_cli:add_model_required_phrases
     build-required-phrases-dataset = scancode_required_phrases.dataset:main
     export-required-phrase-model = scancode_required_phrases.export:main
     train-required-phrase-model = scancode_required_phrases.training:main