Add combined documentation

Signed-off-by: Philippe Ombredanne <pombredanne@aboutcode.org>

Author: pombredanne

docs/source/conf.py

@@ -17,7 +17,7 @@
 
 # -- Project information -----------------------------------------------------
 
-project = "nexb-skeleton"
+project = "purl-validator"
 copyright = "nexB Inc., AboutCode and others."
 author = "AboutCode.org authors and contributors"
 
@@ -79,9 +79,9 @@
 
 html_context = {
     "display_github": True,
-    "github_user": "nexB",
-    "github_repo": "nexb-skeleton",
-    "github_version": "develop",  # branch
+    "github_user": "aboutcode-org",
+    "github_repo": "purl-validator",
+    "github_version": "main",  # branch
     "conf_py_path": "/docs/source/",  # path in the checkout to the docs root
 }
 

docs/source/contribute/contrib_doc.rst

@@ -187,7 +187,7 @@ Style Conventions for the Documentaion
 
     (`Refer <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections>`_)
     Normally, there are no heading levels assigned to certain characters as the structure is
-    determined from the succession of headings. However, this convention is used in Python’s Style
+    determined from the succession of headings. However, this convention is used in Python's Style
     Guide for documenting which you may follow:
 
     # with overline, for parts

docs/source/data-structure-rationale.rst

@@ -0,0 +1,83 @@
+.. _data_structure_rationale:
+
+FST Data Structure Rationale
+=============================
+
+PurlValidator needs exact membership lookup for a large list of base PURLs. The
+lookup data index is built before release and bundled with each library.
+
+
+See https://github.com/aboutcode-org/purl-validator/tree/main/etc/bench for
+actual detailed rationale and bench for the choice of an FST.
+
+
+Why FSTs are used?
+------------------
+
+Finite state transducers store sorted strings in a compact form. PURLs share
+prefixes such as ``pkg:npm/``, ``pkg:pypi/``, and ``pkg:maven/``. This makes an
+FST useful for exact package identity queries.
+
+FST can be memory-mapped and are super compact. They are not as fast as native
+set, but the memory consumption is so much lower than this make them the most
+attractive solution, even if it takes more time to build.
+
+
+Requirements
+---------------
+
+The index structure should provide:
+
+And for the library selection, we have these high level requirements:
+
+- We want exact result without false positives, e.g., no bloom filter.
+- Offline use, with no network is a must: the dataset must be bundled
+  in the releases.
+- With build time index construction, the construction time is not
+  critical.
+- The bundled index should be small enough to ship below crates, and
+  Pypi archive size limits.
+- No rebuild at startup/runtime, and fast enough load time from disk,
+  ideally memory-mapped.
+- Fast enough lookup.
+- Libraries should be maintained, active FOSS for Rust/Go/Python.
+
+
+
+
+Selected FST libraries
+--------------------------
+
+Python uses ``ducer.Map`` with ``mmap``. The map is stored on disk and opened
+without loading the full catalog into Python objects.
+
+Rust uses ``fst::Set``. The generated FST is embedded into the crate.
+
+Go uses Vellum FST. The generated FST is embedded into the module.
+
+Alternatives
+------------
+
+We considered also built-in sets and maps as a baseline:
+
+- Python: ``set`` and ``dict``.
+- Rust: ``HashSet`` and ``HashMap``.
+- Go: ``map[string]struct{}`` and ``map[string]bool``.
+
+These structures are simple and fast. They require loading all keys into
+runtime memory, so they are less useful as the packaged lookup format.
+
+Sorted arrays or slices can use binary search. They are simple and exact, but
+lookup takes repeated string comparisons and the strings still need to be
+loaded.
+
+SQLite can store the PURLs in an indexed table. It gives exact results, but it
+adds a database dependency for a read-only membership check. It has way more
+features than needed and is overkill for our use case.
+
+Bloom filters are small and fast, but they can return false positives. They
+should cannot be used as validation index.
+
+A DAWG can store a set of strings by sharing prefixes and suffixes. It may be a
+valid alternative to an FST (it is very similar to) but there are few maintained
+libraries in the target languages.

docs/source/explanations.rst

@@ -0,0 +1,31 @@
+.. _explanations:
+
+Explanations
+============
+
+Syntax validation and identity validation
+-----------------------------------------
+
+The Package-URL spec defines the PURL format. A PURL can follow the spec
+format and still name a package that is not known in the package ecosystems.
+
+PurlValidator checks the package PURL against reference data of known PURLs. This
+helps find misspelled names, wrong package types, and PURL that
+do not appear in the reference upstream ecosystem package repositories.
+
+
+Offline validation
+------------------
+
+SBOM and compliance workflows may run in CI systems, private networks, or
+air-gapped environments. PurlValidator packages lookup data with each released
+library so validation does not need a network registry access at runtime.
+
+
+Base PURL validation
+--------------------
+
+PURL existence is checked before version existence.
+
+The current libraries validate base PURLs only, no versions. Version support
+can be a future enhancement.

docs/source/how-to-guides.rst

@@ -0,0 +1,36 @@
+.. _how_to_guides:
+
+How-to Guides
+=============
+
+Choose an implementation
+------------------------
+
+Use the implementation that matches the application:
+
+- Use Python for Python scripts, data pipelines, etc.
+- Use Rust for Rust appss.
+- Use Go for Go apps and command-line tools.
+
+All implementations package PURL index data with the released library.
+
+
+Update validation data
+----------------------
+
+PurlValidator index data is released with each package. To update the
+data used by an application, update the PurlValidator package version.
+
+
+Validation results
+--------------------------
+
+Treat validation results in these groups:
+
+- Known: the PURL is valid and exists in the reference data.
+- Unknown: the PURL is valid (parsing) but not present in the reference data.
+- Invalid or unsupported: the input is not a supported or known PURL.
+
+For SBOM checks, you should report unknown and invalid PURLs separately.
+Invalid PURLs are usually an error of the SBOM or SCA producer tool.
+Unknown PURLs could be new packages, or typos, or SCA tools inventions.

docs/source/index.rst

@@ -1,16 +1,57 @@
-Welcome to nexb-skeleton's documentation!
-=========================================
+PurlValidator Documentation
+===========================
 
-.. toctree::
-   :maxdepth: 2
-   :caption: Contents:
+PurlValidator checks whether a base Package-URL (PURL) is present in a known
+package catalog. It works without a network connection after installation.
 
-   skeleton-usage
-   contribute/contrib_doc
+A valid PURL string can still name a package that is not known. PurlValidator
+adds this package identity check for SBOM, VEX, SCA, and compliance workflows.
+
+Documentation overview
+----------------------
+
+Getting started
+~~~~~~~~~~~~~~~
+
+- :ref:`quickstart`
+- :ref:`introduction`
+
+Tutorials
+~~~~~~~~~
+
+- :ref:`tutorials`
+
+How-to guides
+~~~~~~~~~~~~~
+
+- :ref:`how_to_guides`
+
+Reference
+~~~~~~~~~
+
+- :ref:`reference`
+
+Explanations
+~~~~~~~~~~~~
+
+- :ref:`explanations`
+- :ref:`data_structure_rationale`
 
 Indices and tables
-==================
+------------------
 
 * :ref:`genindex`
-* :ref:`modindex`
 * :ref:`search`
+
+.. toctree::
+   :maxdepth: 2
+   :hidden:
+
+   quickstart
+   introduction
+   tutorials
+   how-to-guides
+   reference
+   explanations
+   data-structure-rationale
+   contribute/contrib_doc

docs/source/introduction.rst

@@ -0,0 +1,47 @@
+.. _introduction:
+
+Introduction
+============
+
+PurlValidator checks package identity for Package-URLs (PURLs). It does
+not replace syntax validation. It adds a lookup against an index of packaged
+reference data.
+
+Why this exists?
+-----------------
+
+PURL is used in SBOMs, VEX documents, SCA tools, and vulnerability databases.
+The PURL spec tells tools how to write a package identifier, but does
+not prove that the package exists.
+
+Common PURL data problems include:
+
+- Misspelled package names.
+- Wrong or made up package types.
+- Package that are not present in an ecosystem.
+
+PurlValidator answers this question:
+
+Does this PURL exists for a known package?
+
+Repositories
+------------
+
+We have three implementations in Rust, Go and Python.
+Each repository has language-specific usage notes in its README.
+
+- Python: https://github.com/aboutcode-org/purl-validator
+- Rust: https://github.com/aboutcode-org/purl-validator.rs
+- Go: https://github.com/aboutcode-org/purlvalidator-go
+
+
+Validation scope
+----------------
+
+PurlValidator validates PURLs, ignoring version. A base PURL contains:
+
+- Type, such as ``npm`` or ``pypi``.
+- Optional namespace, such as an npm scope or Maven groupid.
+- Name.
+
+Versions, qualifiers, and subpaths are not part of the lookup query.

docs/source/quickstart.rst

@@ -0,0 +1,84 @@
+.. _quickstart:
+
+Quickstart
+==========
+
+Python
+------
+
+Install the Python package:
+
+.. code-block:: bash
+
+    pip install purl-validator
+
+Validate a PURL:
+
+.. code-block:: python
+
+    from purl_validator import PurlValidator
+
+    validator = PurlValidator()
+
+    print(validator.validate_purl("pkg:nuget/FluentValidation"))
+    print(validator.validate_purl("pkg:nuget/non-existent-foo-bar"))
+
+Rust
+----
+
+Install the Rust crate:
+
+.. code-block:: bash
+
+    cargo add purl_validator
+
+Validate a PURL:
+
+.. code-block:: rust
+
+    use purl_validator::validate;
+
+    fn main() {
+        let exists = validate("pkg:nuget/FluentValidation")
+            .expect("input must be a supported base PURL");
+
+        println!("{exists}");
+    }
+
+Go
+--
+
+Install the Go module:
+
+.. code-block:: bash
+
+    go get github.com/aboutcode-org/purlvalidator-go
+
+Validate a PURL:
+
+.. code-block:: go
+
+    package main
+
+    import (
+        "fmt"
+        "log"
+
+        purlvalidator "github.com/aboutcode-org/purlvalidator-go"
+    )
+
+    func main() {
+        exists, err := purlvalidator.Validate("pkg:nuget/FluentValidation")
+        if err != nil {
+            log.Fatal(err)
+        }
+
+        fmt.Println(exists)
+    }
+
+Next steps
+----------
+
+- Use the Python README for Python-specific helper APIs:  https://github.com/aboutcode-org/purl-validator
+- Use the Rust README for error handling with ``ValidateError``: https://github.com/aboutcode-org/purl-validator.rs
+- Use the Go README for ``Validate`` return values and integration examples: https://github.com/aboutcode-org/purlvalidator-go