cravex2-reachability: Create pipeline for symbol reachability

[pombredanne]

Pipeline(s) that can retrieve vulnerable symbols, collect local symbols (pur2sym) and match them

[ziadhany]

According to my understanding, there are three types of reachability statuses: REACHABLE, POTENTIALLY_REACHABLE, and NOT_REACHABLE.

Suppose we receive a vcs_url, a commit_hash, and ( a patch URL not sure if we need this ) from VulnerableCode API

Steps:

1. Clone the Git repository
2. Get the patch_text and contents of both the vulnerable and fixed files.
3. Extract the vulnerable and fixed function ranges.
4. Extract vulnerable function symbols, fixed function symbols, and resource symbols.
5. Search for matching ( exact_code_matches, imports, direct_calls, ... )
6. Return the reachability result:

{
    "reachability_status": reachability_status,
    "evidence": {
          ...
          ...
    },
    "vulnerable_symbols": vuln_symbols,
    "fixed_symbols": fixed_symbols,
}

Based on my understanding, the vulnerability is REACHABLE when there is a match for both the path and the fingerprint.
It is POTENTIALLY_REACHABLE when there is only a path match,
and NOT_REACHABLE when no matches are found.

[ziadhany]

According to my understanding, we have two types of packages:

• Directly vulnerable package:
  The package itself contains the vulnerable code.
  Flask-2.3.2
  GHSA-68rp-wp8r-4726
  pallets/flask@089cb86

• Package with a (dependency that have a vulnerability ) :
  The package itself is not vulnerable, but one of its dependencies is.
  vulnerablecode v20.10
  GHSA-w2fm-2cpv-w7v5
  aio-libs/aiohttp@0c2e9da

According to my understanding, the vulnerability reachability statuses mean the following:

Directly vulnerable package:

• Reachable: code exists and can be executed ( exact code match )
• Potentially Reachable: can't get a exact code match , but the vulnerable function identifier exist
• Not Reachable: Can't get the vulnerable function identifier

Package with a (dependency that have a vulnerability )

• Reachable: the application import and call the vulnerable dependency function
• Potentially Reachable: the application import the vulnerable dependency ( can't find a direct call )
• Not Reachable: No imports or calls

[ziadhany]

This is done now:

Add Pipeline(s) that can retrieve vulnerable / fix symbols, collect local symbols (pur2sym) and match them
Pipeline Graph:

graph TD
    %% Main Entry Point
    Start([SymbolReachability Pipeline]) --> Step1[get_vulnerabilities_patches]
    
    %% Step 1: Get Patches
    Step1 --> ExtractPatches[Extract fixed_in_patches from vulnerabilities]
    ExtractPatches --> DedupePatches[Deduplicate by vcs_url & commit_hash]
    DedupePatches --> Step2[collect_resource_index]
    
    %% Step 2: Collect Resource Index
    Step2 --> FilterResources[Filter codebaseresources by TS_QUERIES languages]
    FilterResources --> ResLoop{For each candidate_resource}
    ResLoop -->|Next resource| BuildResIndex[ResourceAnalyzer.build_index]
    BuildResIndex --> ExtractDefs[Extract definitions, imports, calls & callers_of]
    ExtractDefs --> SaveResIndex[Store in resource_indexes]
    SaveResIndex --> ResLoop
    ResLoop -->|All resources processed| Step3[collect_patch_symbols]
    
    %% Step 3: Collect Patch Symbols
    Step3 --> GroupPatches[Group patches by vcs_url]
    GroupPatches --> RepoLoop{For each repository}
    RepoLoop -->|Next repo| CloneRepo[Clone repo to temp dir]
    CloneRepo --> PatchLoop{For each patch in repo}
    PatchLoop -->|Next patch| PatchAnalyzer[PatchAnalyzer.collect_patch_symbols]
    PatchAnalyzer --> GetDiff[Get changed files & diff vulnerable/fixed texts]
    GetDiff --> DetectLang[detect_language_with_scancode]
    DetectLang --> ParseAST[LanguageQuery.parse_code_to_ast]
    ParseAST --> ExtractSymbols[SymbolExtractor.extract_changed_symbols]
    ExtractSymbols --> DiffSymbols[diff_changed_symbols]
    DiffSymbols --> SavePatchSyms[Store in patch_symbols by commit_hash]
    SavePatchSyms --> PatchLoop
    PatchLoop -->|All patches checked| RepoLoop
    RepoLoop -->|All repos processed| Step4[collect_and_match_resources]
    
    %% Step 4: Match Resources and Patches
    Step4 --> MatchPatchLoop{For each patch}
    MatchPatchLoop -->|Next patch| MatchResLoop{For each candidate_resource}
    MatchResLoop -->|Next resource| CheckLang{Language matches patch?}
    CheckLang -->|No| MatchResLoop
    CheckLang -->|Yes| Matcher[ResourcePatchMatcher.match]
    
    Matcher --> MatchVuln[Match Vulnerable Symbols]
    Matcher --> MatchFixed[Match Fixed Symbols]
    
    MatchVuln & MatchFixed --> CheckEvidence{Evidence found?}
    CheckEvidence -->|No| MatchResLoop
    CheckEvidence -->|Yes| Classify[classify_reachability]
    Classify --> StatusReach[REACHABLE / UNKNOWN / NOT_REACHABLE]
    StatusReach --> UpdateData[save_resource_reachability_report]
    UpdateData --> MatchResLoop
    
    MatchResLoop -->|All resources checked| MatchPatchLoop
    MatchPatchLoop -->|All patches checked| End([End Process])

    %% Styling
    classDef stepNode fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px;
    classDef functionNode fill:#f1f8e9,stroke:#4caf50,stroke-width:2px;
    classDef logicNode fill:#fff3e0,stroke:#ff9800,stroke-width:2px;
    classDef loopNode fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px;
    
    class Start,End,Step1,Step2,Step3,Step4 stepNode;
    class RepoLoop,PatchLoop,ResLoop,MatchPatchLoop,MatchResLoop loopNode;
    class CheckLang,CheckEvidence logicNode;
    class PatchAnalyzer,Matcher,Classify,BuildResIndex functionNode;

Loading

see:

• Create pipeline for symbol reachability #2151