-
-
Notifications
You must be signed in to change notification settings - Fork 203
Create pipeline for symbol reachability #2151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
86d4dcc
feat: create pipeline for symbol reachability and add a test
ziadhany 87bde58
Add more test for reachability and remove redundant code
ziadhany 1a32c0f
Fix the format bugs and refactor the code
ziadhany a6bec7a
Fix a bug in the import-catching logic and add a test.
ziadhany 680e3b6
Add an unidiff dependency to pyproject.toml file
ziadhany 0c96148
Simplify the pipeline logic for imports and direct calls
ziadhany e7a7056
Add support for vulnerablecode reachability option
ziadhany 605a292
Fix the tests and improve patch extraction performance
ziadhany 0a99680
Fix formatting and linting errors.
ziadhany c752810
Refactor and simplify reachability pipeline
ziadhany 5ddf24f
Add support for getting constant symbols
ziadhany decfbf2
Remove dead code and fix the test
ziadhany 4599f13
Add unidiff to uv.lock
ziadhany 1d5cb02
Add support constant to resource analyzer
ziadhany 5b8d57e
Fix formating error in CI
ziadhany ee31acb
Add a test for constants, extract_imports,collect_imports
ziadhany 60f7963
Remove dependency and copy only the required file
ziadhany c222fed
Update pipeline/functions name
ziadhany 0fd6f07
Fix CI files format
ziadhany 192deab
Fix a typo in patch.py.ABOUT file
ziadhany c8b573e
Skip the test for macOS
ziadhany 08c81ec
Allow reachability by default, for vulnerabilities pipeline
ziadhany 43bd799
Update the code to difflib instead of unidiff library
ziadhany 20b5a86
Add missing docs
ziadhany 626baab
Fix a typo in build_symbol_metadata function signature
ziadhany c380406
Add a test for java
ziadhany fd011f6
Simplify the pipeline test
ziadhany 8f7958d
Generate advisory reachability report ( last step in the pipeline )
ziadhany 7f5aa02
Fix Formating and typo in test
ziadhany 78e2423
Remove type hints
ziadhany 1558768
Remove type hints for symbols.py
ziadhany 2b774c3
Remove type hints for symbols.py
ziadhany 7f196b9
Fix a typo in docs
ziadhany e7941d9
Fix a typo in error message
ziadhany e7f9da5
Update the pipeline to clone repo once and collect_patch_symbols for …
ziadhany bcde743
Try to fix bug in vulnerability dependency
ziadhany 027ab5e
Update the pipeline to have a test for resource_patch_matcher
ziadhany 60ab0e2
Add more test and make sure it returns not if there is no match
ziadhany bde0ec5
Move Business logic to be in the pipes reachability file
ziadhany 2dfce28
fix is_reachable to be by default NOT_REACHABLE
ziadhany 35c3dca
Fix a bug related to vulnerability dependency reachability pipeline
ziadhany 2b17307
Update function docs
ziadhany ca03409
Fix a typo
ziadhany df56dd2
Fix format error
ziadhany cb83ff3
Add a LoopProgress to the most time-consuming functions
ziadhany File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # http://nexb.com and https://github.com/aboutcode-org/scancode.io | ||
| # The ScanCode.io software is licensed under the Apache License version 2.0. | ||
| # Data generated with ScanCode.io is provided as-is without warranties. | ||
| # ScanCode is a trademark of nexB Inc. | ||
| # | ||
| # You may not use this software except in compliance with the License. | ||
| # You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0 | ||
| # Unless required by applicable law or agreed to in writing, software distributed | ||
| # under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR | ||
| # CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations under the License. | ||
| # | ||
| # Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES | ||
| # OR CONDITIONS OF ANY KIND, either express or implied. No content created from | ||
| # ScanCode.io should be considered or used as legal advice. Consult an Attorney | ||
| # for any legal advice. | ||
| # | ||
| # ScanCode.io is a free software code scanning tool from nexB Inc. and others. | ||
| # Visit https://github.com/aboutcode-org/scancode.io for support and download. | ||
|
|
||
| from scanpipe.pipelines import Pipeline | ||
| from scanpipe.pipes import reachability | ||
| from scanpipe.pipes.symbols import TS_QUERIES | ||
|
|
||
|
|
||
| class SymbolReachability(Pipeline): | ||
| """ | ||
| Determine the reachability of vulnerabilities identified in the project. | ||
|
|
||
| Note: You must run `find_vulnerabilities` pipeline before running this pipeline. | ||
|
|
||
| For every patch the git repository is cloned and extract the vulnerable and fixed | ||
| symbols from the patch commit. These symbols are then matched against | ||
| the project's codebase resources to determine if the vulnerable code | ||
| is actually present and reachable. | ||
|
|
||
| The analysis checks if vulnerable symbols are defined, imported, called, | ||
| or exactly match a code within the project files. The results, including | ||
| tool_details and a reachability status (yes, unknown, or no), are stored | ||
| in the `extra_data` of the matching resources under the `symbols_reachability` key. | ||
|
|
||
| Finally, a summary report is generated for each vulnerability | ||
| advisory and saved as a JSON output file. | ||
| """ | ||
|
|
||
| download_inputs = False | ||
| is_addon = True | ||
| results_url = "/project/{slug}/resources/?extra_data=symbol_reachability" | ||
|
|
||
| @classmethod | ||
| def steps(cls): | ||
| return ( | ||
| cls.get_vulnerabilities_patches, | ||
| cls.collect_resource_index, | ||
| cls.collect_patch_symbols, | ||
| cls.collect_and_match_resources, | ||
| cls.generate_advisory_reachability_report, | ||
| ) | ||
|
|
||
| def get_vulnerabilities_patches(self): | ||
| """Get unique patch for all vulnerabilities.""" | ||
| self.patches = reachability.get_vulnerabilities_patches( | ||
| package_vulnerabilities=self.project.package_vulnerabilities, | ||
| dependency_vulnerabilities=self.project.dependency_vulnerabilities, | ||
| ) | ||
|
|
||
| def collect_resource_index(self): | ||
| """Collect resources symbols for each resource""" | ||
| self.candidate_resources = self.project.codebaseresources.files().filter( | ||
|
ziadhany marked this conversation as resolved.
|
||
| is_binary=False, | ||
| is_archive=False, | ||
| is_media=False, | ||
| programming_language__in=TS_QUERIES.keys(), | ||
| ) | ||
| self.resource_indexes = reachability.collect_resource_index( | ||
| candidate_resources=self.candidate_resources, logger=self.log | ||
| ) | ||
|
|
||
| def collect_patch_symbols(self): | ||
| """Collect patch symbols for all related commits.""" | ||
| self.patch_symbols = reachability.collect_patch_symbols( | ||
| patches=self.patches, logger=self.log | ||
| ) | ||
|
|
||
| def collect_and_match_resources(self): | ||
| """Match resource symbols against patch symbols.""" | ||
| reachability.match_patches_to_resources( | ||
| patches=self.patches, | ||
| patch_symbols=self.patch_symbols, | ||
| resource_indexes=self.resource_indexes, | ||
| candidate_resources=self.candidate_resources, | ||
| logger=self.log, | ||
| ) | ||
|
|
||
| def generate_advisory_reachability_report(self): | ||
| """Generate a reachability report summarizing status by advisory.""" | ||
| reachability.generate_advisory_reachability_report( | ||
| project=self.project, | ||
| patches=self.patches, | ||
| candidate_resources=self.candidate_resources, | ||
| ) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.