diff --git a/docs/source/aboutcode-docs/writing_good_commit_messages.rst b/docs/source/aboutcode-docs/writing_good_commit_messages.rst index 1a032f25..8fea78fc 100644 --- a/docs/source/aboutcode-docs/writing_good_commit_messages.rst +++ b/docs/source/aboutcode-docs/writing_good_commit_messages.rst @@ -1,3 +1,5 @@ +.. _good_commit_messages: + Writing good Commit Messages ============================ diff --git a/docs/source/conf.py b/docs/source/conf.py index 61a2097f..5dac4ea8 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -36,8 +36,12 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ +'sphinx.ext.intersphinx' ] +# Temporary Mapping, Once scancode-toolkit.readthedocs.io has the migrated docs, this can be changed to the same. +intersphinx_mapping = {'scancode-toolkit': ('https://sphinx-test-ayan.readthedocs.io/en/scancode-toolkit/', None)} + # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/docs/source/doc_maintenance.rst b/docs/source/doc_maintenance.rst index d60fcab3..beb8c2fa 100644 --- a/docs/source/doc_maintenance.rst +++ b/docs/source/doc_maintenance.rst @@ -32,9 +32,15 @@ Now you can install the dependencies in a virtualenv:: cd aboutcode virtualenv -p /usr/bin/python3.6 docs-venv - source bin/activate + source docs-venv/bin/activate -Now you can install Sphinx and the format theme used by readthedocs:: +Now, the following prerequisites are installed + +- Sphinx +- sphinx_rtd_theme (the format theme used by ReadTheDocs) +- docs8 (style linter) + +:: pip install Sphinx sphinx_rtd_theme doc8 @@ -83,6 +89,7 @@ system before creating a Pull Request. cd docs ./scripts/sphinx_build_link_check.sh + ./scripts/doc8_style_check.sh Share AboutCode Document Improvements ------------------------------------- @@ -99,11 +106,224 @@ examples:: git push git status -The AboutCode webhook with ReadTheDocs should rebuild the documentation. You can review your -results online. +The AboutCode webhook with ReadTheDocs should rebuild the documentation after your Pull Request +is Merged. + +Refer the `Pro Git Book `_ available online for Git tutorials +covering more complex topics on Branching, Merging, Rebasing etc. + +Continuous Integration +---------------------- + +The documentations are checked on every new commit through Travis-CI, so that common errors are +avoided and documentation standards are enforced. Travis-CI presently checks for these 3 aspects +of the documentation : + +1. Successful Builds (By using ``sphinx-build``) +2. No Broken Links (By Using ``link-check``) +3. Linting Errors (By Using ``Doc8``) + +So run these scripts at your local system before creating a Pull Request:: + + cd docs + ./scripts/sphinx_build_link_check.sh + ./scripts/doc8_style_check.sh + +Style Checks Using ``Doc8`` +--------------------------- + +How To Run Style Tests +^^^^^^^^^^^^^^^^^^^^^^ + +In the project root, run the following command:: + + $ doc8 --max-line-length 100 docs/source/ --ignore D000 + +A sample output is:: + + Scanning... + Validating... + docs/source/scancode-toolkit/misc/licence_policy_plugin.rst:37: D002 Trailing whitespace + docs/source/scancode-toolkit/misc/faq.rst:45: D003 Tabulation used for indentation + docs/source/scancode-toolkit/misc/faq.rst:9: D001 Line too long + docs/source/scancode-toolkit/misc/support.rst:6: D005 No newline at end of file + ======== + Total files scanned = 34 + Total files ignored = 0 + Total accumulated errors = 326 + Detailed error counts: + - CheckCarriageReturn = 0 + - CheckIndentationNoTab = 75 + - CheckMaxLineLength = 190 + - CheckNewlineEndOfFile = 13 + - CheckTrailingWhitespace = 47 + - CheckValidity = 1 + +Now fix the errors and run again till there isn't any style error in the documentation. + +What is Checked? +^^^^^^^^^^^^^^^^ + +PyCQA is an Organization for code quality tools (and plugins) for the Python programming language. +Doc8 is a sub-project of the same Organization. Refer this `README `_ for more details. + +What is checked: + + - invalid rst format - D000 + - lines should not be longer than 100 characters - D001 + + - RST exception: line with no whitespace except in the beginning + - RST exception: lines with http or https URLs + - RST exception: literal blocks + - RST exception: rst target directives + + - no trailing whitespace - D002 + - no tabulation for indentation - D003 + - no carriage returns (use UNIX newlines) - D004 + - no newline at end of file - D005 + +Interspinx +---------- + +Aboutcode documentation uses `Intersphinx `_ +to create links to other Sphinx Documentations, to maintain links to other Aboutcode Projects. + +To link sections in the same documentation, standart reST labels are used. Refer +`Cross-Referencing `_ for more information. + +For example:: + + .. _my-reference-label: + + Section to cross-reference + -------------------------- + + This is the text of the section. + + It refers to the section itself, see :ref:`my-reference-label`. + +Now, using Intersphinx, you can create these labels in one Sphinx Documentation and then referance +these labels from another Sphinx Documentation, hosted in different locations. + +You just have to add the following in the ``conf.py`` file for your Sphinx Documentation, where you +want to add the links:: + + extensions = [ + 'sphinx.ext.intersphinx' + ] + + intersphinx_mapping = {'scancode-toolkit': ('https://scancode-toolkit.readthedocs.io/en/latest/', None)} + +To show all Intersphinx links and their targets of an Intersphinx mapping file, run:: + + python -msphinx.ext.intersphinx https://scancode-toolkit.readthedocs.io/en/latest/objects.inv + +.. WARNING:: + + ``python -msphinx.ext.intersphinx https://scancode-toolkit.readthedocs.io/objects.inv`` will + give error. + +This enables you to create links to the ``scancode-toolkit`` Documentation in your own +Documentation, where you modified the configuration file. Links can be added like this:: + + For more details refer :ref:`scancode-toolkit:doc_style_guide`. + +You can also not use the ``scancode-toolkit`` label assigned to all links from +scancode-toolkit.readthedocs.io, if you don't have a label having the same name in your Sphinx +Documentation. Example:: + + For more details refer :ref:`doc_style_guide`. + +If you have a label in your documentation which is also present in the documentation linked by +Intersphinx, and you link to that label, it will create a link to the local label. + +For more information, refer this tutorial named +`Using Intersphinx `_. + +Extra Style Checks +------------------ + +1. Headings + + (`Refer `_) + Normally, there are no heading levels assigned to certain characters as the structure is + determined from the succession of headings. However, this convention is used in Python’s Style + Guide for documenting which you may follow: + + # with overline, for parts + + * with overline, for chapters + + =, for sections + + -, for subsections + + ^, for sub-subsections + + ", for paragraphs + +2. Heading Underlines + + Do not use underlines that are longer/shorter than the title headline itself. As in: + +:: + + Correct : + + Extra Style Checks + ------------------ + + Incorrect : + + Extra Style Checks + ------------------------ + +.. note:: + + Underlines shorter than the Title text generates Errors on sphinx-build. + + +3. Internal Links + + Using ``:ref:`` is advised over standard reStructuredText links to sections (like + ```Section title`_``) because it works across files, when section headings are changed, will + raise warnings if incorrect, and works for all builders that support cross-references. + However, external links are created by using the standard ```Section title`_`` method. + +4. Eliminate Redundancy + + If a section/file has to be repeated somewhere else, do not write the exact same section/file + twice. Use ``.. include: ../README.rst`` instead. Here, ``../`` refers to the documentation + root, so file location can be used accordingly. This enables us to link documents from other + upstream folders. + +5. Using ``:ref:`` only when necessary + + Use ``:ref:`` to create internal links only when needed, i.e. it is referenced somewhere. + Do not create references for all the sections and then only reference some of them, because + this created unnecessary references. This also generates ERROR in ``restructuredtext-lint``. + +6. Spelling + + You should check for spelling errors before you push changes. `Aspell `_ + is a GNU project Command Line tool you can use for this purpose. Download and install Aspell, + then execute ``aspell check `` for all the files changed. Be careful about not + changing commands or other stuff as Aspell gives prompts for a lot of them. Also delete the + temporary ``.bak`` files generated. Refer the `manual `_ for more + information on how to use. + +7. Notes and Warning Snippets + + Every ``Note`` and ``Warning`` sections are to be kept in ``rst_snippets/note_snippets/`` and + ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are + frequently used in multiple files. + +Converting from Markdown +------------------------ -Documentation Style Guides --------------------------- +If you want to convert a ``.md`` file to a ``.rst`` file, this `tool `_ +does it pretty well. You'd still have to clean up and check for errors as this contains a lot of +bugs. But this is definitely better than converting everything by yourself. -The ``scancode-toolkit`` documentation is compliant to our doc style standards, enforced using -``doc8``. For more details refer :ref:`contrib_doc_dev`. +This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for +Sphinx/ReadTheDocs hosting. diff --git a/docs/source/index.rst b/docs/source/index.rst index 390791ca..0c8a5d55 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -25,6 +25,14 @@ Documentation Guide doc_maintenance +Getting Started +*************** + +.. toctree:: + :maxdepth: 2 + + scancode-toolkit/getting-started/newcomer + Tutorial Documents ****************** @@ -65,3 +73,7 @@ Indices and Tables * :ref:`genindex` * :ref:`modindex` + +.. _improve_docs: + +.. include:: /scancode-toolkit/rst_snippets/improve_docs.rst diff --git a/docs/source/scancode-toolkit/cli-reference/basic-options.rst b/docs/source/scancode-toolkit/cli-reference/basic-options.rst index c04d87ad..a817216f 100644 --- a/docs/source/scancode-toolkit/cli-reference/basic-options.rst +++ b/docs/source/scancode-toolkit/cli-reference/basic-options.rst @@ -5,6 +5,10 @@ ---- +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + +---- + ``--generated`` Options ----------------------- @@ -12,7 +16,7 @@ An example of using ``--generated`` in a scan:: - ./scancode -clpieu --json-pp output.json samples --generated + scancode -clpieu --json-pp output.json samples --generated In the results, for each file the following attribute is added with it's corresponding ``true``/``false`` value :: @@ -47,7 +51,7 @@ An example usage:: - ./scancode -clpieu --json-pp output.json samples --max-email 5 + scancode -clpieu --json-pp output.json samples --max-email 5 This only reports 5 email addresses per file and ignores the rest. @@ -71,7 +75,7 @@ An example usage:: - ./scancode -clpieu --json-pp output.json samples --max-url 10 + scancode -clpieu --json-pp output.json samples --max-url 10 This only reports 10 urls per file and ignores the rest. @@ -100,7 +104,7 @@ An example usage:: - ./scancode -clpieu --json-pp output.json samples --license-score 70 + scancode -clpieu --json-pp output.json samples --license-score 70 Here's the license results on setting the integer value to 100, Vs. the default value 0. This is visualized using ScanCode workbench in the License Info Dashboard. @@ -134,7 +138,7 @@ An example Scan:: - ./scancode -cplieu --json-pp output.json samples --license-text + scancode -cplieu --json-pp output.json samples --license-text An example matched text included in the results is as follows:: @@ -178,7 +182,7 @@ A scan example using the ``--license-url-template TEXT`` option :: - ./scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.yml + scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.yml In a normal scan, reference url for "ZLIB License" is as follows:: @@ -212,7 +216,7 @@ An example Scan:: - ./scancode -cplieu --json-pp output.json samples --license-text --license-text-diagnostics + scancode -cplieu --json-pp output.json samples --license-text --license-text-diagnostics Running a scan on the samples directory with ``--license-text --license-text-diagnostics`` options, causes the following difference in the scan result of the file diff --git a/docs/source/scancode-toolkit/cli-reference/core-options.rst b/docs/source/scancode-toolkit/cli-reference/core-options.rst index 8ee6084f..04d1ddbe 100644 --- a/docs/source/scancode-toolkit/cli-reference/core-options.rst +++ b/docs/source/scancode-toolkit/cli-reference/core-options.rst @@ -7,6 +7,10 @@ ---- +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + +---- + Comparing Progress Message Options ---------------------------------- @@ -89,7 +93,7 @@ Comparing Progress Message Options An example scan command using ``--from-json``:: - ./scancode --from-json sample.json --json-pp sample_2.json --classify + scancode --from-json sample.json --json-pp sample_2.json --classify This inputs the scan results from ``sample.json``, runs the post-scan plugin ``--classify`` and outputs the results for this scan to ``sample_2.json``. @@ -114,4 +118,4 @@ Comparing Progress Message Options An example usage:: - ./scancode -clieu --json-pp sample.json samples --max-in-memory -1 + scancode -clieu --json-pp sample.json samples --max-in-memory -1 diff --git a/docs/source/scancode-toolkit/cli-reference/help-text-options.rst b/docs/source/scancode-toolkit/cli-reference/help-text-options.rst index ba616742..11133a0e 100644 --- a/docs/source/scancode-toolkit/cli-reference/help-text-options.rst +++ b/docs/source/scancode-toolkit/cli-reference/help-text-options.rst @@ -20,19 +20,11 @@ displays basic usage, and some simple examples. The command line option for this You can also use the shorter ``-h`` option, which does the same. -For Linux based systems the full command is:: - - $ ./scancode --help - -And for windows, it will be like:: +To see the help text from the Terminal, execute the following command:: $ scancode --help -.. note:: - - Make sure you are in the Scancode Root Directory before carrying out this command. After - extracting the ``.zip`` or ``.tar.bz`` file, the folder for Scancode-Toolkit version 3.1.1 - will be named like "scancode-toolkit-3.1.1". +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst The Following Help Text is displayed, i.e. This is the help text for Scancode Version 3.1.1 :: @@ -235,14 +227,12 @@ of code scanning: The command line option for displaying these basic examples is ``--examples``. -For Linux based systems the full command is:: - - $ ./scancode --examples - -And for windows, it will be like:: +To see the help text from the Terminal, execute the following command:: $ scancode --examples +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + The Following Text is displayed, i.e. This is the examples for Scancode Version 3.1.1 :: Scancode command lines examples: @@ -311,14 +301,12 @@ The command line option for displaying all the plugins is: - ``--plugins`` -For Linux based systems the full command is:: - - $ ./scancode --plugins - -And for windows, it will be like:: +To see the help text from the Terminal, execute the following command:: $ scancode --plugins +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + .. note:: Plugins that are shown by using ``--plugins`` include the following: @@ -772,7 +760,7 @@ This option prints the options selected for one specific scan command. If we run this command:: - ./scancode -clpieu --json-pp sample.json samples --classify --summary --summary-with-details --print-options + scancode -clpieu --json-pp sample.json samples --classify --summary --summary-with-details --print-options The output will be:: diff --git a/docs/source/scancode-toolkit/cli-reference/list-options.rst b/docs/source/scancode-toolkit/cli-reference/list-options.rst index 06ad2e66..8813b178 100644 --- a/docs/source/scancode-toolkit/cli-reference/list-options.rst +++ b/docs/source/scancode-toolkit/cli-reference/list-options.rst @@ -18,6 +18,8 @@ There's also another section for ``extractcode`` options. The order of the sections and all their options is the same as in the :ref:'cli_help_text', available in the command line. +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + .. _cli_basic: .. include:: /scancode-toolkit/rst_snippets/basic_options.rst diff --git a/docs/source/scancode-toolkit/cli-reference/output-filters-and-control.rst b/docs/source/scancode-toolkit/cli-reference/output-filters-and-control.rst index d113f383..e8788a48 100644 --- a/docs/source/scancode-toolkit/cli-reference/output-filters-and-control.rst +++ b/docs/source/scancode-toolkit/cli-reference/output-filters-and-control.rst @@ -7,6 +7,10 @@ Controlling Scancode Output and Filters ---- +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + +---- + ``--strip-root`` Vs. ``--full-root`` ------------------------------------ @@ -17,7 +21,7 @@ Controlling Scancode Output and Filters :: - ./scancode -cplieu --json-pp output.json samples --full-root + scancode -cplieu --json-pp output.json samples --full-root These two changes only the "path" attribute of the file information. For this comparison we compare the "path" attributes of the file ``LICENSE`` inside ``JGroups`` directory. @@ -53,7 +57,7 @@ Controlling Scancode Output and Filters This scan ignores all files with authors matching the string "Apache Software Foundation":: - ./scancode -cplieu --json-pp output.json samples --ignore-author "Apache Software Foundation" + scancode -cplieu --json-pp output.json samples --ignore-author "Apache Software Foundation" More information on :ref:`glob_pattern_matching`. @@ -71,7 +75,7 @@ Controlling Scancode Output and Filters This scan ignores all files with Copyright Holders matching the string "Free Software Foundation":: - ./scancode -cplieu --json-pp output.json samples --ignore-copyright-holder "Free Software Foundation" + scancode -cplieu --json-pp output.json samples --ignore-copyright-holder "Free Software Foundation" More information on :ref:`glob_pattern_matching`. @@ -86,7 +90,7 @@ Controlling Scancode Output and Filters An example Scan:: - ./scancode -cplieu --json-pp output.json samples --only-findings + scancode -cplieu --json-pp output.json samples --only-findings .. note:: diff --git a/docs/source/scancode-toolkit/cli-reference/output-format.rst b/docs/source/scancode-toolkit/cli-reference/output-format.rst index f36b9a81..4f416fff 100644 --- a/docs/source/scancode-toolkit/cli-reference/output-format.rst +++ b/docs/source/scancode-toolkit/cli-reference/output-format.rst @@ -12,6 +12,10 @@ following options. ---- +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + +---- + .. _output_to_stdout: .. include:: /scancode-toolkit/rst_snippets/stdout.rst @@ -30,7 +34,7 @@ following options. The following code performs a scan on the samples directory, and publishes the results in ``json`` format:: - ./scancode -clpieu --json output.json samples + scancode -clpieu --json output.json samples .. include:: /scancode-toolkit/rst_snippets/note_snippets/output_json_ugly.rst @@ -144,7 +148,7 @@ following options. The following code performs a scan on the samples directory, and publishes the results in ``json-pp`` format:: - ./scancode -clpieu --json-pp output.json samples + scancode -clpieu --json-pp output.json samples A sample JSON output for an individual file will look like:: @@ -247,7 +251,7 @@ following options. The following code performs a scan on the samples directory, and publishes the results in ``json-lines`` format:: - ./scancode -clpieu --json-lines output.json samples + scancode -clpieu --json-lines output.json samples Here is a sample line from a report generated by the ``jsonlines`` format:: @@ -342,7 +346,7 @@ Comparing Different ``json`` Output Formats The following code performs a scan on the samples directory, and publishes the results in ``spdx-rdf`` format:: - ./scancode -clpieu --spdx-rdf output.spdx samples + scancode -clpieu --spdx-rdf output.spdx samples Learn more about SPDX specifications `here `_ and in this GitHub `repository `_. @@ -363,7 +367,7 @@ Comparing Different ``json`` Output Formats The following code performs a scan on the samples directory, and publishes the results in ``spdx-tv`` format:: - ./scancode -clpieu --spdx-tv output.spdx samples + scancode -clpieu --spdx-tv output.spdx samples A SPDX-TV file starts with:: @@ -431,7 +435,7 @@ Comparing Different ``json`` Output Formats The following code performs a scan on the samples directory, and publishes the results in HTML format:: - ./scancode -clpieu --html output.html samples + scancode -clpieu --html output.html samples The HTML page generated has these following Tables: @@ -457,10 +461,12 @@ Comparing Different ``json`` Output Formats ScanCode also supports formatting the output in a HTML visualization tool, which is more helpful than the standard HTML format. + .. include:: /scancode-toolkit/rst_snippets/warning_snippets/output_htmlapp_dep.rst + The following code performs a scan on the samples directory, and publishes the results in ``html-app`` format:: - ./scancode -clpieu --csv output.html samples + scancode -clpieu --csv output.html samples The Files scanned are shown in the left sidebar, and the section on the right contains separate tabs for the following: @@ -476,8 +482,6 @@ Comparing Different ``json`` Output Formats .. include:: /scancode-toolkit/rst_snippets/note_snippets/output_htmlapp_search.rst - .. include:: /scancode-toolkit/rst_snippets/warning_snippets/output_htmlapp_dep.rst - .. figure:: data/output_html_app1.png .. figure:: data/output_html_app2.png @@ -494,7 +498,7 @@ Comparing Different ``json`` Output Formats The following code performs a scan on the samples directory, and publishes the results in ``csv`` format:: - ./scancode -lpceiu --csv sample.csv samples + scancode -lpceiu --csv sample.csv samples The first line of the csv file contains the headings, and they are: diff --git a/docs/source/scancode-toolkit/cli-reference/scan-options-post.rst b/docs/source/scancode-toolkit/cli-reference/scan-options-post.rst index a31f0437..82ff546b 100644 --- a/docs/source/scancode-toolkit/cli-reference/scan-options-post.rst +++ b/docs/source/scancode-toolkit/cli-reference/scan-options-post.rst @@ -7,6 +7,10 @@ Post-Scan options activate their respective post-scan plugins which execute the .. include:: /scancode-toolkit/rst_snippets/post_scan_options.rst +---- + +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + To see all plugins available via command line help, use ``--plugins``. .. include:: /scancode-toolkit/rst_snippets/note_snippets/post_scan_plugins.rst @@ -26,7 +30,7 @@ To see all plugins available via command line help, use ``--plugins``. When the following command is executed to scan the ``samples`` directory with this option enabled:: - ./scancode -clpieu --json-pp output.json samples --mark-source + scancode -clpieu --json-pp output.json samples --mark-source Then, the following directories are marked as "Source", i.e. Their "is_source" attribute is changed from "false" to "True". @@ -52,7 +56,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --consolidate + scancode -clpieu --json-pp output.json samples --consolidate The JSON output file is structured as follows:: @@ -129,6 +133,12 @@ To see all plugins available via command line help, use ``--plugins``. [ToDo] Resolve Error and then Add content [ERROR] Check https://github.com/nexB/scancode-toolkit/issues/1758 + .. WARNING:: + + Running the following scan generates an error:: + + ./scancode -clp --json-pp sample_filter_clues.json samples --filter-clues + ---- ``--is-license-text`` Option @@ -146,7 +156,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --license-text --is-license-text + scancode -clpieu --json-pp output.json samples --license-text --is-license-text If the samples directory is scanned with this plugin, the files containing mostly license texts will have the following attribute set to 'true':: @@ -184,7 +194,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --classify --license-clarity-score + scancode -clpieu --json-pp output.json samples --classify --license-clarity-score The "license_clarity_score" will have the following attributes: @@ -243,7 +253,7 @@ To see all plugins available via command line help, use ``--plugins``. Applying License Policies during a ScanCode scan, using the ``--license-policy`` Plugin:: - ./scancode -clipeu --json-pp output.json samples --license-policy policy-file.yml + scancode -clipeu --json-pp output.json samples --license-policy policy-file.yml .. include:: /scancode-toolkit/rst_snippets/note_snippets/post_lic_pol_notsub.rst @@ -307,7 +317,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --summary + scancode -clpieu --json-pp output.json samples --summary The whole JSON file is structured as follows, when the ``--summary`` plugin is applied:: @@ -437,7 +447,7 @@ To see all plugins available via command line help, use ``--plugins``. An example scan using the ``--summary-by-facet`` Plugin:: - ./scancode -clieu --json-pp output.json samples --summary --facet dev="*.java" --facet dev="*.c" --summary-by-facet + scancode -clieu --json-pp output.json samples --summary --facet dev="*.java" --facet dev="*.c" --summary-by-facet .. include:: /scancode-toolkit/rst_snippets/note_snippets/pre_facet_core.rst @@ -584,7 +594,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --classify --summary --summary-key-files + scancode -clpieu --json-pp output.json samples --classify --summary --summary-key-files Running the scan with ``--summary --summary-key-files`` Plugins creates summaries for key files with the same license, copyright and other scan information, at a codebase level (in addition @@ -665,7 +675,7 @@ To see all plugins available via command line help, use ``--plugins``. An example Scan:: - ./scancode -clpieu --json-pp output.json samples --summary-with-details + scancode -clpieu --json-pp output.json samples --summary-with-details .. include:: /scancode-toolkit/rst_snippets/note_snippets/post_summary_details.rst diff --git a/docs/source/scancode-toolkit/cli-reference/scan-options-pre.rst b/docs/source/scancode-toolkit/cli-reference/scan-options-pre.rst index a3c2e1c8..6b6603f8 100644 --- a/docs/source/scancode-toolkit/cli-reference/scan-options-pre.rst +++ b/docs/source/scancode-toolkit/cli-reference/scan-options-pre.rst @@ -7,6 +7,10 @@ Pre-Scan Options ---- +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + +---- + ``--ignore`` Option ------------------- @@ -16,7 +20,7 @@ Pre-Scan Options A sample usage:: - ./scancode --ignore "*.java" samples samples.json + scancode --ignore "*.java" samples samples.json Here, Scancode ignores files ending with `.java`, and continues with other files as usual. @@ -33,7 +37,7 @@ Pre-Scan Options A sample usage:: - ./scancode --include "*.java" samples samples.json + scancode --include "*.java" samples samples.json Here, Scancode selectively scans files that has names ending with `.java`, and ignores all other files. This is basically complementary in behavior to the ``--ignore`` option. @@ -51,6 +55,10 @@ Pre-Scan Options ``--classify``. ``--license-clarity-score`` and ``--summary-key-files`` are Post-Scan Options. + The ``--classify`` option can be used like:: + + scancode -clpieu --json-pp sample_facet.json samples --classify + This option makes ScanCode further classify scanned files/directories, to determine whether they fall in these following categories @@ -97,7 +105,7 @@ Pre-Scan Options You can use the ``--facet`` option in the following manner:: - ./scancode -clpieu --json-pp sample_facet.json samples --facet dev="*.java" --facet dev="*.c" + scancode -clpieu --json-pp sample_facet.json samples --facet dev="*.java" --facet dev="*.c" This adds to the header object, the following attribute:: diff --git a/docs/source/scancode-toolkit/cli-reference/synopsis.rst b/docs/source/scancode-toolkit/cli-reference/synopsis.rst index a7dc33ad..2fb62503 100644 --- a/docs/source/scancode-toolkit/cli-reference/synopsis.rst +++ b/docs/source/scancode-toolkit/cli-reference/synopsis.rst @@ -1,3 +1,5 @@ +.. _cli_synopsis: + Synopsis ======== @@ -5,34 +7,55 @@ ScanCode detects licenses, copyrights, package manifests and direct dependencies in source code and binary files, by scanning the files. This page introduces you to the ScanCode Toolkit Command Line Interface in the following sections: +- Installation - Quickstart - Type of Options - Output Formats - Other Important Documentation +.. _syn_install: + +Installation +------------ + +Scancode-Toolkit installation can be installed from ``pip``, the default Python Package Manager. +However, there are more ways to perform an installation, and refer the following sections for +detailed Instructions on the each of the Installation Methods. + +- :ref:`pip_install` +- :ref:`latest_release_download_install` +- :ref:`source_configure_install` + +.. _synopsis_quickstart: + Quickstart ---------- -The basic usage is:: +The basic command to perform a scan, if Scancode is installed from ``pip``:: - path/to/scancode [OPTIONS] + scancode [OPTIONS] -To scan the ``samples`` directory, the command will be:: +The basic usage in case of a download and configure installation (on Linux/MacOS) is:: - path/to/scancode -clpieu --json-pp path/to/output.json path/to/samples + path/to/scancode [OPTIONS] -.. Note:: +.. include:: /scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst - The includes both the output option and output file name. - For example in ``./scancode -clpieu --json-pp output.json samples``, - ``--json-pp output.json`` is . +Here Scancode scans the file or directory for license, origin and packages and saves +results to FILE(s) using one or more output format option. Error and progress are printed to +stdout. -.. include:: /scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst +To scan the ``samples`` directory distributed with ScanCode-Toolkit, the command will be:: + + scancode -clpieu --json-pp path/to/output.json path/to/samples + +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_output_format.rst -Alternatively, instead of using ``path/to/scancode`` (the path from root of file system) we can -go into the scancode directory (like ``scancode-toolkit-3.1.1``) and then use ``./scancode``. -The same applies for input and output options. To scan a folder ``samples`` inside ScanCode -directory, and output to a file ``output.json`` in the same directory, the command will be:: +Alternatively, in case of download and configure installations, where ``path/to/scancode`` is used +(the path from root of file system) we can go into the scancode directory +(like ``scancode-toolkit-3.1.1``) and then use ``./scancode``. The same applies for input and +output options. To scan a folder ``samples`` inside ScanCode directory, and output to a file +``output.json`` in the same directory, the command will be:: ./scancode -clpieu --json-pp output.json samples @@ -40,12 +63,11 @@ While a scan using absolute paths from the file system root will look like:: home/ayansm/software/scancode-toolkit-3.1.1/scancode -clpieu --json-pp home/ayansm/scan_scan_results/output.json home/ayansm/codebases/samples/ -Throughout the documentation ``./scancode --clpieu --json-pp output.json samples`` will be used -as am example when the terminal is at ``scancode-toolkit-3.1.1`` and we are scanning the -default ``samples`` folder distributed with Scancode-Toolkit. +Commands similar to ``scancode --clpi --json-pp output.json samples`` will be used as examples +throughout the documentation. Here we are inside the ``virtualenv`` where Scancode-Toolkit was +installed by ``pip``, and the default ``samples`` folder is being scanned, which is distributed +by default with Scancode-Toolkit. -Scans the file or directory for license, origin and packages and saves results to -FILE(s) using one or more output format option. Error and progress are printed to stdout. .. _scancode_cli_options: @@ -61,11 +83,16 @@ ScanCode Toolkit Command Line options can be divided into these major sections: - :ref:`cli_pre_scan` - :ref:`cli_post_scan` +Refer the individual pages which are linked to above, for detailed discussions on the Command +Line Options listed under each section. + +.. _synopsis_output: + Output Formats -------------- -The output file format is set by using the various output options. The default output format -is JSON, the entire file being in one line, without whitespace characters. +The output file format is set by using the various output options. The recommended output format +is JSON. If ``--json`` is used, the entire file being in one line, without whitespace characters. The following example scans will show you how to run a scan with each of the result formats. For the scans, we will use the ``samples`` directory provided with the ScanCode Toolkit. @@ -75,9 +102,9 @@ the scans, we will use the ``samples`` directory provided with the ScanCode Tool JSON file output ^^^^^^^^^^^^^^^^ -Scan the ``samples`` directory and save the scan to a JSON file::: +Scan the ``samples`` directory and save the scan to a JSON file (pretty-printed)::: - ./scancode -clpieu --json-pp output.json samples + scancode -clpieu --json-pp output.json samples A sample JSON output file structure will look like:: @@ -264,7 +291,7 @@ file. When the scan is done, open ``samples.html`` in your web browser. :: - ./scancode -clpieu --html output.html samples + scancode -clpieu --html output.html samples .. image:: data/scancode-toolkit-static-html1.png .. image:: data/scancode-toolkit-static-html2.png diff --git a/docs/source/scancode-toolkit/contribute/contrib_dev.rst b/docs/source/scancode-toolkit/contribute/contrib_dev.rst index 580c7479..415ae4bd 100644 --- a/docs/source/scancode-toolkit/contribute/contrib_dev.rst +++ b/docs/source/scancode-toolkit/contribute/contrib_dev.rst @@ -6,6 +6,8 @@ Contributing to Code Development See `CONTRIBUTING.rst `_ for details. +.. _contrib_code_conven: + Code layout and conventions --------------------------- @@ -96,6 +98,8 @@ Another useful option after a test run with some failures is to re-run only the the ``--lf`` option, for instance: ``py.test -vvs --lf`` would only run only test functions that failed in the previous run. +.. _contrib_dev_pip_and_configure: + pip requirements and the configure script ----------------------------------------- diff --git a/docs/source/scancode-toolkit/contribute/contrib_doc.rst b/docs/source/scancode-toolkit/contribute/contrib_doc.rst index 0aa75e90..3a10a7f8 100644 --- a/docs/source/scancode-toolkit/contribute/contrib_doc.rst +++ b/docs/source/scancode-toolkit/contribute/contrib_doc.rst @@ -3,6 +3,88 @@ Contributing to the Documentation ================================= +.. _contrib_doc_setup_local: + +Setup Local Build +----------------- + +To get started, create or identify a working directory on your local machine. + +Open that directory and execute the following command in a terminal session:: + + git clone https://github.com/nexB/scancode-toolkit.git + +That will create an ``/scancode-toolkit`` directory in your working directory. +Now you can install the dependencies in a virtualenv:: + + cd scancode-toolkit + virtualenv -p /usr/bin/python3.6 docs-venv + source docs-venv/bin/activate + +Now, the following prerequisites are installed + +- Sphinx +- sphinx_rtd_theme (the format theme used by ReadTheDocs) +- docs8 (style linter) + +:: + + pip install Sphinx sphinx_rtd_theme doc8 + +Now you can build the HTML documents locally:: + + cd docs + make html + +Assuming that your Sphinx installation was successful, Sphinx should build a local instance of the +documentation .html files:: + + open build/html/index.html + +.. note:: + + In case this command did not work, for example on Ubuntu 18.04 you may get a message like “Couldn’t + get a file descriptor referring to the console”, try: + + :: + + see build/html/index.html + +You now have a local build of the AboutCode documents. + +.. _contrib_doc_share_improvements: + +Share Document Improvements +--------------------------- + +Ensure that you have the latest files:: + + git pull + git status + +Before commiting changes run Continious Integration Scripts locally to run tests. Refer +:ref:`doc_ci` for instructions on the same. + +Follow standard git procedures to upload your new and modified files. The following commands are +examples:: + + git status + git add source/index.rst + git add source/how-to-scan.rst + git status + git commit -m "New how-to document that explains how to scan" + git status + git push + git status + +The Scancode-Toolkit webhook with ReadTheDocs should rebuild the documentation after your +Pull Request is Merged. + +Refer the `Pro Git Book `_ available online for Git tutorials +covering more complex topics on Branching, Merging, Rebasing etc. + +.. _doc_ci: + Continuous Integration ---------------------- @@ -14,6 +96,14 @@ of the documentation : 2. No Broken Links (By Using ``link-check``) 3. Linting Errors (By Using ``Doc8``) +So run these scripts at your local system before creating a Pull Request:: + + cd docs + ./scripts/sphinx_build_link_check.sh + ./scripts/doc8_style_check.sh + +.. _doc_style_docs8: + Style Checks Using ``Doc8`` --------------------------- @@ -22,20 +112,16 @@ How To Run Style Tests In the project root, run the following command:: - $ doc8 --max-line-length 100 docs/source/scancode-toolkit --ignore D000 - -.. note:: - - Only the scancode-toolkit documentation style standards are enforced presently. + $ doc8 --max-line-length 100 docs/source/ --ignore D000 A sample output is:: Scanning... Validating... - docs/source/scancode-toolkit/misc/licence_policy_plugin.rst:37: D002 Trailing whitespace - docs/source/scancode-toolkit/misc/faq.rst:45: D003 Tabulation used for indentation - docs/source/scancode-toolkit/misc/faq.rst:9: D001 Line too long - docs/source/scancode-toolkit/misc/support.rst:6: D005 No newline at end of file + docs/source/misc/licence_policy_plugin.rst:37: D002 Trailing whitespace + docs/source/misc/faq.rst:45: D003 Tabulation used for indentation + docs/source/misc/faq.rst:9: D001 Line too long + docs/source/misc/support.rst:6: D005 No newline at end of file ======== Total files scanned = 34 Total files ignored = 0 @@ -71,8 +157,69 @@ What is checked: - no carriage returns (use UNIX newlines) - D004 - no newline at end of file - D005 -Extra Style Checks ------------------- +.. _doc_interspinx: + +Interspinx +---------- + +ScanCode toolkit documentation uses `Intersphinx `_ +to link to other Sphinx Documentations, to maintain links to other Aboutcode Projects. + +To link sections in the same documentation, standart reST labels are used. Refer +`Cross-Referencing `_ for more information. + +For example:: + + .. _my-reference-label: + + Section to cross-reference + -------------------------- + + This is the text of the section. + + It refers to the section itself, see :ref:`my-reference-label`. + +Now, using Intersphinx, you can create these labels in one Sphinx Documentation and then referance +these labels from another Sphinx Documentation, hosted in different locations. + +You just have to add the following in the ``conf.py`` file for your Sphinx Documentation, where you +want to add the links:: + + extensions = [ + 'sphinx.ext.intersphinx' + ] + + intersphinx_mapping = {'aboutcode': ('https://aboutcode.readthedocs.io/en/latest/', None)} + +To show all Intersphinx links and their targets of an Intersphinx mapping file, run:: + + python -msphinx.ext.intersphinx https://aboutcode.readthedocs.io/en/latest/objects.inv + +.. WARNING:: + + ``python -msphinx.ext.intersphinx https://aboutcode.readthedocs.io/objects.inv`` will give + error. + +This enables you to create links to the ``aboutcode`` Documentation in your own Documentation, +where you modified the configuration file. Links can be added like this:: + + For more details refer :ref:`aboutcode:doc_style_guide`. + +You can also not use the ``aboutcode`` label assigned to all links from aboutcode.readthedocs.io, +if you don't have a label having the same name in your Sphinx Documentation. Example:: + + For more details refer :ref:`doc_style_guide`. + +If you have a label in your documentation which is also present in the documentation linked by +Intersphinx, and you link to that label, it will create a link to the local label. + +For more information, refer this tutorial named +`Using Intersphinx `_. + +.. _doc_style_conv: + +Style Conventions for the Documentaion +-------------------------------------- 1. Headings @@ -97,17 +244,17 @@ Extra Style Checks Do not use underlines that are longer/shorter than the title headline itself. As in: -:: + :: - Correct : + Correct : - Extra Style Checks - ------------------ + Extra Style Checks + ------------------ - Incorrect : + Incorrect : - Extra Style Checks - ------------------------ + Extra Style Checks + ------------------------ .. note:: diff --git a/docs/source/scancode-toolkit/contribute/index.rst b/docs/source/scancode-toolkit/contribute/index.rst index d35ce91e..be58f651 100644 --- a/docs/source/scancode-toolkit/contribute/index.rst +++ b/docs/source/scancode-toolkit/contribute/index.rst @@ -10,3 +10,4 @@ roadmap gsoc17_final_report gsoc19_final_report + long_running_issues diff --git a/docs/source/scancode-toolkit/contribute/long_running_issues.rst b/docs/source/scancode-toolkit/contribute/long_running_issues.rst new file mode 100644 index 00000000..d41db92b --- /dev/null +++ b/docs/source/scancode-toolkit/contribute/long_running_issues.rst @@ -0,0 +1,183 @@ +Long Running Issues - Help Needed +================================= + +Long Running Issues - Scancode Toolkit +-------------------------------------- + +- `Good First Issues List `_ +- `First Timers Only Issues List `_ +- `Add improved documentation for V3.0 `_ +- `Documentation Roadmap `_ +- `Documentation Inconsistencies Tracker `_ + +.. + [Suggestions for LRIs][ToDo] + - People Interested in GSoC (And PRs Opened by Them) + +Roadmaps +-------- + +This section provides a overview of ScanCode developement roadmap, in the sense what maintainers +are considering, what the long-term goals are. + +- :ref:`roadmap` +- `Milestones `_ +- `v3.1 `_ +- `v3.2 `_ +- `Documentation Roadmap `_ + +.. _good_first_issue: + +Good First Issue +---------------- + +A `good first issue `_ +means it's recommended for people who haven't contributed to our codebase before. + +A ``Solving Good First Issue`` is basically an issue based guide for new contributors to +solve a ``Good First Issue``. + +Manitainers of ScanCode Toolkit label small bugs as good first issues for new contributors. + +.. _good_1st_issue_links: + +Important Links +^^^^^^^^^^^^^^^ + +- Issues labeled `good first issue `_ +- The status `list of good first issue `_ +- opening a `Solving Good First Issue `_ + +.. _good_1st_issue_understand_b4_solving: + +What you should understand before Solving a Good First Issue +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- ScanCode Toolkit tracks different kinds of `issues using GitHub `_. +- Some of these issues are labeled ``Good First Issue`` by the Maintainers, as in they + recommend them. +- A list of ``Good First Issue`` is also maintained `here `_. +- The list of ``Good First Issue`` contains a list of Issues, people assigned to them + (or waiting to be assigned) and their status (active/completed). +- Issues here are marked Solved using a `Task List `_. +- Contributors come at this issue (and not at individual issues) and ask the maintainers to + be assigned. +- After being assigned, contributors open another Issue labeled ``Solving Good First Issue`` + so maintainers can Track their Progress on that Issue. +- The task list in the new issue opened (labeled ``Solving Good First Issue``) has to be updated by + the Contributor. + +.. _good_1st_issue_workflow: + +Step by Step Workflow : Contributor's Tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Look at the Issues in `Good First Issues List `_ + These are basically all the `issues labeled Good First Issue `_ + with information on people assigned to the issues, and their status. + +#. Select a ``Good First Issue`` which isn't assigned and comment in the `Good First Issues List `_ + asking the maintainers to assign you. + +#. After you are assigned to that Issue by a maintainer, make sure you are assigned to the one you + requested. It'll look like ``[ ] #1724 @AyanSinhaMahapatra`` where ``#1724`` is the Issue number + and ``AyanSinhaMahapatra`` is your GitHub user ID. + +#. Open an Issue labeled `Solving Good First Issue here `_ . + +#. In this issue you opened, reference the following: + (include ``#[number]`` in comments, like ``#1724``) + + - The Original issue labeled ``Good First Issue`` that you are solving. + - The Pull Request you open later. + - The `Good First Issues List `_ + +#. Follow the Instruction on that Issue (It's a Template you'll just have to mark the Task List + as you complete tasks) + + - **Claim this issue** + - **Reproduce** + - **Create a Unit test** + - **Come up with a Solution** + - **Code your Solution** + - **Run the tests locally** + - **Commit and Push** + - **Start a Pull Request** + - **Make Sure all the Tests Pass** + - **Done** Ask in comments for a review :) + +#. Remind maintainers to close both Issues, after your Pull Request is Merged. + + - The Original issue labeled ``Good First Issue`` + - The issue opened by you labeled ``Solving Good First Issue`` + +.. _first_timers_only: + +First Timers Only +----------------- + +A `first timers only `_. +issue means we've worked to make it more legible to folks who either **haven't contributed to our +codebase before, or even folks who haven't contributed to open source before**. + +Small documenation inconsistencies are first timers only issues for new contributors. + +Important Links +^^^^^^^^^^^^^^^ + +- Issues labeled `first timers only `_ +- The status list of `all first timers only `_ Issues +- All `Documentation Inconsistencies Issue `_. +- opening a `First Timers Only Issue `_ + +What you should understand before Solving a First Timers Only Issue +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- ScanCode Toolkit tracks different kinds of `issues using GitHub `_. +- Some of these issues are documentation inconsistencies `listed here `_. +- A list of ``First Timers Only`` Issues is also `maintained here `_. +- The list of ``First Timers Only`` contains a list of Issues, people assigned to them + (or waiting to be assigned) and their status (active/completed). +- Issues here are marked Solved using a `Task List `_. +- Contributors come at this issue and ask the maintainers to be assigned. +- After being assigned, contributors open another Issue labeled ``First Timers Only`` + so maintainers can Track their Progress. +- The task list has to be updated by the Contributor. + +Step by Step Workflow : Contributor's Tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Look at the Issues in ``First Timers Only`` `List `_ + These are basically all the `issues `_ + labeled ``First Timers Only`` issues, with information on people assigned to the issues, + and their status. + +#. Select a ``First Timers Only`` issue which isn't assigned to anyone yet, and comment on the + `List `_ asking the maintainers to assign + you. + +#. After you are assigned to that Issue by a maintainer, make sure you are assigned to the one you + requested. It'll look like ``[ ] #1724 @AyanSinhaMahapatra`` where ``#1724`` is the Issue number + and ``AyanSinhaMahapatra`` is your GitHub user ID. + +#. Open an Issue labeled `First Timers Only here `_ . + +#. In this issue you opened, reference the following: + (include ``#[number]`` in comments, like ``#1724``) + + - The Original issue labeled ``Good First Issue`` that you are solving. + - The Pull Request you open later. + - The `Good First Issues List `_ + +#. Follow the Instruction on that Issue (It's a Template you'll just have to mark the Task List + as you complete tasks) + + - **Issue Claimed** + - **Review Guidelines** + - **Refer Documentation** + - **Build Docs Locally** + - **Update** + - **Commit and Push** + - **Start a Pull Request** + - **Make Sure all the Tests Pass** + - **Done** Ask in comments for a review :) diff --git a/docs/source/scancode-toolkit/contribute/roadmap.rst b/docs/source/scancode-toolkit/contribute/roadmap.rst index 2ba972a4..167277db 100644 --- a/docs/source/scancode-toolkit/contribute/roadmap.rst +++ b/docs/source/scancode-toolkit/contribute/roadmap.rst @@ -1,3 +1,5 @@ +.. _roadmap: + Roadmap ======= diff --git a/docs/source/scancode-toolkit/explanations/overview.rst b/docs/source/scancode-toolkit/explanations/overview.rst index 0ad19db2..c3df546c 100644 --- a/docs/source/scancode-toolkit/explanations/overview.rst +++ b/docs/source/scancode-toolkit/explanations/overview.rst @@ -1,6 +1,8 @@ Overview ======== +.. _explain_how_scancode_works: + How does ScanCode work? ----------------------- diff --git a/docs/source/scancode-toolkit/getting-started/docs.rst b/docs/source/scancode-toolkit/getting-started/docs.rst deleted file mode 100644 index 930dd5b4..00000000 --- a/docs/source/scancode-toolkit/getting-started/docs.rst +++ /dev/null @@ -1,19 +0,0 @@ -Documentation -============= - -This page provides an index of current ScanCode user documentation. - -Documentation -------------- - -The ScanCode toolkit documentation lives at aboutcode.readthedocs.io/en/latest/scancode-toolkit/. - -Contribute to Docs ------------------- - -See :ref:`contrib_doc_dev` for more details. - -Google Summer of Docs ---------------------- - -See :ref:`GSoD2019` for more details. diff --git a/docs/source/scancode-toolkit/getting-started/ide-config.rst b/docs/source/scancode-toolkit/getting-started/ide-config.rst index 5c9dd80e..9ebfb95e 100644 --- a/docs/source/scancode-toolkit/getting-started/ide-config.rst +++ b/docs/source/scancode-toolkit/getting-started/ide-config.rst @@ -1,3 +1,5 @@ +.. _ide_config: + IDE Configuration ================= diff --git a/docs/source/scancode-toolkit/getting-started/index.rst b/docs/source/scancode-toolkit/getting-started/index.rst index bc736d33..a2f44536 100644 --- a/docs/source/scancode-toolkit/getting-started/index.rst +++ b/docs/source/scancode-toolkit/getting-started/index.rst @@ -7,10 +7,9 @@ home install ide-config - docs whats-new + newcomer .. [ToAdd] - releases Link to Changelog diff --git a/docs/source/scancode-toolkit/getting-started/install.rst b/docs/source/scancode-toolkit/getting-started/install.rst index db6918c4..5ee32c74 100644 --- a/docs/source/scancode-toolkit/getting-started/install.rst +++ b/docs/source/scancode-toolkit/getting-started/install.rst @@ -1,11 +1,33 @@ Comprehensive Installation ========================== -ScanCode requires Python 2.7.x and is tested on Linux, Mac, and Windows. Make sure Python 2.7 -is installed first. +The fastest way to install Scancode-Toolkit is by using ``pip``. You can also install +ScanCode-Toolkit by compiling it from source or by Downloading and Configuring the +latest release from GitHub. + +- :ref:`pip_install` +- :ref:`latest_release_download_install` +- :ref:`source_configure_install` + +.. NOTE:: + + After ``pip install``, you can perform a scan using only:: + + scancode [OPTIONS] + + This is unlike other install methods where path to scancode is provided by using + ``path/to/scancode``, or by using ``./scancode`` inside the Scancode install directory. + +--- + +Before Installing +----------------- + +ScanCode requires either Python 3.6.x or Python 2.7.x and is tested on Linux, Mac, and Windows. +Make sure Python 2.7 or Python 3.6 is installed first. System Requirements -------------------- +^^^^^^^^^^^^^^^^^^^ - Hardware : ScanCode will run best with a modern X86 processor and at least 2GB of RAM and 250MB of disk. @@ -20,13 +42,18 @@ System Requirements .. _install_prerequisites: Prerequisites -------------- -ScanCode needs a Python 2.7 interpreter. +^^^^^^^^^^^^^ -- On Linux: Use your package manager to install python2.7. If Python 2.7 is not available from - your package manager, you must compile it from sources. For instance, visit +ScanCode needs a Python 3.6 or a Python 2.7 interpreter. + +.. Note:: + + ScanCode currently doesn't support Python 3.7.x, though support will be added soon. + +- On Linux: Use your package manager to install ``python2.7`` or ``python3.6``. If they are not + available from your package manager, you must compile it from sources. For instance, visit https://github.com/dejacode/about-code-tool/wiki/BuildingPython27OnCentos6 for instructions - to compile Python from sources on Centos. + to compile Python 2.7 from sources on Centos. - On Ubuntu 12.04, 14.04 and 16.04, you will need to install these packages first: ``python-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev`` @@ -48,34 +75,85 @@ ScanCode needs a Python 2.7 interpreter. python just in c:\python27\python.exe). See the Windows installation section for more installation details. +.. Note:: + + 64-bit Python interpreters (x86) are currently not supported by Scancode for Python 2.7 in + Windows. Use 32-bit Python isntead, even with 64-bit Windows. + - On Mac: Download and install Python from this url: https://www.python.org/ftp/python/2.7.13/python-2.7.13-macosx10.6.pkg -Do not use Unicode, non-ASCII in your installation Path -------------------------------------------------------- -There is a bug in underlying libraries that prevent this. +.. WARNING:: + + Do not use Unicode, non-ASCII in your installation Path if you are using a Python 2.7 interpreter. + +--- + +.. _pip_install: + +Installation by ``pip`` +----------------------- + +Scancode Toolkit can be easily installed using ``pip``. The steps are: + +#. Create a Python 2.7 or Python 3.6 Virtual Environment:: + + virtualenv -p /usr/bin/python3.6 venv-scancode + +#. Activate the Virtual Environment you just created:: + + source venv-scancode/bin/activate + +#. Run ``pip install scancode-toolkit`` to Install Scancode. + +.. NOTE:: + + If you use Python 2.7, scancode-toolkit Version 3.0.2 is installed by default. For Python 3 + the latest version of Scancode Toolkit is installed by default. + +.. WARNING:: + + Requesting a specific version through ``pip install`` for Python 3 will give Errors if the + Version isn't 3.1.x or later. + +To uninstall, run ``pip uninstall scancode-toolkit``. -.. _install_scancode: +--- -.. - [ToDo] - Add "pip isntall" option - Update to Python 3.6 +.. _latest_release_download_install: + +Download and Configure latest Release +------------------------------------- Installation on Linux and Mac ------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Get the Scancode Toolkit tarball or zip archive of a specific Version by going to the +`GitHub Release Page `_ + +For example, Version 3.1.1 tarball or .zip archive can be obtained from +`Toolkit Release 3.1.1 `_ +under assets options. Download and extract the Archive from command line: + +For ``.zip`` archive:: + + unzip scancode-toolkit-3.1.1.zip + +For ``.tar.bz2`` archive:: + + tar -xvf scancode-toolkit-3.1.1.tar.bz2 -Download and extract the latest ScanCode release from: -https://github.com/nexB/scancode-toolkit/releases/ +Or Right Click and select "Extract Here". -Open a terminal in the extracted directory and run:: +Check whether the :ref:`install_prerequisites` are installed. Open a terminal in the extracted +directory and run:: ./scancode --help -This will configure ScanCode and display the command line help. +This will configure ScanCode and display the command line :ref:`cli_help_text`. Installation on Windows ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ - Download the latest ScanCode release zip file from https://github.com/nexB/scancode-toolkit/releases/ @@ -136,7 +214,59 @@ Installation on Windows - The installation is complete. Un-installation ---------------- +^^^^^^^^^^^^^^^ - Delete the directory in which you extracted ScanCode. - Delete any temporary files created in your system temp directory under a ScanCode directory. + +--- + +.. _source_configure_install: + +Build From Source +----------------- + +You can also download the Scancode Toolkit Source Code and build from it yourself. This is how you +would want to do it if: + +- You are Adding new patches to Scancode and want to test it. +- You want to test a specific Version/Checkpoint/Branch from the VCS + + +Download the ScanCode-Toolkit Source Code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you don't have the ScanCode Toolkit Source Code downloaded, get it from it's official Repository +(Downloaded as a .zip file) or run the following:: + + git clone https://github.com/nexB/scancode-toolkit.git + cd scancode-toolkit + +Now, by default the files are checked out to the develop branch, but you can jump to any checkpoint +using the following command:: + + git checkout master + +Here, ``master`` branch has the latest release of Scancode-Toolkit. You can also check out to any +of the following: + +- Branches (Locally created or already present) +- Tags (essentially Version Numbers) [Example - ``v3.1.1``, ``v3.1.0`` etc] +- Commits (use the shortened commit hash) [Example - ``4502055``, ``f276398`` etc] + +Configure the build +^^^^^^^^^^^^^^^^^^^ + +ScanCode use the Configure scripts to install a virtualenv, install required packaged dependencies +as pip requirements and more configure tasks such that ScanCode can be installed in a +self-contained way with no network connectivity required. + +Open a terminal, clone the scancode-toolkit repository, cd to the clone directory and run:: + + ./configure + +On Windows open a command prompt, cd to the clone directory and run instead:: + + configure + +Now you are ready to use the freshly configured scancode-toolkit. diff --git a/docs/source/scancode-toolkit/getting-started/newcomer.rst b/docs/source/scancode-toolkit/getting-started/newcomer.rst new file mode 100644 index 00000000..ec537ec4 --- /dev/null +++ b/docs/source/scancode-toolkit/getting-started/newcomer.rst @@ -0,0 +1,311 @@ +Are you new to Scancode-Toolkit? +================================ + +This is the perfect place to start, if you are new to ScanCode-Toolkit. Have a quick look at the +table of contents below, as these are the main sections you might need help on. These sections +have extensive links to other important documentation pages, and make sure you go through them +all. + +Table of Contents +----------------- + +#. :ref:`newcomer_try_scancode` + + - :ref:`newcomer_before` + - :ref:`newcomer_scan_codebase` + - :ref:`newcomer_use_scancode_better` + - :ref:`newcomer_all_tutorials` + - :ref:`newcomer_whats_new` + +#. :ref:`newcomer_learn_scancode` + + - :ref:`newcomer_cli_ref` + - :ref:`newcomer_explanations` + - :ref:`newcomer_plugins` + +#. :ref:`newcomer_contribute` + + - :ref:`newcomer_contribute_general_info` + - :ref:`newcomer_code` + - :ref:`newcomer_good_first_issue` + - :ref:`newcomer_add_functionalirty` + - :ref:`newcomer_update_docs` + - :ref:`newcomer_gsoc_gsod` + +---- + +.. _newcomer_try_scancode: + +Try ScanCode Toolkit +-------------------- + +This section is about using the Scancode-Toolkit, i.e. Performing a scan on a codebase/files to +determine their license, copyrights and other information, according to your requirements. + +#. The :ref:` newcomer_scan_codebase` section helps you with configuring your virtual environment, + installing Scancode and performing a basic scan, and subsequently visualize the results. + +#. The :ref:`newcomer_use_scancode_better` section helps you customize the scan according to your + requirements, and better understand the advanced features you can use. + +#. The :ref:`newcomer_all_tutorials` is essentially an exhaustive list of all Tutorials and How To's + with a brief description on what they help you to achieve. + +.. _newcomer_before: + +Before you start using Scancode +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. You need to make sure :ref:`install_prerequisites` are installed, and a `virtualenv `_ + is created. + +#. Now you can either run ``pip install scancode-toolkit`` like that in the + :ref:`pip_install` documentation, or follow the instructions for the + :ref:`latest_release_download_install` method. Alternatively, you can also + :ref:`source_configure_install`. + +#. Run ``scancode -h`` to make sure Scancode was installed properly. + If this shows any Error, refer the `Common Installation Errors Issue `_ + for common errors. The documentation also has tips on :ref:`ide_config`. + +.. note:: + + Refer :ref:`synopsis_quickstart` to make sure you are using the scan command correctly. + +.. _newcomer_scan_codebase: + +Scan a Codebase +^^^^^^^^^^^^^^^ + +Once you are all set up with Scancode Toolkit, i.e. Running ``scancode -h`` shows the +:ref:`cli_help_text`, you can start scanning files or a codebase. + +#. Refer :ref:`synopsis_quickstart` for commonly used scan commands, and commonly used + :ref:`synopsis_output`. (The recommended output format is ``JSON``) + +#. Refer `this section `_ for Extractcode Options. + +#. :ref:`how_to_run_a_scan` is a sample tutorial for absolute beginners, to walk them through the + process of running a scan. Follow this tutorial and perform a scan on the ``sample`` folder + distributed with ScanCode, or any file/folder of your choice. Avoid advanced options, and just + follow the basic instructions. + +#. ScanCode generates output files with scan results. You can visualize ``JSON`` result files using + `Scancode Workbench `_. Follow this tutorial :ref:`how_to_visualize_scan_results` + to visualize the scan results. + +.. _newcomer_use_scancode_better: + +Use ScanCode Better +^^^^^^^^^^^^^^^^^^^ + +#. Go through all the options in the page :ref:`cli_list_options`, to know about Scancode Command + Line options. You can then modify the Scan according to your requirements. + +.. _newcomer_all_tutorials: + +All Tutorials/How-Tos +^^^^^^^^^^^^^^^^^^^^^ + +The Tutorials are: + +#. :ref:`how_to_run_a_scan` +#. :ref:`how_to_visualize_scan_results` +#. :ref:`how_to_set_what_scan_detects` +#. :ref:`how_to_extract_archives` +#. :ref:`how_to_specify_output_format` +#. :ref:`how_to_add_post_scan_plugin` + +The How-To's are: + +#. :ref:`add_new_license_for_det` +#. :ref:`add_new_license_det_rule` + +.. _newcomer_whats_new: + +ScanCode Versions +^^^^^^^^^^^^^^^^^ + +#. You can see all Scancode Toolkit versions on the `GitHub release page `_. +#. Refer :ref:`whats_new_this_release` to know more about the latest release. +#. You can also refer the `CHANGELOG `_ for more information on specific releases. +#. If you want to use/test a specific version of Scancode Toolkit, you can follow the instructions + in :ref:`source_configure_install` docs. + +---- + +.. _newcomer_learn_scancode: + +Learn more about ScanCode Toolkit +--------------------------------- + +Here we give an introduction on the Scancode Toolkit Documentation Sections that can help you to +learn more about Scancode Toolkit. + +.. _newcomer_cli_ref: + +CLI Reference +^^^^^^^^^^^^^ + +This section contains a complete guide to ScanCode Toolkit Command Line options, i.e. What the +command-line options are, how different options affect the scan and outputs, how to use these +options and examples of their use cases. + +Now this section has three types of pages: + +#. The :ref:`cli_synopsis` page and the :ref:`how_to_run_a_scan` page as summaries. +#. An exhaustive list of all Command Line Options at :ref:`cli_list_options` +#. All the other pages detailing the :ref:`scancode_cli_options` + +Note that the page for one type of options also has a short list of all the options detailed on +that page in the beginning. The :ref:`cli_list_options` page just has all of them together, and +also the extractcode options. + +.. _newcomer_explanations: + +How Scancode Works +^^^^^^^^^^^^^^^^^^ + +This section has documentation on :ref:`explain_how_scancode_works`. + +.. _newcomer_plugins: + +Plugins +^^^^^^^ + +Plugins are an integral part of ScanCode Toolkit in the sense they are used to easily extend +Scancode capabilities, and developers can code their own plugins according to their requirements. + +This section has documentation on: + +#. The :ref:`plugin_arch` +#. The :ref:`license_policy_plugin` +#. All :ref:`plugin_tutorials` + +---- + +.. _newcomer_contribute: + +Contribute +---------- + +If you are looking to Contribute to Scancode Toolkit, this is where you start. + +.. _newcomer_contribute_general_info: + +General Information +^^^^^^^^^^^^^^^^^^^ + +#. Also refer the `Contribution `_ page here. +#. For more Project Ideas, refer :ref:`contributor_project_ideas`. +#. Before committing your work, make sure you have read this post on :ref:`good_commit_messages`. + +.. _newcomer_code: + +Contribute Code +^^^^^^^^^^^^^^^ + +If you haven't contributed to Scancode Toolkit refer :ref:`newcomer_good_first_issue`. + +To determine where to contribute, you can refer: + +#. ScanCode Toolkit tracks issues via the `GitHub Issue tracker `_ +#. Broad `milestones `_ for upcoming versions are also maintained. + +And documentation related to contributing code can be referred at :ref:`contrib_code_dev`. The main +sections are: + +#. :ref:`contrib_code_conven` +#. :ref:`scancode_toolkit_developement_running_tests` +#. :ref:`contrib_dev_pip_and_configure` + +.. _newcomer_good_first_issue: + +Good First Issues +^^^^^^^^^^^^^^^^^ + +A `good first issue `_ +means it's recommended for people who haven't contributed to Scancode Toolkit before. + +#. Refer the detailed documentation for :ref:`good_first_issue`. +#. :ref:`good_1st_issue_links` for Good First issues are also compiled. +#. :ref:`good_1st_issue_understand_b4_solving` +#. :ref:`good_1st_issue_workflow` + +A `first timers only `_ +issue means we've worked to make it more legible to folks who either **haven't contributed to our +codebase before, or even folks who haven't contributed to open source before**. + +Refer the detailed documentation for :ref:`first_timers_only`. + +.. _newcomer_add_functionalirty: + +Add new Functionality/Enhancement to ScanCode +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There are two main paths you can follow to add a new functionality to Scancode. +They are: + +#. Add the functionality to Scancode itself. +#. Add plugins if the functionality is very much application dependent. + +Refer `enhancement issues `_ for the first type of +enhancements. If you want to add a plugin to implement the functionality, refer all the +:ref:`plugin_tutorials`. + +.. _newcomer_update_docs: + +Update our Documentation +^^^^^^^^^^^^^^^^^^^^^^^^ + +Maintaining a comprehensive, accurate, updated and effective documentation is very important +as that directly affects the acceptability of Scancode Toolkit. + +To contribute to Scancode Toolkit Documentation, first refer the :ref:`contrib_doc_dev` section. + +The sections in this page cover the following: + +#. :ref:`contrib_doc_setup_local` +#. :ref:`contrib_doc_share_improvements` +#. :ref:`doc_ci` system for the Documentation +#. :ref:`doc_style_docs8` +#. :ref:`doc_interspinx` +#. :ref:`doc_style_conv` + +You can contribute to the following Open Issues on documentation. + +#. `Issues with label Documentation `_ +#. `Documentation Inconsistencies Tracker `_ +#. `ScanCode Toolkit Documentation Roadmap `_ +#. `First Timers Only Issues List `_ + +.. note:: + + Refer :ref:`improve_docs` to report Documentation Errors or to request Improvements. + +Also, consider contributing to other Aboutcode Project Documentations, as they need more support. + +.. _newcomer_gsoc_gsod: + +Participate in GSoC/GSoD +^^^^^^^^^^^^^^^^^^^^^^^^ + +If you want to participate in any of the two programs: + +- `Google Summer of Code `_ +- `Google Season of Docs `_ + +Then: + +#. Keep an eye out for Application Timelines. +#. Solve multiple of these :ref:`good_first_issue` to demonstrate your skills, and improve your + chances of selection. +#. Refer the Projects Ideas list for details on tentative projects. + + - :ref:`GSoC2019` + - :ref:`GSoD2019` + +#. Remain active in Gitter and talk with the organization mentors well ahead of the deadlines. +#. Select projects according to your skills and finalize project proposals. +#. Discuss your proposals extensively with corresponding mentors. +#. Apply for the Programs well before the Deadline. diff --git a/docs/source/scancode-toolkit/getting-started/whats-new.rst b/docs/source/scancode-toolkit/getting-started/whats-new.rst index 192d2176..e847af3c 100644 --- a/docs/source/scancode-toolkit/getting-started/whats-new.rst +++ b/docs/source/scancode-toolkit/getting-started/whats-new.rst @@ -1,3 +1,5 @@ +.. _whats_new_this_release: + What's New in This Release? =========================== @@ -20,7 +22,7 @@ This is the first 3.1 release with the best, fastest and most efficient ScanCode This release contains many improvements, fixes and new features including breaking API changes (when compared to 2.2.x). See the `CHANGELOG `_ for details. This release also comes with a Full Documentation hosted at -aboutcode.readthedocs.io/en/latest/scancode-toolkit/. +`aboutcode.readthedocs.io `_. To install, download scancode-toolkit-3.1.1.zip or scancode-toolkit-3.1.1.tar.bz2 from the Downloads section below and follow installation instructions in the `README `_. @@ -38,19 +40,32 @@ Brief Summary Of Changes Documentation Support ^^^^^^^^^^^^^^^^^^^^^ -``pip install`` -^^^^^^^^^^^^^^^ +From Scancode 3.1.1 Scancode comes with improved and comprehensive Documentation Support. + +The documentation is available both offline (distributed with Scancode) and online, +at ``scancode-toolkit.readthedocs.io/en/latest/``. The documentation in GitHub wiki's will be +deprecated hereafter. The new documentation has improved support in terms of: + +#. Consistent with the Latest Version +#. Command Line Interface Reference +#. Getting Started Support for Newcomers +#. Restructured to fit into the recommended `4 Category Doc Format `_ +#. Updated Tutorials/How To's +#. Updated Plugin Support + +This results in a much better documentation experience in Users and even contributors. + +``pip install`` Support +^^^^^^^^^^^^^^^^^^^^^^^ + +Now, ``pip install`` is the recommended install method, across all platforms. This greatly +simplifies the install process, and is much faster and easier than the Download and Configure +method, for non-developer users. Python 3 Support ^^^^^^^^^^^^^^^^ -Explanations ------------- - -.. - [ToDo] - The above sub-sections will have to be structured and elaborated in the following manner:- +Python 3 is now officially supported by Scancode-Toolkit. - 1. Main new feature upgrades from the last release (Identical to changelog) - 2. A brief summary of the changes (Explaining previous changes, links to other material) - 3. "If your work includes *this* function you should upgrade to this release" +This also means improved Unicode support, so it's easy to translate strings from Unicode to +other languages. And as Scancode has users in more than 100 languages, this is a major improvement. diff --git a/docs/source/scancode-toolkit/how-to-guides/add_new_license.rst b/docs/source/scancode-toolkit/how-to-guides/add_new_license.rst index 6e73632c..b509f995 100644 --- a/docs/source/scancode-toolkit/how-to-guides/add_new_license.rst +++ b/docs/source/scancode-toolkit/how-to-guides/add_new_license.rst @@ -1,3 +1,5 @@ +.. _add_new_license_for_det: + How To Add a New License for Detection ====================================== diff --git a/docs/source/scancode-toolkit/plugins/index.rst b/docs/source/scancode-toolkit/plugins/index.rst index 3091df17..01d4ac3c 100644 --- a/docs/source/scancode-toolkit/plugins/index.rst +++ b/docs/source/scancode-toolkit/plugins/index.rst @@ -1,5 +1,5 @@ **Plugins** -================= +=========== .. toctree:: :maxdepth: 2 diff --git a/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst b/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst index d6b31e67..59bad566 100644 --- a/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst +++ b/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst @@ -1,3 +1,5 @@ +.. _plugin_tutorials: + Plugin Tutorials ================ diff --git a/docs/source/scancode-toolkit/rst_snippets/custom_output_format.rst b/docs/source/scancode-toolkit/rst_snippets/custom_output_format.rst index 8e921dc7..c5ba21f8 100644 --- a/docs/source/scancode-toolkit/rst_snippets/custom_output_format.rst +++ b/docs/source/scancode-toolkit/rst_snippets/custom_output_format.rst @@ -36,7 +36,7 @@ Now I can run ScanCode using my newly created template: :: - $ ./scancode -clpeui --custom-output output.json --custom-template template.html samples + $ scancode -clpeui --custom-output output.json --custom-template template.html samples Scanning files... [####################################] 46 Scanning done. diff --git a/docs/source/scancode-toolkit/rst_snippets/extract.rst b/docs/source/scancode-toolkit/rst_snippets/extract.rst index ad7c62f2..89e8aa76 100644 --- a/docs/source/scancode-toolkit/rst_snippets/extract.rst +++ b/docs/source/scancode-toolkit/rst_snippets/extract.rst @@ -9,7 +9,7 @@ To extract the packages in the ``samples`` directory :: - ./extractcode samples + extractcode samples This extracts the zlib.tar.gz package: diff --git a/docs/source/scancode-toolkit/rst_snippets/improve_docs.rst b/docs/source/scancode-toolkit/rst_snippets/improve_docs.rst new file mode 100644 index 00000000..04a3a2d6 --- /dev/null +++ b/docs/source/scancode-toolkit/rst_snippets/improve_docs.rst @@ -0,0 +1,7 @@ +Something Missing? +------------------ + +If something is missing in the documentation or if you found some part confusing, please file +an `issue `_ with your suggestions for +improvement. Use the "Documentation Improvement" template. +Your help makes ScanCode docs better, we love hearing from you! diff --git a/docs/source/scancode-toolkit/rst_snippets/note_snippets/basic_clpieu.rst b/docs/source/scancode-toolkit/rst_snippets/note_snippets/basic_clpieu.rst index 2c5b39a5..2ad8b8e7 100644 --- a/docs/source/scancode-toolkit/rst_snippets/note_snippets/basic_clpieu.rst +++ b/docs/source/scancode-toolkit/rst_snippets/note_snippets/basic_clpieu.rst @@ -2,10 +2,10 @@ Unlike previous 2.x versions, -c, -l, and -p are not default. If any of combination of these options are used, ScanCode only performs that specific task, and not the others. - ``./scancode -e`` only scans for emails, and doesn't scan for copyright/license/packages/general + ``scancode -e`` only scans for emails, and doesn't scan for copyright/license/packages/general information. .. note:: These options, i.e. -c, -l, -p, -e, -u, and -i can be used together. As in, instead of - ``./scancode -c -i -p``, you can write ``./scancode -cip`` and it will be the same. + ``scancode -c -i -p``, you can write ``scancode -cip`` and it will be the same. diff --git a/docs/source/scancode-toolkit/rst_snippets/note_snippets/output_samples.rst b/docs/source/scancode-toolkit/rst_snippets/note_snippets/output_samples.rst index 50aac0b9..dbea3b44 100644 --- a/docs/source/scancode-toolkit/rst_snippets/note_snippets/output_samples.rst +++ b/docs/source/scancode-toolkit/rst_snippets/note_snippets/output_samples.rst @@ -1,7 +1,7 @@ .. note:: You can Output Scan Results in two different file formats simultaniously in one Scan. An - example - ``./scancode -clpieu --json-pp output.json --html output.html samples``. + example - ``scancode -clpieu --json-pp output.json --html output.html samples``. .. note:: diff --git a/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst b/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst new file mode 100644 index 00000000..dd6f0cab --- /dev/null +++ b/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst @@ -0,0 +1,4 @@ +.. note:: + + For more information on the Scan Command for Various Installation Methods/Operating Systems, + refer :ref:`syn_install`. diff --git a/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_output_format.rst b/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_output_format.rst new file mode 100644 index 00000000..eff52527 --- /dev/null +++ b/docs/source/scancode-toolkit/rst_snippets/note_snippets/synopsis_output_format.rst @@ -0,0 +1,10 @@ +.. Note:: + + The includes both the output option and output file name. + For example in the command ``scancode -clpieu --json-pp output.json samples``, + ``--json-pp output.json`` is . + +.. WARNING:: + + There isn't a "Default" output option in Versions 3.x onwards, you have to + specify explicitly. diff --git a/docs/source/scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst b/docs/source/scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst index 48b7fc47..3656acc7 100644 --- a/docs/source/scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst +++ b/docs/source/scancode-toolkit/rst_snippets/tip_snippets/synopsis_quickstart.rst @@ -1,8 +1,4 @@ .. Tip:: - On Windows use ``scancode`` instead of ``path/to/scancode``. - -.. WARNING:: - - There isn't a "Default" output option in Versions 3.x onwards, you have to - specify explicitly. + On Windows use ``scancode`` instead of ``path/to/scancode``, irrespective of + the installation method. diff --git a/docs/source/scancode-toolkit/tutorials/how_to_extract_archives.rst b/docs/source/scancode-toolkit/tutorials/how_to_extract_archives.rst index 207a93eb..39c77b11 100644 --- a/docs/source/scancode-toolkit/tutorials/how_to_extract_archives.rst +++ b/docs/source/scancode-toolkit/tutorials/how_to_extract_archives.rst @@ -1,3 +1,5 @@ +.. _how_to_extract_archives: + How To Extract Archives ======================= @@ -13,6 +15,6 @@ Usage: :: - ./extractcode [OPTIONS] + extractcode [OPTIONS] .. include:: /scancode-toolkit/rst_snippets/extract.rst diff --git a/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst b/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst index 76d6a3bb..4d3452c8 100644 --- a/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst +++ b/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst @@ -1,3 +1,5 @@ +.. _how_to_specify_output_format: + How to specify Scancode Output Format ===================================== @@ -5,17 +7,19 @@ A basic overview of formatting Scancode Output is presented here. More information on :ref:`cli_output_format`. +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + JSON ---- If you want JSON output of ScanCode results, you can pass the ``--json`` argument to ScanCode. The following commands will output scan results in a formatted json file: -* ``./scancode --json /path/to/output.json /path/to/target/dir`` +* ``scancode --json /path/to/output.json /path/to/target/dir`` -* ``./scancode --json-pp /path/to/output.json /path/to/target/dir`` +* ``scancode --json-pp /path/to/output.json /path/to/target/dir`` -* ``./scancode --json-lines /path/to/output.json /path/to/target/dir`` +* ``scancode --json-lines /path/to/output.json /path/to/target/dir`` To compare the JSON output in different formats refer :ref:`comparing_json`. @@ -27,9 +31,9 @@ HTML If you want HTML output of ScanCode results, you can pass the ``--html`` argument to ScanCode. The following commands will output scan results in a formatted HTML page or simple web application: -* ``./scancode --html /path/to/output.html /path/to/target/dir`` +* ``scancode --html /path/to/output.html /path/to/target/dir`` -* ``./scancode --html-app /path/to/output.html /path/to/target/dir`` +* ``scancode --html-app /path/to/output.html /path/to/target/dir`` For more details on the HTML output format refer :ref:`output_html`. diff --git a/docs/source/scancode-toolkit/tutorials/how_to_run_a_scan.rst b/docs/source/scancode-toolkit/tutorials/how_to_run_a_scan.rst index b3925f57..e7ec7235 100644 --- a/docs/source/scancode-toolkit/tutorials/how_to_run_a_scan.rst +++ b/docs/source/scancode-toolkit/tutorials/how_to_run_a_scan.rst @@ -4,56 +4,43 @@ How to Run a Scan In this simple tutorial example, we perform a basic scan on the ``samples`` directory distributed by default with Scancode. -.. - [ToDo] - Add Windows/MacOS Support and remove this WARNING. +Prerequisites +------------- -.. WARNING:: - - This tutorial is for Linux based systems presently. Additional Help for Windows/MacOS will be - added. +Refer :ref:`install_prerequisites` to make sure the correct Python Interpreters and other +prerequisites are satisfied. Setting up a Virtual Environment -------------------------------- -Scancode Toolkit 3.1.1 and Workbench 3.1.0 is not compatible with python 3.x so we will create a -`virtual environment `_ using the ``Virtualenv`` -tool with a python 2.7 interpreter. +ScanCode Toolkit supports Python 3 in 3.2.x and later versions, so if you are using 3.2.x or later +versions, you should create a `virtual environment `_ +using the ``Virtualenv`` tool with a python 3.6 interpreter. -The following commands set up and activate the Virtual Environment ``venv-scan3.1.1``: +The following commands set up and activate the Virtual Environment ``venv-scancode-py3``: :: - virtualenv -p /usr/bin/python2.7 venv-scan3.1.1 - source venv-scan3.1.1/bin/activate - -.. - [ToDo] - Update from Python 2.7 to Python 3.6 - -Setting up Scancode Toolkit ---------------------------- + virtualenv -p /usr/bin/python3.6 venv-scancode-py3 + source venv-scancode-py3/bin/activate -Get the Scancode Toolkit Version 3.1.1 tarball or .zip archive from the -`Toolkit GitHub Release `_ Page under -assets options. Download and extract the Archive from command line: -For .zip archive:: +If you are using Scancode Toolkit 3.1.0 and earlier versions, they are not compatible with +Python 3.x so you should create the virtual environment with a python 2.7 interpreter:: - unzip scancode-toolkit-3.1.1.zip + virtualenv -p /usr/bin/python2.7 venv-scancode-py2 + source venv-scancode-py2/bin/activate -For .tar.bz2 archive:: - - tar -xvf scancode-toolkit-3.1.1.tar.bz2 +Setting up Scancode Toolkit +--------------------------- -Or Right Click and select "Extract Here". +Get ScanCode Toolkit from ``pip``:: -Check whether the :ref:`install_prerequisites` are installed. Open a terminal in the extracted -directory and run:: + pip install scancode-toolkit - ./scancode --help +.. Note:: -This will configure ScanCode and display the command line :ref:`cli_help_text`. + You can install a specific version of Scancode Toolkit like ``scancode-toolkit==3.1.1``. Looking into Files ------------------ @@ -71,7 +58,7 @@ Performing Extraction To extract the packages inside ``samples`` directory:: - ./extractcode samples + extractcode samples This extracts the zlib.tar.gz package: @@ -118,7 +105,7 @@ Running The Scan Now, run the scan with the options decided:: - ./scancode -clpeui -n 2 --ignore "*.java" --json-pp sample.json samples + scancode -clpeui -n 2 --ignore "*.java" --json-pp sample.json samples A Progress report is shown:: diff --git a/docs/source/scancode-toolkit/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst b/docs/source/scancode-toolkit/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst index a938a2a9..a9d3787c 100644 --- a/docs/source/scancode-toolkit/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst +++ b/docs/source/scancode-toolkit/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst @@ -1,3 +1,5 @@ +.. _how_to_set_what_scan_detects: + How to set what will be detected in Scan ======================================== @@ -16,6 +18,8 @@ be saved in the JSON format, which can be loaded into Scancode Workbench for vis :ref:`how_to_visualize_scan_results` for more information. Another output format option is a static html file. See :ref:`cli_output_format` for more information. +.. include:: /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst + Scan for all clues: ^^^^^^^^^^^^^^^^^^^ @@ -23,7 +27,7 @@ To scan for licenses, copyrights, urls, emails, package information, and file in :: - ./scancode -clipeu --json output.json samples + scancode -clipeu --json output.json samples Scan for license and copyright clues: @@ -31,7 +35,7 @@ Scan for license and copyright clues: :: - ./scancode -cl --json-pp output.json samples + scancode -cl --json-pp output.json samples Scan for emails and URLs: @@ -39,7 +43,7 @@ Scan for emails and URLs: :: - ./scancode -eu --json-pp output.json samples + scancode -eu --json-pp output.json samples Scan for package information: @@ -47,7 +51,7 @@ Scan for package information: :: - ./scancode -p --json-pp output.json samples + scancode -p --json-pp output.json samples Scan for file information: @@ -55,7 +59,7 @@ Scan for file information: :: - ./scancode -i --json-pp output.json samples + scancode -i --json-pp output.json samples To see more example scans: @@ -63,6 +67,6 @@ To see more example scans: :: - ./scancode --examples + scancode --examples For more information, refer :ref:`cli_list_options`.