aboutcode-org · steven-esser · Nov 25, 2019 · Nov 4, 2019 · Nov 7, 2019 · Nov 7, 2019
diff --git a/docs/source/aboutcode-docs/writing_good_commit_messages.rst b/docs/source/aboutcode-docs/writing_good_commit_messages.rst
@@ -1,3 +1,5 @@
+.. _good_commit_messages:
+
 Writing good Commit Messages
 ============================
 

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -36,8 +36,12 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+'sphinx.ext.intersphinx'
 ]
 
+# Temporary Mapping, Once scancode-toolkit.readthedocs.io has the migrated docs, this can be changed to the same.
+intersphinx_mapping = {'scancode-toolkit': ('https://sphinx-test-ayan.readthedocs.io/en/scancode-toolkit/', None)}
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

diff --git a/docs/source/doc_maintenance.rst b/docs/source/doc_maintenance.rst
@@ -32,9 +32,15 @@ Now you can install the dependencies in a virtualenv::
 
     cd aboutcode
     virtualenv -p /usr/bin/python3.6 docs-venv
-    source bin/activate
+    source docs-venv/bin/activate
 
-Now you can install Sphinx and the format theme used by readthedocs::
+Now, the following prerequisites are installed
+
+- Sphinx
+- sphinx_rtd_theme (the format theme used by ReadTheDocs)
+- docs8 (style linter)
+
+::
 
     pip install Sphinx sphinx_rtd_theme doc8
 
@@ -83,6 +89,7 @@ system before creating a Pull Request.
 
     cd docs
     ./scripts/sphinx_build_link_check.sh
+    ./scripts/doc8_style_check.sh
 
 Share AboutCode Document Improvements
 -------------------------------------
@@ -99,11 +106,224 @@ examples::
     git push
     git status
 
-The AboutCode webhook with ReadTheDocs should rebuild the documentation. You can review your
-results online.
+The AboutCode webhook with ReadTheDocs should rebuild the documentation after your Pull Request
+is Merged.
+
+Refer the `Pro Git Book <https://git-scm.com/book/en/v2/>`_ available online for Git tutorials
+covering more complex topics on Branching, Merging, Rebasing etc.
+
+Continuous Integration
+----------------------
+
+The documentations are checked on every new commit through Travis-CI, so that common errors are
+avoided and documentation standards are enforced. Travis-CI presently checks for these 3 aspects
+of the documentation :
+
+1. Successful Builds (By using ``sphinx-build``)
+2. No Broken Links   (By Using ``link-check``)
+3. Linting Errors    (By Using ``Doc8``)
+
+So run these scripts at your local system before creating a Pull Request::
+
+    cd docs
+    ./scripts/sphinx_build_link_check.sh
+    ./scripts/doc8_style_check.sh
+
+Style Checks Using ``Doc8``
+---------------------------
+
+How To Run Style Tests
+^^^^^^^^^^^^^^^^^^^^^^
+
+In the project root, run the following command::
+
+    $ doc8 --max-line-length 100 docs/source/ --ignore D000
+
+A sample output is::
+
+    Scanning...
+    Validating...
+    docs/source/scancode-toolkit/misc/licence_policy_plugin.rst:37: D002 Trailing whitespace
+    docs/source/scancode-toolkit/misc/faq.rst:45: D003 Tabulation used for indentation
+    docs/source/scancode-toolkit/misc/faq.rst:9: D001 Line too long
+    docs/source/scancode-toolkit/misc/support.rst:6: D005 No newline at end of file
+    ========
+    Total files scanned = 34
+    Total files ignored = 0
+    Total accumulated errors = 326
+    Detailed error counts:
+        - CheckCarriageReturn = 0
+        - CheckIndentationNoTab = 75
+        - CheckMaxLineLength = 190
+        - CheckNewlineEndOfFile = 13
+        - CheckTrailingWhitespace = 47
+        - CheckValidity = 1
+
+Now fix the errors and run again till there isn't any style error in the documentation.
+
+What is Checked?
+^^^^^^^^^^^^^^^^
+
+PyCQA is an Organization for code quality tools (and plugins) for the Python programming language.
+Doc8 is a sub-project of the same Organization. Refer this `README <https://github.com/PyCQA/doc8/blob/master/README.rst>`_ for more details.
+
+What is checked:
+
+    - invalid rst format - D000
+    - lines should not be longer than 100 characters - D001
+
+        - RST exception: line with no whitespace except in the beginning
+        - RST exception: lines with http or https URLs
+        - RST exception: literal blocks
+        - RST exception: rst target directives
+
+    - no trailing whitespace - D002
+    - no tabulation for indentation - D003
+    - no carriage returns (use UNIX newlines) - D004
+    - no newline at end of file - D005
+
+Interspinx
+----------
+
+Aboutcode documentation uses `Intersphinx <http://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html>`_
+to create links to other Sphinx Documentations, to maintain links to other Aboutcode Projects.
+
+To link sections in the same documentation, standart reST labels are used. Refer
+`Cross-Referencing <http://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#ref-role>`_ for more information.
+
+For example::
+
+    .. _my-reference-label:
+
+    Section to cross-reference
+    --------------------------
+
+    This is the text of the section.
+
+    It refers to the section itself, see :ref:`my-reference-label`.
+
+Now, using Intersphinx, you can create these labels in one Sphinx Documentation and then referance
+these labels from another Sphinx Documentation, hosted in different locations.
+
+You just have to add the following in the ``conf.py`` file for your Sphinx Documentation, where you
+want to add the links::
+
+    extensions = [
+    'sphinx.ext.intersphinx'
+    ]
+
+    intersphinx_mapping = {'scancode-toolkit': ('https://scancode-toolkit.readthedocs.io/en/latest/', None)}
+
+To show all Intersphinx links and their targets of an Intersphinx mapping file, run::
+
+    python -msphinx.ext.intersphinx https://scancode-toolkit.readthedocs.io/en/latest/objects.inv
+
+.. WARNING::
+
+    ``python -msphinx.ext.intersphinx https://scancode-toolkit.readthedocs.io/objects.inv`` will
+    give error.
+
+This enables you to create links to the ``scancode-toolkit`` Documentation in your own
+Documentation, where you modified the configuration file. Links can be added like this::
+
+    For more details refer :ref:`scancode-toolkit:doc_style_guide`.
+
+You can also not use the ``scancode-toolkit`` label assigned to all links from
+scancode-toolkit.readthedocs.io, if you don't have a label having the same name in your Sphinx
+Documentation. Example::
+
+    For more details refer :ref:`doc_style_guide`.
+
+If you have a label in your documentation which is also present in the documentation linked by
+Intersphinx, and you link to that label, it will create a link to the local label.
+
+For more information, refer this tutorial named
+`Using Intersphinx <https://my-favorite-documentation-test.readthedocs.io/en/latest/using_intersphinx.html>`_.
+
+Extra Style Checks
+------------------
+
+1. Headings
+
+    (`Refer <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections>`_)
+    Normally, there are no heading levels assigned to certain characters as the structure is
+    determined from the succession of headings. However, this convention is used in Python’s Style
+    Guide for documenting which you may follow:
+
+    # with overline, for parts
+
+    * with overline, for chapters
+
+    =, for sections
+
+    -, for subsections
+
+    ^, for sub-subsections
+
+    ", for paragraphs
+
+2. Heading Underlines
+
+    Do not use underlines that are longer/shorter than the title headline itself. As in:
+
+::
+
+    Correct :
+
+    Extra Style Checks
+    ------------------
+
+    Incorrect :
+
+    Extra Style Checks
+    ------------------------
+
+.. note::
+
+    Underlines shorter than the Title text generates Errors on sphinx-build.
+
+
+3. Internal Links
+
+    Using ``:ref:`` is advised over standard reStructuredText links to sections (like
+    ```Section title`_``) because it works across files, when section headings are changed, will
+    raise warnings if incorrect, and works for all builders that support cross-references.
+    However, external links are created by using the standard ```Section title`_`` method.
+
+4. Eliminate Redundancy
+
+    If a section/file has to be repeated somewhere else, do not write the exact same section/file
+    twice. Use ``.. include: ../README.rst`` instead. Here, ``../`` refers to the documentation
+    root, so file location can be used accordingly. This enables us to link documents from other
+    upstream folders.
+
+5. Using ``:ref:`` only when necessary
+
+    Use ``:ref:`` to create internal links only when needed, i.e. it is referenced somewhere.
+    Do not create references for all the sections and then only reference some of them, because
+    this created unnecessary references. This also generates ERROR in ``restructuredtext-lint``.
+
+6. Spelling
+
+    You should check for spelling errors before you push changes. `Aspell <http://aspell.net/>`_
+    is a GNU project Command Line tool you can use for this purpose. Download and install Aspell,
+    then execute ``aspell check <file-name>`` for all the files changed. Be careful about not
+    changing commands or other stuff as Aspell gives prompts for a lot of them. Also delete the
+    temporary ``.bak`` files generated. Refer the `manual <http://aspell.net/man-html/>`_ for more
+    information on how to use.
+
+7. Notes and Warning Snippets
+
+    Every ``Note`` and ``Warning`` sections are to be kept in ``rst_snippets/note_snippets/`` and
+    ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are
+    frequently used in multiple files.
+
+Converting from Markdown
+------------------------
 
-Documentation Style Guides
---------------------------
+If you want to convert a ``.md`` file to a ``.rst`` file, this `tool <https://github.com/chrissimpkins/md2rst>`_
+does it pretty well. You'd still have to clean up and check for errors as this contains a lot of
+bugs. But this is definitely better than converting everything by yourself.
 
-The ``scancode-toolkit`` documentation is compliant to our doc style standards, enforced using
-``doc8``. For more details refer :ref:`contrib_doc_dev`.
+This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for
+Sphinx/ReadTheDocs hosting.
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -25,6 +25,14 @@ Documentation Guide
    doc_maintenance
 
 
+Getting Started
+***************
+
+.. toctree::
+   :maxdepth: 2
+
+   scancode-toolkit/getting-started/newcomer
+
 Tutorial Documents
 ******************
 
@@ -65,3 +73,7 @@ Indices and Tables
 
 * :ref:`genindex`
 * :ref:`modindex`
+
+.. _improve_docs:
+
+.. include::  /scancode-toolkit/rst_snippets/improve_docs.rst
diff --git a/docs/source/scancode-toolkit/cli-reference/basic-options.rst b/docs/source/scancode-toolkit/cli-reference/basic-options.rst
@@ -5,14 +5,18 @@
 
 ----
 
+.. include::  /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst
+
+----
+
 ``--generated`` Options
 -----------------------
 
     The ``--generated`` option classifies automatically generated code files with a flag.
 
     An example of using ``--generated`` in a scan::
 
-        ./scancode -clpieu --json-pp output.json samples --generated
+        scancode -clpieu --json-pp output.json samples --generated
 
     In the results, for each file the following attribute is added with it's corresponding
     ``true``/``false`` value ::
@@ -47,7 +51,7 @@
 
     An example usage::
 
-        ./scancode -clpieu --json-pp output.json samples --max-email 5
+        scancode -clpieu --json-pp output.json samples --max-email 5
 
     This only reports 5 email addresses per file and ignores the rest.
 
@@ -71,7 +75,7 @@
 
     An example usage::
 
-        ./scancode -clpieu --json-pp output.json samples --max-url 10
+        scancode -clpieu --json-pp output.json samples --max-url 10
 
     This only reports 10 urls per file and ignores the rest.
 
@@ -100,7 +104,7 @@
 
     An example usage::
 
-        ./scancode -clpieu --json-pp output.json samples --license-score 70
+        scancode -clpieu --json-pp output.json samples --license-score 70
 
     Here's the license results on setting the integer value to 100, Vs. the default value 0. This is
     visualized using ScanCode workbench in the License Info Dashboard.
@@ -134,7 +138,7 @@
 
     An example Scan::
 
-        ./scancode -cplieu --json-pp output.json samples --license-text
+        scancode -cplieu --json-pp output.json samples --license-text
 
     An example matched text included in the results is as follows::
 
@@ -178,7 +182,7 @@
 
     A scan example using the ``--license-url-template TEXT`` option ::
 
-        ./scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.yml
+        scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.yml
 
     In a normal scan, reference url for "ZLIB License" is as follows::
 
@@ -212,7 +216,7 @@
 
     An example Scan::
 
-        ./scancode -cplieu --json-pp output.json samples --license-text --license-text-diagnostics
+        scancode -cplieu --json-pp output.json samples --license-text --license-text-diagnostics
 
     Running a scan on the samples directory with ``--license-text --license-text-diagnostics`` options,
     causes the following difference in the scan result of the file

diff --git a/docs/source/scancode-toolkit/cli-reference/core-options.rst b/docs/source/scancode-toolkit/cli-reference/core-options.rst
@@ -7,6 +7,10 @@
 
 ----
 
+.. include::  /scancode-toolkit/rst_snippets/note_snippets/synopsis_install_quickstart.rst
+
+----
+
 Comparing Progress Message Options
 ----------------------------------
 
@@ -89,7 +93,7 @@ Comparing Progress Message Options
 
     An example scan command using ``--from-json``::
 
-        ./scancode --from-json sample.json --json-pp sample_2.json --classify
+        scancode --from-json sample.json --json-pp sample_2.json --classify
 
     This inputs the scan results from ``sample.json``, runs the post-scan plugin ``--classify`` and
     outputs the results for this scan to ``sample_2.json``.
@@ -114,4 +118,4 @@ Comparing Progress Message Options
 
     An example usage::
 
-        ./scancode -clieu --json-pp sample.json samples --max-in-memory -1
+        scancode -clieu --json-pp sample.json samples --max-in-memory -1