Skip to content

Commit e57ddb1

Browse files
AyanSinhaMahapatraAyanSinhaMahapatra
authored
Merge pull request #112 from nexB/restructure-gsoc-pages
Restructure GSoC/GSoD archive
2 parents 3496af8 + 4115f44 commit e57ddb1

17 files changed

Lines changed: 725 additions & 12 deletions

‎docs/source/archive/gsoc-toc.rst‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,55 @@ student participation in open source software development.
88
It was started by Google in 2005.
99
More about GSoc - <https://summerofcode.withgoogle.com/about/>_
1010

11+
GSoC 2022
12+
---------
1113

14+
.. toctree::
15+
:maxdepth: 2
16+
17+
gsoc/reports/2022/scancode_workbench_omkar
18+
19+
GSoC 2021
20+
---------
21+
22+
.. toctree::
23+
:maxdepth: 2
24+
25+
gsoc/reports/2021/deltacode_pratik
26+
gsoc/reports/2021/scancode_toolkit_akanksha
27+
gsoc/reports/2021/vulnerablecode_hritik
28+
29+
GSoC 2020
30+
---------
31+
32+
.. toctree::
33+
:maxdepth: 2
34+
35+
gsoc/reports/2020/scancode_toolkit_abhishek
36+
37+
GSoC 2019
38+
---------
39+
40+
.. toctree::
41+
:maxdepth: 2
42+
43+
gsoc/reports/2019/deltacode_arnav
44+
gsoc/reports/2019/scancode_toolkit_abhishek
45+
46+
GSoC 2017
47+
---------
48+
49+
.. toctree::
50+
:maxdepth: 2
51+
52+
gsoc/reports/2017/scancode_toolkit_yash
53+
54+
GSoC project ideas pages
55+
------------------------
1256

1357
.. toctree::
1458
:maxdepth: 2
1559

16-
gsoc/gsoc21_final_report
17-
gsoc/gsoc19_final_report
18-
gsoc/gsoc_2019
19-
gsoc/gsoc_2018
20-
gsoc/gsoc_2017
60+
gsoc/org_pages/gsoc_2019
61+
gsoc/org_pages/gsoc_2018
62+
gsoc/org_pages/gsoc_2017

docs/source/archive/gsoc/gsoc_2017.rst renamed to docs/source/archive/gsoc/org_pages/gsoc_2017.rst

File renamed without changes.

docs/source/archive/gsoc/gsoc_2018.rst renamed to docs/source/archive/gsoc/org_pages/gsoc_2018.rst

File renamed without changes.

docs/source/archive/gsoc/gsoc_2019.rst renamed to docs/source/archive/gsoc/org_pages/gsoc_2019.rst

File renamed without changes.

‎docs/source/archive/gsoc/reports/2017/scancode_toolkit_yash.rst‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
Plugin architecture
2+
======================
3+
4+
**Project: Plugin architecture for ScanCode**
5+
---------------------------------------------
6+
7+
Yash D. Saraf `yashdsaraf@gmail.com <mailto:yashdsaraf@gmail.com>`_
8+
9+
----
10+
11+
This project's purpose was to create a decoupled plugin architecture for
12+
`ScanCode <https://github.com/nexB/scancode-toolkit>`_ such that it can handle plugins at different
13+
stages of a scan and can be coupled at runtime. These stages were,
14+
15+
1. `Format <https://github.com/nexB/scancode-toolkit/issues/639>`_ :
16+
---------------------------------------------------------------------
17+
18+
In this stage, the plugins are supposed to run **after** the scanning is done and ``post-scan``
19+
plugins are called. These plugins could be used for:
20+
21+
22+
- **converting the scanned output to the given format (say csv, json, etc.)**
23+
24+
**HOWTO**
25+
26+
Here, a plugin needs to add an entry in the ``scancode_output_writers`` entry point in the following
27+
format : ``'<format> = <module>:<function>'``.
28+
29+
30+
- ``<format>`` is the format name which will be used as the command line option name
31+
(e.g ``csv`` or ``json`` ).
32+
- ``<module>`` is a python module which implements the ``output`` hook specification.
33+
- ``<function>`` is the function to which the scan output will be passed if this plugin is called.
34+
35+
The ``<format>`` name will be automatically added to the ``--format`` command line option and
36+
(if called) the scanned data will be passed to the plugin.
37+
38+
2. `Post-scan <https://github.com/nexB/scancode-toolkit/issues/704>`_ :
39+
------------------------------------------------------------------------
40+
41+
In this stage, the plugins are supposed to run **after** the scanning is done. Some uses for these
42+
plugins were:
43+
44+
45+
- **summarization of scan outputs**
46+
47+
e.g A post-scan plugin for marking ``is_source`` to true for directories with ~90% of source
48+
files.
49+
50+
- **simplification of scan outputs**
51+
52+
e.g The ``--only-findings`` option to return files or directories with findings for the
53+
requested scans. Files and directories without findings are omitted (not considering basic file
54+
information as findings)).
55+
56+
This option already existed, I just ported it to a post-scan plugin.
57+
58+
**HOWTO**
59+
60+
Here, a plugin needs to add an entry in the ``scancode_post_scan`` entry point in the following
61+
format ``'<name> = <module>:<function>'``
62+
63+
- ``<name>`` is the command line option name (e.g **only-findings**).
64+
- ``<module>`` is a python module which implements the ``post_scan`` hook specification.
65+
- ``<function>`` is the function to which the scanned files will be passed if this plugin is called
66+
67+
The command line option for this plugin will be automatically created using the ``<function>`` 's
68+
doctring as its help text and (if called) the scanned files will be passed to the plugin.
69+
70+
3. `Pre-scan <https://github.com/nexB/scancode-toolkit/issues/719>`_ :
71+
-----------------------------------------------------------------------
72+
73+
In this stage, the plugins are supposed to run **before** the scan starts. So the potential uses
74+
for these types of plugins were to:
75+
76+
- **ignore files based on a given pattern (glob)**
77+
- **ignore files based on their info i.e size, type etc.**
78+
- **extract archives before scanning**
79+
80+
**HOWTO**
81+
82+
Here, a plugin needs to add an entry in the ``scancode_pre_scan`` entry point in the following
83+
format : ``'<name> = <module>:<class>'``
84+
85+
86+
* ``<name>`` is the command line option name (e.g **ignore** ).
87+
* ``<module>`` is a python module which implements the ``pre_scan`` hook specification.
88+
* ``<class>`` is the class which is instantiated and its appropriate method is invoked if this
89+
plugin is called. This needs to extend the ``plugincode.pre_scan.PreScanPlugin`` class.
90+
91+
The command line option for this plugin will be automatically created using the ``<class>`` 's
92+
doctring as its help text. Since there isn't a single spot where ``pre-scan`` plugins can be
93+
plugged in, more methods to ``PreScanPlugin`` class can be added which can represent different
94+
hooks, say to add or delete a scan there might be a method called ``process_scan``.
95+
96+
If a plugin's option is passed by the user, then the ``<class>`` is instantiated with the user
97+
input and its appropriate aforementioned methods are called.
98+
99+
4. Scan (proper):
100+
-----------------
101+
102+
In this stage, the plugins are supposed to run **before** the scan starts and **after** the
103+
``pre-scan`` plugins are called. These plugins would have been used for
104+
105+
- **adding or deleting scans**
106+
- **adding dependency scans (whose data could be used in other scans)**
107+
108+
No development has been done for this stage, but it will be quite similar to ``pre-scan``.
109+
110+
5. Other work:
111+
--------------
112+
113+
`Group cli options in cli help <https://github.com/nexB/scancode-toolkit/issues/709>`_
114+
115+
Here, the goal was to add command line options to pre-defined groups such that they are displayed
116+
in their respective groups when ``scancode -h`` or ``scancode --help`` is called. This helped to
117+
better visually represent the command line options and determine more easily what context they
118+
belong to.
119+
120+
`Add a Resource class to hold all scanned info <https://github.com/nexB/scancode-toolkit/issues/738>`_
121+
* ``Ongoing`` *
122+
123+
Here, the goal was to create a ``Resource`` class, such that it holds all the scanned data for a
124+
resource (i.e a file or a directory). This class would go on to eventually encapsulate the caching
125+
logic entirely. For now, it just holds the ``info`` and ``path`` of a resource.
126+
127+
6. What's left?
128+
---------------
129+
130+
- Pre-scan plugin for archive extractions
131+
- Scan (proper) plugins
132+
- More complex post-scan plugins
133+
- Support plugins written in languages other than python
134+
135+
**Additionally, all my commits can be found** `here <https://github.com/nexB/scancode-toolkit/commits/develop?author=yashdsaraf>`_.

docs/source/archive/gsoc/gsoc19_final_report.rst renamed to docs/source/archive/gsoc/reports/2019/deltacode_arnav.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Google Summer of Code 2019 - Final report
1+
Approximately similar file detection
22
=========================================
33

44
Project: Approximately similar file detection in DeltaCode

‎docs/source/archive/gsoc/reports/2019/scancode_toolkit_abhishek.rst‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
Port scancode-toolkit to Python 3
2+
=========================================
3+
4+
Project: **scancode-toolkit** to Python 3
5+
-----------------------------------------
6+
7+
**Owner:** `Abhishek Kumar <https://github.com/Abhishek-Dev09>`_
8+
9+
**Mentor:** `Philippe Ombredanne <https://github.com/pombredanne>`_
10+
11+
Overview
12+
--------
13+
14+
**Problem:** Since Python 2.7 will retire in few months and will not be maintained any longer.
15+
16+
**Solution:** `Scancode <https://github.com/nexB/scancode-toolkit/>`__ needs to be ported to
17+
python 3 and all test suites must pass on both version of Python. The main difference that
18+
makes Python 3 better than Python 2.x is that the support for unicode is greatly improved in
19+
Python 3. This will also be useful for scancode as scancode has users in more than 100 languages
20+
and it's easy to translate strings from unicode to other languages.
21+
22+
**Objective**: To make scancode-toolkit installable on on Python 3.6 and higher, as presently it
23+
installs with Python 2.7 only.
24+
25+
Implementation
26+
--------------
27+
28+
- It was started in development mode(editable mode) and then it was moved to work in virtual
29+
environments.
30+
- I have worked module by module according to the order of hierarchy of modules. For example :All
31+
module is dependent on commoncode, so it must be ported first. In this way we have created the
32+
Porting order:
33+
34+
1. commoncode
35+
2. plugincode
36+
3. typecode
37+
4. extractcode
38+
5. textcode
39+
6. scancode basics (some tests are integration tests and will have to wait to be ported)
40+
7. formattedcode, starting with JSON (some tests are integration tests and will have to wait
41+
to be ported)
42+
8. cluecode
43+
9. licensedcode
44+
10. packagedcode (depends on licensecode)
45+
11. summarycode
46+
12. fixup the remaining bits and tests
47+
48+
After porting each module, I have marked these modules as ported ``scanpy3`` with help of
49+
**conffest** plugin (created by `@pombredanne <https://github.com/pombredanne>`_). **Conffest**
50+
plugin is heart of this project. Without this, it was very difficult to do. Dependencies was fixed
51+
at the time of porting the module where it was used.
52+
53+
Challenging part of Project
54+
---------------------------
55+
56+
It is very difficult to deal with paths on different operating systems.The issue is around
57+
macOS/Windows/Linux. The first two OS handle unicode paths comfortably on Python 2 and 3 but not
58+
completely on macOS Mojave because its filesystem encoding is APFS. Linux paths are bytes and
59+
os.listdir is broken on Python 2. As a result you can only sanely handle Linux paths as bytes
60+
on Python 2. But on Python 3 path seems to be corrected as ``unicode`` on Linux.
61+
62+
For more details visit here :
63+
64+
- https://vstinner.github.io/painful-history-python-filesystem-encoding.html
65+
- `jaraco/path.py#130 <https://github.com/jaraco/path.py/issues/130>`__
66+
67+
We came with various Solution:
68+
69+
- To use pathlib which generally handle paths correctly across platforms. And for backports we use
70+
pathlib 2. But this solution also fails because pathlib 2 does not work as expected wrt unicode
71+
vs bytes. And os.listdir also doesn't work properly.
72+
73+
- To use `path.py <https://pypi.org/project/path.py/>`__ which handles the paths across all the
74+
platforms even on macOS Mojave .
75+
76+
- Use ``bytes`` on linux and python 3 and ``unicode`` everywhere.
77+
78+
We choose the third solution because it is most fundamental and simple and easy to use.
79+
80+
Project was tracked in this ticket `nexB/scancode-toolkit#295 <https://github.com/nexB/scancode-toolkit/issues/295>`__
81+
82+
**Project link :** `Port Scancode to Python 3 <https://summerofcode.withgoogle.com/organizations/6118953540124672/>`__
83+
84+
..
85+
[Org Link] https://summerofcode.withgoogle.com/organizations/6118953540124672/
86+
[Project Link] https://summerofcode.withgoogle.com/projects/#5969926387400704
87+
88+
**My contribution :** `List of Commits <https://github.com/nexB/scancode-toolkit/commits?author=Abhishek-Dev09>`__
89+
90+
**Note :** Please give your feedback `here <https://github.com/nexB/scancode-toolkit/issues/295>`_
91+
92+
Outcome
93+
-------
94+
95+
Now we have liftoff on Python 3 . We are able to run basic scans without errors on develop branch.
96+
You check it by running ``scancode -clipeu samples/ --json-pp - -n4`` .
97+
98+
At last I would like to thanks my Mentor **@pombredanne** aka
99+
`Philippe Ombredanne <https://github.com/pombredanne>`__ . He has helped lot in completing this
100+
project. He is very supportive and responsive. I have learned a lot from him. By his encouragement
101+
and motivation, I am very improving day by day, building and developing my skills. I have completed
102+
all the tasks that were in the scope of this GSoC project.

‎docs/source/archive/gsoc/reports/2020/scancode_toolkit_abhishek.rst‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
Support Python 3.x Part 2
2+
==========================
3+
4+
Goals
5+
6+
- Remove legacy Python code
7+
- Support latest Python 3.x
8+
- Improve 3rd party package provisioning
9+
10+
ScanCode has been ported on Python 3 but still supports Python 2.7.
11+
The goal of this project is to remove all the code that provides Python 2-specific support.
12+
13+
In addition, the way we deal with 3rd party packages needs to be refactored:
14+
today we embed a copy of all our third-party dependencies as pre-built wheels
15+
in the ScanCode repository. This creates some problems:
16+
17+
1. We supported Python 2 and now support Python 3.6.
18+
Each version may need a pre-built wheel if there is native code.
19+
With 3.7, 3.8 and beyond this means creating and storing in the ScanCode git
20+
repo 4 variants for each dependency.
21+
22+
2. We support Windows, Linux, macOS and possibly FreeBSD each with some possible
23+
architecture variants. For each of these we are storing a variant for each of the
24+
Python versions above.
25+
26+
This creates way too many wheels that end up making ScanCode checkouts and tarballs too big.
27+
We need to define a new improved way to handle third-party dependencies including:
28+
29+
- Make the build automation more prominent
30+
- Ensure secured pinning of wheel versions and actual files (e.g. using checksums)
31+
- Do not bundle these wheels anymore.
32+
33+
For more info see Disk quota and Limitations.
34+
35+
Basically it is an extension of previous GSoC 2019 Project: Port scancode-toolkit to Python 3
36+
37+
Assigned to
38+
-----------
39+
40+
Abhishek Kumar: https://github.com/Abhishek-Dev09
41+
42+
Level
43+
-----
44+
45+
Advanced
46+
47+
Tech
48+
----
49+
50+
Python, build scripts, packaging
51+
52+
URLS
53+
----
54+
55+
https://github.com/nexB/scancode-toolkit/issues/295
56+
57+
Project Board
58+
--------------
59+
60+
https://github.com/nexB/scancode-toolkit/projects/8
61+
62+
Mentors
63+
--------
64+
65+
@majurg https://github.com/majurg
66+
@pombredanne https://github.com/pombredanne
67+
68+
Commits
69+
-------
70+
71+
List of commits(merged): https://github.com/nexB/scancode-toolkit/commits?author=Abhishek-Dev09
72+
https://github.com/nexB/scancode-toolkit/pull/2184 (open)
73+
https://github.com/nexB/scancode-toolkit/pull/2167 (open)

docs/source/archive/gsoc/gsoc21_final_report.rst renamed to docs/source/archive/gsoc/reports/2021/deltacode_pratik.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
Google Summer of Code 2021 - Final report
2-
=========================================
1+
Virtual Codebase support
2+
=========================
33

44
Project: Virtual Codebase support in DeltaCode
55
----------------------------------------------------------

0 commit comments

Comments
 (0)