Add OVAL parser for Ubuntu

[sbs2001]

TODOs :

• Use typed python instead of comments

How is this implemented ?

Oval files basically contain the following elements : OvalObjects, OvalStates , OvalTests, OvalDefinitions, OvalVariables .

What the parser does, is take in the OVAL document(xml file), get all the definitions. Then iterate over each definition, while doing so it does the following for every definition.
0.Every definition has enclosed within a vulnerability id and the vulnerability description, we collect these.
1.Collect all the OvalTests which are referred to , within the OvalDefinition's body
2.This OvalTest has a OvalObject and OvalState nested within.
3.From (2) go take the OvalObject, this has generally a OvalVariable nested within (or it may directly has a single package name enclosed within )
4.Iterate over the children of the OvalVariable element, each child encloses a package name within we collect this.
5.From (2) go take the OvalState element, this encloses a version range,, which denotes vulnerable versions we also collect this.
6.Repeat this until we cover all the definitions.

I really suggest to take a look at vulnerabilities/tests/test_data/ubuntu_oval._data.xml to get a quick idea of how a OVAL document is structured.

High level overview
This is a repaste from chat, which provides a gist of the parser's intended implementation:

The way I am trying to implement the OVAL parser is , by creating a OvalParser base class which has abstract methods required to import data from an Oval document , these need to be implemented for the specific provider(eg ubuntu,suse etc), and a extract_data method which basically orchestrates these methods to return the data in appropriate form.

Best case scenario : All the implementations of the parser for each provider, end up looking similar , in that case we could have a single OvalParser in which we could just throw in a OVAL file and get the data back.

Worst case scenario : We end up having UbuntuOvalParser SUSEOvalParser with slight modifications, which is not bad.

The following is a link to interactive google colab notebook https://colab.research.google.com/drive/1-NVbGwF80bYxzDYLXk5A4yvn0L-Qdgpe?usp=sharing for the OVAL parser.

Fixes #145

[sbs2001]

So the good news is, it's working for all Ubuntu OVAL files, bad news is it's really slow. It took over 6 minutes to parse https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml.bz2 which is about 40 MB . Here are the stats

...: p = pstats.Stats('perf.txt') 
   ...: p.sort_stats('cumulative').print_stats(10)                                                                           
Sat May 16 15:19:18 2020    perf.txt

         769968949 function calls (769934359 primitive calls) in 383.945 seconds

   Ordered by: cumulative time
   List reduced from 469 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     27/1    0.000    0.000  383.945  383.945 {built-in method builtins.exec}
        1    0.000    0.000  383.945  383.945 profiler.py:1(<module>)
        1    0.434    0.434  382.571  382.571 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/oval_parser.py:22(get_data)
    84661   12.240    0.000  205.191    0.002 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/lib_oval.py:408(getElementByID)
    33373    9.235    0.000  195.203    0.006 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/oval_parser.py:115(get_object_state_of_test)
    15267   59.637    0.004  162.846    0.011 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/oval_parser.py:95(get_tests_of_definition)
289921428   75.677    0.000  123.842    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/lib_oval.py:734(getId)
    33373    2.101    0.000  112.006    0.003 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/lib_oval.py:356(getStates)
    33373   52.569    0.002  109.479    0.003 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/lib_oval.py:377(<listcomp>)
110130900   56.909    0.000   56.909    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/lib_oval.py:1358(__init__)

I have few ideas which could greatly improve the getElementById method which would probably decrease the time for each of these top time consuming function calls.

[sbs2001]

Update After some tweaks , the same test case takes 5 seconds to complete, that's a massive performance gain.

Sat May 16 17:18:32 2020    perf3.txt

         4094392 function calls (4059802 primitive calls) in 5.144 seconds

   Ordered by: cumulative time
   List reduced from 476 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     27/1    0.000    0.000    5.144    5.144 {built-in method builtins.exec}
        1    0.000    0.000    5.144    5.144 profiler.py:1(<module>)
        1    0.140    0.140    3.727    3.727 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/oval_parser.py:60(get_data)
    33373    0.072    0.000    1.665    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/vulnerabilities/scraper/oval_parser.py:173(get_versionsrngs_from_state)
    33371    0.110    0.000    1.581    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/range_specifier.py:20(__init__)
        1    0.000    0.000    1.325    1.325 /usr/lib/python3.8/xml/etree/ElementTree.py:1192(parse)
        1    0.000    0.000    1.325    1.325 /usr/lib/python3.8/xml/etree/ElementTree.py:571(parse)
        1    1.325    1.325    1.325    1.325 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}
    33371    0.107    0.000    1.300    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/range_specifier.py:44(_parse)
    33382    0.152    0.000    0.808    0.000 /home/shivam/coding/opensource/oriv/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/specifier.py:51(__init__)