Merge pull request #82 from napalm-automation/develop

Release 0.0.7

.gitignore

@@ -65,3 +65,5 @@ test/unit/test_devices.py
 tags
 
 report.json
+
+prof/

CHANGELOG.rst

@@ -0,0 +1,20 @@
+0.0.2
++++++
+
+    - Translators now accept ``continue_negating`` option
+    - YAML files can now include other files via the ``!include relative/path/to/file.yaml`` directive
+    - ``TextParser``, ``list - block`` supports manual keys via the ``key`` argument
+    - ``TextParser``, ``list - block`` now supports flat list of commands (i.e. BGP neighbors and static routes) via the ``flat`` argument
+    - ``TextParser``, ``list - block`` now supports composite keys via the ``composite_key`` argument
+    - ``TextParser``, ``list - block`` now supports creating elements manually via the ``mandatory`` argument
+
+    - Move mandatory elements previously on the default action to a dedicated action
+    - from is optional, by default it will always follow the parent
+    - from is now a pointer, no need to keep serializing/deserializing
+    - mode is optional. All parsers have a main "default" action now.
+    - JSONParser added
+
+0.0.1
++++++
+
+    - Initial version

docs/index.rst

@@ -29,3 +29,4 @@ Documentation
    yang/translators/TextTranslator
    yang/api
    yang/jinja_filters
+   yang/faq

docs/yang/faq.rst

@@ -0,0 +1,24 @@
+FAQ
+===
+
+Some YAML files are insanely largely. Can I break them down into multiple files?
+________________________________________________________________________________
+
+    Yes, you can with the ``!include relative/path/to/file.yaml`` directive. For example::
+
+        # ./main.yaml
+        my_key:
+            blah: asdasdasd
+            bleh: !include includes/bleh.yaml
+
+        # ./includes/bleh.yaml
+        qwe: 1
+        asd: 2
+
+    Will result in the final object::
+
+        my_key:
+            blah: asdasdasd
+            bleh:
+                qwe: 1
+                asd: 2

docs/yang/parsers.rst

@@ -3,6 +3,52 @@ Parsers
 
 Parsers are responsible for mapping native configuration/show_commands to a YANG model.
 
+Special actions
+===============
+
+Most actions depend on the parser you are using, however, some are common to all of them:
+
+unnecessary
+-----------
+
+This makes the parser skip the field and continue processing the tree.
+
+not_implemented
+---------------
+
+This makes the parser stop processing the tree underneath this value. For example::
+
+    field_1:
+        process: unnecessary
+    field_2:
+        process: not_implemented
+        subfield_1:
+            process: ...
+        subfield_2:
+            process: ...
+    field_3:
+        ...
+
+The ``not_implemented`` action will stop the parser from processing ``subfield_1`` and ``subfield_2``
+and move directly onto ``field_3``.
+
+gate
+----
+
+Works like ``not_implemented`` but accepts a condition. For example::
+
+    protocols:
+        protocol:
+            bgp:
+                _process:
+                  - mode: gate
+                    when: "{{ protocol_key != 'bgp bgp' }}"
+                global:
+                    ...
+
+The snippet above will only process the ``bgp`` subtree if the condition is **not** met.
+
+
 Special fields
 ==============
 
@@ -17,7 +63,7 @@ mode
 * **Example**: Parse the description field with a simple regular expression::
 
     _process:
-        mode: search
+      - mode: search
         regexp: "description (?P<value>.*)"
         from: "{{ bookmarks.interface[interface_key] }}"
 
@@ -52,7 +98,7 @@ from
 
     address:
         _process:
-            mode: xpath
+          - mode: xpath
             xpath: "family/inet/address"
             key: name
             from: "{{ bookmarks['parent'] }}"
@@ -112,14 +158,14 @@ Some actions let's you provide additional information for later use. Those will
 
     address:
         _process:
-            mode: block
+          - mode: block
             regexp: "(?P<block>ip address (?P<key>(?P<ip>.*))\\/(?P<prefix>\\d+))(?P<secondary> secondary)*"
             from: "{{ bookmarks['parent'] }}"
         config:
             _process: unnecessary
             ip:
                 _process:
-                    mode: value
+                  - mode: value
                     value: "{{ extra_vars.ip }}"
 
 The first regexp captures a bunch of vars that later can be used by just reading them from
@@ -137,13 +183,15 @@ the device. For example::
         parser: XMLParser
         execute:
             - method: _rpc
-              args:
+              args: []
+              kwargs:
                   get: "<get-configuration/>"
 
 * **execute** is a list of calls to do to from the device to extract the data.
 
   * **method** is the method from the device to call.
-  * **args** are arguments that will be passed to the method.
+  * **args** are the numbered/ordered arguments for the method
+  * **kwargs** are the keyword arguments for the method
 
 In addition, some methods like ``parse_config`` and ``parse_state`` may have mechanisms to pass the
 information needed to the parser instead of relying on a live device to obtain it. For parsers, you

docs/yang/parsers/TextParser.rst

@@ -30,14 +30,21 @@ Arguments:
  * **regexp** (mandatory) - Regular expression to apply. Note that it must capture two things at least;
    ``block``, which will be the entire block of configuration relevant for the interface and
    ``key``, which will be the key of the element.
-
+ * **mandatory** (optional) will force the creation of one or more elements by specifying them manually
+   in a dict the ``key``, ``block`` (can be empty string) and any potential ``extra_vars`` you may want to specify.
+ * **composite_key** (optional) is a list of attributes captured in the regexp to be used as the key for the element.
+ * **flat** (optional) if set to ``true`` (default is ``false``) the parser will understand the configuration for the
+   element consists of flat commands instead of nested (for example BGP neighbors or static routes)
+ * **key** (optional) set key manually
+ * **post_process_filter** (optional) - Modify the key with this Jinja expression. ``key`` and ``extra_vars``
+   variables are available.
 
 Example 1
 
   Capture the interfaces::
 
     _process:
-        mode: block
+      - mode: block
         regexp: "(?P<block>interface (?P<key>(\\w|-)*\\d+)\n(?:.|\n)*?^!$)"
         from: "{{ bookmarks.interfaces }}"
 
@@ -66,22 +73,94 @@ Example 2
 
     subinterface:
         _process:
-            mode: block
+          - mode: block
             regexp: "(?P<block>interface {{interface_key}}\\.(?P<key>\\d+)\\n(?:.|\\n)*?^!$)"
             from: "{{ bookmarks.interfaces }}"
 
 
-Example 3.
+Example 3
 
   Sometimes we can get easily more information in one go than just the ``key`` and the ``block``. For
   those cases we can capture more groups and they will be stored in the ``extra_vars`` dictionary::
 
         address:
             _process:
-                mode: block
+              - mode: block
                 regexp: "(?P<block>ip address (?P<key>(?P<ip>.*))\\/(?P<prefix>\\d+))(?P<secondary> secondary)*"
                 from: "{{ bookmarks['parent'] }}"
 
+Example 4
+
+  In some cases native configuration might be "flat" but nested in a YANG model. This is the case of the `global`
+  or `default` VRF, in those cases, it is hard you may want to ensure that `global` VRF is always created::
+
+        _process:
+          - mode: block
+            regexp: "(?P<block>vrf definition (?P<key>(.*))\n(?:.|\n)*?^!$)"
+            from: "{{ bookmarks['network-instances'][0] }}"
+            mandatory:
+                - key: "global"
+                  block: ""
+                  extra_vars: {}
+
+Example 5
+
+  Some list elements have composite keys, if that's the case, use the composite key to tell the parser how to map
+  captured elements to the composite key::
+
+        protocols:
+            _process: unnecessary
+            protocol:
+                _process:
+                    - mode: block
+                      regexp: "(?P<block>router (?P<protocol_name>(bgp))\\s*(?P<process_id>\\d+)*\n(?:.|\n)*?)^(!|   vrf \\w+)$"
+                      from: "{{ bookmarks['network-instances'][0] }}"
+                      composite_key: [protocol_name, protocol_name]
+                      when: "{{ network_instance_key == 'global' }}"
+
+Example 6
+
+  Some list elements (like static routes or BGP neighbors) are configured as a flat list of commands instead of
+  nested. By default, if you would try to parse each command individually the parser would try to create
+  a new element with each line and fail as multiple lines belong to the same element but they are treated independently.
+  By setting ``flat: true`` this behavior is changed and subsequent commands will update an already created object::
+
+    bgp:
+        neighbors:
+            neighbor:
+                _process:
+                    - mode: block
+                      regexp: "(?P<block>neighbor (?P<key>\\d+.\\d+.\\d+.\\d+).*)"
+                      from: "{{ bookmarks['protocol'][protocol_key] }}"
+                      flat: true
+
+Example 7
+
+  In some rare cases you might not be able to extract the key directly from the configuration. For example,
+  the ``static`` protocol consists of ``ip route`` commands. In that case you can set the key yourself::
+
+    protocols:
+        protocol:
+            _process:
+                - mode: block
+                  regexp: "(?P<block>ip route .*\n(?:.|\n)*?^!$)"
+                  from: "{{ bookmarks['network-instances'][0] }}"
+                  key: "static static"
+
+Example 8
+
+  Sometimes you need to transform the key value. For example, static routes require the prefix in CIDR format,
+  but Cisco IOS outputs routes in ``<network> <mask>`` format. In that case you can use ``post_process_filter`` to
+  apply additional filters::
+
+    static:
+        _process:
+            - mode: block
+               regexp: "(?P<block>ip route (?P<key>\\d+\\S+ \\d+\\S+).*)"
+               from: "{{ bookmarks['network-instances'][0] }}"
+               post_process_filter: "{{ key|addrmask_to_cidr }}"
+
+
 Leaf - search
 -------------
 
@@ -99,7 +178,7 @@ Example.
 
     description:
         _process:
-            mode: search
+          - mode: search
             regexp: "description (?P<value>.*)"
             from: "{{ bookmarks.interface[interface_key] }}"
 
@@ -118,7 +197,7 @@ Example.
 
     secondary:
         _process:
-            mode: value
+          - mode: value
             value: "{{ extra_vars.secondary != None }}"
 
 Leaf - is_absent
@@ -136,7 +215,7 @@ Example.
             _process: unnecessary
             enabled:
                 _process:
-                    mode: is_absent
+                  - mode: is_absent
                     regexp: "(?P<value>^\\W*switchport$)"
                     from: "{{ bookmarks['parent'] }}"
 
@@ -151,7 +230,7 @@ Example.
 
     enabled:
         _process:
-            mode: is_present
+          - mode: is_present
             regexp: "(?P<value>no shutdown)"
             from: "{{ bookmarks.interface[interface_key] }}"
 
@@ -172,7 +251,7 @@ Example.
   Check type of interface by extracting the name and doing a lookup::
 
     _process:
-        mode: map
+      - mode: map
         regexp: "(?P<value>(\\w|-)*)\\d+"
         from: "{{ interface_key }}"
         map:
@@ -181,4 +260,3 @@ Example.
             Loopback: softwareLoopback
             Port-Channel: ieee8023adLag
             Vlan: l3ipvlan
-

docs/yang/parsers/XMLParser.rst

@@ -27,6 +27,7 @@ Arguments:
 
 * **xpath** (mandatory): elements to traverse
 * **key** (mandatory): which element is the key of the list
+* **post_process_filter** (optional): modify the key with this Jinja2 expression
 
 Example:
 
@@ -35,7 +36,7 @@ Example:
 
     interface:
         _process:
-            mode: xpath
+          - mode: xpath
             xpath: "interfaces/interface"
             key: name
             from: "{{ bookmarks.interfaces }}"
@@ -73,7 +74,7 @@ Example:
 
     description:
         _process:
-            mode: xpath
+          - mode: xpath
             xpath: description
             from: "{{ bookmarks['parent'] }}"
 
@@ -92,7 +93,7 @@ Example:
 
     name:
         _process:
-            mode: value
+          - mode: value
             value: "{{ interface_key }}"
 
 Leaf - map
@@ -114,7 +115,7 @@ Example:
 
     type:
         _process:
-            mode: map
+          - mode: map
             xpath: name
             regexp: "(?P<value>[a-z]+).*"
             from: "{{ bookmarks['parent'] }}"
@@ -137,7 +138,7 @@ Example:
 
     enabled:
         _process:
-            mode: is_absent
+          - mode: is_absent
             xpath: "disable"
             from: "{{ bookmarks['parent'] }}"
 
@@ -148,4 +149,3 @@ Leaf - is_present
 -----------------
 
 Works exactly like ``xpath`` but if the evaluation is ``None``, it will return ``False``.
-