Add TTP Parser with template overloads from Git

README.md

@@ -19,15 +19,15 @@ Regardless, the Onboarding App greatly simplifies the onboarding process by allo
 
 ### Support Matrix (Sync Devices From Network)
 
-|     Data Attribute      | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  | HP Comware |
-| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: | :-: |
-| Hostname                | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Platform                | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Manufacturer            | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Serial Number           | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Device Type             | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Mgmt Interface          | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
-| Mgmt IP Address         | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 |
+|     Data Attribute      | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  | HP Comware | Palo Alto Panos |
+| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: | :-: | :-: |
+| Hostname                | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Platform                | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Manufacturer            | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Serial Number           | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Device Type             | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Mgmt Interface          | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
+| Mgmt IP Address         | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | 🧪 | 🧪 |
 
 ### Support Matrix (Sync Data From Network)
 

changes/274.added

@@ -0,0 +1,5 @@
+Added TTP Parser support.
+Added ability to load TTP templates from Git Repository.
+Added TTP template precedence loading from Git Repository.
+Added Sync Device from Network support for Palo Alto Panos.
+Added a ability to use nautobot-app-nornir connection option extras in Sync Devices job.

development/nautobot_config.py

@@ -147,5 +147,13 @@
                 },
             },
         },
+        "connection_options": {
+            "netmiko": {
+                "extras": {
+                    "fast_cli": False,
+                    "read_timeout_override": 30,
+                },
+            },
+        },
     },
 }

docs/dev/arch_decision.md

@@ -4,6 +4,6 @@ The intention is to document deviations from a standard Model View Controller (M
 
 ## Handling the Nornir Inventory
 
-In order for Nornir to function, an inventory is created. There are multiple supported inventory sources that fit many needs; however there is a unique requirement that this plugin is trying to solve. The problem is specifically around the first SSoT job (Sync Devices from Network); how can we create an inventory when there is no source "yet"? Our solution to this problem is to generate a empty inventory, and then process the ip addresses from the job form to create a inventory in an on demand fashion and inject the credentials into the inventory based on the secrets group selected.
+In order for Nornir to function, an inventory is created. There are multiple supported inventory sources that fit many needs; however there is a unique requirement that this plugin is trying to solve. The problem is specifically around the first SSoT job (Sync Devices from Network); how can we create an inventory when there is no source "yet"? Our solution to this problem is to generate a empty inventory, and then process the ip addresses from the job form to create a inventory in an on demand fashion and inject the credentials into the inventory based on the secrets group selected. This inventory does support loading in Nornir connection options `extras` by using the `nautobot-app-nornir` PLUGIN_CONFIG.
 
 For the general application constraint for this ADR see the [Credentials Section](../user/app_getting_started.md#device-credentials-functionality).

docs/user/app_detailed_design.md

@@ -21,9 +21,10 @@ To learn how to extend the app, or update its default YAML definitions visit [Ex
     - Other required fields are selected in the job inputs form.
 
 2. The SSoT framework loads the Nautobot adapter information.
-3. The SSoT frameworks network adapter `load()` method calls nornir functionality.
+3. The SSoT frameworks network adapter `load()` method calls Nornir functionality.
     - The job inputs data is passed to the InitNornir initializer, because we only have basic information a custom `EmptyInventory` Nornir inventory plugin is packaged with the App. This get initialized in the `InitNornir` function, but actually initializes a true inventory that is empty.
     - Since `Platform` information may need to be auto-detected before adding a Nornir `Host` object to the inventory, a `create_inventory` function is executed that uses the SSH-Autodetect via Netmiko to try to determine the platform so it can be injected into the `Host` object.
+    - Load in the `PLUGIN_CONFIG` to see if extra connection options need to be added to the `Host` connection_option definition.
     - Finally, all the platform specific commands to run, along with all the jpath, `post_processor` information loaded from the platform specific YAML files must be injected into the Nornir data object to be accessible later in the extract, transform functions.
 4. Within the context of a Nornir `with_processor` context manager call the `netmiko_send_commands` Nornir task.
     - Access the loaded platform specific YAML data and deduplicate commands to avoid running the same command multiple times. E.g. Multiple required data attributes come from the same show command.

docs/user/app_getting_started.md

@@ -49,6 +49,14 @@ The new SSoT based jobs each use their own Nornir inventories.
                 },
             },
         },
+        "connection_options": {
+            "netmiko": {
+                "extras": {  # <==== passed into the connection setup.
+                    "fast_cli": False,
+                    "read_timeout_override": 30,
+                },
+            },
+        },
     },
 ```
 !!! info

docs/user/app_overview.md

@@ -15,7 +15,7 @@ This section descibes the new implementaiton (SSoT Implementation) and the Origi
 
 The new implementation of device onboarding in this app is utilizing the SSoT framework; the main reasons for providing the new mechanisms were to solve the following challenges:
 
-- Make it easier to extending and add additonal vendor/OS support.
+- Make it easier to extending and add additional vendor/OS support.
 - Collapse this app and the external [Network Importer](https://github.com/networktocode/network-importer) into the same Nautobot app for simplified device onboarding with more object support.'
     - Remove the Batfish dependency.
 - Re-use backend plugins and libraries such as `nautobot-app-nornir` to provide the a similar feeling to other plugins like `nautobot-app-golden-config`.
@@ -24,7 +24,7 @@ The new implementation of device onboarding in this app is utilizing the SSoT fr
 
 Expose two new SSoT based Nautobot jobs to perform the syncing of data.
 
-1. `Sync Devices From Network` - Takes mininum inputs nearly identical to the original job (IP, Locaiton, SecretGroup), and create a device with bare minium information to be able to manage a device. This job syncs data from the network itself and creates a device with the follow attributes.
+1. `Sync Devices From Network` - Takes minimum inputs nearly identical to the original job (IP, Location, SecretGroup), and create a device with bare minium information to be able to manage a device. This job syncs data from the network itself and creates a device with the follow attributes.
     - Hostname
     - Serial Number
     - Device Type
@@ -56,7 +56,7 @@ Additional References:
 
 - For more information see [App Use Cases](./app_use_cases.md).
 - To understand the lower level details of how the Network-SSoT framework is designed see [Network-SSoT Design](./app_detailed_design.md)
-- To learn how to add additonal platform/OS support visit [Extending](./external_interactions.md).
+- To learn how to add additional platform/OS support visit [Extending](./external_interactions.md).
 
 ### Original Implementation
 

docs/user/app_use_cases.md

@@ -55,6 +55,29 @@ The Onboarding App will automatically create Platforms for vendor operating syst
 ![cisco_ios_platform](../images/platform_cisco_ios.png)
 ![juniper_junos_platform](../images/platform_juniper_junos.png)
 
+### Passing Custom Nornir Connection Options
+
+Device Onboarding 4.0 uses Netmiko as the automation engine that queries the devices for information; more specifically, nornir-netmiko. To extend the device onboarding app to pass `extras` to the connection options the following can be added to `nautobot_plugin_nornir` `PLUGIN_CONFIG`.
+
+```python
+PLUGINS_CONFIG = {
+    "nautobot_device_onboarding": {},
+    "nautobot_plugin_nornir": {
+        "nornir_settings": {
+            "... omitted ..."},
+        "connection_options": {
+            "netmiko": {
+                "extras": {  # <==== passed into the connection setup.
+                    "fast_cli": False,
+                    "read_timeout_override": 30,
+                },
+            },
+        },
+    },
+}
+```
+
+When the on-demand inventory is created for the `Sync Device from Network` job, the extras in the `netmiko` connection dictionary are added to the connection setup.
 
 # Use-cases and common workflows
 
@@ -145,6 +168,28 @@ Required Fields:
 
 ## Using Git(Datasources) to Override the Apps Defaults
 
+### YAML Overrides
+
 By utilizing the Nautobot core feature `Datasource` the command mappers, jpaths, post_processors for each platform can be overridden. This also gives an easy way for a user to add platform support without having to get those fixes directly upstreamed into this application.
 
 The format of these YAML files are and how to extend this application is covered in [App YAML Overrides](./app_yaml_overrides.md).
+
+
+### Parser Templates
+
+As this App continues to mature, support has been added to support `TTP`; with this addition the ability to add and/or override templates was required. This follows a similar pattern to the YAML overrides.
+
+!!! info
+    To avoid overly complicating the merge logic, the App will always prefer the template files loaded in from the git repository.
+
+File structure:
+```bash
+.
+├── README.md
+└── onboarding_command_mappers
+    └── parsers
+        └── ttp
+            └── <network_driver>_<command seperated by underscores>.ttp
+```
+
+When loading from a Git Repository this App is expecting a root directory called `onboarding_command_mappers`. Parser files should be located in a `parsers` directory followed by one additional directory; e.g., `ttp`. The template file names must be named `<network_driver>_<command_seperated_by_underscores>.ttp`.

docs/user/app_use_cases_original.md

@@ -125,7 +125,7 @@ The status of onboarding jobs can be viewed via the UI (Jobs > Job Results) or r
 To run an onboarding task Job via the api:
 
 
-Post to `/api/extras/jobs/Perform%20Device%20Onboarding/run/` with the relevent onboarding data: 
+Post to `/api/extras/jobs/Perform%20Device%20Onboarding/run/` with the relevant onboarding data: 
 
 ```bash
 curl -X "POST" <nautobot URL>/api/extras/jobs/Perform%20Device%20Onboarding/run/ -H "Content-Type: application/json" -H "Authorization: Token $NAUTOBOT_TOKEN" -d '{"data": {"location": "<valid location UUID>", "ip_address": "<reachable IP to onboard>", "port": 22, "timeout": 30}}

docs/user/app_yaml_overrides.md

@@ -15,9 +15,9 @@ This App provides sane defaults that have been tested, the command mapper files
 The YAML file names must be named `<network_driver>.yml`.  Where network_driver must exist in the netutils mapping exposed from Nautobot core.
 
 ## File Placement
-The override files can either be placed directly into the python plugin command mappers directory (by default: `/opt/nautobot/lib64/python<python version>/site-packages/nautobot_device_onboarding/command_mappers/`) or by using a Git Data Source.
+The override files can either be placed directly into the python plugin command mappers directory (by default: `/opt/nautobot/lib64/python<python version>/site-packages/nautobot_device_onboarding/command_mappers/`) or by using a Git Datasources.
 
-### Git Data Source
+### Git Datasources
 
 File structure:
 ```bash

docs/user/faq.md

@@ -49,7 +49,7 @@ Optional arguments are often used to define a `secret` for Cisco devices and oth
 
 ## Does this app support the discovery and the creation of all interfaces and IP Addresses?
 
-**Yes**. The original Deivce Onboarding job/SSot Sync Devices will only discover and create the management interface and the management IP address. Importing all interfaces and IP addresses is available from the SSoT job (Sync Network Data).
+**Yes**. The original Device Onboarding job/SSot Sync Devices will only discover and create the management interface and the management IP address. Importing all interfaces and IP addresses is available from the SSoT job (Sync Network Data).
 
 ## Does this app support the discovery of device based on fqdn?
 
@@ -81,3 +81,25 @@ It's expected that any changes done asynchronously in Nautobot currently (within
 - Several assumptions made for Nornir inventory that would be different in all other Nornir inventory jobs.
 - An inventory created for each device.
     - Causes additional SQL connections which may benefit from the use of `serial` runner.
+
+## How do I pass in extra connection options into Netmiko for the SSoT based jobs?
+
+Device Onboarding 4.0 uses Netmiko as the automation engine that queries the devices for information; more specifically, nornir-netmiko. To extend the device onboarding app to pass `extras` to the connection options the following can be added to `nautobot_plugin_nornir` `PLUGIN_CONFIG`.
+
+```python
+PLUGINS_CONFIG = {
+    "nautobot_device_onboarding": {},
+    "nautobot_plugin_nornir": {
+        "nornir_settings": {
+            "... omitted ..."},
+        "connection_options": {
+            "netmiko": {
+                "extras": {  # <==== passed into the connection setup.
+                    "fast_cli": False,
+                    "read_timeout_override": 30,
+                },
+            },
+        },
+    },
+}
+```

nautobot_device_onboarding/command_mappers/paloalto_panos.yml

@@ -0,0 +1,33 @@
+---
+sync_devices:
+  hostname:
+    commands:
+      - command: "show system info"
+        parser: "ttp"
+        jpath: "[*].hostname"
+        post_processor: "{{ obj[0] }}"
+  serial:
+    commands:
+      - command: "show system info"
+        parser: "ttp"
+        jpath: "[*].serial"
+        post_processor: "{{ obj[0] }}"
+  device_type:
+    commands:
+      - command: "show system info"
+        parser: "ttp"
+        jpath: "[*].model"
+        post_processor: "{{ obj[0] }}"
+  mgmt_interface:
+    commands:
+      - command: "show interface management | match Name"
+        parser: "raw"
+        jpath: "raw"
+        post_processor: "{{ obj.split('Name: ')[1] }}"
+  mask_length:
+    commands:
+      - command: "show system info"
+        parser: "ttp"
+        jpath: "[*].netmask"
+        post_processor: "{{ obj[0] | netmask_to_cidr }}"
+        iterable_type: "int"

nautobot_device_onboarding/constants.py

@@ -3,6 +3,13 @@
 from django.conf import settings
 from nautobot.dcim.utils import get_all_network_driver_mappings
 
+NETMIKO_EXTRAS = (
+    settings.PLUGINS_CONFIG.get("nautobot_plugin_nornir", {})
+    .get("connection_options", {})
+    .get("netmiko", {})
+    .get("extras", {})
+)
+
 PLUGIN_CFG = settings.PLUGINS_CONFIG["nautobot_device_onboarding"]
 
 # This mapping is only used for the original onboarding job.
@@ -19,9 +26,8 @@
 # This is used in the new SSoT based jobs.
 SUPPORTED_NETWORK_DRIVERS = list(get_all_network_driver_mappings().keys())
 
-# This is used in the new SSoT based jobs. Soon TPP, PYATS should be supported.
-# SUPPORTED_COMMAND_PARSERS = ["textfsm", "ttp", "pyats"]
-SUPPORTED_COMMAND_PARSERS = ["textfsm"]
+# This is used in the new SSoT based jobs. Soon PYATS should be supported.
+SUPPORTED_COMMAND_PARSERS = ["textfsm", "ttp"]
 
 # This should potentially be removed and used nautobot core directly choices.
 # from nautobot.dcim.choices import InterfaceTypeChoices

nautobot_device_onboarding/nornir_plays/command_getter.py

@@ -1,6 +1,7 @@
 """CommandGetter."""
 
 import json
+import os
 from typing import Dict
 
 from django.conf import settings
@@ -16,13 +17,16 @@
 from nornir.core.task import Result, Task
 from nornir_netmiko.tasks import netmiko_send_command
 from ntc_templates.parse import parse_output
+from ttp import ttp
 
 from nautobot_device_onboarding.constants import SUPPORTED_COMMAND_PARSERS, SUPPORTED_NETWORK_DRIVERS
 from nautobot_device_onboarding.nornir_plays.empty_inventory import EmptyInventory
 from nautobot_device_onboarding.nornir_plays.inventory_creator import _set_inventory
 from nautobot_device_onboarding.nornir_plays.logger import NornirLogger
 from nautobot_device_onboarding.nornir_plays.processor import CommandGetterProcessor
-from nautobot_device_onboarding.nornir_plays.transform import add_platform_parsing_info
+from nautobot_device_onboarding.nornir_plays.transform import add_platform_parsing_info, load_files_with_precedence
+
+PARSER_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.dirname(__file__)), "parsers"))
 
 InventoryPluginRegister.register("nautobot-inventory", NautobotORMInventory)
 InventoryPluginRegister.register("empty-inventory", EmptyInventory)
@@ -155,6 +159,26 @@ def netmiko_send_commands(
                             except Exception:  # https://github.com/networktocode/ntc-templates/issues/369
                                 task.results[result_idx].result = []
                                 task.results[result_idx].failed = False
+                        if command["parser"] == "ttp":
+                            try:
+                                # Parsing ttp ourselves instead of using netmikos use_<parser> function to be able to handle exceptions
+                                # ourselves.
+                                ttp_template_files = load_files_with_precedence(
+                                    filesystem_dir=f"{PARSER_DIR}/ttp", parser_type="ttp"
+                                )
+                                template_name = f"{task.host.platform}_{command['command'].replace(' ', '_')}.ttp"
+                                parser = ttp(
+                                    data=current_result.result,
+                                    template=ttp_template_files[template_name],
+                                )
+                                parser.parse()
+                                parsed_result = parser.result(format="json")[0]
+                                # task.results[result_idx].result = json.loads(json.dumps(parsed_result))
+                                task.results[result_idx].result = json.loads(parsed_result)
+                                task.results[result_idx].failed = False
+                            except Exception:
+                                task.results[result_idx].result = []
+                                task.results[result_idx].failed = False
             else:
                 if command["parser"] == "raw":
                     raw = {"raw": current_result.result}

nautobot_device_onboarding/nornir_plays/inventory_creator.py

@@ -3,6 +3,8 @@
 from netmiko import SSHDetect
 from nornir.core.inventory import ConnectionOptions, Host
 
+from nautobot_device_onboarding.constants import NETMIKO_EXTRAS
+
 
 def guess_netmiko_device_type(hostname, username, password, port):
     """Guess the device type of host, based on Netmiko."""
@@ -50,6 +52,7 @@ def _set_inventory(host_ip, platform, port, username, password):
                 username=username,
                 password=password,
                 platform=platform,
+                extras=NETMIKO_EXTRAS,
             )
         },
     )