Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat(cv_deploy): Expose the details of the Workspace build validation errors #4629

Open
wants to merge 28 commits into
base: devel
Choose a base branch
from

Conversation

alexeygorbunov
Copy link
Contributor

@alexeygorbunov alexeygorbunov commented Oct 23, 2024

Change Summary

Explicitly expose the details of the:

  • Config Validation errors and warnings raised when part(s) of the proposed DC is rejected by EoS during validation via Config Session
  • Image Validation errors and warnings raised when part(s) of the proposed image bundle is rejected by EoS during validation
  • (descoped) Time taken by CV to process specific Workspace lifecycle stages

Related Issue(s)

Component(s) name

arista.avd.cv_deploy

Proposed changes

  • Fix NTP settings of CI EoS
  • Change to make molecule test results more readable (task names, CVP WS and CC naming convention)
  • Expose undercv_deploy_results.workspace.build_id an ID of the latest build attempt. Needed as WS in Pending state maybe reused and may have multiple Build attempts (with each attempt producing different set of errors)
  • Expose under cv_deploy_results.workspace.build_results:
    • Errors and warnings related to the validation on the proposed configuration.
    • Errors and warnings related to the proposed image changes (not yet implemented in AVD).
    • Details of the specific device associated with faced error/warning
    • (descoped) Time taken by different cv_workflow to execute specific activities on CV (will help identify if delays are related to the specific aspect of the workflow/CV)
  • Allow to:
    • Fully suppress returned build warnings
    • Suppress warnings based on custom regex patterns
    • Suppress warnings based on pre-defined use-cases (like enabling portfast on switchport)
  • Extensive molecule tests

How to test

  • Run proposed updated cv_deploy molecule test feeding in proper Access Token for CI CVaaS tenant
    CVAAS_AAWG_CI=`cat <path/to/secret>` molecule test -s cv_deploy
  • Validate cv_deploy_results outputs (cv_deploy_results.workspace.build_id and cv_deploy_results.workspace.build_results) of the invalid_config step. Compare with other steps (which don't have incorrect DC/image and therefore don't trigger WS Build failures)

Example with default warning suppression settings (no suppression of warnings enabled):

        "workspace": {
            "build_id": "req-0a4c4607-93ed-4291-b290-90c69eaef0b2",
            "build_results": [
                {
                    "config_validation": {
                        "errors": [
                            {
                                "configlet_name": "avd-A7D46613D44DE45D68D5B5C5CBA06B0D",
                                "error_code": "DEVICE_ERROR",
                                "error_msg": "> spanning-tree mode spine2 % Invalid input (at token 2: 'spine2')",
                                "line_num": 17
                            }
                        ],
                        "warnings": []
                    },
                    "device": {
                        "_exists_on_cv": true,
                        "hostname": "avd-ci-spine2",
                        "serial_number": "A7D46613D44DE45D68D5B5C5CBA06B0D",
                        "system_mac_address": "50:00:00:cb:38:c2"
                    },
                    "image_validation": {
                        "errors": [],
                        "image_input_error": "",
                        "warnings": []
                    },
                    "stages_states": [
                        {
                            "stage": "BUILD_STAGE_CONFIG_VALIDATION",
                            "state": "FAIL"
                        },
                        {
                            "stage": "BUILD_STAGE_CONFIGLET_BUILD",
                            "state": "SUCCESS"
                        },
                        {
                            "stage": "BUILD_STAGE_IMAGE_VALIDATION",
                            "state": "SKIPPED"
                        },
                        {
                            "stage": "BUILD_STAGE_INPUT_VALIDATION",
                            "state": "SUCCESS"
                        }
                    ]
                },
                {
                    "config_validation": {
                        "errors": [
                            {
                                "configlet_name": "avd-DCC816CEAC4BBD6319385043AD318362",
                                "error_code": "DEVICE_ERROR",
                                "error_msg": "> spanning-tree mode spine1 % Invalid input (at token 2: 'spine1')",
                                "line_num": 17
                            },
                            {
                                "configlet_name": "avd-DCC816CEAC4BBD6319385043AD318362",
                                "error_code": "DEVICE_ERROR",
                                "error_msg": ">    seqq 10 permit 10.20.0.0/24 eq 32 % Invalid input (at token 0: 'seqq')",
                                "line_num": 73
                            }
                        ],
                        "warnings": [
                            {
                                "configlet_name": "avd-DCC816CEAC4BBD6319385043AD318362",
                                "error_code": "DEVICE_WARNING",
                                "error_msg": "! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION.",
                                "line_num": 41
                            },
                            {
                                "configlet_name": "avd-DCC816CEAC4BBD6319385043AD318362",
                                "error_code": "DEVICE_WARNING",
                                "error_msg": "! /32 IPv4 address is not configured on the interface",
                                "line_num": 49
                            }
                        ]
                    },
                    "device": {
                        "_exists_on_cv": true,
                        "hostname": "avd-ci-spine1",
                        "serial_number": "DCC816CEAC4BBD6319385043AD318362",
                        "system_mac_address": "50:00:00:d7:ee:0b"
                    },
                    "image_validation": {
                        "errors": [],
                        "image_input_error": "",
                        "warnings": []
                    },
                    "stages_states": [
                        {
                            "stage": "BUILD_STAGE_CONFIG_VALIDATION",
                            "state": "FAIL"
                        },
                        {
                            "stage": "BUILD_STAGE_CONFIGLET_BUILD",
                            "state": "SUCCESS"
                        },
                        {
                            "stage": "BUILD_STAGE_IMAGE_VALIDATION",
                            "state": "SKIPPED"
                        },
                        {
                            "stage": "BUILD_STAGE_INPUT_VALIDATION",
                            "state": "SUCCESS"
                        }
                    ]
                },
                {
                    "config_validation": {
                        "errors": [],
                        "warnings": [
                            {
                                "configlet_name": "avd-13C20F1EDCCED2D85F6DB2FB9E3AC5B6",
                                "error_code": "DEVICE_WARNING",
                                "error_msg": "! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION.",
                                "line_num": 59
                            }
                        ]
                    },
                    "device": {
                        "_exists_on_cv": true,
                        "hostname": "avd-ci-leaf1",
                        "serial_number": "13C20F1EDCCED2D85F6DB2FB9E3AC5B6",
                        "system_mac_address": "50:00:00:72:8b:31"
                    },
                    "image_validation": {
                        "errors": [],
                        "image_input_error": "",
                        "warnings": []
                    },
                    "stages_states": [
                        {
                            "stage": "BUILD_STAGE_CONFIGLET_BUILD",
                            "state": "SUCCESS"
                        },
                        {
                            "stage": "BUILD_STAGE_IMAGE_VALIDATION",
                            "state": "SKIPPED"
                        },
                        {
                            "stage": "BUILD_STAGE_INPUT_VALIDATION",
                            "state": "SUCCESS"
                        },
                        {
                            "stage": "BUILD_STAGE_CONFIG_VALIDATION",
                            "state": "SUCCESS"
                        }
                    ]
                }
            ],
            "build_warnings": true,
            "build_warnings_suppress_patterns": [],
            "build_warnings_suppress_portfast": false,
            "change_control_id": null,
            "description": "AVD-CV-DEPLOY-WORKSPACE-INVALID-CONFIGS-RX0B-CONVERGE",
            "force": false,
            "id": "ws-7c93e948-7676-4ce1-9667-7328e098643c",
            "name": "avd-cv-deploy-workspace-invalid-configs-rx0b-converge",
            "requested_state": "built",
            "state": "build failed"
        }

Checklist

User Checklist

  • N/A

Repository Checklist

  • [ x] My code has been rebased from devel before I start
  • [ x] I have read the CONTRIBUTING document.
  • My change requires a change to the documentation and documentation have been updated accordingly.
  • [ x] I have updated molecule CI testing accordingly. (check the box if not applicable)

Copy link

Review docs on Read the Docs

To test this pull request:

# Create virtual environment for this testing below the current directory
python -m venv test-avd-pr-4629
# Activate the virtual environment
source test-avd-pr-4629/bin/activate
# Install all requirements including PyAVD
pip install "pyavd[ansible] @ git+https://github.com/alexeygorbunov/avd.git@cv_deploy_ci#subdirectory=python-avd" --force
# Point Ansible collections path to the Python virtual environment
export ANSIBLE_COLLECTIONS_PATH=$VIRTUAL_ENV/ansible_collections
# Install Ansible collection
ansible-galaxy collection install git+https://github.com/alexeygorbunov/avd.git#/ansible_collections/arista/avd/,cv_deploy_ci --force
# Optional: Install AVD examples
cd test-avd-pr-4629
ansible-playbook arista.avd.install_examples

@github-actions github-actions bot added the state: CI Updated CI scenario have been updated in the PR label Oct 23, 2024
@github-actions github-actions bot added type: documentation Improvements or additions to documentation state: Documentation role Updated labels Oct 29, 2024
@alexeygorbunov alexeygorbunov changed the title Feat(cv_deploy): Expose the details of config validation WS build errors Feat(cv_deploy): Expose the details of the Workspace build validation errors Nov 1, 2024
@alexeygorbunov alexeygorbunov marked this pull request as ready for review November 19, 2024 20:55
@alexeygorbunov alexeygorbunov requested review from a team as code owners November 19, 2024 20:55
Comment on lines 35 to 37
build_warnings: "{{ cv_workspace_build_warnings.enabled | arista.avd.default(true) }}"
build_warnings_suppress_patterns: "{{ cv_workspace_build_warnings.suppress_patterns | arista.avd.default([]) }}"
build_warnings_suppress_portfast: "{{ cv_workspace_build_warnings.suppress_portfast | arista.avd.default(false) }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need the default filter here since you already set them in the role defaults.

Copy link
Contributor Author

@alexeygorbunov alexeygorbunov Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, but had to fallback to plain-level variables now (vs dict), otherwise defining only one key (like 'enabled') when calling role would make other dict attributes undefined

was:

cv_workspace_build_warnings:
  enabled: true
  suppress_patterns: []
  suppress_portfast: false

now:

cv_workspace_build_warnings_enabled: true
cv_workspace_build_warnings_suppress_patterns: []
cv_workspace_build_warnings_suppress_portfast: false

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise go back to dict and just remove it from default/main.yml and add back the arista.avd.default filters here. Up to you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep current flatten approach (used by most of the other cv_deploy's vars anyway) as long as you are OK with it

python-avd/pyavd/_cv/workflows/finalize_workspace_on_cv.py Outdated Show resolved Hide resolved
python-avd/pyavd/_cv/workflows/finalize_workspace_on_cv.py Outdated Show resolved Hide resolved
python-avd/pyavd/_cv/workflows/finalize_workspace_on_cv.py Outdated Show resolved Hide resolved
python-avd/pyavd/_cv/workflows/models.py Outdated Show resolved Hide resolved
python-avd/pyavd/_cv/workflows/models.py Outdated Show resolved Hide resolved
@ClausHolbechArista
Copy link
Contributor

Moving to draft until comments have been addressed.

@ClausHolbechArista ClausHolbechArista marked this pull request as draft December 16, 2024 07:56
@github-actions github-actions bot added the state: conflict PR with conflict label Dec 16, 2024
Copy link

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@github-actions github-actions bot removed the state: conflict PR with conflict label Dec 17, 2024
Copy link

Conflicts have been resolved. A maintainer will review the pull request shortly.

@alexeygorbunov alexeygorbunov marked this pull request as ready for review December 19, 2024 00:05
@@ -12,46 +12,50 @@
structured_dir: "{{ playbook_dir }}/intended/structured_configs/test_configs"
intended_tag_device: avd-ci-leaf1
intended_tags: "{{ lookup('file', structured_dir ~ '/' ~ intended_tag_device ~ '.yml')| from_yaml }}"
test_id: "cc-false"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add tags with this name on all the tasks related to this test? To make it easier to just run some of them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. See commit 'PR comments - tag cv_deploy molecule play tasks for flexible scope control'

!
ntp server vrf MGMT 1.pool.ntp.org iburst local-interface Management1
!
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure all config files have a closing new-line. It seems like some do and some dont.

Copy link
Contributor Author

@alexeygorbunov alexeygorbunov Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 'PR comments - end-of-file-fixer for avd-ci-core1.cfg'.
Easy to miss as pre-commit excludes these molecule files

@@ -0,0 +1,2 @@
hostname: avd-ci-leaf2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove these files now, since they only contain the default values.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit PR comments - Delete structure_config files containing only default settings

build_id: str,
time: datetime | None = None,
timeout: float = DEFAULT_API_TIMEOUT,
) -> WorkspaceBuildDetailsStreamResponse:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please try to normalise the output a bit, so we don't return raw response objects. The resonses object is an async generator, so we need to process it inside this function to catch errors correctly. You could do return [response.to_dict(Casing.SNAKE) async for response in reponses]

Copy link
Contributor Author

@alexeygorbunov alexeygorbunov Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 'PR Comments - normalize return of get_workspace_build_details'. Tested in CI environment with cv_deploy molecule scenario

@alexeygorbunov alexeygorbunov marked this pull request as draft January 7, 2025 18:19
@alexeygorbunov alexeygorbunov marked this pull request as ready for review January 8, 2025 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state: CI Updated CI scenario have been updated in the PR state: Documentation role Updated type: documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants