[WIP] Wait for and print the results of databricks pipelines test

.nextchanges/cli/pipelines-test-results.md

@@ -0,0 +1 @@
+* `databricks pipelines test` waits for the test results, prints them, and fails when a test fails or the results are incomplete. ([#7272](https://github.com/databricks/cli/pull/7272))

.nextchanges/cli/pipelines-test.md

@@ -0,0 +1 @@
+* Add the `databricks pipelines test` command, which runs a pipeline's tests in a test-only update. ([#7271](https://github.com/databricks/cli/pull/7271))

acceptance/pipelines/databricks-cli-help/output.txt

@@ -30,6 +30,7 @@ Main Commands
   run                   Run a pipeline
   show                  Preview a table's columns and sample rows
   stop                  Stop a pipeline
+  test                  *Preview* Run the tests of a pipeline
 
 Management Commands
   apply-environment     *Public Preview* Apply the latest environment to the pipeline.

acceptance/pipelines/test/arguments/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/arguments/output.txt

@@ -0,0 +1,97 @@
+
+>>> [CLI] pipelines test --help
+Run the tests of a pipeline in a test-only update.
+
+PIPELINE_KEY is the unique name of the pipeline, as defined in its YAML file.
+It is required when test selections are given. If there is only one pipeline
+in the project and no arguments are given, it is selected automatically.
+Outside a project, identify the pipeline with --pipeline-id; all positional
+arguments are then test selections.
+
+TEST_SELECTION is a test directory, file, or pytest test ID, for example
+tests/test_orders.py::test_backfill. In a project, a relative path is a local
+path, relative to the working directory, and is converted to its deployed
+workspace path. With --pipeline-id, a relative path is relative to the
+pipeline's root_path. Paths that start with / are workspace paths. Without a
+selection, all of the pipeline's tests run.
+
+Each --pytest-arg passes one argument to pytest, in the order given. Only -k
+and -m are supported, each followed by its value in the next --pytest-arg, for
+example:
+
+  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
+
+The command waits for the results and succeeds only if at least one test was
+collected and none failed or errored. --no-wait returns after the update starts.
+--timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
+to read the results later. To stop the update, run 'pipelines stop'.
+
+Usage:
+  databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
+
+Pipeline Test Flags:
+      --no-wait                  Return after the test update starts, without waiting for results.
+      --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
+      --resume string            Read the results of an existing test update instead of starting one.
+      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
+
+Flags:
+  -h, --help   help for test
+
+Global Flags:
+      --debug            enable debug logging
+  -o, --output type      output type: text or json (default text)
+  -p, --profile string   ~/.databrickscfg profile
+  -t, --target string    bundle target to use (if applicable)
+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
+
+=== pytest arguments must be passed with --pytest-arg
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
+Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -msmoke
+Error: -m is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
+Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
+
+=== Only -k and -m are supported
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
+Error: unsupported pytest argument "-x": only -k and -m are supported
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-k
+Error: pytest argument -k requires a value: pass it in the next --pytest-arg
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=tests/test_orders.py
+Error: unsupported pytest argument "tests/test_orders.py": pass test paths and test IDs as TEST_SELECTION arguments
+
+=== -k and -m take their value in the next --pytest-arg
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-korders
+Error: pytest argument "-korders" must be split: pass --pytest-arg=-k --pytest-arg=orders
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-k --pytest-arg=-m
+Error: pytest argument -k requires a value, not "-m": pass it in the next --pytest-arg
+
+=== Incompatible run-control flags
+>>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
+Error: --resume cannot be used with test selections
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
+Error: --resume cannot be used with --pytest-arg
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
+Error: --resume cannot be used with --no-wait
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
+Error: --timeout cannot be used with --no-wait
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
+Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 
+Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
+
+=== JSON output is not supported
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
+Error: --output json is not supported by pipelines test

acceptance/pipelines/test/arguments/script

@@ -0,0 +1,26 @@
+trace $CLI pipelines test --help
+
+title "pytest arguments must be passed with --pytest-arg"
+trace musterr $CLI pipelines test --pipeline-id 1234 -k orders
+trace musterr $CLI pipelines test --pipeline-id 1234 -msmoke
+trace musterr $CLI pipelines test --pipeline-id 1234 -- -k orders
+
+title "Only -k and -m are supported"
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-x
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-k
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=tests/test_orders.py
+
+title "-k and -m take their value in the next --pytest-arg"
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-korders
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-k --pytest-arg=-m
+
+title "Incompatible run-control flags"
+trace musterr $CLI pipelines test --pipeline-id 1234 tests --resume 5678
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --no-wait
+trace musterr $CLI pipelines test --pipeline-id 1234 --no-wait --timeout 10m
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume "5678' OR '1'='1"
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume ""
+
+title "JSON output is not supported"
+trace musterr $CLI pipelines test --pipeline-id 1234 -o json

acceptance/pipelines/test/immutable/databricks.yml

@@ -0,0 +1,13 @@
+bundle:
+  name: test-pipeline-test-immutable
+
+workspace:
+  sync_mode: immutable
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-test-immutable
+      libraries:
+        - file:
+            path: pipeline_file.py

acceptance/pipelines/test/immutable/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/immutable/output.txt

@@ -0,0 +1,53 @@
+
+>>> [CLI] pipelines deploy
+Created internal_immutable_snapshots.immutable
+Created pipelines.my_pipeline
+Files: 0 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
+
+=== Local test paths are not supported with workspace.sync_mode immutable
+>>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
+
+=== Absolute workspace paths and running all tests still work
+>>> [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> [CLI] pipelines test my_pipeline --no-wait
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
+      ]
+    }
+  }
+}
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true
+  }
+}

acceptance/pipelines/test/immutable/pipeline_file.py

@@ -0,0 +1,6 @@
+from pyspark import pipelines as dp
+
+
+@dp.table
+def orders():
+    return spark.range(10)

acceptance/pipelines/test/immutable/script

@@ -0,0 +1,12 @@
+trace $CLI pipelines deploy
+rm "$OUT_REQUESTS"
+
+title "Local test paths are not supported with workspace.sync_mode immutable"
+trace musterr $CLI pipelines test my_pipeline tests/test_orders.py::test_backfill
+
+title "Absolute workspace paths and running all tests still work"
+trace $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
+# Like the service, the test server rejects a new update while one is active.
+$CLI pipelines stop my_pipeline &>> LOG.stop
+trace $CLI pipelines test my_pipeline --no-wait
+trace print_requests.py //updates

acceptance/pipelines/test/immutable/tests/test_orders.py

@@ -0,0 +1,2 @@
+def test_backfill():
+    assert True

acceptance/pipelines/test/many-cases/databricks.yml

@@ -0,0 +1,11 @@
+bundle:
+  name: test-pipeline-test-many-cases
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-test-many-cases
+      root_path: .
+      libraries:
+        - file:
+            path: pipeline_file.py

acceptance/pipelines/test/many-cases/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/many-cases/output.txt

@@ -0,0 +1,13 @@
+
+>>> [CLI] pipelines deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...
+Created pipelines.my_pipeline
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
+
+=== Every case is counted once across pages, polls, and the final read
+>>> tail -n 1 LOG.many
+200 tests: 200 passed, 0 failed, 0 skipped, 0 errors
+Event pages read: 8
+Pages read with page_token: 4

acceptance/pipelines/test/many-cases/pipeline_file.py

@@ -0,0 +1,6 @@
+from pyspark import pipelines as dp
+
+
+@dp.table
+def orders():
+    return spark.range(10)

acceptance/pipelines/test/many-cases/script

@@ -0,0 +1,9 @@
+trace $CLI pipelines deploy
+rm "$OUT_REQUESTS"
+
+title "Every case is counted once across pages, polls, and the final read"
+$CLI pipelines test my_pipeline &> LOG.many
+trace tail -n 1 LOG.many
+print_requests.py --get //events --oneline > LOG.events
+echo "Event pages read: $(wc -l < LOG.events | tr -d ' ')"
+echo "Pages read with page_token: $(grep -c page_token LOG.events)"