Skip to content

Commit

Permalink
Forbid non-str tagged values (#4)
Browse files Browse the repository at this point in the history
* WIP

* ScalarNode -> Node

* Simplify

* Simplify

* Simplify

* Update docstring

* WIP

* WIP

* Revert UWYAMLTag class hierarchy

* Revert unnecessary class change

* Rename function

* WIP

* convert() -> @Property convert

* WIP

* Tests pass

* Improve test-function names

* Improve test-function names

* Unit tests @ 100%

* Doc updates

* Simplify

* Update notebooks

* Revert change to pyproject.toml

* Remove commented-out breakpoint

* Update

* Improve non-str hint

* Improve non-str hint

* Doc update

* Custom UWYAMLConvert __repr__()
  • Loading branch information
maddenp-noaa authored Jan 6, 2025
1 parent 63b70bd commit 986226f
Show file tree
Hide file tree
Showing 13 changed files with 238 additions and 197 deletions.
56 changes: 28 additions & 28 deletions docs/sections/user_guide/cli/tools/config/realize-verbose.out
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
[2024-11-27T05:24:34] DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2024-11-27T05:24:34] DEBUG Reading input from stdin
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG hello: '{{ recipient }}'
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: {{ recipient }}
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG hello: world
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG Dereferencing, final value:
[2024-11-27T05:24:34] DEBUG hello: world
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG Writing output to stdout
[2025-01-05T21:15:07] DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2025-01-05T21:15:07] DEBUG Reading input from stdin
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: '{{ recipient }}'
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: {{ recipient }}
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: world
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, final value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: world
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG Writing output to stdout
hello: world
recipient: world
42 changes: 21 additions & 21 deletions docs/sections/user_guide/cli/tools/config/validate-verbose.out
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[2024-11-27T05:24:34] DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2024-11-27T05:24:34] DEBUG Using schema file: schema.jsonschema
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG values:
[2024-11-27T05:24:34] DEBUG greeting: Hello
[2024-11-27T05:24:34] DEBUG recipient: World
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: values
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: values
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: greeting
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: greeting
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: Hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: Hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: World
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: World
[2024-11-27T05:24:34] DEBUG Dereferencing, final value:
[2024-11-27T05:24:34] DEBUG values:
[2024-11-27T05:24:34] DEBUG greeting: Hello
[2024-11-27T05:24:34] DEBUG recipient: World
[2024-11-27T05:24:34] INFO 0 schema-validation errors found in config
[2025-01-05T21:15:07] DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2025-01-05T21:15:07] DEBUG Using schema file: schema.jsonschema
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] values:
[2025-01-05T21:15:07] DEBUG [dereference] greeting: Hello
[2025-01-05T21:15:07] DEBUG [dereference] recipient: World
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: values
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: values
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: greeting
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: greeting
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: Hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: Hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: World
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: World
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, final value:
[2025-01-05T21:15:07] DEBUG [dereference] values:
[2025-01-05T21:15:07] DEBUG [dereference] greeting: Hello
[2025-01-05T21:15:07] DEBUG [dereference] recipient: World
[2025-01-05T21:15:07] INFO 0 schema-validation errors found in config
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[2024-08-26T23:39:19] DEBUG Command: uw rocoto realize --config-file rocoto.yaml --verbose
[2024-08-26T23:39:19] DEBUG Dereferencing, current value:
[2024-08-26T23:39:19] DEBUG workflow:
[2024-08-26T23:39:19] DEBUG attrs:
[2024-08-26T23:39:19] DEBUG realtime: false
[2024-08-26T23:39:19] DEBUG scheduler: slurm
[2024-08-26T23:39:19] DEBUG cycledef:
[2024-08-26T23:39:19] DEBUG - attrs:
[2024-08-26T23:39:19] DEBUG group: howdy
[2024-08-26T23:39:19] DEBUG spec: 202209290000 202209300000 06:00:00
[2025-01-05T21:15:07] DEBUG Command: uw rocoto realize --config-file rocoto.yaml --verbose
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] workflow:
[2025-01-05T21:15:07] DEBUG [dereference] attrs:
[2025-01-05T21:15:07] DEBUG [dereference] realtime: false
[2025-01-05T21:15:07] DEBUG [dereference] scheduler: slurm
[2025-01-05T21:15:07] DEBUG [dereference] cycledef:
[2025-01-05T21:15:07] DEBUG [dereference] - attrs:
[2025-01-05T21:15:07] DEBUG [dereference] group: howdy
[2025-01-05T21:15:07] DEBUG [dereference] spec: 202209290000 202209300000 06:00:00
...
[2024-08-26T23:39:20] DEBUG cycledefs: howdy
[2024-08-26T23:39:20] DEBUG account: '&ACCOUNT;'
[2024-08-26T23:39:20] DEBUG command: echo hello $person
[2024-08-26T23:39:20] DEBUG jobname: hello
[2024-08-26T23:39:20] DEBUG native: --reservation my_reservation
[2024-08-26T23:39:20] DEBUG nodes: 1:ppn=1
[2024-08-26T23:39:20] DEBUG walltime: 00:01:00
[2024-08-26T23:39:20] DEBUG envars:
[2024-08-26T23:39:20] DEBUG person: siri
[2024-08-26T23:39:20] INFO 0 Rocoto XML validation errors found
[2025-01-05T21:15:07] DEBUG [dereference] cycledefs: howdy
[2025-01-05T21:15:07] DEBUG [dereference] account: '&ACCOUNT;'
[2025-01-05T21:15:07] DEBUG [dereference] command: echo hello $person
[2025-01-05T21:15:07] DEBUG [dereference] jobname: hello
[2025-01-05T21:15:07] DEBUG [dereference] native: --reservation my_reservation
[2025-01-05T21:15:07] DEBUG [dereference] nodes: 1:ppn=1
[2025-01-05T21:15:07] DEBUG [dereference] walltime: 00:01:00
[2025-01-05T21:15:07] DEBUG [dereference] envars:
[2025-01-05T21:15:07] DEBUG [dereference] person: siri
[2025-01-05T21:15:07] INFO 0 Rocoto XML validation errors found
6 changes: 4 additions & 2 deletions docs/sections/user_guide/yaml/tags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ Or explicit:
integer: !!int "3"
float: !!float "3.14"
Additionally, UW defines the following tags to support use cases not covered by standard tags:
Additionally, UW defines the following tags to support use cases not covered by standard tags. Where standard YAML tags are applied to their values immediately, application of UW YAML tags is delayed until after Jinja2 expressions in tagged values are dereferenced.

**NB** Values tagged with UW YAML tags must be strings. Use quotes as necessary to ensure that they are.

``!bool``
^^^^^^^^^
Expand All @@ -32,7 +34,7 @@ Converts the tagged node to a Python ``bool`` object. For example, given ``input
flag1: True
flag2: !bool "{{ flag1 }}"
flag3: !bool 0
flag3: !bool "0"
.. code-block:: text
Expand Down
10 changes: 8 additions & 2 deletions notebooks/config.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@
"text": [
"Help on function realize in module uwtools.api.config:\n",
"\n",
"realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[str, int]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n",
"realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[bool, float, int, str]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n",
" Realize a config based on a base input config and an optional update config.\n",
"\n",
" The input config may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it\n",
Expand Down Expand Up @@ -1491,6 +1491,12 @@
" | :param src: The dictionary with new data to use.\n",
" |\n",
" | ----------------------------------------------------------------------\n",
" | Readonly properties inherited from uwtools.config.formats.base.Config:\n",
" |\n",
" | config_file\n",
" | Return the path to the config file from which this object was instantiated, if applicable.\n",
" |\n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors inherited from uwtools.config.formats.base.Config:\n",
" |\n",
" | __dict__\n",
Expand Down Expand Up @@ -1555,7 +1561,7 @@
" |\n",
" | update(self, other=(), /, **kwds)\n",
" | D.update([E, ]**F) -> None. Update D from mapping/iterable E and F.\n",
" | If E present and has a .keys() method, does: for k in E: D[k] = E[k]\n",
" | If E present and has a .keys() method, does: for k in E.keys(): D[k] = E[k]\n",
" | If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v\n",
" | In either case, this is followed by: for k, v in F.items(): D[k] = v\n",
" |\n",
Expand Down
64 changes: 32 additions & 32 deletions notebooks/rocoto.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -256,13 +256,13 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] ERROR 3 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] ERROR Error at workflow -> attrs:\n",
"[2024-11-19T23:15:43] ERROR 'realtime' is a required property\n",
"[2024-11-19T23:15:43] ERROR Error at workflow -> tasks -> task_greet:\n",
"[2024-11-19T23:15:43] ERROR 'command' is a required property\n",
"[2024-11-19T23:15:43] ERROR Error at workflow:\n",
"[2024-11-19T23:15:43] ERROR 'log' is a required property\n"
"[2025-01-05T21:26:43] ERROR 3 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] ERROR Error at workflow.attrs:\n",
"[2025-01-05T21:26:43] ERROR 'realtime' is a required property\n",
"[2025-01-05T21:26:43] ERROR Error at workflow.tasks.task_greet:\n",
"[2025-01-05T21:26:43] ERROR 'command' is a required property\n",
"[2025-01-05T21:26:43] ERROR Error at workflow:\n",
"[2025-01-05T21:26:43] ERROR 'log' is a required property\n"
]
},
{
Expand Down Expand Up @@ -388,8 +388,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -577,8 +577,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -722,8 +722,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -925,7 +925,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -1001,22 +1001,22 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] ERROR 4 Rocoto XML validation errors found\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_ATTRVALID: Element workflow failed to validate attributes\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_NOELEM: Expecting an element cycledef, got nothing\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_INTERSEQ: Invalid sequence in interleave\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element workflow failed to validate content\n",
"[2024-11-19T23:15:43] ERROR Invalid Rocoto XML:\n",
"[2024-11-19T23:15:43] ERROR 1 <?xml version='1.0' encoding='utf-8'?>\n",
"[2024-11-19T23:15:43] ERROR 2 <workflow realtime=\"False\">\n",
"[2024-11-19T23:15:43] ERROR 3 <log>logs/test.log</log>\n",
"[2024-11-19T23:15:43] ERROR 4 <task name=\"greet\">\n",
"[2024-11-19T23:15:43] ERROR 5 <cores>1</cores>\n",
"[2024-11-19T23:15:43] ERROR 6 <walltime>00:00:10</walltime>\n",
"[2024-11-19T23:15:43] ERROR 7 <command>echo Hello, World!</command>\n",
"[2024-11-19T23:15:43] ERROR 8 <jobname>greet</jobname>\n",
"[2024-11-19T23:15:43] ERROR 9 </task>\n",
"[2024-11-19T23:15:43] ERROR 10 </workflow>\n"
"[2025-01-05T21:26:44] ERROR 4 Rocoto XML validation errors found\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_ATTRVALID: Element workflow failed to validate attributes\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_NOELEM: Expecting an element cycledef, got nothing\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_INTERSEQ: Invalid sequence in interleave\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element workflow failed to validate content\n",
"[2025-01-05T21:26:44] ERROR Invalid Rocoto XML:\n",
"[2025-01-05T21:26:44] ERROR 1 <?xml version='1.0' encoding='utf-8'?>\n",
"[2025-01-05T21:26:44] ERROR 2 <workflow realtime=\"False\">\n",
"[2025-01-05T21:26:44] ERROR 3 <log>logs/test.log</log>\n",
"[2025-01-05T21:26:44] ERROR 4 <task name=\"greet\">\n",
"[2025-01-05T21:26:44] ERROR 5 <cores>1</cores>\n",
"[2025-01-05T21:26:44] ERROR 6 <walltime>00:00:10</walltime>\n",
"[2025-01-05T21:26:44] ERROR 7 <command>echo Hello, World!</command>\n",
"[2025-01-05T21:26:44] ERROR 8 <jobname>greet</jobname>\n",
"[2025-01-05T21:26:44] ERROR 9 </task>\n",
"[2025-01-05T21:26:44] ERROR 10 </workflow>\n"
]
},
{
Expand Down
9 changes: 4 additions & 5 deletions src/uwtools/config/formats/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,13 @@
from collections import UserDict
from copy import deepcopy
from io import StringIO
from math import inf
from pathlib import Path
from typing import Optional, Union

import yaml

from uwtools.config import jinja2
from uwtools.config.support import INCLUDE_TAG, depth, log_and_error, yaml_to_str
from uwtools.config.support import INCLUDE_TAG, depth, dict_to_yaml_str, log_and_error
from uwtools.exceptions import UWConfigError
from uwtools.logging import INDENT, MSGWIDTH, log
from uwtools.utils.file import str2path
Expand Down Expand Up @@ -87,8 +86,8 @@ def _compare_config_get_lines(d: dict) -> list[str]:
:param d: A dict object.
"""
sio = StringIO()
yaml.safe_dump(d, stream=sio, default_flow_style=False, indent=2, width=inf)
return sio.getvalue().splitlines(keepends=True)
sio.write(dict_to_yaml_str(d, sort=True))
return sio.getvalue().splitlines(keepends=False)

@staticmethod
def _compare_config_log_header() -> None:
Expand Down Expand Up @@ -224,7 +223,7 @@ def dereference(self, context: Optional[dict] = None) -> None:

def logstate(state: str) -> None:
jinja2.deref_debug("Dereferencing, %s value:" % state)
for line in yaml_to_str(self.data).split("\n"):
for line in dict_to_yaml_str(self.data).split("\n"):
jinja2.deref_debug("%s%s" % (INDENT, line))

while True:
Expand Down
Loading

0 comments on commit 986226f

Please sign in to comment.