Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add !dict and !list UW YAML tags #682

Merged
merged 13 commits into from
Jan 7, 2025
56 changes: 28 additions & 28 deletions docs/sections/user_guide/cli/tools/config/realize-verbose.out
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
[2024-11-27T05:24:34] DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2024-11-27T05:24:34] DEBUG Reading input from stdin
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG hello: '{{ recipient }}'
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: {{ recipient }}
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG hello: world
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: world
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: world
[2024-11-27T05:24:34] DEBUG Dereferencing, final value:
[2024-11-27T05:24:34] DEBUG hello: world
[2024-11-27T05:24:34] DEBUG recipient: world
[2024-11-27T05:24:34] DEBUG Writing output to stdout
[2025-01-05T21:15:07] DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2025-01-05T21:15:07] DEBUG Reading input from stdin
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: '{{ recipient }}'
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: {{ recipient }}
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: world
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: world
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: world
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, final value:
[2025-01-05T21:15:07] DEBUG [dereference] hello: world
[2025-01-05T21:15:07] DEBUG [dereference] recipient: world
[2025-01-05T21:15:07] DEBUG Writing output to stdout
hello: world
recipient: world
42 changes: 21 additions & 21 deletions docs/sections/user_guide/cli/tools/config/validate-verbose.out
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[2024-11-27T05:24:34] DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2024-11-27T05:24:34] DEBUG Using schema file: schema.jsonschema
[2024-11-27T05:24:34] DEBUG Dereferencing, current value:
[2024-11-27T05:24:34] DEBUG values:
[2024-11-27T05:24:34] DEBUG greeting: Hello
[2024-11-27T05:24:34] DEBUG recipient: World
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: values
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: values
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: greeting
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: greeting
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: Hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: Hello
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: recipient
[2024-11-27T05:24:34] DEBUG [dereference] Rendering: World
[2024-11-27T05:24:34] DEBUG [dereference] Rendered: World
[2024-11-27T05:24:34] DEBUG Dereferencing, final value:
[2024-11-27T05:24:34] DEBUG values:
[2024-11-27T05:24:34] DEBUG greeting: Hello
[2024-11-27T05:24:34] DEBUG recipient: World
[2024-11-27T05:24:34] INFO 0 schema-validation errors found in config
[2025-01-05T21:15:07] DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2025-01-05T21:15:07] DEBUG Using schema file: schema.jsonschema
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] values:
[2025-01-05T21:15:07] DEBUG [dereference] greeting: Hello
[2025-01-05T21:15:07] DEBUG [dereference] recipient: World
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: values
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: values
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: greeting
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: greeting
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: Hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: Hello
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: recipient
[2025-01-05T21:15:07] DEBUG [dereference] Rendering: World
[2025-01-05T21:15:07] DEBUG [dereference] Rendered: World
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, final value:
[2025-01-05T21:15:07] DEBUG [dereference] values:
[2025-01-05T21:15:07] DEBUG [dereference] greeting: Hello
[2025-01-05T21:15:07] DEBUG [dereference] recipient: World
[2025-01-05T21:15:07] INFO 0 schema-validation errors found in config
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[2024-08-26T23:39:19] DEBUG Command: uw rocoto realize --config-file rocoto.yaml --verbose
[2024-08-26T23:39:19] DEBUG Dereferencing, current value:
[2024-08-26T23:39:19] DEBUG workflow:
[2024-08-26T23:39:19] DEBUG attrs:
[2024-08-26T23:39:19] DEBUG realtime: false
[2024-08-26T23:39:19] DEBUG scheduler: slurm
[2024-08-26T23:39:19] DEBUG cycledef:
[2024-08-26T23:39:19] DEBUG - attrs:
[2024-08-26T23:39:19] DEBUG group: howdy
[2024-08-26T23:39:19] DEBUG spec: 202209290000 202209300000 06:00:00
[2025-01-05T21:15:07] DEBUG Command: uw rocoto realize --config-file rocoto.yaml --verbose
[2025-01-05T21:15:07] DEBUG [dereference] Dereferencing, current value:
[2025-01-05T21:15:07] DEBUG [dereference] workflow:
[2025-01-05T21:15:07] DEBUG [dereference] attrs:
[2025-01-05T21:15:07] DEBUG [dereference] realtime: false
[2025-01-05T21:15:07] DEBUG [dereference] scheduler: slurm
[2025-01-05T21:15:07] DEBUG [dereference] cycledef:
[2025-01-05T21:15:07] DEBUG [dereference] - attrs:
[2025-01-05T21:15:07] DEBUG [dereference] group: howdy
[2025-01-05T21:15:07] DEBUG [dereference] spec: 202209290000 202209300000 06:00:00
...
[2024-08-26T23:39:20] DEBUG cycledefs: howdy
[2024-08-26T23:39:20] DEBUG account: '&ACCOUNT;'
[2024-08-26T23:39:20] DEBUG command: echo hello $person
[2024-08-26T23:39:20] DEBUG jobname: hello
[2024-08-26T23:39:20] DEBUG native: --reservation my_reservation
[2024-08-26T23:39:20] DEBUG nodes: 1:ppn=1
[2024-08-26T23:39:20] DEBUG walltime: 00:01:00
[2024-08-26T23:39:20] DEBUG envars:
[2024-08-26T23:39:20] DEBUG person: siri
[2024-08-26T23:39:20] INFO 0 Rocoto XML validation errors found
[2025-01-05T21:15:07] DEBUG [dereference] cycledefs: howdy
[2025-01-05T21:15:07] DEBUG [dereference] account: '&ACCOUNT;'
[2025-01-05T21:15:07] DEBUG [dereference] command: echo hello $person
[2025-01-05T21:15:07] DEBUG [dereference] jobname: hello
[2025-01-05T21:15:07] DEBUG [dereference] native: --reservation my_reservation
[2025-01-05T21:15:07] DEBUG [dereference] nodes: 1:ppn=1
[2025-01-05T21:15:07] DEBUG [dereference] walltime: 00:01:00
[2025-01-05T21:15:07] DEBUG [dereference] envars:
[2025-01-05T21:15:07] DEBUG [dereference] person: siri
[2025-01-05T21:15:07] INFO 0 Rocoto XML validation errors found
44 changes: 43 additions & 1 deletion docs/sections/user_guide/yaml/tags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ Or explicit:
integer: !!int "3"
float: !!float "3.14"

Additionally, UW defines the following tags to support use cases not covered by standard tags:
Additionally, UW defines the following tags to support use cases not covered by standard tags. Where standard YAML tags are applied to their values immediately, application of UW YAML tags is delayed until after Jinja2 expressions in tagged values are dereferenced.

**NB** Values tagged with UW YAML tags must be strings. Use quotes as necessary to ensure that they are.

``!bool``
^^^^^^^^^
Expand All @@ -32,12 +34,14 @@ Converts the tagged node to a Python ``bool`` object. For example, given ``input

flag1: True
flag2: !bool "{{ flag1 }}"
flag3: !bool "0"

.. code-block:: text

$ uw config realize -i ../input.yaml --output-format yaml
flag1: True
flag2: True
flag3: False


``!datetime``
Expand All @@ -58,6 +62,26 @@ Converts the tagged node to a Python ``datetime`` object. For example, given ``i

The value provided to the tag must be in :python:`ISO 8601 format<datetime.html#datetime.datetime.fromisoformat>` to be interpreted correctly by the ``!datetime`` tag.

``!dict``
^^^^^^^^^

Converts the tagged node to a Python ``dict`` value. For example, given ``input.yaml``:

.. code-block:: yaml

d1: {'k0': 0, 'k1': 1, 'k2': 2}
d2: !dict "{ k0: 0, k1: 1, k2: 2 }"
d3: !dict "{{ '{' }}{% for n in range(3) %} k{{ n }}:{{ n }},{% endfor %}{{ '}' }}"
d4: !dict "[{% for n in range(3) %}[k{{ n }},{{ n }}],{% endfor %}]"

.. code-block:: text

$ uw config realize --input-file input.yaml --output-format yaml
d1: {'k0': 0, 'k1': 1, 'k2': 2}
d2: {'k0': 0, 'k1': 1, 'k2': 2}
d3: {'k0': 0, 'k1': 1, 'k2': 2}
d4: {'k0': 0, 'k1': 1, 'k2': 2}

``!float``
^^^^^^^^^^

Expand Down Expand Up @@ -140,6 +164,24 @@ Converts the tagged node to a Python ``int`` value. For example, given ``input.y
f2: 11
f2: 140

``!list``
^^^^^^^^^

Converts the tagged node to a Python ``list`` value. For example, given ``input.yaml``:

.. code-block:: yaml

l1: [1, 2, 3]
l2: !list "[{% for n in range(3) %} a{{ n }},{% endfor %} ]"
l3: !list "[ a0, a1, a2, ]"

.. code-block:: text

$ uw config realize --input-file input.yaml --output-format yaml
l1: [1, 2, 3]
l2: ['a0', 'a1', 'a2']
l3: ['a0', 'a1', 'a2']

``!remove``
^^^^^^^^^^^

Expand Down
10 changes: 8 additions & 2 deletions notebooks/config.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@
"text": [
"Help on function realize in module uwtools.api.config:\n",
"\n",
"realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[str, int]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n",
"realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[bool, float, int, str]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n",
" Realize a config based on a base input config and an optional update config.\n",
"\n",
" The input config may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it\n",
Expand Down Expand Up @@ -1491,6 +1491,12 @@
" | :param src: The dictionary with new data to use.\n",
" |\n",
" | ----------------------------------------------------------------------\n",
" | Readonly properties inherited from uwtools.config.formats.base.Config:\n",
" |\n",
" | config_file\n",
" | Return the path to the config file from which this object was instantiated, if applicable.\n",
" |\n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors inherited from uwtools.config.formats.base.Config:\n",
" |\n",
" | __dict__\n",
Expand Down Expand Up @@ -1555,7 +1561,7 @@
" |\n",
" | update(self, other=(), /, **kwds)\n",
" | D.update([E, ]**F) -> None. Update D from mapping/iterable E and F.\n",
" | If E present and has a .keys() method, does: for k in E: D[k] = E[k]\n",
" | If E present and has a .keys() method, does: for k in E.keys(): D[k] = E[k]\n",
" | If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v\n",
" | In either case, this is followed by: for k, v in F.items(): D[k] = v\n",
" |\n",
Expand Down
64 changes: 32 additions & 32 deletions notebooks/rocoto.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -256,13 +256,13 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] ERROR 3 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] ERROR Error at workflow -> attrs:\n",
"[2024-11-19T23:15:43] ERROR 'realtime' is a required property\n",
"[2024-11-19T23:15:43] ERROR Error at workflow -> tasks -> task_greet:\n",
"[2024-11-19T23:15:43] ERROR 'command' is a required property\n",
"[2024-11-19T23:15:43] ERROR Error at workflow:\n",
"[2024-11-19T23:15:43] ERROR 'log' is a required property\n"
"[2025-01-05T21:26:43] ERROR 3 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] ERROR Error at workflow.attrs:\n",
"[2025-01-05T21:26:43] ERROR 'realtime' is a required property\n",
"[2025-01-05T21:26:43] ERROR Error at workflow.tasks.task_greet:\n",
"[2025-01-05T21:26:43] ERROR 'command' is a required property\n",
"[2025-01-05T21:26:43] ERROR Error at workflow:\n",
"[2025-01-05T21:26:43] ERROR 'log' is a required property\n"
]
},
{
Expand Down Expand Up @@ -388,8 +388,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -577,8 +577,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -722,8 +722,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 schema-validation errors found in Rocoto config\n",
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -925,7 +925,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] INFO 0 Rocoto XML validation errors found\n"
"[2025-01-05T21:26:43] INFO 0 Rocoto XML validation errors found\n"
]
},
{
Expand Down Expand Up @@ -1001,22 +1001,22 @@
"name": "stderr",
"output_type": "stream",
"text": [
"[2024-11-19T23:15:43] ERROR 4 Rocoto XML validation errors found\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_ATTRVALID: Element workflow failed to validate attributes\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_NOELEM: Expecting an element cycledef, got nothing\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_INTERSEQ: Invalid sequence in interleave\n",
"[2024-11-19T23:15:43] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element workflow failed to validate content\n",
"[2024-11-19T23:15:43] ERROR Invalid Rocoto XML:\n",
"[2024-11-19T23:15:43] ERROR 1 <?xml version='1.0' encoding='utf-8'?>\n",
"[2024-11-19T23:15:43] ERROR 2 <workflow realtime=\"False\">\n",
"[2024-11-19T23:15:43] ERROR 3 <log>logs/test.log</log>\n",
"[2024-11-19T23:15:43] ERROR 4 <task name=\"greet\">\n",
"[2024-11-19T23:15:43] ERROR 5 <cores>1</cores>\n",
"[2024-11-19T23:15:43] ERROR 6 <walltime>00:00:10</walltime>\n",
"[2024-11-19T23:15:43] ERROR 7 <command>echo Hello, World!</command>\n",
"[2024-11-19T23:15:43] ERROR 8 <jobname>greet</jobname>\n",
"[2024-11-19T23:15:43] ERROR 9 </task>\n",
"[2024-11-19T23:15:43] ERROR 10 </workflow>\n"
"[2025-01-05T21:26:44] ERROR 4 Rocoto XML validation errors found\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_ATTRVALID: Element workflow failed to validate attributes\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_NOELEM: Expecting an element cycledef, got nothing\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_INTERSEQ: Invalid sequence in interleave\n",
"[2025-01-05T21:26:44] ERROR <string>:2:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element workflow failed to validate content\n",
"[2025-01-05T21:26:44] ERROR Invalid Rocoto XML:\n",
"[2025-01-05T21:26:44] ERROR 1 <?xml version='1.0' encoding='utf-8'?>\n",
"[2025-01-05T21:26:44] ERROR 2 <workflow realtime=\"False\">\n",
"[2025-01-05T21:26:44] ERROR 3 <log>logs/test.log</log>\n",
"[2025-01-05T21:26:44] ERROR 4 <task name=\"greet\">\n",
"[2025-01-05T21:26:44] ERROR 5 <cores>1</cores>\n",
"[2025-01-05T21:26:44] ERROR 6 <walltime>00:00:10</walltime>\n",
"[2025-01-05T21:26:44] ERROR 7 <command>echo Hello, World!</command>\n",
"[2025-01-05T21:26:44] ERROR 8 <jobname>greet</jobname>\n",
"[2025-01-05T21:26:44] ERROR 9 </task>\n",
"[2025-01-05T21:26:44] ERROR 10 </workflow>\n"
]
},
{
Expand Down
Loading
Loading