[Enhancement] [zos_copy] [zos_job_submit] [zos_script] Add option to disable autoescape for Jinja templates #1810

rexemin · 2024-11-20T16:38:58Z

SUMMARY

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

module_utils/template.py
zos_copy
zos_job_submit
zos_script

ADDITIONAL INFORMATION

This adds a new option autoescape to all the modules that currently support templates. Its default value is True just to err on the side of caution, but it allows users to disable autoescaping whenever they need to.

…ithub.com/ansible-collections/ibm_zos_core into enhancement/1198/autoescape_option_jinja2

rexemin · 2024-11-23T00:14:24Z

Results for the pipeline

ddimatos

General questions:

How does this impact users who have been piping values to safe?
We probably want to update the changelog fragment so it also includes a behavior change so the one releasing can properly highlight this. This could be a known issues type fragment.

//{{template_values | safe }}

ddimatos · 2024-11-26T07:44:04Z

plugins/module_utils/template.py

+        if autoescape:
+            self.templating_env = jinja2.Environment(autoescape=True, **environment_args)
+        else:
+            self.templating_env = jinja2.Environment(autoescape=jinja2.select_autoescape(), **environment_args)


Question, the branched logic here, else, is relying on jinja2.select_autoescape() which will return false as it is today, would it be safer to be explicit since this could change in the future where maybe it would be true thus if/else would return true should the behavior change.

Also note the documentation:

Explicit in what way? I could change the function so it specifies the kinds of files that will commonly be templated, but just putting false in autoescape generates an error in Bandit.

@rexemin Is that the case if you use the following?

Suggested change

self.templating_env = jinja2.Environment(autoescape=jinja2.select_autoescape(), **environment_args)

self.templating_env = jinja2.Environment(autoescape=autoescape, **environment_args)

Just set autoescape parameter value to whatever autoescape variable has?

Yes, that is the case, it was the original code that caused the bandit error in the first place

yup, is gonna trigger that scan issue no matter what. This is ok for me as long as we have a test which tests autoescape False and True. That means, if this gets changed in the future, then will be caught in the test cases.

rexemin · 2024-11-27T18:04:39Z

This change should not affect users already using the safe filter. Tested without issues with a template that used it.

fernandofloresg

Just requesting a little addition to test

fernandofloresg · 2024-12-02T15:42:46Z

plugins/module_utils/template.py

+        if autoescape:
+            self.templating_env = jinja2.Environment(autoescape=True, **environment_args)
+        else:
+            self.templating_env = jinja2.Environment(autoescape=jinja2.select_autoescape(), **environment_args)


@rexemin Is that the case if you use the following?

Suggested change

self.templating_env = jinja2.Environment(autoescape=jinja2.select_autoescape(), **environment_args)

self.templating_env = jinja2.Environment(autoescape=autoescape, **environment_args)

Just set autoescape parameter value to whatever autoescape variable has?

fernandofloresg · 2024-12-02T15:45:49Z

tests/functional/modules/test_zos_job_submit_func.py

@@ -158,6 +158,18 @@
 /*
 //SYSUT2   DD SYSOUT=*
 //
+""",
+    "Autoescape": """//{{ pgm_name }}    JOB (T043JM,JM00,1,0,0,0),{{ parameter }},CLASS=R,
+//             MSGCLASS=X,MSGLEVEL=1,NOTIFY=S0JM


In the issue related users complained both about & and ' could you add those as part of the tests?

oh I see single quotes is added down there so just &

Co-authored-by: Fernando Flores <[email protected]>

tests/functional/modules/test_zos_copy_func.py

…e_shell_call_cp_to_python_module [Enabler][1652]migrate_shell_call_cp_to_python_module

Swapped old zoau v1.1.0 to use as zoau 1.3.4 ga build

… resource types (#1822) * Added fix for zos_find * Added tests for fix * Updated changelog

* Add enhance error message * Add fragment * Update plugins/modules/zos_copy.py Co-authored-by: Fernando Flores <[email protected]> * Update 1821-Add_error_message.yml * Update zos_copy.py --------- Co-authored-by: Fernando Flores <[email protected]>

…_init_portability [Enabler][1589]volume_init_test_portability

…C tool (#1829) * Update minimum version of z/OS for the GA of 1.12 Signed-off-by: ddimatos <[email protected]> * Update AAP link Signed-off-by: ddimatos <[email protected]> * Venv for 2.18 Signed-off-by: ddimatos <[email protected]> * remove merge flags Signed-off-by: ddimatos <[email protected]> * Merged release into dev * fixed test case * Fixed mvs raw test --------- Signed-off-by: ddimatos <[email protected]> Co-authored-by: Fernando Flores <[email protected]>

…1827) * Change flag used to run specific test * Add more env vars needed to run tests * Add verbose option to ac_test * Add mark and stop options to ac_test * Refactor pytest call Also changes how verbosity is handled, as well as adds printing of each pytest command run. * Update config to support new way of handling env vars in tests * Add profile option to ac_lint * Add support to run multiple files in ac_test * Change boolean options to flags Options `debug` (ac_test, test_concurrent) and `stop` (ac_test) can be now used simply as flags, without the need to add a `"true"` string after the option. * Document and clean up arg parser * Update more boolean options to be flags * Refactor pytest call * Add option 'volumes' to ac_test * Add user option and tweak verbosity * Add ac-test-required command * Add changelog fragment

fernandofloresg

lgtm

… API (#1831) * Replace calls to cp for dcp * Replace copy functions in copy.py * Replace copy calls in modules * Fix typo * Fix return statement * Add changelog fragment * Fix sanity issues

fernandofloresg · 2025-01-02T15:49:33Z

Please check why some fragments are being added, maybe the branch has diverted a bit from dev

rexemin · 2025-01-02T19:51:54Z

Please check why some fragments are being added, maybe the branch has diverted a bit from dev

Done, it looks like it just needed to merge recent changes in dev.

AndreMarcel99 and others added 13 commits August 13, 2024 15:07

Remove Hardcoded hlq

3fb4313

Fix removal

ccff1e4

Add some solution

4f6b373

Add autoescape argument to templating class

8036674

Update template doc fragment

977d5c6

Add autoescape support to zos_copy

fedf210

Add autoescape support to zos_script

ef51aa9

Add autoescape support to zos_job_submit

8a60e21

Add marks to tests

d58e3d7

Merge branch 'dev' into enhancement/1198/autoescape_option_jinja2

a78a22b

Update templating tests

8e13ab6

Update docs

b12c4ab

Add changelog fragment

1e8b032

rexemin marked this pull request as ready for review November 22, 2024 19:29

rexemin added 4 commits November 22, 2024 16:00

Fix bandit error

26066a8

Merge branch 'dev' into enhancement/1198/autoescape_option_jinja2

d8562b0

Fix pep8 issue

b20adde

Merge branch 'enhancement/1198/autoescape_option_jinja2' of https://g…

a240a98

…ithub.com/ansible-collections/ibm_zos_core into enhancement/1198/autoescape_option_jinja2

rexemin requested review from ketankelkar, fernandofloresg, ddimatos, richp405 and AndreMarcel99 November 25, 2024 15:38

ddimatos requested changes Nov 26, 2024

View reviewed changes

AndreMarcel99 added 3 commits November 27, 2024 14:36

Remove cp

9be927a

Fix dir_path

164a996

Add fragment

0d48999

fernandofloresg requested changes Dec 2, 2024

View reviewed changes

AndreMarcel99 and others added 5 commits December 2, 2024 10:40

Update plugins/modules/zos_fetch.py

f8d5255

Co-authored-by: Fernando Flores <[email protected]>

Fix execpton handle

cfcfc36

Fix sanity

5da0601

Adjust best practices

9f04361

Merge branch 'dev' into enabler/1589/volume_init_portability

4e444c6

fernandofloresg reviewed Dec 2, 2024

View reviewed changes

tests/functional/modules/test_zos_copy_func.py Show resolved Hide resolved

Add '&' to autoescape test

1d8ea2c

rexemin requested a review from fernandofloresg December 3, 2024 20:52

AndreMarcel99 and others added 12 commits December 4, 2024 11:20

Merge pull request #1816 from ansible-collections/enabler/1652/migrat…

04c3770

…e_shell_call_cp_to_python_module [Enabler][1652]migrate_shell_call_cp_to_python_module

Update mounts.env (#1826)

cb6d498

Swapped old zoau v1.1.0 to use as zoau 1.3.4 ga build

[zos_mvs_raw] Find VSAM cluster when DISP=OLD and find data and index…

1c64254

… resource types (#1822) * Added fix for zos_find * Added tests for fix * Updated changelog

Merge branch 'dev' into enabler/1589/volume_init_portability

b79ce3d

End portability

7a420e8

Add fragment

3b813bc

Merge branch 'dev' into enabler/1589/volume_init_portability

e40ccfb

Merge pull request #1830 from ansible-collections/enabler/1589/volume…

d8faad9

…_init_portability [Enabler][1589]volume_init_test_portability

Merge branch 'dev' into enhancement/1198/autoescape_option_jinja2

2faa9a9

fernandofloresg approved these changes Dec 16, 2024

View reviewed changes

rexemin added 3 commits December 16, 2024 10:49

[Enabler] [module_utils/copy.py] Replace calls to cp for dcp's Python…

816167c

… API (#1831) * Replace calls to cp for dcp * Replace copy functions in copy.py * Replace copy calls in modules * Fix typo * Fix return statement * Add changelog fragment * Fix sanity issues

Merge branch 'dev' into enhancement/1198/autoescape_option_jinja2

8287605

Add warning when turning off autoescaping

b1397b4

rexemin requested a review from ddimatos December 18, 2024 00:02

fernandofloresg force-pushed the dev branch 2 times, most recently from 2660c02 to f5376bf Compare December 18, 2024 17:08

Merge branch 'dev' into enhancement/1198/autoescape_option_jinja2

546efce

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] [zos_copy] [zos_job_submit] [zos_script] Add option to disable autoescape for Jinja templates #1810

[Enhancement] [zos_copy] [zos_job_submit] [zos_script] Add option to disable autoescape for Jinja templates #1810

rexemin commented Nov 20, 2024

rexemin commented Nov 23, 2024

ddimatos left a comment

ddimatos Nov 26, 2024

rexemin Nov 27, 2024

fernandofloresg Dec 2, 2024

rexemin Dec 2, 2024

fernandofloresg Dec 2, 2024

rexemin commented Nov 27, 2024

fernandofloresg left a comment

fernandofloresg Dec 2, 2024

fernandofloresg Dec 2, 2024

fernandofloresg Dec 2, 2024

fernandofloresg left a comment

fernandofloresg commented Jan 2, 2025

rexemin commented Jan 2, 2025

	self.templating_env = jinja2.Environment(autoescape=jinja2.select_autoescape(), **environment_args)
	self.templating_env = jinja2.Environment(autoescape=autoescape, **environment_args)

[Enhancement] [zos_copy] [zos_job_submit] [zos_script] Add option to disable autoescape for Jinja templates #1810

Are you sure you want to change the base?

[Enhancement] [zos_copy] [zos_job_submit] [zos_script] Add option to disable autoescape for Jinja templates #1810

Conversation

rexemin commented Nov 20, 2024

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

rexemin commented Nov 23, 2024

ddimatos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rexemin commented Nov 27, 2024

fernandofloresg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fernandofloresg left a comment

Choose a reason for hiding this comment

fernandofloresg commented Jan 2, 2025

rexemin commented Jan 2, 2025