Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft implementation of proposed syntax changes (feedback requested) #766

Draft
wants to merge 133 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
133 commits
Select commit Hold shift + click to select a range
8f06ac4
reworked collect outputs so callable's are used over simply copying v…
tclose Jun 3, 2024
f06171d
if bool field argstr includes a string template, do templating instea…
tclose Jun 5, 2024
2b33473
bugfix for argstr formatting for bool types
tclose Jul 23, 2024
b177091
wrote cmd_out arg and unittest based on "ls" for shell_cmd
tclose Jul 23, 2024
5758eb3
added small note to docs
tclose Jul 23, 2024
0f0485c
renamed cmd_arg and cmd_out to shell_arg and shell_out
tclose Jul 23, 2024
16cecf1
touched up docs
tclose Jul 23, 2024
6ee5b86
pass through kwargs to attrs.field in shell_arg and shell_out
tclose Jul 23, 2024
47806ff
added shell_task decorator
tclose Jul 23, 2024
c0d7011
implemented shell_task and basic unittests. Generated tasks do not wo…
tclose Jul 23, 2024
0da3b46
shell command unittests pass
tclose Jul 23, 2024
c4d1875
renamed pydra.mark.shell_commands to pydra.mark.shell
tclose Jul 23, 2024
d25eab0
fixed up inheritance of Inputs and Outputs
tclose Jul 23, 2024
e4095a7
added tests for output_file_template/output_field_name and various un…
tclose Jul 23, 2024
e9a58bc
converted to hatch build system and made proper namespace package
tclose Jul 23, 2024
e8bce86
initial structure of 'design' sub-package
tclose Aug 1, 2024
e1b677e
implementing new decorator syntax
tclose Nov 9, 2024
1a0ffff
got all new python interface unittests to pass
tclose Nov 9, 2024
3a159bb
fixed up checking of wrapped type in python.task decorator
tclose Nov 11, 2024
dabe2ef
added shell.task decorator implementation and working through tests
tclose Nov 11, 2024
0aa12cf
debugging shell interface syntax
tclose Nov 12, 2024
8581f55
debugging type hints
tclose Nov 12, 2024
4b408a3
debugged test_shell unittests
tclose Nov 14, 2024
6cf4e1b
added tests for shell command templates
tclose Nov 15, 2024
8cbeade
cleaning up shell tests
tclose Nov 15, 2024
79e98f6
capture defaults in pydra.design
tclose Nov 16, 2024
bba8be7
added function to python.interface fields instead of static
tclose Nov 16, 2024
7ab7713
reworked and debugged shell design tests
tclose Nov 16, 2024
e606459
all python and shell design tests pass
tclose Nov 16, 2024
b38ced6
added initialization tests to test_python
tclose Nov 16, 2024
a9356d1
renamed Interface->TaskSpec and interface->define
tclose Nov 16, 2024
b9594ee
converted to inputs and outputs to dicts from lists within design mak…
tclose Nov 17, 2024
91b4116
added workflow design module
tclose Nov 17, 2024
8130802
added check for symbols defined outside to the scope of the function
tclose Nov 17, 2024
5aa9fbf
enhanced get_undefined_symbols to skip decorator and signature type h…
tclose Nov 17, 2024
e4da149
debugged workflow construction tests
tclose Nov 18, 2024
acde206
added check for NOTHING values and auto added to ext
tclose Nov 18, 2024
07cf5a8
added new workflow tests
tclose Nov 18, 2024
9f7a42e
added check for unrecognised inputs
tclose Nov 18, 2024
2d4f030
added custom hash for function types based on function source
tclose Nov 18, 2024
35c952d
added shell workflow and debugged shell task definitions
tclose Nov 18, 2024
ac08c18
added cononical workflow test
tclose Nov 19, 2024
a77be6a
added lazy inputs test
tclose Nov 19, 2024
476f7db
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 19, 2024
cd97c11
Debugged splitting and combining of lazy fields
tclose Dec 3, 2024
d825056
updated python versions to include 3.13 and drop <3.10
tclose Dec 4, 2024
03e6951
resolved gnarly circular imports
tclose Dec 5, 2024
384e57d
fixed up lazy out splitting
tclose Dec 5, 2024
553bb2f
fixing up state maintenance in workflow nodes
tclose Dec 6, 2024
032fd4e
restored functionality from specs
tclose Dec 8, 2024
bdee4c8
cleaned up specs.py so that it works with new syntax
tclose Dec 9, 2024
d2f0bf3
updated attrs required version to latest
tclose Dec 9, 2024
35f489c
debugged reworking of specs module
tclose Dec 9, 2024
a9071e4
renamed OutSpec to Outputs
tclose Dec 9, 2024
1b50350
renamed FunctionTask -> PythonTask and ShellCommandTask -> ShellTask …
tclose Dec 9, 2024
51c911b
debugged design unittests
tclose Dec 9, 2024
f227987
added dataclass_transform decorators to define and outputs
tclose Dec 9, 2024
0cf7d35
Added the use of '?' to signify optional shell template fields
tclose Dec 9, 2024
4be0945
added in "modify|" syntax as discussed with @satra
tclose Dec 10, 2024
328e0b1
implemented defaults in shell command template using '=' syntax
tclose Dec 10, 2024
cf7b331
cleaned up insertion of special stdout, stderr and return_code output…
tclose Dec 10, 2024
c6b7cd7
debugged setting of state in split and combine
tclose Dec 10, 2024
0b8ac66
added tests to demonstrate nested workflows
tclose Dec 10, 2024
000140e
added check for lazy vals to non-lazy inputs in workflow construct
tclose Dec 10, 2024
5e898af
shell tasks now execute
tclose Dec 13, 2024
7df9a2b
implemented python task execution
tclose Dec 13, 2024
860002f
moved boutiques into design pacakge
tclose Dec 16, 2024
5d08b0e
added typing to Result to specify output type
tclose Dec 16, 2024
70d0e06
reworked dockertask test
tclose Dec 16, 2024
1daec25
got tests to load
tclose Dec 16, 2024
0dc7ec6
renamed docs to old-docs
tclose Dec 16, 2024
e254177
added new docs structure
tclose Dec 16, 2024
0077f68
added simple installation instructions
tclose Dec 16, 2024
c79905c
touched up new docs
tclose Dec 16, 2024
8c08f9e
more docs restructuring
tclose Dec 16, 2024
718e42a
reworked new docs structure
tclose Dec 18, 2024
8e1d83c
Update pydra/engine/tests/test_boutiques.py
tclose Dec 19, 2024
c3f8249
touching up docs structure
tclose Dec 19, 2024
602c8fe
fixing up docs
tclose Dec 21, 2024
b4d2b16
removed state references from execution code (they will be handled be…
tclose Dec 21, 2024
fb86964
debugging test utils
tclose Dec 21, 2024
97c6f11
removed implicit extraction of output names from return values
tclose Dec 21, 2024
0cfcc5f
cleaned up all @mark.task to @python.define
tclose Dec 21, 2024
4908d21
changed supported version to Py3.11, use Exception.add_note instead o…
tclose Dec 24, 2024
44d8b67
got execution notebook example to work
tclose Dec 24, 2024
e193bb8
reworking testing tasks
tclose Dec 24, 2024
53f28bd
Added shell-task tutorial
tclose Dec 24, 2024
75d429b
fixed up templating defaults
tclose Dec 26, 2024
92fe97c
debugged shell and python task tutorias
tclose Dec 26, 2024
78f51eb
moved docs back to original location, renamed new docs to "new-docs"
tclose Dec 26, 2024
363ac5d
fixed import of version
tclose Dec 26, 2024
bc030f5
added rtd theme to docs deps
tclose Dec 26, 2024
a3724bd
fixing up new docs build action
tclose Dec 26, 2024
9383ccd
added docs dep
tclose Dec 27, 2024
ee6844a
revert main docs to latest tag
tclose Dec 27, 2024
137965b
reworking docs CI
tclose Dec 27, 2024
2574d8e
touching up docs deps
tclose Dec 27, 2024
be5d8b5
more docs deps
tclose Dec 27, 2024
6491756
fixed docs ci
tclose Dec 27, 2024
571ba9c
docs ci touch up
tclose Dec 27, 2024
c4d7468
removed restriction on release for docs deploy
tclose Dec 27, 2024
4a8c42d
include ipython in docs deps
tclose Dec 27, 2024
1a6b067
renamed Outputs to TaskOutputs
tclose Dec 27, 2024
a8ed05e
finished workflow tutorial and reworked some of the docs structure
tclose Dec 28, 2024
8e88ed2
finished off the workflow tutorial for now
tclose Dec 28, 2024
6f1c8e3
restructuring new docs
tclose Dec 28, 2024
03546ed
more restructuring of new docs
tclose Dec 29, 2024
35641b1
writing getting-starting notebook
tclose Dec 29, 2024
ffdfce1
renamed Spec and specification to Def and definition
tclose Dec 29, 2024
e317e8e
finished getting-started (apart from debugging) tutorial
tclose Dec 29, 2024
720efdb
imported examples from tutorials
tclose Dec 29, 2024
7a7cffd
touched up errors in first-level-glm
tclose Dec 29, 2024
edd682f
reinstated changes to getting-started
tclose Dec 29, 2024
f5276b9
touched up getting-started
tclose Dec 29, 2024
8170777
converted two-level glm to new syntax
tclose Dec 30, 2024
02f6a1c
more touching up of documentation
tclose Dec 31, 2024
4502260
finished python task and canonical form tuts
tclose Dec 31, 2024
d71542f
added dependencies required for examples
tclose Dec 31, 2024
70edf85
touching up tutorials
tclose Dec 31, 2024
9594e75
renamed spec to definition across the board
tclose Dec 31, 2024
940f004
fixed mrtrix import
tclose Dec 31, 2024
e15cb35
finished playing with tutorials for now
tclose Dec 31, 2024
96e0b92
fixed doc dependency
tclose Dec 31, 2024
3495b50
download tags to docs build
tclose Dec 31, 2024
ec669e1
more fixes to docs CI
tclose Dec 31, 2024
e95e8ff
commented out pydra-mrtrix3 dep for docs until new release
tclose Jan 1, 2025
b5b95d3
combined typing and hashing
tclose Jan 1, 2025
1e81774
renamed help_string to help
tclose Jan 1, 2025
ac7ab7f
renamed make_task_spec to make_task_def
tclose Jan 4, 2025
7c47eef
touch up docs
tclose Jan 6, 2025
83cbbb6
touching up docs
tclose Jan 7, 2025
e6c488f
renamed tutorials to give natural order
tclose Jan 7, 2025
88a05bb
filled out explanation sections with subheadings
tclose Jan 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ cov.xml

# This can be generated in-tree. We never want to commit it.
pydra/_version.py
pydra/engine/_version.py
11 changes: 6 additions & 5 deletions docs/input_spec.rst
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,9 @@ In the example we used multiple keys in the metadata dictionary including `help_
(a specific input field will be sent).


Validators
----------
Pydra allows for using simple validator for types and `allowev_values`.
The validators are disabled by default, but can be enabled by calling
`pydra.set_input_validator(flag=True)`.
`shell_arg` Function
--------------------

For convenience, there is a function in `pydra.mark` called `shell_arg()`, which will
takes the above metadata values as arguments and inserts them into the metadata passed
to `attrs.field`. This can be especially useful when using an IDE with code-completion.
49 changes: 0 additions & 49 deletions pydra/__init__.py

This file was deleted.

6 changes: 6 additions & 0 deletions pydra/design/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from .base import TaskSpec, list_fields
from . import python
from . import shell


__all__ = ["TaskSpec", "list_fields", "python", "shell"]
Loading
Loading