feat(rfc): Write RFC 0001 to propose task chunking support

[mwiebe]

Request for comment on task chunking proposal

Tracking Issue: #53

Chunking frames together is a commonly supported solution for amortizing application and scene loading overhead, and Open Job Description should support it. This RFC adds chunking to job templates with the goal that it be easy to author and read templates that work well with both contiguous frame ranges and pick up renders, and is not to much work to implement.

Here's a link directly to the rendered RFC: https://github.com/mwiebe/openjd-specifications/blob/task-chunking-rfc/rfcs/0001-task-chunking.md

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[Ahuge]

Love this RFC, thanks Mark.

Question for you, do we have a reference to the rangeConstraint value to determine if the {{Task.Param.Frame}} value is a NONCONTIGUOUS or a CONTIGUOUS value?

Perhaps that isn't something we need to worry about, I am just getting my head around this RFC but I wanted to ask if that had been considered.

**Edit the rangeConstraint determines how OpenJD will pass chunks to the task, that makes sense.
The CONTIGUOUS vs NONCONTIGUOUS value refers to if the task/adaptor supports NONCONTIGUOUS ranges.

I was implementing a POC for the NukeAdaptor and by default it currently only supports (\d+)-(\d+) frame ranges (CONTIGUOUS) but because we're running an adaptor workflow, we could add NONCONTIGUOUS functionality

Please correct me if I am wrong in that assumption.

**Perhaps having a third example would help. Where the frame range passed to the job parameter is a complex/NONCONTIGUOUS range but setting the rangeConstraint value to be CONTIGUOUS shows that the task only ever receives CONTIGUOUS chunks

[mwiebe]

Love this RFC, thanks Mark.

Awesome, thanks!

Question for you, do we have a reference to the rangeConstraint value to determine if the {{Task.Param.Frame}} value is a NONCONTIGUOUS or a CONTIGUOUS value?

I expect that the person setting the rangeConstraint and the person writing the step script are the same, so for a particular step it would always be one or the other. Do you have an idea about when you might reach for that value programmatically from the step script, versus just implementing one or the other behavior directly?

Perhaps that isn't something we need to worry about, I am just getting my head around this RFC but I wanted to ask if that had been considered.

I had considered a slightly different angle, of whether it makes sense for someone submitting a job to control this option. My conclusion was it's better not to, and the "@fmtstring selection" Design Rationale section talks about it.

[mwiebe]

I was implementing a POC for the NukeAdaptor and by default it currently only supports (\d+)-(\d+) frame ranges (CONTIGUOUS) but because we're running an adaptor workflow, we could add NONCONTIGUOUS functionality

Please correct me if I am wrong in that assumption

Yes this is exactly the idea. Then when someone submits a pickup render with a few scattered frames, it can run with chunks like "7,12,25".

[mwiebe]

**Perhaps having a third example would help. Where the frame range passed to the job parameter is a complex/NONCONTIGUOUS range but setting the rangeConstraint value to be CONTIGUOUS shows that the task only ever receives CONTIGUOUS chunks

This is a good idea, maybe I can do that in the first example by setting a non-contiguous default there with a comment explaining how it will get chunked.

UPDATE: I've changes the first example's default frames to something noncontiguous and added a description of likely chunk selection. Thanks for the suggestion!

[jairaws]

Why are we using the default prefix? Is it because this value can be overridden at runtime based on targetRuntimeSeconds or by the remaining frames in a range? If so, is this the same as maxTaskCount?

[mwiebe]

Thanks for reviewing the proposal! It's called default to indicate the scheduler should use this value if it doesn't have a better value to use. It's not the same as maxTaskCount because the scheduler can choose larger values, like when chunks are finishing quicker than in targetRuntimeSeconds.

[ryyakobe]

Maybe we could consider renaming it to initialTaskCount or even initialChunkSize to emphasize that it's the starting point for chunk sizing and avoid confusion with optional parameters that have predefined defaults?

[crowecawcaw]

Would it make sense to make rangeContraint optional and have a default value of NONCONTIGUOUS? Pedantically, the NONCONTIGUOUS option is a lack of a constraint. I'd guess that NONCONTIGUOUS would be the much more common option, so it'd simplify the parameter definition for many templates.

[mwiebe]

I made it required so that templates will be more understandable without having to know what the default is. I would have picked CONTIGUOUS as the default, and NONCONTIGUOUS as the more general option you can opt into. I was thinking of CLI commands that have start/end frame options.

[eherozhao]

I am not sure if I misunderstand this.
Does these 15 tasks mean 15 frames here? or openjd Tasks? I am thinking that we are talking about 3 openjd tasks with 6 frames chunk size but I am a bit confused about the terminalogy here.

[mwiebe]

Yes, task == frame here, so it means 15 tasks and 15 frames. A chunk of 6 tasks is then also a chunk of 6 frames.