Fix read permission denied on train script when run as non-root

[astefanutti]

What this PR does / why we need it:

This PR writes the train function script into a tmp directory so it can be read when TrainJob containers run as non-root.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #2372.

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 12695578194

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 12692161092:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[andreyvelich]

If we create this file under HOME directory, will it still fail for non-root containers ?

I intentionally write the script into the same file, so the file system between users' workspace and Kubernetes Pod is the same. Which means users can use standard Python imports while developing ML training code. Similar to how we showcase at KubeCon demo: https://youtu.be/Lgy4ir1AhYw?t=446

Related: #2347.

cc @shravan-achar

[astefanutti]

With the stricter security policy, containers are run with random UID with no fixed HOME directory.

I need to research a bit more what could be the other possible options and get back to you.

[astefanutti]

I've given this another cycle and it doesn't seem possible to change the working directory permissions easily.

On the other hand, mounting an emptyDir volume as the /workspace directory in the training runtime works as expected with the stricter security policy.

My understanding is this approach aligns well with the initializer and worker pods coordinating via PVCs. The default emptyDir fits with the initializers being currently run as init containers, but this can be adapted to mount a PVC when they'll run as separate Jobs.