Allow manually saving checkpoints

[stohrendorf]

Given that it can take quite some time to train 2500 steps, it would be convenient to create manual checkpoints so training can resume from that.

[DanRuta]

checkpoints should be written to disk every 50 iterations, not 2500. Unless I'm mis-understanding something

[stohrendorf]

This looks like 2500 to me (or 50*50), unless I'm misunderstanding something.

Some of the checkpoints need 1.5 hours or more (I guess it depends on the train data samples length).

[DanRuta]

Ah, what is your batch size set to? 1.5 hours sounds like a lot, between checkpoints. Should be a few minutes - depends on your hardware also

[stohrendorf]

I don't have top-notch hardware, it's an RTX 2060 with 12 GB of VRAM. The batch size which maximizes VRAM usage is 8. Anything above that results in an OOM at some time. I usually see the first checkpoint to be saved after 35-40 minutes, the next one after about 1h15m, and then it stays around 1h30-1h45 for each checkpoint. It's not a rig built for AI, so it's fine if you're not considering it to be an issue. I just like to fool around with AI stuff.