Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proper support for Mochi and temporal vae #312

Open
AustinMroz opened this issue Nov 6, 2024 · 0 comments
Open

Proper support for Mochi and temporal vae #312

AustinMroz opened this issue Nov 6, 2024 · 0 comments

Comments

@AustinMroz
Copy link
Collaborator

Mochi is now available in core, but it's temporal vae introduces some technical difficulties that are not easily solved as the number of frames must be congruent to 1 mod 6.

  • Batching vae opperations is difficult since frames in latent space interpolate between one another.
    • More testing is needed, but I suspect that batching will have sideeffects on output even with forced overlap.
  • vae encode will truncate frames if not passed a number congruent to 1 mod 6. I don't foresee a sane way to restrict selections in the UI, but am considering adding a hint that a selected number is valid for mochi instead of preventing selection in the widget.
    • Under basic workflows, an invalid frame count is a minor annoyance.
    • Under Meta Batch workflows, this results in skips in the output and audio desync.

Ideally, any solutions would not be tied to mochi in case other temporally aware vae are added in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant