feat: set up turborepo

[cprussin]

This PR sets up turborepo for monorepo task orchestration.

Scope

This PR mostly just adds turborepo to the repository, configures it and some packages to hook their tasks into top level commands test, build, start:dev, start:prod, and fix, and converts commands that used to use lerna to use turbo instead. To do this correctly I had to restructure a bit of how some packages were built.

But why?

A monorepo orchestration tool enables us to declaratively configure task orchestration between and within packages within the monorepo. It makes it much easier to define and reproduce complex build pipelines. It also enables caching task execution and re-running only what needs to run based on what's changing in the repo.

With a good monorepo orchestration tool properly configured, the typical development and CI commands are solely concerned with what you want to do (i.e. "start the dev servers") and not the steps required to accomplish that, and the tool ensures only necessary work is done and is cached and re-used as possible.

What about lerna?

We already had lerna in the monorepo, but lerna lacks many modern features of a monorepo task orchestration framework, including task caching and defining complex task dependency relationships (e.g. you must run build on all dependencies of a package before you can test it, or you must pull the .env.local from Vercel before running a nextjs dev server, or that dependencies must be built before you can run typescript type checking, etc)

Lerna isn't going away in this PR and is still useful for managing publishing and versioning packages. However, we are no longer using lerna for task orchestration.

What about nx?

I evaluated using nx which is used by lerna under the hood, but I found myself fighting a lot of UX issues with nx that made turborepo feel like a better fit here:

• nx's recommended monorepo structure is to use a single package.json at the root, see https://nx.dev/deprecated/integrated-vs-package-based. While nx claims this is no longer the case, I found many issues where things just wouldn't work correctly or it was unclear how to use them in a repo where each package has it's own package.json. It's clear that our style of monorepo is not nx's primary intended use case.
• the nx documentation felt quite poor compared to turbo repo's documentation
• nx's task execution UI felt poor and clunky compared to turborepo's terminal UI
• I was having consistent issues with persistent tasks (i.e. dev servers) hanging and going unresponsive in nx; these issues did not happen in turbo. I'm not convinced nx is designed to handle persistent tasks correctly.

What about bazel/buck/etc?

I've used these tools in the past and while they're extremely powerful, they're also extremely complex and hard to configure correctly, and require a ton of maintenance and in many cases fairly significant changes to how packages are built.

Where those tools really shine is dealing inter-dependencies between packages from various programming languages. That could become valuable to us eventually, although it is possible to use turborepo in a polyglot environment (however it's a bit clunky). But for now, we are only managing javascript/typescript packages and bazel/buck/etc would be an extremely expensive lift compared to turborepo.

Perhaps this will be worth revisiting eventually if our needs for complex task orchestration between packages of different languages grows.

What's next?

There are a ton more things we can do to improve things, which I'll hold off as follow ups. Significant follow-up improvements would be:

• Including docker image building as part of the build task, and running the build task in CI, to ensure changes don't break docker builds
• Moving prettier to each package and out of pre-commit, and hooking up pre-commit to run fix
• Exploring integration with packages in the monorepo that aren't typescript or javascript. For instance, the hermes client relies on the Swagger openapi spec to build, but currently there's no way to run the task to generate the spec file in the hermes server package when building the hermes client since we don't have task orchestration working between Rust and JS. So for a long time, we've relied on the hack of pulling this file from a hermes server as part of the build. However, if a change in the hermes server code required the client to change, this wouldn't be caught until the server went live. Having turborepo set up to be usable in non-ts / js packages would solve this problem nicely.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
api-reference	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2024 5:55pm
component-library	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2024 5:55pm
proposals	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2024 5:55pm
staking	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2024 5:55pm

[vercel]

@cprussin is attempting to deploy a commit to the Pyth Network Team on Vercel.

A member of the Team first needs to authorize it.

[keyvankhademi]

Nice!

[ali-bahjati]

Very nice!