Apple Silicon M1 / Linux aarch64 releases?

[Manouchehri]

Is there any internal work going on to support native Apple Silicon binaries, instead of running Java under Rosetta 2? (Given that some stages of CodeQL are single threaded bound, the M1 Max will probably be the "best" CPU for many of us, especially when a new Mac Mini and/or Mac Pro comes out.)

Linux aarch64 binaries would be very useful too (for my use-case, actually more useful than native Apple Silicon).

Related #89.

[turbo]

Hi,

Thanks for the question. We recognise the increasing importance of ARM-based platforms and are continually evaluating enhanced support for them. However, we can’t commit to a timeline for Linux/ARM support yet.

The CodeQL support for Apple Silicon (which we recently announced) is in beta, and we’ll spend some time collecting feedback before committing to a native implementation. If you encounter any blockers with the beta, please open an issue.

You can track upcoming features on our public roadmap, and work towards ARM support will show up there once we have a schedule in place.

Thanks

[edoardopirovano]

A couple of further notes on progress we are making towards better ARM support in CodeQL:

• In 2.7.3, we expect our Mac OS distribution to bundle a native JDK for M1 macs and this will be used by default when appropriate. Rosetta will still be required as some parts of the CodeQL engine that run outside Java will still not be natively-compiled. However, query compilation and evaluation (which are the two performance-critical parts of our code), will now be running natively which should give a reasonable performance boost.
• We do not currently have plans to ship a native JDK for Linux aarch64. The problem here is that we do not have any CI infrastructure with this architecture, so we cannot adequately test things as we would need to in order to officially support this. However, I note that you can supply your own JDK and have CodeQL use it by setting the CODEQL_JAVA_HOME environment variable. This should allow you to run most of the CodeQL engine (as I mentioned above, there are some parts of CodeQL that are outside Java which will not work) using a native JDK that you have downloaded yourself. I would recommend the Temurin JDK, which is the one we bundle.

If you do go ahead with the second option, I note that while we do not officially support that setup I'd be very interested to hear how you get along and may be able to offer some debugging advice if anything doesn't work.

[Manouchehri]

Thank you for the information!

Rosetta will still be required as some parts of the CodeQL engine that run outside Java will still not be natively-compiled. However, query compilation and evaluation (which are the two performance-critical parts of our code), will now be running natively which should give a reasonable performance boost.

I haven't done any in-depth profiling, but it appears that for my build targets, the most time in spent in the JDK. So this should help a ton. =)

We do not currently have plans to ship a native JDK for Linux aarch64. The problem here is that we do not have any CI infrastructure with this architecture, so we cannot adequately test things as we would need to in order to officially support this.

Could you use Docker for Apple Silicon? That's my plan. https://docs.docker.com/desktop/mac/apple-silicon/

If you do go ahead with the second option, I note that while we do not officially support that setup I'd be very interested to hear how you get along and may be able to offer some debugging advice if anything doesn't work.

I'll let you know if I do try it out. Figured it made more sense to ask first before I end up redoing some of the (better) work that the CodeQL team has planned.

[edoardopirovano]

Could you use Docker for Apple Silicon? That's my plan. https://docs.docker.com/desktop/mac/apple-silicon/

This could certainly be an option to resolve our CI issue, thanks for bringing it to our attention.

The other issue is that as I understand things, in most Linux ARM64 systems there is no Rosetta-style emulator, so all the code has to be aarch64. Currently, the non-Java parts of our code are not compiled for aarch64 and it would take some work to get them compiling there. I believe query compilation and evaluation should just work with a native JDK since that part of our code is Java, but database creation and build tracing requires native code, so I expect that to not work without some more engineering effort to get that native code compiled for aarch64. I don't think we would ever ship a partially broken CLI where certain functions don't work - if we commit to supporting aarch64 on Linux we'd want everything to be working before we released it.

It's certainly on our radar, but not currently something we can commit to any release date for as we have not yet decided when we will be able to dedicate resources to this. Still, thank you for expressing an interest in this part of our work! Knowing there's users out there who would benefit from this is definitely helpful in deciding how to prioritise it.

[Manouchehri]

This could certainly be an option to resolve our CI issue, thanks for bringing it to our attention.

In theory you can also run Linux natively on the M1, but it's an uphill battle. I would just use Docker on the M1, since they already have a ton of developers focusing on supporting that workflow.

The other issue is that as I understand things, in most Linux ARM64 systems there is no Rosetta-style emulator, so all the code has to be aarch64.

binfmt_misc / qemu-user-binfmt is basically the Linux version of Rosetta 2. That said, using binfmt_misc in containers might not be ideal depending on your CI/CD setup (binfmt_misc is a "host" kernel feature, not something that unprivileged containers can adjust afaik), so I'd guess that manually using qemu-x86_64 in your scripts would be the preferable solution.

The nice thing is if you go with QEMU user mode and eventually start to ship more aarch64 native binaries, you/CodeQL developers can also run qemu-aarch64 on your x86_64 laptops for quick local tests.

https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/
https://azeria-labs.com/arm-on-x86-qemu-user/

It's certainly on our radar, but not currently something we can commit to any release date for as we have not yet decided when we will be able to dedicate resources to this.

No problem, I'm not in a rush.

Still, thank you for expressing an interest in this part of our work! Knowing there's users out there who would benefit from this is definitely helpful in deciding how to prioritise it.

Heck, I find the work interesting enough that I applied for a position at Semmle a couple years ago. =P

[edoardopirovano]

Thanks for sharing the above information, running the native parts of our code in QEMU may well provide a way forward. This is a little bit outside my area of expertise (I work primarily on the query evaluation engine which lives in the Java part of our codebase), but once we do start properly looking into Linux aarch64 support I'll be sure to share your suggestions with whoever picks up that work.

Heck, I find the work interesting enough that I applied for a position at Semmle a couple years ago. =P

Well, if you find yourself looking for work again I'd certainly encourage you to consider applying again. Our hiring has certainly not slowed down since the acquisition by GitHub - quite the opposite I think we hiring faster than we ever did in the Semmle days.

[Manouchehri]

Ran into a similar issue (no aarch64 support in our automated GitHub Action pipeline) with an unrelated product that my team is working on, but thought you might find this answer useful:

https://stackoverflow.com/a/70312558/2079814

[planetf1]

Just to add a note, I'm working with some open source libraries. I develop on mac, but when I try a new library I often start with linux - it's usually easier to compile for example. Similarly when working on a PR for that upstream project I really want to make sure linux is good. Running a linux VM on something like UTM on a m1 mac is very fast & easy.

The linux VM of course is running with the aarch64 architecture. Currently there is no codeQL binary compiled for this. I do note the point about about no CI, and the option of using java for much of the function - though in practice I'll likely just use a cloud intel linux vm, or native arm on mac - or as part of a github action of course.

But thought it worth mentioning.

[hsm207]

for my use case, i work inside Linux devcontainers on an M1 mac so a Linux aarch64 binary would be very helpful.

[sgleske-ias]

Shipping AWS services using their Graviton architecture (aarch64) is what AWS recommends and there's compute cost savings with it. Ideally, the compiling and testing happen from Graviton if that's where it is being shipped to in production.

CodeQL not operating on ARM64 Linux means we have a huge security hole and need to make a choice:

• Shop for an alternative to GitHub advanced security and CodeQL.
• Or use something in addition to CodeQL.
• Or live without security scanning (least ideal).

[patriknyblad]

We have a project that is horribly slow when running through rosetta, since CodeQL spawns processes in x86 with no possibility to force it back by prefixing with arch -arm64 so we can't really use CodeQL effectively 😬

I see that this issue has been open since 2021, what is the roadmap for when we could see a native arm64 binary?

[jketema]

@turbo's answer is still valid. For an up-to-date public roadmap see: GitHub Public Roadmap (view)

[patriknyblad]

@turbo's answer is still valid. For an up-to-date public roadmap see: https://github.com/orgs/github/projects/4247/views/1

Not much on the public roadmap for this issue 😢

[dym-ok]

It would be great if the issue of linux-aarch64 support get the well-deserved attention.

I wonder if GitHub sustainability leaders could advocate for getting some resources on this issue as ARM64 architecture is well known for energy efficiency.
For the scale of deployment of GitHub Advanced Security tools this is sure to have a massive impact on making the GitHub platform greener 🌲