[mbg]

Traced analysis is unsupported on macOS 27 (or newer) or with Xcode 27 (or newer). This PR adds a check for traced analyses which checks whether the macOS version is < 27 and the Xcode version is < 27.

If either is >= 27, then we report this issue to the user with a suitable diagnostic. I have also added a FF which controls whether this is just a warning or a fatal error which aborts the analysis.

Further, there is a new ToolsFeature for Swift which allows the CLI to indicate to us when it is ready to support swift on all platforms. Once the CLI reports that it supports this, we skip all of the checks for swift.

There are two new FFs: one which controls whether the new checks are performed and one which controls whether a failing check is a fatal error or not.

Notes for reviewers

• We have to be careful about the checks in checkRunnerSupportsTracedAnalysis with respect to determining whether an analysis is traced.

Risk assessment

For internal use only. Please select the risk level of this change:

• Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Workflow types:

• Advanced setup - Impacts users who have custom CodeQL workflows.
• Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

• Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.
• Code Quality - The changes impact analyses when analysis-kinds: code-quality.
• Other first-party - The changes impact other first-party analyses.

Environments:

• Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.
• GHES - Impacts CodeQL workflows on GitHub Enterprise Server.

How did/will you validate this change?

• Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).
• End-to-end tests - I am depending on PR checks (i.e. tests in pr-checks).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

• Feature flags - All new or changed code paths can be fully disabled with corresponding feature flags.

How will you know if something goes wrong after this change is released?

• Telemetry - I rely on existing telemetry or have made changes to the telemetry.
  • Dashboards - I will watch relevant dashboards for issues after the release. Consider whether this requires this change to be released at a particular time rather than as part of a regular release.
  • Alerts - New or existing monitors will trip if something goes wrong with this change.

Are there any special considerations for merging or releasing this change?

• Special considerations - This change should only be merged once certain preconditions are met. Please provide details of those or link to this PR from an internal issue.

Merge / deployment checklist

• Confirm this change is backwards compatible with existing workflows.
• Consider adding a changelog entry for this change.
• Confirm the readme and docs have been updated if necessary.

[Copilot]

🟡 Changes recommended

Swift capability handling can still reject supported configurations, workflow conditions let cancellation bypass the Dependabot exclusion, and missing feature flags prevent key CI checks and regression tests from exercising the intended paths.

6 open issues

• Enable unsupported-analysis checking in the generated workflow · New
• Parenthesize supplied conditions to preserve the exclusion · New
• Strengthen regression test with flags and a traced language · New
• Enable both unsupported-analysis flags in exemption tests · New
• Exclude Swift when the CLI supports all Swift platforms · New
• Apply Swift capability exemption to language selection

5 resolved since last review

• Detect the effective Xcode toolchain version
• Use macOS product version instead of Darwin kernel release
• Skip traced-language checks on non-macOS runners
• Attach compatibility diagnostics only to affected languages
• Make feature-flagged init failure test deterministic

2 files not reviewed

• .github/workflows/__unsupported-macos.yml: Excluded by your content exclusion settings
• lib/entry-points.js: Excluded by your content exclusion settings

_{Review effort: Max}

[henrymercer]

I'd like to dig deeper into the robustness of this method of checking the Xcode version. For instance, my machine has Xcode installed, but xcodeVersion would report "Xcode is probably not installed" because /private/var/db/xcode_select_link is missing. Also, Xcode is installed at /Applications/Xcode.app on my machine so even if Xcode was found, so the check would report that the path does not contain the expected pattern and we'd fail to determine the version number. I'm running macOS 27, and perhaps the behaviour would be different on a previous OS version, but I think we need to consider the variance in how Xcode might be installed if we're going to ship this to self-hosted runners.

Would you be able to share some discussion on why you chose this method? For instance, given we only care about checking whether the major version is >= 27, I'm curious how the check in the PR weighs up against e.g. xcrun --sdk macosx --show-sdk-version.

It might also be helpful to ship this more incrementally, e.g. we could ship the macOS 27 check as is in one PR, and we could ship the Xcode check as is but for hosted runners only in another. We could also add telemetry to measure whether the detection mechanism works in production. For instance, it might be useful to record if a Swift analysis completes successfully but the detection mechanism says Xcode is not installed.