Repository navigation
fix(api): refuse an upstream host that no sync run resolves as given - #903
Conversation
POST /repos stored upstream hosts that the registration check could not vet and that the run could not use as given: - a hostname that Go's resolver refuses before it asks DNS, such as `<<<.com` or an IPv4 literal with a zone (`https://1.2.3.4%25eth0`): the courtesy check failed open, the create answered 201, and every run failed; - a name of only digits and dots that is no IP address, such as `2130706433`, `1.2.3` or `01.2.3.4`: curl reads them as 127.0.0.1, 1.2.0.3 and 1.2.3.4, so the run dialed an address that the registration check never saw (the dial-time guard still vets it); - a port outside 1-65535; - a base URL with a query or a fragment, even an empty one (`https://h?`), which ends the path of every fetch URL. validUpstreamHost now also requires a dialable authority: an IP address without a zone, or a name that passes the rule of Go's resolver (net.isDomainName: at most 253 bytes, labels of 1 to 63 letters, digits, '_' and '-', not only digits and dots); a port from 1 to 65535; and no '?' or '#' in a base URL. Such a host answers 400 `is not a valid host` in both host fields, as any other invalid host does. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: I962873fa0be93de3eb41a79ac0805a745e5200a2 Signed-off-by: Thomas Kosiewski <tk@coder.com>
handlers_sync.go is covered code: validUpstreamHost and the new dialableAuthority and dialableHostname. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: I5138f4324d2d709f49fab59b06c14a9308a4d86c Signed-off-by: Thomas Kosiewski <tk@coder.com>
curl refuses bücher.example as a bad URL, so the host rule says so, and the dialableHostname comment says what bounds a zone the egress answer would repeat. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: I410340caebd88cf2c0996dbc8b4c69cfbbfb51c3 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b1a6a1f4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
curl reads a host such as 0x7f000001, 0x7f.1 or 1.2.3.0x4 as an IPv4 address (127.0.0.1, 1.2.3.4), as it does 2130706433, but the letter x let it pass the resolver rule. curlIPv4Name refuses a name of one to four labels that are each a decimal, octal or hexadecimal number. A base URL must start with `http://` or `https://` in lower case (GIT_ALLOW_PROTOCOL refuses `HTTPS`, and the worker would not route it through the egress proxy) and hold no space, which curl refuses. The non-ASCII rule now gives the reason that holds everywhere: the registration check cannot look such a name up, and whether git and curl convert it depends on the locale of the server's git. The docs list the rules, say that an empty port is the default, and that a name may have 254 bytes with a final dot. Tests add the boundary cases that four mutants survived. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: I07e32b3cc42a2039da7d52b8eb3b7948205bc89e Signed-off-by: Thomas Kosiewski <tk@coder.com>
handlers_sync.go is covered code: validUpstreamHost, dialableHostname, and the new resolverName and curlIPv4Name. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: If37f3da4f5d67fdab58a35b91559b468a662832d Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aeba00c736
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
curlIPv4Name refused any one to four labels made of digits or 0x hex, so names that only look numeric (0x100.1, 0x100000000) answered 400 although curl keeps them names. It now parses each label as curl's URL parser does: decimal, octal (leading 0) or hexadecimal (0x, 0X), at most 32 bits, all but the last at most 255 and the last within the bits the others leave. Against curl 8.22 it agrees on 33 forms, from 0x7f.1. and 0377.0377.0377.0377 to 0400.1.1.1 and 4294967296. Tests add upper-case hex digits, the range boundaries, names that only look numeric, and 64-byte labels of digits and of dashes; the docs say which number forms are refused. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Idf4ed0893728a8e3c43baba6eda5a5647f7eecad Signed-off-by: Thomas Kosiewski <tk@coder.com>
Main gained #896, #901 and #902. #902 dropped the strconv import of handlers_sync.go with quoteValue; this branch's port and IPv4 checks use it, so the merge keeps it. The other conflicts are the lane-schedule fingerprints and sync commit; the next commit re-records them on the merged tree. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Ib2c61d6a44b3a74698bce02f2eef91f7fe0568a3 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Ib69e3836b0cd3da64045e0fdff26182c2c9bec88 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8dfa9daff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
POST /reposstored upstream hosts that the registration check could not vet and that no sync run could use as given. The create answered 201, and the trouble came later, run by run:upstream.hostorbase_repo.upstream_host)<<<.com,a..b.example, a label of 64 bytes, a name of 254 bytesis not a valid hosthttps://1.2.3.4%25eth02130706433,1.2.3,01.2.3.4,0x7f000001,0x7f.1,1.2.3.0x4127.0.0.1,1.2.0.3,1.2.3.4,127.0.0.1,127.0.0.1and1.2.3.4, an address that the check never saw (the dial-time guard still vets it)forge.example:0,forge.example:65536https://forge.example?,https://forge.example/git#xhttps://forge.example?/o/n.gitasks the wrong pathhttps://forge.example/a b) or a scheme that is not lower case (HTTPS://forge.example)GIT_ALLOW_PROTOCOLrefusesHTTPS, and the worker'shttps://prefix check would not route it through the egress proxyvalidUpstreamHost(internal/api/handlers_sync.go) now also requires a dialable authority (dialableAuthority,dialableHostname): an IP address without a zone, or a name that passes the rule of Go's resolver,net.isDomainName(resolverName: at most 253 bytes, or 254 with a final dot; labels of 1 to 63 letters, digits,_and-that do not start or end with-; not only digits and dots) and that curl does not read as an IPv4 address (curlIPv4Name: one to four labels, a final dot aside, each a decimal, octal or hexadecimal number of at most 32 bits, all but the last at most 255 and the last within the bits the others leave, as curl'sipv4_normalizereads them;0x100.1and0x100000000stay names); a port from 1 to 65535, or an empty one (the default); and, in a base URL, a lower-casehttp://orhttps://and no?,#or space. A sync run resolves through the egress proxy, whose Go resolver refuses any other name. The.root name, which Go accepts, is refused: it names no host.A non-ASCII name such as
bücher.example→ 400; give its ASCII form,xn--bcher-kva.example. Go's resolver refuses the Unicode form, so the registration check cannot look it up, and whether git and curl convert it to the ASCII form depends on the locale that the server's git runs in. This is a change for a deployment where that conversion worked: such a create answered 201 before.Hosts that keep working include
forge.example,forge.example.,a_b.example,1-1,xn--bcher-kva.example,localhost,0x1.0x2.0x3.0x4.0x5and0x100.1(names for curl),10.0.0.1:1,[::1]:65535,forge.example:,https://forge.example/sub/path/,https://[2001:db8::1]:443/gitandhttps://forge.example/a%3Fb.The docs (
docs/reference/http-api.md, thehostbullet) andinternal/api/README.mdstate the rule.Evidence
TestValidUpstreamHostlists 40 accepted and 54 refused hosts, including the boundaries (a 63-byte label first and last, a 253-byte name, ports 1 and 65535, five numeric labels).TestSyncCreateRefusesUndialableHostssends seven of them in both host fields: 400is not a valid hostnaming the field, nothing stored.?#check, no space check, an upper-case scheme admitted, numeric names admitted, a last label of 64 bytes admitted, one of 63 refused,-or_not counted as a letter, no label limit, no length limit, a zone admitted, nocurlIPv4Name, three labels instead of four, no hex, only lower-case0x, any number of labels, a label that starts with-.curl -v http://2130706433:9/tries127.0.0.1:9;1.2.3tries1.2.0.3;01.2.3.4tries1.2.3.4;<<<.comis rejected as a bad hostname. The review found that0x7f000001and0x7f.1go to127.0.0.1, and thatbücher.exampleis converted toxn--bcher-kva.exampleunder a UTF-8 locale but refused in the C locale.go test -race -tags=openstorage_viewcheck ./internal/api/...without pg passes at 2b1a6a1 (388 s), at bfffd09 (349 s) and at the merge 2e98dad (359 s). The sync worker is unchanged. PostgreSQL and S3-WAL rely on CI.handlers_sync.gois covered by the lane-schedule model, so its fingerprints are re-recorded (c5b0d36, aeba00c, b8dfa9d).Reviews
resolverNameagainst Go'sisDomainName(6.7M exhaustive names, 2M random, 60 s of fuzzing: no difference but.):0x7f000001) passed (Codex and the review): bfffd09 addscurlIPv4Name.bücher.example" held only in the C locale (review), and Codex asked to keep IDN hosts working. bfffd09 keeps the refusal with the reason that holds everywhere (above), and the docs say to give the ASCII form; keeping Unicode names would need IDNA conversion for the registration check (a directgolang.org/x/net/idnadependency) and would still depend on the server's locale.internal/api/README.mdstates the rule.0x100.1,0x100000000) were refused although curl keeps them names, and a fresh review found two test gaps (upper-case hex digits, 64-byte labels of digits or dashes) and three P3s. 78f3a1a parses each label as curl does, with IPv4's ranges, and agrees with curl 8.22 on 33 forms (0x7f.1.,0377.0377.0377.0377,0xff.0xffffffread as addresses;0400.1.1.1,0xff.0x1000000,4294967296read as names); the tests add those boundaries, and the docs name the number forms and Go's all-digits rule. The first table row no longer listsbücher.exampleas failing every run. 8 more overlay mutants: 7 killed, and the 64-bit parse is equivalent (the range checks cut the same values).strconvimport ofhandlers_sync.gowithquoteValue, and this branch needs it, so the merge keeps it; b8dfa9d re-records the lane-schedule fingerprints on the merged tree.curlIPv4Namewith libcurl 8.22 (the one git links) on about 4.3 million names, exhaustive over the listed labels in one to five labels with and without a final dot, plus random and boundary names, and through git and a capture proxy: no difference. curl 8.18 keeps a final-dot or0Xform a name where the PR refuses it, which is over-refusal of all-numeric names only. Its 66 mutants: 59 killed, 6 equivalent. The Codex review (P2) said[forge.example]passes; Go 1.27.4'surl.Parse(the versiongo.modand the release image pin) refuses a bracketed host that is not IPv6 (invalid host,invalid IP-literal), sovalidUpstreamHostnever reaches the name. All six Codex requests of this PR are used./etc/hostsholds and that Go's DNS rule refuses (such as one with~) is refused, although the run's resolver reads/etc/hostsfirst. Older links stored with any of these hosts keep failing on each run; nothing rewrites them.Found by the reviews of #900 (desk follow-up).
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh