Repository navigation
feat(devkit): corporate validation kit - #61
Conversation
A self-contained, PS 5.1-compatible probe for locked-down corporate Windows: pinned installs of the published 0.2.1/0.3.1/0.1.3/0.2.0 line into an isolated venv, synthetic-fixture functional checks (doctor, qa-pack fail-closed, search index/query, judge backend fail-closed, bridge args validation, memory quarantine), proxy/TLS-inspection diagnostics, VS Code CLI detection, an offline --no-index reinstall proof, and a documented Restricted Mode procedure left as NOT_TESTED until run by a human. Emits sanitized results.json + report.md (user paths and token-shaped strings redacted) with explicit result classes. Never touches proxy settings, the trust store, the registry, credentials or real session data. Tested under pwsh 7 on Linux: 22 PASS / 0 FAIL / 2 NOT_TESTED. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
github-actions
Bot
added
the
documentation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| $lr = Invoke-Proc $code @('--list-extensions') 60 | ||
| $exts = ($lr.StdOut -split "`n" | Where-Object { $_ -match 'icaro0310\.' }) -join ', ' | ||
| Add-Result 'vscode.cli' 'PASS' ("VS Code $($r.StdOut.Split("`n")[0]); installed icaro0310 extensions: " + $(if ($exts) { $exts } else { 'none' })) $lr.StdOut.Substring(0, [Math]::Min(400, $lr.StdOut.Length)) |
There was a problem hiding this comment.
🔍 Extension discovery failure appears successful
vscode.cli ignores code --list-extensions failures. An empty error result becomes PASS with no matching extensions reported.
Was this helpful? React with 👍 or 👎 to provide feedback.
| $skipHit = $Skip -contains $Id | ||
| if ($skipHit) { Add-Result $Id 'NOT_TESTED' 'skipped by -Skip'; return } | ||
| if (@($Only).Count -gt 0) { | ||
| $onlyHit = $Only -contains $Id | ||
| if (-not $onlyHit) { return } | ||
| } |
There was a problem hiding this comment.
🔍 Skipped checks escape the selection filter
Invoke-Check records -Skip results before applying -Only. An unrelated skipped check appears in the selected run's report.
Was this helpful? React with 👍 or 👎 to provide feedback.
- PS5.1 compat: replace $IsWindows/$IsLinux (inexistent on PS 5.1, fatal under StrictMode) with OSVersion.Platform detection - Invoke-Proc: route .cmd/.bat shims through cmd.exe so npm, code and package bins actually launch on Windows - pins: read install_spec from the bundled devkit manifest when present so the kit validates what devkit installs; standalone ZIP falls back to latest published versions (pin source recorded in evidence) - classify "index cannot serve spec" as BLOCKED_ENVIRONMENT (covers incomplete -WheelDir bundles and filtered mirrors) - sanitize summary and command fields in Add-Result, closing the exception-message leak path in Invoke-Check - Get-TlsIssuer captures SslPolicyErrors: untrusted chains report UNVERIFIED with the issuer, never PASS - per-run temp root, fully removed at exit (unless OutputDir is inside it) so repeated runs stop accumulating venvs/npm trees/wheels Tests: +6 structural assertions covering each fix; pwsh execution suite still green (17 kit tests, 123 total, 1 skipped).
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| $mf = Get-Content $mc -Raw | ConvertFrom-Json | ||
| foreach ($t in @($mf.tools)) { | ||
| $spec = "$($t.install_spec)" | ||
| if ($spec -match '^@icaro0310/devin-bridge@\S+$') { $BridgeSpec = $spec; continue } | ||
| foreach ($pkg in $PyPackages) { | ||
| if ($spec -match ('^' + [regex]::Escape($pkg.Name) + '(\[[^\]]*\])?==(\S+)$')) { | ||
| $pkg.Spec = $spec | ||
| $pkg.Want = $Matches[2] | ||
| } | ||
| } | ||
| } | ||
| $PinSource = 'bundled devkit manifest' |
There was a problem hiding this comment.
🔴 Checkout probes test obsolete releases
In a checkout, $PyPackages and $BridgeSpec inherit older manifest pins instead of the releases targeted by the regression probes. The qa-pack and judge probes can fail on their fixed regressions, falsely reporting that the published toolchain is broken.
Learn more
The kit defines fallback pins for the releases its smoke checks intend to validate. In a checkout, the manifest is found and replaces those pins before the checks run. The checked-in manifest pins devin-qa-pack==0.2.0, poordjaevin[local,mcp]==0.1.2, devin-search==0.2.0, and bridge 0.1.0, while the fallback pins are 0.2.1, 0.1.3, 0.2.1, and 0.2.0. The qa-pack probe and judge probe exercise regressions targeted at the later versions, so a checkout can report a product failure for a release the standalone kit does not test.
Example: With the current checkout and an index serving both qa-pack 0.2.0 and 0.2.1, the kit installs 0.2.0. Its bad-database probe tests that older build and can flag the known regression despite 0.2.1 working.
Recommended fix: Keep the manifest pins aligned with the validated releases, or separate manifest compatibility checks from release-specific regression probes. Test the effective installed specs and probe results from an actual checkout run.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if ($script:IsWindowsHost -and $File -match '\.(cmd|bat)$') { | ||
| $cmdExe = if ($env:ComSpec) { $env:ComSpec } else { 'cmd.exe' } | ||
| $psi.FileName = $cmdExe | ||
| $psi.Arguments = '/c "' + '"' + $File + '"' + $(if ($argLine) { ' ' + $argLine } else { '' }) + '"' |
There was a problem hiding this comment.
🟥 Batch shim paths execute unintended commands
On Windows, Invoke-Proc passes paths with cmd metacharacters to .cmd shims without shell-safe quoting. A crafted -WheelDir or temp path can execute commands outside the intended validation probe.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Corporate validation kit
environments/corporate-windows/validation-kit/Invoke-CorporateValidation.ps1— a single self-contained PS 5.1-compatible script that proves whether the published devin-* distribution works on a locked-down corporate machine (Zscaler/TLS inspection, restricted index, no admin, VS Code Restricted Mode).What it checks (24 probes)
NOT_DETECTED_VALIDwhen absent)devin-search==0.2.1,devin-doctor==0.3.1,devin-qa-pack==0.2.1,poordjaevin==0.1.3,devin-memory==0.3.1,@icaro0310/devin-bridge@0.2.0— into an isolated temp venv/prefixBRIDGE_AGENT_ARGSvalidation, memory quarantine on a synthetic storepip --no-indexreinstall--install-extension(opt-in), documented Restricted Mode procedure (NOT_TESTEDuntil run manually)Guarantees
PASS/FAIL_PRODUCT/BLOCKED_ENVIRONMENT/NOT_DETECTED_VALID/NOT_TESTED/UNVERIFIEDresults.json+report.md; exit 1 only onFAIL_PRODUCTVerified in this session (pwsh 7, Linux)
22 PASS · 0 FAIL_PRODUCT · 0 BLOCKED · 2 NOT_TESTED (Restricted Mode + opt-in ext install).
-Offlineand-Onlymodes exercised. 11 pytest cases cover PS5.1 syntax constraints, schema, sanitization and real execution under pwsh.Does not claim a Windows-corporate pass — the kit must still be run on the target machine; that is its purpose.
Generated with Devin