Skip to content

feat: add hub-completeness guard to prevent instruction/prompt README drift - #3322

Open
pratik wayase (PratikWayase) wants to merge 2 commits into
microsoft:mainfrom
PratikWayase:feat/hub-completeness-guard
Open

pratik wayase (PratikWayase) wants to merge 2 commits into
microsoft:mainfrom
PratikWayase:feat/hub-completeness-guard

Conversation

@PratikWayase

Copy link
Copy Markdown
Contributor

Description

Adds a CI-style guard that fails when an instruction or prompt artifact exists on disk but isn't referenced in its hub README. The two hub indexes (.github/instructions/README.md and .github/prompts/README.md) are hand-maintained and excluded from lint:md coverage, so drift between the README and the actual files goes undetected until someone notices manually.

The validator (scripts/linting/Test-DocHubCompleteness.ps1) follows the same dot-sourceable core + CLI guard pattern used by Test-ExtensionArtifactNaming.ps1. It enumerates both hubs, applies the dt-method-* exclusion for per-method coaching prompts, and checks each artifact's filename as a case-insensitive substring in the README text. Supports -ChangedFilesOnly for PR-scoped validation and exports JSON results with CI annotations + step summary via the shared CIHelpers/LintingHelpers modules.

Also reconciles 10 instruction artifacts that were already missing from the instructions README Directory Structure tree (caught by the guard on first run).

Related Issue(s)

Closes #2435

Type of Change

Code & Documentation:

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Infrastructure & Configuration:

  • GitHub Actions workflow
  • Linting configuration (markdown, PowerShell, etc.)
  • Security configuration
  • DevContainer configuration
  • Dependency update

Other:

  • Script/automation (.ps1, .sh, .py)

Testing

  • npm run lint:doc-hubs — passes against current repo (59 artifacts checked, 0 missing)
  • Invoke-Pester -Path scripts/tests/linting/Test-DocHubCompleteness.Tests.ps1 -Output Detailed — 25/25 tests pass
  • npm run lint:ps — PSScriptAnalyzer clean
  • npm run lint:md — passes
  • npm run lint:frontmatter — 0 errors
  • Integration test verifies the guard passes against the current repository state

Checklist

Required Checks

  • Documentation is updated (if applicable)
  • Files follow existing naming conventions
  • Changes are backwards compatible (if applicable)
  • Tests added for new functionality (if applicable)

Required Local Checks

  • Local validation aggregate: npm run validate:local
  • Documentation validation (if docs changed): npm run validate:docs
  • Spell checking: npm run spell-check
  • Link validation: npm run lint:md-links

Security Considerations

  • This PR does not contain any sensitive or NDA information
  • Any new dependencies have been reviewed for security issues
  • Security-related scripts follow the principle of least privilege

Additional Notes

The guard is wired into validate:local immediately after lint:extension-artifact-naming. The dt-method-* exclusion matches the prompts hub convention of summarizing per-method coaching prompts in a note rather than linking them individually.

Duplicate basenames (e.g., pull-request.instructions.md exists at both the instructions root and under hve-core/) are detected and reported as non-fatal warnings rather than failures, since the current repo has this case and pure basename matching would false-pass if at least one copy is referenced. This can be tightened to path-aware matching in a follow-up if needed.

@PratikWayase
pratik wayase (PratikWayase) requested a review from a team as a code owner November 2, 2026 14:36
@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.22917% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.28%. Comparing base (7d2dde5) to head (5a5053a).

Files with missing lines Patch % Lines
scripts/linting/Test-DocHubCompleteness.ps1 93.22% 13 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3322      +/-   ##
==========================================
+ Coverage   85.25%   85.28%   +0.03%     
==========================================
  Files         198      199       +1     
  Lines       40650    40842     +192     
  Branches      201      201              
==========================================
+ Hits        34655    34833     +178     
- Misses       5924     5938      +14     
  Partials       71       71              
Flag Coverage Δ
pester 86.96% <93.22%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
scripts/linting/Test-DocHubCompleteness.ps1 93.22% <93.22%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding the completeness check and tests. Please address the repository-root handling issue noted inline and add regression coverage before merging.

$changedFiles = @()
if ($ChangedFilesOnly) {
try {
$changedFiles = @(Get-ChangedFilesFromGit -BaseBranch $BaseBranch -FileExtensions @('*.instructions.md', '*.prompt.md'))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding the completeness check and tests; all 25 submitted tests and the normal lint command passed when rerun.

$changedFiles = @(Get-ChangedFilesFromGit -BaseBranch $BaseBranch -FileExtensions @('*.instructions.md', '*.prompt.md'))

Could you normalize RepoRoot to an absolute path and run this Git lookup from that directory?

In a controlled fixture, -RepoRoot . -ChangedFilesOnly returned success with zero artifacts checked, while the same target with an absolute root found its missing entry; invoking from another checkout also used that checkout's changes instead of the selected target's.

Please add regression tests for both forms.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add a hub-completeness guard to prevent instruction/prompt index README drift

4 participants