Skip to content

feat(cli): add configurable Insights analysis and evidence workflows - #316

Draft
liam-langchain wants to merge 5 commits into
mainfrom
codex/cli-insights
Draft

liam-langchain wants to merge 5 commits into
mainfrom
codex/cli-insights

Conversation

@liam-langchain

@liam-langchain liam-langchain commented Sep 15, 2026

Copy link
Copy Markdown

Summary

Adds configurable Insights analysis and evidence drill-down using the existing LangSmith Go SDK. No API or SDK changes.

Command Capability
insights create Start a bounded report from flags, an analysis JSON file, or a saved configuration ID.
insights runs ID Read a page of report evidence, optionally restricted to a category.
insights list Read a bounded page, with offset and saved-config filtering.
insights get ID Read status and results with project/workspace context.

Creation supports --preview-run with --dry-run, --file/-f, --categories, --attributes, --cluster-model, --summary-model, --config-id, --dry-run, and --output/-o. Categories/attributes flags accept JSON file paths. Analysis files use API field names and reject unknown fields, scheduler controls, and credential-validation overrides. File and saved-config modes reject conflicting analysis flags.

Samples accept 1–1000. Custom summary prompts require trace variables; categories reject duplicate trimmed names, and attribute names cannot contain whitespace. JSON files reject duplicate keys and excessive nesting.

Dry-run creates no job. Optional --preview-run UUID reads one project-scoped root run and returns variable bindings, missing paths, and unchecked paths without generating a summary. It does not verify the run matches the analysis window/filter or will be sampled; thread messages and feedback remain unchecked. It validates the local request but does not resolve a saved configuration, authorize a write, verify model availability, freeze a trace sample, or guarantee cost. Creation is not automatically retried after an ambiguous response.

Evidence includes extracted summaries/attributes and available run metadata; IO is explicitly marked as previews. Full IO uses run get --full. Attribute sorting is page-local, and evidence pagination exposes next_offset.

Listing behavior change: default limit is now 20, with a maximum page size of 100 and explicit --offset. The existing JSON array format is preserved. Full pages do not guarantee another page.

Saved-config authoring, generation, scheduling, and cancellation are not added. This independent PR targets main; #308 is the reference-only combined draft.

Manual testing

Build with make build, select an authenticated profile/workspace, and replace placeholders. See the README for analysis.json and the full workflow.

bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json --dry-run
bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json --dry-run --preview-run RUN_ID
# Review first: the next command incurs workspace model usage.
bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json
bin/langsmith --format json insights get JOB_ID --project-id PROJECT_ID
bin/langsmith --format json insights runs JOB_ID --project-id PROJECT_ID --cluster-id CLUSTER_ID --limit 20
bin/langsmith --format json insights list --project-id PROJECT_ID --config-id CONFIG_ID --limit 20 --offset 0

Test Plan

  • Request construction, category/attribute validation, strict bounded file parsing, conflicting modes, and saved-config-only payloads.
  • Prompt/sample boundaries and preview flag conflicts; preview performs only a scoped run query and preserves null/false/zero with explicit missing and unchecked paths.
  • Dry-run sends no creation request; creation retains no-retry behavior.
  • Evidence pagination, project/category scope, page-local sorting parameters, and false/zero preservation.
  • Bounded report listing preserves its JSON array and makes one page request.
  • Live paid creation with manual categories/custom models and restricted-role/deployment coverage.

Corridor analysis was blocked by session policy. Local lint is deferred to CI because golangci-lint is unavailable.

@liam-langchain liam-langchain changed the title feat(cli): create and wait for Insights reports feat(cli): add configurable Insights analysis and evidence workflows Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant