feat(cli): add configurable Insights analysis and evidence workflows - #316
Draft
liam-langchain wants to merge 5 commits into
Draft
liam-langchain wants to merge 5 commits into
liam-langchain wants to merge 5 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds configurable Insights analysis and evidence drill-down using the existing LangSmith Go SDK. No API or SDK changes.
insights createinsights runs IDinsights listinsights get IDCreation supports
--preview-runwith--dry-run,--file/-f,--categories,--attributes,--cluster-model,--summary-model,--config-id,--dry-run, and--output/-o. Categories/attributes flags accept JSON file paths. Analysis files use API field names and reject unknown fields, scheduler controls, and credential-validation overrides. File and saved-config modes reject conflicting analysis flags.Samples accept 1–1000. Custom summary prompts require trace variables; categories reject duplicate trimmed names, and attribute names cannot contain whitespace. JSON files reject duplicate keys and excessive nesting.
Dry-run creates no job. Optional
--preview-run UUIDreads one project-scoped root run and returns variable bindings, missing paths, and unchecked paths without generating a summary. It does not verify the run matches the analysis window/filter or will be sampled; thread messages and feedback remain unchecked. It validates the local request but does not resolve a saved configuration, authorize a write, verify model availability, freeze a trace sample, or guarantee cost. Creation is not automatically retried after an ambiguous response.Evidence includes extracted summaries/attributes and available run metadata; IO is explicitly marked as previews. Full IO uses
run get --full. Attribute sorting is page-local, and evidence pagination exposesnext_offset.Listing behavior change: default limit is now 20, with a maximum page size of 100 and explicit
--offset. The existing JSON array format is preserved. Full pages do not guarantee another page.Saved-config authoring, generation, scheduling, and cancellation are not added. This independent PR targets
main; #308 is the reference-only combined draft.Manual testing
Build with
make build, select an authenticated profile/workspace, and replace placeholders. See the README foranalysis.jsonand the full workflow.bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json --dry-run bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json --dry-run --preview-run RUN_ID # Review first: the next command incurs workspace model usage. bin/langsmith --format json insights create --project-id PROJECT_ID --file analysis.json bin/langsmith --format json insights get JOB_ID --project-id PROJECT_ID bin/langsmith --format json insights runs JOB_ID --project-id PROJECT_ID --cluster-id CLUSTER_ID --limit 20 bin/langsmith --format json insights list --project-id PROJECT_ID --config-id CONFIG_ID --limit 20 --offset 0Test Plan
Corridor analysis was blocked by session policy. Local lint is deferred to CI because
golangci-lintis unavailable.