Repository navigation
Conversation
Add Outcrawl (https://outcrawl.ai), a scrape and search API for AI agents, as a third web search emulation provider next to Brave and Tavily. - backend: OutcrawlProvider (POST /v1/search with a Bearer key), factory case, ProviderTypeOutcrawl and config validation - frontend: provider type union and the provider select option - tests: request/response mapping, auth header, default limit, HTTP error
Contributor
|
Thank you for your contribution! Before we can merge this PR, we need you to sign our Contributor License Agreement (CLA). To sign, please reply with the following comment:
You only need to sign once — it will be valid for all your future contributions to this project. I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Outcrawl as a third provider for web search emulation, next to Brave and Tavily. Outcrawl (https://outcrawl.ai) is a scrape and search API for AI agents. The provider calls
POST https://api.outcrawl.ai/v1/searchwith a Bearer key and mapshits[].url/title/snippettoSearchResult. Admins can pick it in the existing provider select; quota, proxy, expiry and the weighted selection all work as they do for the other providers.Disclosure: I work on Outcrawl.
Changes
backend/internal/pkg/websearch/outcrawl.go(new):OutcrawlProvider, modeled onTavilyProvider(same body limit, error format and result mapping). The endpoint is a package variable so tests can point it at anhttptestserver, like the Brave tests do.backend/internal/pkg/websearch/outcrawl_test.go(new): name, request JSON, response parsing, a fullSearchagainsthttptest(method,Authorizationheader, body, mapping), default limit, HTTP error, invalid JSON.backend/internal/pkg/websearch/types.go:ProviderTypeOutcrawl.backend/internal/pkg/websearch/manager.go:buildProvidercase.backend/internal/service/websearch_config.go: accepted invalidProviderTypes;websearch_config_test.gocovers it in the valid config case.frontend/src/api/admin/settings.ts: type union;frontend/src/views/admin/SettingsView.vue:Outcrawloption in the provider select.provider.goand two comments updated to list the new type.Testing
From
backend/:gofmt -l internal/pkg/websearch internal/service: no outputgo vet ./internal/pkg/websearch/: cleango test -count=1 -tags=unit ./internal/pkg/websearch/: ok (7 newTestOutcrawlProvider_*tests pass)go test -count=1 -tags=unit -run WebSearch ./internal/service/: okgolangci-lint run ./internal/pkg/websearch/... ./internal/service/...(v2.15.0): nothing in the touched files; it reports 3 errcheck findings in existing test files this PR does not touch (openai_ws_client_read_test.go,openai_ws_passthrough_turn_pricing_test.go,openai_ws_v2_passthrough_lifecycle_test.go)From
frontend/:pnpm run typecheck(vue-tsc --noEmit): passespnpm exec eslint src/api/admin/settings.ts src/views/admin/SettingsView.vue: cleanLive check: a throwaway test (not committed) built a
Managerwith a singleoutcrawlprovider and ranSearchWithBestProviderfor "golang context package" withMaxResults: 3. It returned provideroutcrawlwith 3 results (pkg.go.dev/context, go.dev/blog/context, pkg.go.dev/golang.org/x/net/context), each with a title and snippet.The free tier includes 1,000 credits a month; a search costs 1 credit. Docs: https://outcrawl.ai/docs/search/overview
This PR was prepared with AI assistance (omp with Claude): the provider, tests, frontend option and this description were written with it, and the commands above plus the live search call were run as part of preparing it.