Skip to content

feat(websearch): add Outcrawl search provider - #8475

Open
heydryft wants to merge 1 commit into
Wei-Shaw:mainfrom
heydryft:feat/outcrawl-provider
Open

heydryft wants to merge 1 commit into
Wei-Shaw:mainfrom
heydryft:feat/outcrawl-provider

Conversation

@heydryft

@heydryft heydryft commented Nov 1, 2026

Copy link
Copy Markdown

Summary

Adds Outcrawl as a third provider for web search emulation, next to Brave and Tavily. Outcrawl (https://outcrawl.ai) is a scrape and search API for AI agents. The provider calls POST https://api.outcrawl.ai/v1/search with a Bearer key and maps hits[].url/title/snippet to SearchResult. Admins can pick it in the existing provider select; quota, proxy, expiry and the weighted selection all work as they do for the other providers.

Disclosure: I work on Outcrawl.

Changes

  • backend/internal/pkg/websearch/outcrawl.go (new): OutcrawlProvider, modeled on TavilyProvider (same body limit, error format and result mapping). The endpoint is a package variable so tests can point it at an httptest server, like the Brave tests do.
  • backend/internal/pkg/websearch/outcrawl_test.go (new): name, request JSON, response parsing, a full Search against httptest (method, Authorization header, body, mapping), default limit, HTTP error, invalid JSON.
  • backend/internal/pkg/websearch/types.go: ProviderTypeOutcrawl.
  • backend/internal/pkg/websearch/manager.go: buildProvider case.
  • backend/internal/service/websearch_config.go: accepted in validProviderTypes; websearch_config_test.go covers it in the valid config case.
  • frontend/src/api/admin/settings.ts: type union; frontend/src/views/admin/SettingsView.vue: Outcrawl option in the provider select.
  • provider.go and two comments updated to list the new type.

Testing

From backend/:

  • gofmt -l internal/pkg/websearch internal/service: no output
  • go vet ./internal/pkg/websearch/: clean
  • go test -count=1 -tags=unit ./internal/pkg/websearch/: ok (7 new TestOutcrawlProvider_* tests pass)
  • go test -count=1 -tags=unit -run WebSearch ./internal/service/: ok
  • golangci-lint run ./internal/pkg/websearch/... ./internal/service/... (v2.15.0): nothing in the touched files; it reports 3 errcheck findings in existing test files this PR does not touch (openai_ws_client_read_test.go, openai_ws_passthrough_turn_pricing_test.go, openai_ws_v2_passthrough_lifecycle_test.go)

From frontend/:

  • pnpm run typecheck (vue-tsc --noEmit): passes
  • pnpm exec eslint src/api/admin/settings.ts src/views/admin/SettingsView.vue: clean

Live check: a throwaway test (not committed) built a Manager with a single outcrawl provider and ran SearchWithBestProvider for "golang context package" with MaxResults: 3. It returned provider outcrawl with 3 results (pkg.go.dev/context, go.dev/blog/context, pkg.go.dev/golang.org/x/net/context), each with a title and snippet.

The free tier includes 1,000 credits a month; a search costs 1 credit. Docs: https://outcrawl.ai/docs/search/overview

This PR was prepared with AI assistance (omp with Claude): the provider, tests, frontend option and this description were written with it, and the commands above plus the live search call were run as part of preparing it.

Add Outcrawl (https://outcrawl.ai), a scrape and search API for AI agents,
as a third web search emulation provider next to Brave and Tavily.

- backend: OutcrawlProvider (POST /v1/search with a Bearer key), factory
  case, ProviderTypeOutcrawl and config validation
- frontend: provider type union and the provider select option
- tests: request/response mapping, auth header, default limit, HTTP error
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Before we can merge this PR, we need you to sign our Contributor License Agreement (CLA).

To sign, please reply with the following comment:

I have read the CLA Document and I hereby sign the CLA

You only need to sign once — it will be valid for all your future contributions to this project.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant