Create a recurring crawl seed

curl --request POST \
  --url https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: '<string>'})
};

fetch('https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources"

payload = { "url": "<string>" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "id": "<string>",
  "object": "knowledge_base.web_source",
  "knowledge_base_id": "<string>",
  "url": "<string>",
  "created_at": 123,
  "crawler_options": {},
  "last_crawl_at": 123,
  "next_crawl_at": 123,
  "metrics": {
    "pages_total": 1,
    "pages_indexed": 1,
    "pages_failed": 1,
    "pages_pending": 1,
    "pages_skipped": 1,
    "last_run_started_at": 123,
    "last_run_completed_at": 123
  },
  "updated_at": 123
}

Web Sources

Create a recurring crawl seed

Registers a start URL + per-domain crawler options. The first crawl is scheduled immediately unless the knowledge base is paused (crawl_settings.paused - creating a seed never resumes the crawler). Discovered pages arrive as documents (origin: web_source, web_source_id set), deduped against manual documents by source_key. The engine-global crawl settings (webpages_limit, periodicity, paused, plus decorative depth) are rejected here with 400 - set them via the KB PATCH crawl_settings. The URL is the idempotency key: re-POSTing an already-registered URL returns the existing seed with 200 rather than creating a duplicate. Requires editor+.

POST

knowledge_bases

{knowledgeBaseId}

web_sources

Create a recurring crawl seed

curl --request POST \
  --url https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: '<string>'})
};

fetch('https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://{host}/v2/workspaces/slug:storage/webhooks/v1/knowledge_bases/{knowledgeBaseId}/web_sources"

payload = { "url": "<string>" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "id": "<string>",
  "object": "knowledge_base.web_source",
  "knowledge_base_id": "<string>",
  "url": "<string>",
  "created_at": 123,
  "crawler_options": {},
  "last_crawl_at": 123,
  "next_crawl_at": 123,
  "metrics": {
    "pages_total": 1,
    "pages_indexed": 1,
    "pages_failed": 1,
    "pages_pending": 1,
    "pages_skipped": 1,
    "last_run_started_at": 123,
    "last_run_completed_at": 123
  },
  "updated_at": 123
}

Authorizations

Authorization

string

header

required

User session JWT or instance API key (iak_*). Send as Authorization: Bearer <token>.

Path Parameters

knowledgeBaseId

string

required

Knowledge base id. Legacy physical prefix vs_ (the kb_ rename is deferred).

Maximum string length: 128

Pattern: ^vs_[A-Za-z0-9-]+$

Body

application/json

url

string<uri>

required

crawler_options

object

Response

Existing seed returned (the URL was already registered).

Recurring web crawl seed. A seed is a generator of documents, not a document: pages it discovers materialize as knowledge_base.document rows (source_type: web_page, origin: web_source, web_source_id set) deduped in the same source_key space as manual documents. A seed carries only its URL and per-domain crawler_options - the engine-global crawl settings (webpages_limit, periodicity, paused) live on the knowledge base's crawl_settings object (one crawler per KB).

string

required

Maximum string length: 128

Pattern: ^seed_[A-Za-z0-9-]+$

object

enum<string>

required

Available options:

knowledge_base.web_source

knowledge_base_id

string

required

Pattern: ^vs_[A-Za-z0-9-]+$

url

string<uri>

required

Seed URL the crawler starts from.

created_at

integer

required

crawler_options

object | null

Provider-specific crawl configuration passed through to the crawler (e.g. include/exclude URL patterns, auth headers, render mode). Validated by the crawler, not by this API. Projects PER DOMAIN on the store-level crawler: sources on the same hostname share these options.

last_crawl_at

integer | null

Reserved. Currently always null - there is no per-seed crawl-run ledger yet.

next_crawl_at

integer | null

Reserved. Currently always null (no per-seed scheduled-run computation yet).

metrics

object

Per-seed page counts, DB-derived from documents attributed to this seed via parent_web_source_ids (multi-attribution, §F-5). Independent of crawler reachability - these are always available even when the crawler is unreachable.

Hide child attributes

metrics.pages_total

integer

Required range: x >= 0

metrics.pages_indexed

integer

Required range: x >= 0

metrics.pages_failed

integer

Required range: x >= 0

metrics.pages_pending

integer

Required range: x >= 0

metrics.pages_skipped

integer

Reserved. Currently always 0 (no per-seed "skipped" status).

Required range: x >= 0

metrics.last_run_status

enum<string> | null

Reserved. Currently always null (no per-seed run ledger).

Available options:

running,

completed,

failed,

partial,

null

metrics.last_run_started_at

integer | null

Reserved. Currently always null.

metrics.last_run_completed_at

integer | null

Reserved. Currently always null.

updated_at

integer

List crawl seeds Retrieve a crawl seed (config + metrics + last/next run)

⌘I

Introduction

Platform API

Agent Creator API

LLM Gateway API

Governance API

Knowledges API

Create a recurring crawl seed

Authorizations

Path Parameters

Body

Response