API referenceKnowledge BaseCrawl jobs

Create crawl job

Create a crawl job to crawl the given URL with specified depth and page limits.

Headers

xi-api-keystringOptional

Request

This endpoint expects an object.
urlstringRequired
URL to a page of documentation that the agent will have access to in order to interact with users.
max_depthintegerOptional1-5Defaults to 3

Maximum depth for crawling (1-5), defaults to 3.

max_pagesintegerOptional1-10000Defaults to 1000

Maximum number of pages to crawl (1-10,000), defaults to 1000.

patternstringOptional<=2048 characters
If set, only URLs that match this pattern are included.
sitemap_urlslist of stringsOptional

List of URLs to crawl from sitemap (optional, overrides automatic URL discovery).

parent_folder_idstringOptional
If set, the created document or folder will be placed inside the given folder.
enable_auto_syncbooleanOptionalDefaults to false

Whether to enable auto-sync for this URL document.

auto_removebooleanOptionalDefaults to false

Whether to automatically remove the document if the URL becomes unavailable. Only applicable when auto-sync is enabled.

minimum_frequency_daysintegerOptional1-180

Minimum frequency (in days) at which the underlying eligible documents are refreshed. The actual interval may be shorter, never longer. Defaults to 7, tightened to the parent folder’s frequency if that is stricter. Only applicable when auto-sync is enabled.

Response

Successful Response
idstring
typeenumDefaults to discovery
root_folder_idstring
statusstring
created_atinteger
folder_pathlist of objectsOptional
The folder path segments leading to the root folder, from root to parent folder.

Errors

422
Crawl Jobs Create Request Unprocessable Entity Error