For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Create a crawl job to crawl the given URL with specified depth and page limits.
Nagłówki
xi-api-keystringOpcjonalny
Żądanie
This endpoint expects an object.
urlstringWymagany
URL to a page of documentation that the agent will have access to in order to interact with users.
max_pagesintegerOpcjonalny1-10000Domyślnie 1000
Maximum number of pages to crawl (1-10,000), defaults to 1000.
patternstringOpcjonalny<=2048 characters
If set, only URLs that match this pattern are included.
sitemap_urlslist of stringsOpcjonalny
List of URLs to crawl from sitemap (optional, overrides automatic URL discovery).
parent_folder_idstringOpcjonalny
If set, the created document or folder will be placed inside the given folder.
enable_auto_syncbooleanOpcjonalnyDomyślnie false
Whether to enable auto-sync for this URL document.
auto_removebooleanOpcjonalnyDomyślnie false
Whether to automatically remove the document if the URL becomes unavailable. Only applicable when auto-sync is enabled.
auto_discoverbooleanOpcjonalnyDomyślnie false
Automatically discover and add new pages linked from already-crawled pages during auto-sync. Requires enable_auto_sync=true.
minimum_frequency_daysintegerOpcjonalny1-180
Minimum frequency (in days) at which the underlying eligible documents are refreshed. The actual interval may be shorter, never longer. Defaults to 7, tightened to the parent folder’s frequency if that is stricter. Only applicable when auto-sync is enabled.
max_depthintegerOpcjonalnyDomyślnie 3Deprecated
Deprecated - this field is a no-op and will be removed in a future version.
Odpowiedź
Successful Response
idstring
typeenumDomyślnie discovery
Dozwolone wartości:
root_folder_idstring
statusstring
created_atinteger
folder_pathlist of objectsOpcjonalny
The folder path segments leading to the root folder, from root to parent folder.
Błędy
422
Crawl Jobs Create Request Unprocessable Entity Error
