Web
The Web source ingests pages from a website. Each page becomes a record in the
pipeline — its contents are uploaded to a new agent session for processing. The
pages_source field selects how URLs are discovered: from a sitemap, by
crawling links, or both.
Authentication
auth is optional. Omit it for public sites. If the target site requires
authentication, obtain the credential before you create the pipeline, then supply
it as a RemoteAuth credential whose resolved header is sent
with every request.
auth.type | Fields | Resulting header |
|---|---|---|
bearer | token | Authorization: Bearer <token> |
header | header, value | The value sent verbatim under the named header |
oauth_client_credentials | Client credentials and token endpoint | A fetched Authorization: Bearer <token> |
The credential to use depends entirely on the target site; there is no single provider page to obtain it from.
Configuration
Politeness, limits, and authentication are configured on the source; the
pages_source object selects and configures the discovery mode.
SOURCE FIELD (WEB, CRAWL MODE)
Code example with json syntax.1
Fields
| Field | Required | Description |
|---|---|---|
type | Yes | web. |
pages_source | Yes | How URLs are discovered. See Page discovery. |
requests_per_second | No | Maximum sustained requests per second per host. Between 0.1 and 20. Defaults to 2.0. |
max_pages | No | Maximum pages fetched per run. Between 1 and 1000000. Defaults to 10000. Enforced as a hard cap in sitemap mode; in crawl mode it is a soft hint. |
js_rendering | No | Render pages with a headless browser (slower). Defaults to false. Turn on if pages return empty content without rendering. |
user_agent | No | User-Agent header sent with every request. Defaults to Vectara-Crawler/1.0 (+https://www.vectara.com/crawler). |
max_page_bytes | No | Maximum page body bytes to download. Oversized pages are not truncated: they are dropped during discovery, or the fetch fails when the response is too large. Between 65536 and 104857600. Defaults to 10485760. |
auth | No | Authentication for the website. Omit for public sites. See Authentication. |
Page discovery
The pages_source object is discriminated by its type.
Sitemap (sitemap)
Read pages enumerated by one or more sitemaps (sitemap.xml,
sitemap-index.xml, or gzipped variants). No link-following.
| Field | Required | Description |
|---|---|---|
type | Yes | sitemap. |
sitemap_urls | Yes | Sitemap URLs to read. Between 1 and 100. |
Crawl (crawl)
Discover pages by breadth-first link-following from one or more seed URLs.
| Field | Required | Description |
|---|---|---|
type | Yes | crawl. |
urls | Yes | Seed URLs for the crawl. Between 1 and 1000. |
max_depth | No | Maximum link-following depth from each seed URL. Between 0 and 10. Defaults to 3. |
same_domain_only | No | If true, links outside the seed URL's registered domain are not followed. Defaults to true. |
pos_regex | No | Regex patterns matched against the full URL; a URL must fully match at least one to be crawled. Empty list means no positive filter. |
neg_regex | No | Regex patterns matched against the full URL; URLs fully matching any are skipped. |
Sitemap and crawl (sitemap_crawl)
Read pages from sitemaps and crawl outward from their entries. Sitemap entries serve as the initial crawl seeds.
| Field | Required | Description |
|---|---|---|
type | Yes | sitemap_crawl. |
sitemap_urls | Yes | Sitemap URLs to read; their entries become the initial crawl seeds. Between 1 and 100. |
urls | No | Additional explicit seed URLs. |
max_depth, same_domain_only, pos_regex, neg_regex | No | The crawl controls described under Crawl. |
How records are fetched
Each URL that passes the discovery rules becomes one record.
Requests are rate-limited per host by requests_per_second.
When js_rendering is true, pages are rendered with a headless browser. This
requires the web-renderer service; if it is unavailable, rendering fails closed
rather than falling back to an unrendered fetch.
Source metadata
Each record carries source metadata that the connector resolves at fetch time.
system_metadata depends on the discovery mode:
| Key | Description |
|---|---|
lastmod | The sitemap's <lastmod> value for the page. Populated in sitemap mode. |
etag | The page's ETag response header. Populated in crawl mode. |
last_modified | The page's Last-Modified response header. Populated in crawl mode. |
user_metadata is empty for Web.
acl_metadata is not populated by the Web source. The
ACL metadata buckets are left
null.
Incremental sync
When sync_mode is incremental (the default), the pipeline tracks a watermark
per run. See Sync mode.
In sitemap mode, per-URL change detection uses the sitemap's <lastmod>, so
only pages whose <lastmod> advanced since the last successful run are
reprocessed.
In crawl mode, pages do not carry a per-URL change signal, so the crawl runs each time and refetches the pages it discovers.
Deletes are not propagated. A page the connector can no longer retrieve (for example, one removed from the sitemap) stops appearing in new runs. The pipeline emits no delete signal, so anything a previous run already produced downstream is left in place.