Web
The Web source ingests pages from a website. Each page becomes a record in the
pipeline — its contents are uploaded to a new agent session for processing. The
pages_source field selects how URLs are discovered: from a sitemap, by
crawling links, or both.
Authentication
auth is optional. Omit it for public sites. If the target site requires
authentication, obtain the credential before you create the pipeline, then supply
it as a RemoteAuth credential whose resolved header is sent
with every request.
auth.type | Fields | Resulting header |
|---|---|---|
bearer | token | Authorization: Bearer <token> |
header | header, value | The value sent verbatim under the named header |
oauth_client_credentials | Client credentials and token endpoint | A fetched Authorization: Bearer <token> |
The credential to use depends entirely on the target site; there is no single provider page to obtain it from.
Configuration
Politeness, limits, and authentication are configured on the source; the
pages_source object selects and configures the discovery mode.
SOURCE FIELD (WEB, CRAWL MODE)
Code example with json syntax.1
Fields
| Field | Required | Description |
|---|---|---|
type | Yes | web. |
pages_source | Yes | How URLs are discovered. See Page discovery. |
requests_per_second | No | Maximum sustained requests per second, applied to each concurrent fetch independently. Between 0.1 and 50. Defaults to 2.0. |
max_concurrent_fetches | No | Maximum pages fetched concurrently per host. Between 1 and 4. Defaults to 2. |
max_concurrent | No | Deprecated and ignored. Set max_concurrent_fetches instead. |
max_pages | No | Maximum pages fetched per run. Between 1 and 1000000. Defaults to 10000. Enforced as a hard cap in sitemap mode; in crawl mode it is a soft hint. |
js_rendering | No | Render pages with a headless browser (slower). Defaults to false. Turn on if pages return empty content without rendering. |
user_agent | No | User-Agent header sent with every request. Defaults to Vectara-Crawler/1.0 (+https://www.vectara.com/crawler). |
max_page_bytes | No | Maximum page body bytes to download. Oversized pages are not truncated: they are dropped during discovery, or the fetch fails when the response is too large. Between 65536 and 104857600. Defaults to 10485760. |
auth | No | Authentication for the website. Omit for public sites. See Authentication. |
Page discovery
The pages_source object is discriminated by its type.
Sitemap (sitemap)
Read pages enumerated by one or more sitemaps (sitemap.xml,
sitemap-index.xml, or gzipped variants). No link-following.
| Field | Required | Description |
|---|---|---|
type | Yes | sitemap. |
sitemap_urls | Yes | Sitemap URLs to read. Between 1 and 100. |
Crawl (crawl)
Discover pages by breadth-first link-following from one or more seed URLs.
| Field | Required | Description |
|---|---|---|
type | Yes | crawl. |
urls | Yes | Seed URLs for the crawl. Between 1 and 1000. |
max_depth | No | Maximum link-following depth from each seed URL. Between 0 and 10. Defaults to 3. |
same_domain_only | No | If true, links outside the seed URL's registered domain are not followed. Defaults to true. |
pos_regex | No | Regex patterns matched against the full URL; a URL must fully match at least one to be crawled. Empty list means no positive filter. |
neg_regex | No | Regex patterns matched against the full URL; URLs fully matching any are skipped. |
Sitemap and crawl (sitemap_crawl)
Read pages from sitemaps and crawl outward from their entries. Sitemap entries serve as the initial crawl seeds.
| Field | Required | Description |
|---|---|---|
type | Yes | sitemap_crawl. |
sitemap_urls | Yes | Sitemap URLs to read; their entries become the initial crawl seeds. Between 1 and 100. |
urls | No | Additional explicit seed URLs. |
max_depth, same_domain_only, pos_regex, neg_regex | No | The crawl controls described under Crawl. |
How records are fetched
Each URL that passes the discovery rules becomes one record.
Pages are downloaded before records are processed: at most
max_concurrent_fetches pages are fetched concurrently from a host, and each
concurrent fetch is rate-limited by requests_per_second independently — the
worst-case rate against a host is max_concurrent_fetches × requests_per_second. Processing a record does not contact the host again.
Raise these values only for sites you are authorized to crawl at that
intensity, such as your own domain.
A page that was fetched but could not be staged falls back to a live fetch at
processing time. This happens when the body exceeds the content cache's ceiling,
when the staging write fails, and when the staged copy is gone by the time the
record is processed — through cache TTL or eviction. The fallback fetch
re-checks robots.txt and passes through the same host rate limit, and the body
is streamed under max_page_bytes rather than buffered in memory.
Pages that fail during the download pass are dropped and produce no record at
all. A page is dropped on a non-2xx response, on a read error, when a redirect
leaves the registered domain while same_domain_only is set, and when a
redirect lands on a URL that robots.txt disallows.
When js_rendering is true, pages are rendered with a headless browser. This
requires the web-renderer service; if it is unavailable, rendering fails closed
rather than falling back to an unrendered fetch.
Source metadata
Each record carries source metadata that the connector resolves at fetch time.
system_metadata depends on the discovery mode:
| Key | Description |
|---|---|
lastmod | The sitemap's <lastmod> value for the page. Populated in sitemap mode. |
etag | The page's ETag response header. Populated in crawl mode. |
last_modified | The page's Last-Modified response header. Populated in crawl mode. |
user_metadata is empty for Web.
acl_metadata is not populated by the Web source. The
ACL metadata buckets are left
null.
Incremental sync
When sync_mode is incremental (the default), the pipeline tracks a watermark
per run. See Sync mode.
In sitemap mode, per-URL change detection uses the sitemap's <lastmod>, so
only pages whose <lastmod> advanced since the last successful run are
reprocessed.
In crawl mode, pages do not carry a per-URL change signal, so the crawl runs each time and refetches the pages it discovers.
Deletes are not propagated. A page the connector can no longer retrieve (for example, one removed from the sitemap) stops appearing in new runs. The pipeline emits no delete signal, so anything a previous run already produced downstream is left in place.