Skip to main content

Connect your data

Choose how your content reaches a corpus. Three paths copy content in; a fourth leaves it in the source system and queries it at request time.

Pull from a source system on a schedule​

A pipeline fetches records from a source system on a trigger and hands each record to an agent in its own session. The agent decides what happens to the record: index it into a corpus, extract fields from it, or discard it. Sources include file stores such as Amazon S3, enterprise content systems such as SharePoint and Confluence, and the web.

A pipeline needs an agent that already knows how to process a record, so build the agent first.

See every supported data source and its configuration, or set up your first pipeline.

Upload files or index documents yourself​

Upload a file straight to a corpus and let Vectara extract and chunk it, or index documents you have already structured. This is the right path for one-off loads, for testing, and for ingestion you drive from your own code.

Add content to a corpus.

Crawl a source a pipeline does not cover yet​

vectara-ingest is an open-source Python crawler you run on your own infrastructure. It reaches far more systems than the first-class sources do, at the cost of operating it yourself, and it is provided without official support. Reach for it when no pipeline source fits.

Query a system instead of indexing it​

Box, Confluence, and Google Drive are available both as pipeline sources and as agent tools. A pipeline copies content into a corpus. Corpus-level access controls who can query it, but per-document permissions from the source system are not inherited automatically. A tool reaches the system at request time, so the source system stays the live point of access control.

Compare the two patterns before you choose.