Skip to main content
Version: 2.0

Create a corpus

POST 
/v2/corpora

Creates a corpus to store and manage your documents. A corpus is a container for documents and their associated metadata. You can set the corpus key, name, description, encoder, and filter attributes.

Corpus object​

The key property is required and uniquely identifies the corpus. The name property is optional and defaults to the value of key. The optional description property provides additional information about the corpus. You can choose a key that follows a naming convention of your choice. An identifiable key makes a corpus easier to manage and reference in your application.

Use the queries_are_answers and documents_are_questions boolean properties to treat queries or documents in the corpus as questions or answers. These settings affect the semantics of the encoder at query time and indexing time.

Add metadata as filter attributes​

Define metadata fields with the filter_attributes object when you create a corpus with this endpoint or the Vectara Console. The corpus then supports filtering on specific metadata attributes at the document level or the part level.

Filter attributes attach metadata to your data at the document (doc) or part level. You can use this metadata later in filter expressions to narrow the scope of your queries. A filter attribute must specify a unique name (up to 64 characters long) and a level of doc or part. At indexing time, the platform extracts metadata with this name and makes it available to filter expressions. Learn more

Doc and part filter levels​

The doc attribute applies to the entire document. Use this for metadata that is consistent across the whole document, such as author, publication date, and document ID.

The part attribute applies to specific sections or chunks within a document. Use this for metadata that varies within different parts of the document, such as sections, page numbers, and sentiment scores.

If indexed is true, the platform builds an index on the extracted values. This index improves the performance of filter expressions that use the attribute.

Filter attributes must specify a type. The platform validates the type when documents are indexed. The four supported types are:

  • integer: stores signed whole-number values up to eight bytes in length.
  • real: stores floating point values in [IEEE 754 8-byte format].
  • text: stores textual strings in [UTF-8 encoding].
  • boolean: stores true/false values.

After you define filter attributes, you can use them within your queries. For example:

  • Document-level attribute: doc.publication_year > 2020
  • Part-level attribute: part.sentiment_score > 0.7

Custom dimensions​

Custom dimensions add user-defined values to your data, in addition to what the platform automatically extracts and stores from the text. For example, upvotes can be a custom dimension. For an example, see Add custom dimensions to boost content.

Request​

Responses​

The response includes a unique id that you use to reference the corpus. The name does not need to be unique within an account.

Last updated on by Ayesha Saddiqua