SCIM

The SCIM source ingests user records from a SCIM 2.0 service provider through the Users endpoint. Each user becomes a record in the pipeline, rendered as one HTML profile document whose contents are uploaded to a new agent session for processing. Identity fields become document metadata, and group memberships plus enterprise attributes become filterable metadata. The source works against any SCIM 2.0 service provider, such as Okta or Microsoft Entra ID.

Authentication

auth is required. Before you create the pipeline, obtain a credential with read access to the provider's Users endpoint, then supply it as a RemoteAuth credential.

auth.type	Fields	Resulting header
bearer	token	Authorization: Bearer <token>
header	header, value	The value sent verbatim under the named header
oauth_client_credentials	client_id, client_secret, token_endpoint, and optional scopes	A fetched Authorization: Bearer <token>

Where to obtain the credential depends on the service provider. For a provider that issues long-lived SCIM API tokens, use bearer. For a provider that exchanges client credentials for access tokens, use oauth_client_credentials. Credential secrets are encrypted at rest and never returned in responses.

Configuration

SOURCE FIELD (SCIM)

Code example with json syntax.json

1

Fields

Field	Required	Description
type	Yes	scim.
base_url	Yes	Base URL of the SCIM 2.0 service provider, for example https://example.okta.com/scim/v2. The Users resource path is appended automatically and must not be included.
auth	Yes	Authentication for the service provider. See Authentication.
filter	No	SCIM filter expression applied to the user listing, restricting which users are ingested. Requires a service provider that supports the filter query parameter. When incremental_strategy is last_modified_filter, the change window is combined with this expression.
attributes	No	Attribute names to request from the service provider, passed as the SCIM attributes query parameter. The meta attribute is always requested so that incremental sync can read each user's meta.lastModified timestamp. When omitted, the service provider returns its default attribute set. Returned attributes are rendered into the user's profile document and also populate the record's source metadata, so narrowing this list narrows both what is indexed and what is available for filtering. See Source metadata. An attribute the service provider marks as returned only on request is included only when named here.
incremental_strategy	No	How incremental syncs detect changed users, either last_modified_filter or client_side. Defaults to last_modified_filter. See Incremental sync.

How records are fetched

Each run pages through the provider's Users endpoint. Each user becomes one record, and its content is an HTML profile document listing the user's name, email, title, user type, active state, enterprise attributes, group memberships, roles, and entitlements. A user whose resource carries no parseable meta.lastModified timestamp is ingested on every run rather than skipped, since without a timestamp its change state is unknowable.

Source metadata

Each record carries source metadata that the connector resolves at fetch time.

system_metadata:

Key	Description
user_name	The SCIM userName.
display_name	The user's display name.
external_id	The provider's external ID for the user.
email	The user's primary email address, or the first email when none is marked primary.
created_at	The resource's created timestamp in RFC 3339 format.
updated_at	The resource's last-modified timestamp in RFC 3339 format.
url	The resource's location URL.

user_metadata carries the classification attributes useful for attribute-based filtering. Absent attributes are skipped, and multi-valued attributes are joined into comma-separated strings:

Key	Description
active	Whether the user is active.
user_type	The SCIM userType.
title	The user's job title.
groups	Display names of the user's group memberships, or the group reference value for a membership without a display name.
group_ids	IDs of the user's group memberships.
roles	The user's roles.
entitlements	The user's entitlements.
employee_number	Enterprise extension employee number.
department	Enterprise extension department.
organization	Enterprise extension organization.
division	Enterprise extension division.
cost_center	Enterprise extension cost center.
manager	The manager's display name, or the manager reference value when no display name is set.

acl_metadata reports org-wide read access with the principal buckets left null. SCIM user resources describe identities, not per-principal document grants, and a user directory is an audience-level asset.

Incremental sync

When sync_mode is incremental, which is the default, the pipeline tracks a watermark based on each user's meta.lastModified timestamp. See Sync mode. On the next run, only users modified since the stored watermark are reprocessed.

incremental_strategy selects how the change window is applied:

• last_modified_filter, the default, pushes the change window into the SCIM filter query parameter as a meta.lastModified range, so only changed users are enumerated. This requires a service provider that supports filtering on meta.lastModified.
• client_side enumerates every user on each run and evaluates the change window locally. This works against any service provider. Use it when the service provider does not support filtering on meta.lastModified, which the SCIM 2.0 specification leaves optional.

The window is always also enforced client-side, so a provider that silently ignores an unsupported filter still yields a correct incremental run.

Deletes are not propagated. A user the connector can no longer retrieve stops appearing in new runs. The pipeline emits no delete signal, so anything a previous run already produced downstream, for example a document indexed into a corpus, is left in place.

Related

• Sources overview
• Pipeline concepts
• Session metadata