PushUploader

PushUploader is the SDK's client for the Glean Indexing API. Connector base classes use it internally, so most connectors never touch it directly. You reach for it when you need to push data outside the fetch → transform → upload lifecycle: event-driven updates, one-off backfills, or targeted deletes.

from glean.indexing.push import PushUploader

uploader = PushUploader(datasource="company_wiki")
uploader.bulk_index_documents(documents)

Credentials come from GLEAN_SERVER_URL and GLEAN_INDEXING_API_TOKEN, same as everywhere else in the SDK.

Constructor

Argument	Default	Purpose
datasource	—	Datasource name sent with every call.
retries	None	Generated-client retry configuration.
server_url	None	Per-call server URL override.
timeout_ms	None	Per-call timeout override.
http_headers	None	Extra HTTP headers.
observability	None	A ConnectorObservability to record upload logs and metrics.
upload_max_workers	5	Concurrent middle-page uploads. Must be greater than zero.

Documents

Bulk replace

bulk_index_documents() replaces the datasource's documents. This is what a full crawl runs, and it is what deletes stale documents.

uploader.bulk_index_documents(
    documents,
    batch_size=1000,
    max_batch_bytes=5 * 1024 * 1024,
)

Argument	Default	Purpose
upload_id	generated	Groups batches into one upload session.
batch_size	1000	Maximum documents per batch.
max_batch_bytes	5 MiB	Maximum serialized bytes per batch. Set None to batch by count only.
force_restart_upload	None	Discards a previous incomplete session.
disable_stale_document_deletion_check	None	Forces synchronous stale deletion.

Batching applies both limits: a batch closes when it hits batch_size documents or max_batch_bytes serialized bytes, whichever comes first. That keeps one unusually large document from pushing a batch past the API's payload limit.

danger

bulk_index_documents() is a replacement. Documents absent from the call are deleted as stale. Never call it with a partial result set — see Indexing modes.

Incremental updates

To add or update documents without touching anything else, use index_documents():

uploader.index_documents(documents)

This is the right call for event-driven connectors reacting to a webhook.

Deleting

uploader.delete_document(object_type="article", doc_id="page_123")

Pre-supplied batches

If you're already producing batches — a streaming connector, or your own chunking — hand them over directly:

uploader.bulk_index_document_batches(batches, batch_count=len(batches))

Pass batch_count when you know it. The uploader uses it to mark the final page, which is what triggers stale-document cleanup.

Identities

Document ACLs only evaluate if Glean knows about the users and groups they reference. See Permissions.

uploader.bulk_index_users(users=users, batch_size=1000)
uploader.bulk_index_groups(groups=groups, batch_size=1000)
uploader.bulk_index_memberships(memberships=memberships, batch_size=1000)

Single-item and delete variants exist too: index_user(), index_group(), index_membership(), delete_user(), delete_group(), delete_membership().

info

Push groups and memberships together. A group with no memberships produces ACLs that can never match anyone. The connector base classes raise InconsistentDataError when get_identities() returns groups without memberships.

Employees

For people data, separate from document ACLs:

uploader.bulk_index_employees(employees=employees, batch_size=1000)

Parallelism

bulk_index_document_batches() uploads middle pages concurrently via a thread pool sized by upload_max_workers (default 5). The first and last pages are always sequential — the first opens the upload session, the last closes it and triggers stale deletion, so neither can race.

uploader = PushUploader(datasource="company_wiki", upload_max_workers=10)

Raise it for many small batches over a high-latency link; lower it to 1 to make an upload fully sequential when debugging.

warning

Parallel uploads are on by default. If you wire up InMemoryMetricsProvider for local testing, be aware it has a data race under concurrent uploads and can undercount. The default NoOpMetricsProvider and the cloud providers are unaffected. Tracked in issue #107.

Parameter naming

Document methods take disable_stale_document_deletion_check; user, group, membership, and employee methods take disable_stale_data_deletion_check. The behavior is analogous. Tracked in issue #116.