Custom Properties for Datasources

In this document, we discuss how to define custom properties for a datasource and how to upload them with each document.

What are custom properties?

Fundamentally, a custom property is any additional information that is attached to a document. These properties can be used in 3 different ways:

The above 3 use-cases are explained in respective sections below. But first, we explain how you can set up and upload custom properties for your datasource.

Note

By default, no further action is needed after configuring and uploading custom properties to use them as facets/operators and to use them for improving searchability. Rendering requires a few additional steps (described below).

Setting up custom properties

Important

It is highly recommended to configure custom properties first and then upload documents with the custom properties. Expect delays in indexing if it is done the other way round!

Before custom properties can be uploaded for documents, we must configure the datasource to accept the specific custom properties. You need to define them under an object definition.

The configuration can be done in 2 ways:

  1. Admin console ( recommended )
  2. API calls to /adddatasource

Using admin console

1. From the app setup page in the admin console, go to object definitions.

custom-properties-setup-page

2. Define your object (if you haven't already) and click on property definitions.

An object is any logical categorization of documents within your datasource (for example, documents/spreadsheets/slides can be objects for a drive datasource).

custom-properties-object-definition

3. Configure the properties in the property definition modal.

custom-properties-property-definitions

Using API Calls

Here is a sample API call for the above example:

API Samples
curlpython
Copy
Copied
curl -X POST https://customer-be.glean.com/api/index/v1/adddatasource  \
{
   "name": “testDatasource”
   "objectDefinitions": [
        {
            "name": "Account",
            "displayLabel": "Account",
            "propertyDefinitions": [
                {
                "name": "priority",
                "displayLabel": "Priority",
                "displayLabelPlural": "Priorities",
                "propertyType": "TEXT",
                "hideUiFacet": false
                }
            ]
        }
   ]
 ... 
}
Copy
Copied
from glean_indexing_api_client.api import datasources_api
from glean_indexing_api_client.model.custom_datasource_config import CustomDatasourceConfig
datasource_api = datasources_api.DatasourcesApi(api_client)

datasource_config = CustomDatasourceConfig(
    name="testDatasource",
    object_definitions=[
        ObjectDefinition(
            name="Account",
            display_label= "Account",
            property_definitions=[
                PropertyDefinition(
                    name="priority",
                    display_label="Priority",
                    display_label_plural="Priorities",
                    property_type="TEXT",
                    hide_ui_facet=False
                    ...
                )
            ],
        )
    ],
    ...
)
try:
  datasource_api.adddatasource_post(datasource_config)
except indexing_api.ApiException as e:
  print('Exception when calling DatasourcesApi->adddatasource_post: %s\\n' % e)

The above API calls define a document type Account for our datasource testDatasource. For Account we define an operator called priority. This operator will become a facet if we set hideUiFacet to false. Note: propertyTypes "DATE" and "INT" are not faceted.

Uploading custom properties

After you have defined the properties, you can set the properties values while uploading the document. A sample API call for /indexdocument(link) where we illustrate how this is done is:

API Samples
curlpython
Copy
Copied
curl -X POST https://customer-be.glean.com/api/index/v1/indexdocument \
{
   "document": [
        {
            "title": "Account 1",
            "id": "Account_1",
            "customProperties": [
                {
                    "name": "priority",
                    "value": "High",
                }
            ]
            ...
        }
    ... 
   ]
}
Copy
Copied
from glean_indexing_api_client.api import documents_api
from glean_indexing_api_client.model.index_document_request import IndexDocumentRequest
from glean_indexing_api_client.model.document_definition import DocumentDefinition
from glean_indexing_api_client.model.content_definition import ContentDefinition
from glean_indexing_api_client.model.user_reference_definition import (
    UserReferenceDefinition,
)
from glean_indexing_api_client.model.document_permissions_definition import (
    DocumentPermissionsDefinition,
)

request = IndexDocumentRequest(
    # DocumentDefinition has many fields, we show the usage of a few basic ones.
    document=DocumentDefinition(
        title='Account 1',
        id='Account_1',
        custom_properties=[
            CustomProperty(
                name='priority', 
                value='High'
            )
        ]
        ...
    )
    ...
)
documents_api = documents_api.DocumentsApi(api_client)
try:
    documents_api.indexdocument_post(request)
except indexing_api.ApiException as e:
    print("Exception when calling DocumentsApi->indexdocument_post: %s\n" % e)

In the above API call we set the value for the custom property Priority as High for the document Account_1.

Operators and Facets

What are operators?

operators

Operators are a powerful way to filter information in Glean. In the above example we use the operators app: and owner: with query Glean. This tells Glean to only look for documents which match “Glean” and belong to app:drive and have owner:”Steve Smith”. This allows you to find the relevant information faster.

What are Facets?

facets

Facets are special kinds of operators which are shown as visual filters in the Glean UI. You can find them on the right hand side of the search result page. This removes the burden of remembering the operators and gives you a nice and interactive way of filtering results.

Default Facets

Glean also has a default faceting behavior. Facets are automatically created of the same name for the following document fields if they are set for a document:

  1. author
  2. owner
  3. tags
  4. updatedAt
  5. objectType

Example

Once you follow the steps above in defining custom properties and uploading them for a document, you should be able to use the priority operator and see it as a facet!

Note

If you wish to hide the facet from showing up in the UI, you may select the 'Hide UI Facet' option in the admin console for the property (or) set hideUiFacet to true in the API call.

Rendering

Apart from the above mentioned steps for configuring and uploading custom properties, if you would like to show custom properties in the search result rendering - the following 2 steps are required:

  1. In the UI options for the custom property, select Search Result (screenshot below) (or) set uiOptions for the custom property in the API call to SEARCH_RESULT .

custom-properties-ui-option

  1. Contact Glean support to show the custom property to the search result as per your preference. An example is shown below.

rendering-custom-result

For more details on rendering of search result refer here

Warning

If you have already uploaded documents with a custom property, before setting the UI options for the property to include the Search result - please expect a delay (could be multiple hours, depending on the volume of documents) before the documents' properties are visible in the search results.

To propagate this update quicker - it is recommended to re-upload the documents (or) call the /processalldocuments endpoint for the datasource.

Searchability and Ranking

All custom properties are automatically used to enrich the document in the index - thereby improving searchability and ranking.

For example, if we have a ticketing datasource and include a ticket ID as a custom property - we can then search by the ticket ID to fetch the relevant ticket (without requiring it to be in the title or the body!).