Enterprise Search with Azure Cognitive Search – A Beginner’s Guide

Azure Cognitive Search is a PaaS search engine, which serves as a solid and great foundation for search and search-based applications. It offers a high degree of flexibility in configuring ranking and scoring models, linguistics, as well as query and content processing. Due to this flexibility, it is often used as a search engine for enterprise search and here we will briefly explain how to get started.

Azure Cognitive Search as a Foundation for an Enterprise Search

Digital workplaces offer their embedded and built-in search engines, such as Microsoft Search as the search experience for Microsoft 365. If such search experiences do not satisfy your business requirements, there are good reasons to choose an external search engine.

Such an external search engine can be Azure Cognitive Search. Azure Cognitive Search is a scalable search engine and has been offered as platform as a service for a decade already. It generally runs in failover setups and is capable of indexing millions of documents, in particular for enterprise search scenarios. However, as it is just the search engine as a core, you need to integrate some components around it.

An enterprise search based on Azure Cognitive Search comprises the following components:

1. Displaying search results. A search interface, which displays the search results, offers verticals, filters, result blocks, a type-ahead and more. This is also where in enterprise search scenarios users need to authenticate against.
2. Indexing data. Connectors are responsible for crawling documents and data from the content sources, which should be indexed.
3. Security trimming and secure search. A security token store, such as the Raytion Custom Security Manager, is needed to implement secure search or security trimming, respectively.

How do you Index Content Sources?

It is quite easy to integrate third-party sources, such as Atlassian Confluence, ServiceNow or OpenText Content Server, in order to make Azure Cognitive Search a fully featured enterprise search. Azure Cognitive Search comes with a few built-in connectors.  Through its APIs it also easily offers to index more content sources. Thus, you can implement your own crawler or data connector, rely on open-source connectors or on commercial ones, such as our Raytion Enterprise Search Connectors.

In order to integrate new content sources into Cognitive Search, you need to set up and configure the following components:

1. Within Azure Cognitive Search you need to create a search index with its schema.
2. Unless the security models of the content sources can be fully built on Azure AD, you need to have a security token store up and running, e.g., the Raytion Custom Security Manager.
3. You need to deploy and configure the connectors, which index the content sources.

Search Experience and User Journeys

Having the data of the content source indexed is the first step towards a great enterprise search. But this does not yet display the data to your users. Therefore, you need to have a search experience or a search interface at hand.

In the context of Azure Cognitive Search, the search interface has multiple responsibilities. In enterprise search scenarios it needs to make sure that only authenticated users are able to search. So it needs to support authentication providers, such as Azure AD, Google Cloud Identity, Okta and even Active Directory or similar.

The search interface has to offer a query pipeline. Within this query pipeline the token store will be queried with the user ID of the searching user. Based on the response the original search query is transformed so that it adds an ACL filter (access control list filter), which is needed for secure search (i.e., security trimming). Furthermore, you can integrate synonym expansions, natural language understanding, ranking hints and more within the query pipeline.

After communicating with Azure Cognitive Search, the search interface needs to render the search results, dispay filters and more. A common aim is that users can quickly grasp the information displayed or to further drill down into the data. This way, the search solution helps users solving their task at hand.

We have customers, who implement the search interface either on their own or with our help. There are open-source frameworks available, which can be used as a starting point for your search interface implementation. Alternatively, our commercial framework Raytion Search & Retrieval Interface is a compatible turnkey solution for Azure Cognitive Search.

The Outcome

Azure Cognitive Search is a solid and PaaS foundation for a great enterprise search experience. It offers a high degree of flexibility when it comes to customizing the query processing, content processing and search experience. But you need to build or purchase some components around this great search engine, such as the search interface, connectors, and security trimmer. Our customers use the search engine in their enterprise search scenarios and for site search scenarios.

Azure Cognitive Search comes with a service cost. However, in turn, you do not need to operate the search engine itself and take care of updates or patches, which is a huge benefit in particular for large search installations.

If you are interested in more information on how to build an enterprise search based on Azure Cognitive Search, then please reach out to our experts.