A COVID-19 bot tale starring Azure Bot Service and Cognitive Services

April 2020. The corona virus was spreading across the globe. And so was fake news and overwhelming amounts of information available on the internet that was making it difficult for people to distinguish fact from fiction. The World Health Organization (WHO) had just declared a “massive infodemic
“, citing an overabundance of reported information, some inaccurate, about the virus that made it “hard for people to find trustworthy sources and reliable guidance” when they needed it. If there was something more disturbing than the virus itself, it was the perception of the virus.

Context is key in such trying times. With all the social media hysteria – the risk of losing this essential context was real. People, innocently, fed on social media forwards and these laced their daily conversations, from the dinner table at home to the virtual tea breaks at work. 5G Mobile networks do not spread COVID-19! It was evident that to fight the pandemic, we would need to fight the infodemic.

My team and I were transitioning to a remote work culture. As we settled into our home workspaces, we brainstormed the possibilities of a solution. The problem statement was clear. People had questions, lots of them. The media and social channels, they relied on, capitalized on their emotional churn. What people really needed were quick, accurate, authentic answers to their queries- to address their fears and anxieties and help them bust myths in their fight against the virus. We looked at the options that were available. We wanted to build a system that spared the user the hassle of reading through long COVID19 FAQs on websites and navigating between predetermined commands on menu (read IVR) styled chatbots. We envisioned a faster and easier way for the user to get reliable guidance via an intuitive interface. Hence, we decided on using the most natural mode of communication ever known to mankind – conversation.

Leveraging the guidelines, tools and services available from Microsoft Bot Framework and Azure Bot Service we planned, designed, built, tested and deployed CoVA – A Virtual Assistant to help with COVID19 queries. This is how we did it.

Video walkthrough of CoVA [Credits: Vaishakh Krishan]

QnA Maker: Building the Knowledgebase

The North Star for CoVA was to ensure that we collated data only from credible sources like government websites e.g.: WHO, CDC and government sources. We used QnAMaker – a cloud based API service that helps you answer user’s questions, based on the data in the knowledgebase (KB). As first step, we curated data from the FAQ pages, site contents, documents, manuals etc. to create our own knowledgebase (KB). We added the pre-vetted content to the knowledge base as QnA pairs.

QnAMaker supported Markdown and provided a complete authoring experience to manage the content using an easy to use interface. Therefore, in addition to plain text, CoVA was able to render rich UI elements – Video, Image and Hyperlinks in its responses through Adaptive Cards. This has become easier with the recent BUILD 2020 updates that introduced a Rich Text Editor in QnA Maker.

In contrast to Rule Based chatbots that strictly adhered to a pattern and presented a menu of options, we wanted CoVA to entertain open ended, natural language queries. QnAMaker can easily create a conversational layer over your data.  This enabled the users of CoVA to interact with QnA Maker using natural language input. That said, Natural Language Processing (NLP) is a tricky space when there are multiple variations of the same user utterance. There are several ways the same question can be phrased. To improve the likelihood of answering the user query, we added Alternate Questions that pointed to the same Intent but accounted for variations in user utterances.  We also increased the confidence score from the default 0 to 3, in addition to other best practices, to better ensure the appropriateness of the answers.

Figure: Alternate questions configured in QnA Maker for the same answer

The domain itself posed additional challenges for CoVA. Government restrictions and guidelines for COVID19, helpline numbers etc. varied across the globe, from country to country. To facilitate such customized responses, CoVA needed to be routed appropriately to multiple KBs.

When designing CoVA, we applied a lot of thought around its persona. CoVA’s personality, we believed, would be a great driver to user engagement and an influencer on the user experience. Luckily, QnAMaker provided 5 different personality types out of the box – Professional, Friendly, Witty, Caring and Enthusiastic. We reviewed the datasets and found that each personality covered ~100 chitchat scenarios. We thought CoVA’s persona identified best with Caring and hence we set it as the base tone but tweaked several responses based on our needs.

Figure: Chitchat QnA for Caring personality

Once the KB was populated, the Test panel within QnA served the purpose of verifying the correctness of the responses. Post several successful test runs, we published the knowledgebase – thereby creating an API endpoint that could be invoked by the chatbot client. The KB was Live.

Microsoft Bot Framework and Azure Bot Service: Giving the bot a name, face and address

Now that we had the KB published from QnA Maker, we needed to create a bot that integrated with it. There were different ways we could do this but the easiest was to hit the “Create Bot” button right from the Publish page in the QnA Maker portal. This redirected us to the Azure portal.  The new Web App bot creation pane was prefilled with the attributes for QnA Auth Key, regions etc. In a few minutes from clicking “Create”, we had a web app bot deployed to an Azure App Service.

Figure: Creating a bot connected to QnA KB

Azure Bot Service also provides the ability to Test in Web Chat and further download the bot source code to work on it locally. Hence, we setup the source repository and continued our development locally in VS IDE. Continous deployment was setup to enable automatic code deployments to Azure. Bot Framework Emulator was used to test CoVA – both when run locally on our machine and when running remotely from Azure, via tunneling. This way, we used Microsoft Bot Framework v4.0 to encapsulate the chatbot logic and when hosted in Azure, it exposed an API that could be consumed by external applications or channels.

For CoVA, the primary channel was Web Chat. To integrate the Web Chat to the site, we configured the Direct Line channel, obtained the Direct Line secret from the Azure Portal and used it to generate a token that we passed to the Web Chat. We also customized the Web Chat control styling via styleOptions.

Figure: Integrating webchat control in website

One of my favorite things about the Microsoft Bot Framework is the ability to create a single code base for the bot and yet host it on multiple channels. In Azure, we configured Teams and Facebook in addition to Web Chat, in Channels under Bot Management. Additionally, we setup the Direct Line Speech channel that enabled users to provide audio input by button press of the microphone icon in the Web chat.

Figure: CoVA hosted in Teams

 Figure: CoVA hosted in Web Chat

Cognitive Services: Adding intelligence to the bot

Intelligent chatbots are cognitive. They can see, hear, speak and reason. Microsoft Cognitive Services is an ever-growing rich suite of APIs that help developers leverage powerful AI algorithms to add such intelligence to the apps we build. We used some of these to enhance CoVA’s capabilities.

One of the challenges for CoVA was to improve its ability to respond to maximum user queries, without compromising on the quality of the results. Though we had connected to static online data sources, we deployed Automation scripts that detected updates in these sources and reflected these in our knowledgebase by invoking QnA APIs. But despite our best efforts to keep the QnA KB updated, there seemed no respite from the inevitable “Sorry, I couldn’t answer that” response. Keeping up with the growing and changing guidelines and FAQs from government sources was a challenge. In our quest to reduce the number of failed responses from the bot and increase the probability of accurate responses, we banked on Bing Custom Search. Unlike Bing Web Search API that allows you to search over the entire web, Bing Custom Search allows you to search over your targeted web space. We created an instance of this ad free, commercial grade service and preconfigured the websites we wanted to include in it. It perfectly suited our purpose of delivering only relevant results from a subset of the web, our trusted sites. Now, CoVA had a contingency plan. If user queries could not be answered based off the data in QnA maker, plan B would be to fire up Bing Custom Search.

Figure: Bing Custom search invoked when QnA Maker yields no result

So, at this point, a user could engage with CoVA and more so using free form text. QnA Maker not only helped to build a personalized FAQ service but also helped respond to user’s questions in a natural conversational way. But we needed more help. In a practical scenario, the user’s text would be far cry from the perfectly crafted query that’s grammatically sound, spell error free and perfectly punctuated. Moreover, as we paced through the implementation, we were apprehensive of opening the portal to an audience with no gated checks. Content Moderation APIs came to our rescue.  It’s content filtering capabilities helped detect potential profanity in language, it flagged text that it deemed inappropriate and checked for personally identifiable information (PII). Also, thanks to its Autocorrect feature that enabled real time corrections, CoVA was far more forgiving of spelling mistakes in user utterances.

Figure: Text Moderation APIs screening user utterances

Bot Analytics and Power BI: Providing Insights

Intelligent bots are also teachable. They can be trained to perform better. We provisioned AppInsights and Bot Analytics, added custom telemetry and connected it to a Power BI dashboard to report on conversation metrics, chat transcripts, the traffic per channel, queries that the bot failed to respond to etc.

Figure: Bot Analytics shown in PowerBI report

Putting it all together

This is what the final architecture of CoVA looks like. Futuristically, we could move AppInsights data to a DataLake to mitigate its limited retention period, integrate with LUIS and Translator APIs to make CoVA more powerful etc.

Figure: Architecture of CoVA

Yet the journey thus far was exhilarating and gratifying, not only for having worked on a socially impactful solution but also for the learnings we gathered in the process. Most of all, it was reassuring to know that all hope is not lost. Even to fight such an unprecedented and exceptional enemy, we have never had more powerful tools. Information is Power. Stay Informed. Stay safe.

One thought on “A COVID-19 bot tale starring Azure Bot Service and Cognitive Services

Add yours

  1. Wow that was unusual. I just wrote an very long comment but after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say superb blog!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: