When Halinka met Watson
A story about a tough love in the world of bots.
Halinka the Bot
As you probably figured it out, Halinka is a bot, a chatbot. It (or should I call her “She”?) supports our hiring process in automation of some common and boring tasks. And yes, we hire software programmers that’s why we hired the bot to handle the process smoothly ;-)
Who’s Watson
Watson is a very smart guy developed at IBM — right, not a guy but an AI, yet in Halinka’s world it’s a guy. Watson is a very broad term as it consists of many services that you can use in your application. Here I will focus on just one of them — Natural Language Processing, in short NLP, which is called Conversation.
… and the story goes
Halinka started as a Hubot clone and on the very beginning she was just like many other bots — only responding to a predefined set of commands. Not very clever. As more and more commands were added to the bot and more and more people at SoftwareMill were engaged in the hiring process, it was very hard to figure out how to use the bot, what commands are supported, and what the params should be.
So I had decided that we need to teach Halinka a better understanding of what are the people’s needs. The idea was simple: all uncaught commands will be send to an AI to be processed and matched with a predefined set of intents. Next, the AI will tell Halinka what the user wants.
A marriage with Watson was obvious but not at all easy.
First, Halinka was supporting only Polish language, because it’s our native language at SoftwareMill. Watson can understand only a few languages and Polish isn’t one of them. Choosing English was the only reasonable choice.
Second, I had to find a way to still allow use commands but catch unmatched commands/sentences and send them to Watson for further processing. Surprisingly, Hubot allows to define a custom listener to catch all the commands directed to him. Yet, the predefined commands must have a marker that the command was matched and processed.
Technicals
To resolve the first issue I had to translate all the commands into English, easy thing. However, it took the users some time to get used to the new commands’ language.
The second issue was also easy to resolve — Hubot has a built-in function to mark a message as handled, it looks like this:
where you must call .finish() to mark the message as handled by the bot. Another thing is a custom listener that will catch all the messages, check if a message was handled (marked with .finish()) and, if not, send the message to Watson for further processing. This is easily done by the following code:
The first callback is used to check if a message should be send to Watson and the second callback is called only when the first callback returned a non-false value. The returned value is a text that should be processed by Watson.
The smart guy
Having the whole setup on the Halinka’s end done, I was able to start training Watson how to handle given messages, how to understand a free form text and how to transform it into a command that can be used by Halinka.
First thing to do is to define your Intents. In a basic form, an intent is a command, an action you want to express in free form sentences. That will train Watson how to match the incoming message with a command you want to perform, see the screenshot below:
An intent consists of a name (followed by #) and a set of sentences which express an action you want to perform. You should provide at least 5 such sentences — it gives enough information for Watson to learn what you meant. In the future you will be able to add more to narrow Watson’s understanding and improve accuracy of given answers.
The next step is a Dialog — a way to match an intent with an action, see the screenshot:
As you see, on the left you select an intent and on the right you define a respond — super easy. This is the simplest form, the answer will be sent back to the bot and it can be a help message. You can also check accuracy of the matched intent, it’s a level of confidence which represents Watson’s understanding. If the confidence is too low (less than 50% as in the example above) you can answer with a common text that will help you identify intents that should be improved by adding more sentences. Or simply create a new intent. And a raw text is not the only option, you can use JSON to respond with more complicated answers.
You can also test your setup without connecting a bot. Watson has a test console built-in where you can post your sentences and see if Watson was able to recognise and match them with one of the defined intents.
You can dynamically pair existing intents with the sentence by selecting an intent from the dropdown. It is the easiest way to teach Watson and improve his accuracy.
A grand finale
Each and every relationship needs time to grow and become mature. Over the last two years I’ve been slowly improving Halinka and Watson by adding more sentences and more intents and introducing entities to seal the relationship. Halinka learned to use Slack Dialogs, Watson learned to understand Polish names and surnames. The process continues and their relationship is growing stronger day by day.
