Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore.

15 St Margarets, NY 10033
(+381) 11 123 4567



The 5 Step Journey to Becoming a Chatbot Tester | by Florian Treml | Apr, 2021

One question we hear a lot is “What’s the best way to start testing a chatbot ?” The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

Five Step Journey

Go the whole way from starting the project to publishing a chatbot on a preview website — everything you need is included in those cloud platforms.

I do not recommend to start with a free, self-hosted stack like Rasa, as it might distract from the journey to become a chatbot tester by tearing your focus more towards infrastructure topics and actual chatbot development.

  • Intents, Entities, Utterances
  • Accuracy, Sensitivity, Specificity, Precision, Recall, F1-Score
  • Confidence and Confidence Threshold
  • Confusion Matrix

Here are some links to introductory articles:

1. Chatbot Trends Report 2021

2. DO’s and 3 DON’Ts for Training a Chatbot NLP Model

3. Concierge Bot: Handle Multiple Chatbots from One Chat Screen

4. An expert system: Conversational AI Vs Chatbots

Wrote about this conflict in one of my previous blog posts.

So invest some days to educate yourself with BotiumScript, an easy-to-learn scripting language for describing chatbot test cases. You can start at the official Botium Docs or watch Shama Ugale in one of her awesome talks:

Testing Conversational AI by Shama Ugale

To get started in Botium Box, read through the Botium Box Wiki with tutorials, howtos, technical background and reference docs.

You can read more in one of my previous blog posts.

Because of this it is even more important to understand the required test levels to meet at least the minimum quality standards in software projects.

  • Regression Testing on API level— Identify flaws in the conversation flow before going to production
  • NLP Testing — Improve your chatbot understanding
  • E2E Testing — Verifying the end-user experience
  • Voice Testing — Understand your users on voice channels
  • Performance Testing — Ensure your chatbot is responsive under high load
  • Security Testing — Making your chatbot secure
  • Monitoring — Get notified when problems arise

Read about some important pitfulls here: 4 DO’s and 3 DON’Ts for Chatbot Testing Strategies

The good news is that you can shine and partially re-use your test automation skills gathered in your previous projects in chatbot testing as well.

Credit: Source link

Previous Next
Test Caption
Test Description goes like this