Troubleshoot the Safetynet

Teneo Query Language is a versatile tool that can help you maintain and improve your system. In this section, we'll walk through some strategies for exposing weak points in a bot's repertoire

View inputs to the Safetynet

The query to view user inputs that hit the Safetynet could look like this:

lu t.e1.userInput: t.e2.fname ~= ".*[s|S]afetynet.*"

This query will print all unique user inputs for which the name of the subsequently triggered flow contains the string 'Safetynet'. This is the default name for the Safetynet in Teneo solutions using the Teneo Dialogue Resources. If you're using a Safetynet with a different name, be sure to adjust the query accordingly.

The results will typically include two types of input:

1. Questions not yet in the solution's domain. In this case, you can decide whether to add new content to the solution to include these questions.
2. Variants of questions that the solution should answer, but that were not assigned to the correct intent. In this case of the solution, you can adjust your training data in the relevant flows, or expand the condition to recognize the same sentence.

!f you want to view inputs to other flows, you can rewrite the query above using the exact name of the particular flow: lu t.e1.userInput: t.e2.fname == "Name of your flow here"

View flows preceding the Safetynet

If a user input hits the Safetynet, you may want to go one step further and zoom out to get the bigger picture of what happened. A useful query for this is to list the flow and output that preceded the Safetynet input. Maybe the bot's answer was unclear, causing the user to ask a follow-up question for clarification. Or perhaps users did not like the answer they got and reacted with a complaint. In any case, these inputs can help to point out parts of your solution where you may want to take action, either by improving the bot's response or perhaps by adding a follow-up response in an individual flow.

A query that gives you this information can look like this:

lu t1.id as "Session ID", t1.e1.fname as "Preceding Flow", t1.e2.answerText as "Preceding Bot Response", t2.e1.userInput as "Unrecognized User Input" :
    t1.index == t2.index - 1,
    t1.e1.fname !~ ".*[s|S]afetynet.*",
    t2.e2.fname ~= ".*[s|S]afetynet.*"

Let's break down what this query does.

1. The command and selection lu t1.id as "Session ID", t1.e1.fname as "Preceding Flow", t1.e2.answerText as "Preceding Bot Response", t2.e1.userInput as "Unrecognized User Input" tells TQL to generate a report with the following information: transaction ID's, flow names, and responses for all transactions that we found. The header column should include the labels as given, e.g. "Preceding Flow", etc.
2. The constraint t1.index == t2.index - 1 tells TQL to only include pairs of transactions that are immediately consecutive. T1, the first transaction, should happen immediately before T2, the second transaction. We can force this by referencing the indices of the two transactions. The index of T1 should be 1 less than the index of T2.
3. The constraint t1.e.fname !~ ".[s|S]afetynet." is optional. It specifies that we want to ignore all transaction pairs in which the Safetynet response itself preceded a second response of the Safetynet; we chose to omit these transactions as they could just be noise, but if you want to see them anyway, you can leave the constraint out of the query.
4. The final constraint t2.e2.fname ~= ".[s|S]afetynet." makes sure that the second transaction is one in which the Safetynet was triggered.

After running the query your results will look like something like this:

A note on events

You may have noticed that we refer to items from different events in the query above.

In the first transaction we look at:

• t1.e1.fname
• t1.e2.answerText

In the second transaction we look at:

• t2.e1.userInput
• t2.e2.fname

There is an important reason for this. Consider the first case, where we are querying t1: here, we are first accessing the raised flow (with fname) and then the response (answerText). These two are from the same transaction, but different events. The same goes for the second case, where we query the user input, which is associated with the request event, followed by the name of the triggered flow, which is recorded in one of the later path events. If you are writing a query that you know should get results, and it returns nothing, make sure that you are referring to the correct events.