How I built an educational Google Assistant to help kids practice times tables
My youngest son is learning times tables at school so I started thinking of giving him a vocal assistant to help him practice.
I made a search within the educational Google Assistants but I didn’t find anything suitable for my purpose, so I decided to build a simple Google Assistant project to implement a fully-automated vocal tool to let users practice times tables.
Chatbot Design
As kids start learning times tables from the easiest ones and it normally takes them some time to become comfortable will all the tables, I define four types of exercise, which the user will be able to request the Assistant:
- Limit questions up to a given times table: for instance, if a kid only knows up to 5 times table, he can ask the bot to limit questions accordingly.
- Limit questions to one given times table only: this is useful when studying a new table for the first time, as the kid can request to be trained on that table only.
- Limit questions from a given times table onward (e.g. from 6 onwards): this exercise is helpful to focus on upper times tables when kids progress in their study.
- Ask questions about all times tables (up to 9) with no restriction, helpful when kids have completed the study and want to practice on all of them.
The bot asks the exercise type in the greeting message, and then it keeps asking questions accordingly; on top of that, the user can switch to another exercise type at any time during the dialogue.
Another capability I want to add to the Assistant is to keep track of right and wrong answers and to be able to provide a summary and a score whenever asked.
Let’s now see how I did it.
Google Actions Project
The place to start off is the Google Actions Console, where all begins with creating a new project. Be aware you may be asked to accept Google’s Terms of Service.
I give my project the name Tabelline which is the Italian for “Times Tables”. At this stage, I don’t care about making it multi-language but it turns out to be an easy exercise in Google Actions. At the time of creating the project, I also have to pick a language and a country/region.
The Actions Console walks you through the Action creation wizard, so at the next step, I select the “Custom” project and click the “Next” button.
Then I select to build the project Actions using the Dialogflow platform.
Using Dialogflow to build the Actions means creating a DialogFlow agent linked to the Actions project which takes care of understanding the user intents and providing corresponding adequate responses. You can imagine the Actions project as “wrapping” the Dialogflow agent: the former owns the engagement channel through the Google Assistant platform, while the latter implements all the conversational aspects of the project.
We’ll work on the DialogFlow agent just shortly, before that we are landing on the Overview page of your newly created Actions project. You will see several tabs at the top of the screen:
- Overview contains links and shortcuts to other screens belonging to other tabs.
- Develop focuses on the Actions and their execution.
- Test shows the Assistant in action while you are building it.
- Deploy manages the publication to Alpha, Beta and Production environments.
Clicking on Decide how your Action is invoked redirects you to the Invocation screen under the Develop tab, where you have to provide a Display Name to your Assistant: on my side, I call it Esercizi di Tabelline (“Times Tables Exercises”).
The Display Name determines how the user will open it, e.g. saying “talk to” followed by the Display Name. You can find more details on the trigger phrases for each language on this Google Docs page.
Still within the Develop tab, let’s move to the Actions screen and click either on the Add Action or on the Get Started button.
This opens up the Create Action dialog: I tap on the Build button and since we selected to build our actions using Dialogflow, I am redirected to the DialogFlow console (again, you may have to accept the Terms of Service to be allowed to proceed).
Dialogflow Project
Once landed in the Dialogflow console, I create the agent providing the name of my choice (I am using the same as in the Actions project); at this stage, I am also required to set language and timezone.
Clicking on the small Settings wheel at the top of the left-side navbar, you access the agent configuration, where you can select Export and Import and restore the entire Dialogflow project from my ZIP file available at this link: https://github.com/foppini975/Tabelline_GoogleAssistant/blob/main/DialogFlow/Tabelline.zip
Once restored, you can explore the project screens and start looking at the Intents list.
The chatbot is made of 9 intents:
- The Default Welcome Intent greets the user, requests to pick an exercise type and advises the user to request help if needed.
- The aiuto (help) intent captures help requests and responds with instructions to the user.
- The four intents richiestaTabellina, richiestaTabellinaDa, richiestaTabellinaFinoA and richiestaTutteTabelline correspond to the four available exercise types: the user can trigger any of them at any time during the dialog to determine the type of the following questions.
- The rispostaNumero (numeric response) intent captures the user’s attempts to respond to the bot’s question: I trained it with numeric answers only, but you may be more creative and add some more textual frills.
- The statistiche (statistics) intent can be used at any time by the user to ask for the summary and the score.
- The Default fallback Intent, which is matched when the bot doesn’t recognize the user’s expression. I left it in its default form.
I created a question context to keep track of the process of “asking a question and replying a number”, and assigned it to those intents which are related to the process of exercising (all except Welcome, Fallback and Help). The fulfillment API uses this context to keep track of the latest asked question and of the past results, which represent the status (i.e. the memory) of the chatbot.
Note that only some intents can start the question context (e.g. those selecting the exercise type), while the numeric response intent and the statistics intent are only available once the context has been entered.
A possible enhancement could be of keeping track of recurrent mistakes made by the user to determine points of weakness and return them in response to the statistics intent: all that information should be added to the question context.
I assigned the question context a lifespan of 5, meaning that the user may ask up to 5 questions outside of the current context without exiting the context. Given the fact only 3 intents (welcome, fallback and help) are outside of the context, I believe it’s a very conservative value.
The chatbot requires only one entity, which I called tabellinaNumero (table number) and I trained it with both numeric and alphabetic numbers from 1 to 10.
The last step of the chatbot creation is to setup the Fulfillment part. This part requires coding, and specifically you have to expose a webhook compatible with Dialogflow input/output format. You can find some Google documentation at this link.
Fulfillment Webhook
I coded my webhook in Python and I made my code available in a github repository: https://github.com/foppini975/Tabelline_GoogleAssistant
The python code is built with Flask. There is an only endpoint called /tabelline exposing a single POST method which takes care of the entire fulfillment.
The webhook code is organized within two files:
- tabelline.py: it implements the Flask app and the post endpoint scheleton.
- dyncontest.py: it implements the context app which generate the response based upon the received intent and entities and the context parameters.
In tabelline.py, the Tabelline class first processes the received json request (request.json object) and checks whether the extracted intent (request.json[‘queryResult’][‘intent’][‘displayName’]) belongs to the question-context: in that case it lets the DynContest class elaborate the response, which is straightforwardly returned back to the user.
from flask_restful import Resource...class Tabelline(Resource):
def post(self):
json_data = request.json
responseId = json_data['responseId']
intentName = json_data['queryResult']['intent']['displayName']... if intentName in ["richiestaTabellina", "richiestaTabellinaDa", "richiestaTabellinaFinoA", "richiestaTutteTabelline", "rispostaNumero", "statistiche"]:
dc = dyncontest.DynContest(json_data, logging.getLogger())
return make_response(jsonify(dc.respond()))...
Otherwise, if the intent doesn’t belong to the question-context, the Tabelline class returns a fallback string:
unrecognized_string = "Non ho capito"fullfillmentJson = { "fulfillmentMessages": [{"text": { "text": [ unrecognized_string ] }}],
"fulfillmentText": "Unrecognized intent",
"payload": {
"google": {
"expectUserResponse": True,
"richResponse": {"items": [{"simpleResponse": {"textToSpeech": unrecognized_string}}]}}
},
}return make_response(jsonify(fullfillmentJson))
In the DynContest class we have a few methods:
- The constructor __init__() extracts some data from the json request and prepares that for future usage.
def __init__(self, json_data, logger = None):
self.json_data = json_data
self.session = self.json_data['session']
self.responseId = self.json_data['responseId']
self.queryText = self.json_data['queryResult']['queryText']
self.intentName = self.json_data['queryResult']['intent']['displayName']
self.logger = logger
random.seed(datetime.now())- getContextParameters() returns the value corresponding to the given key from the context parameters (which represent the data persistent while the question-context is active).
def getContextParameters(self, context):
# first of all, let's look for the context name
try:
contexts = self.json_data["queryResult"]["outputContexts"]
except KeyError:
return {}
context_name = os.path.join(self.session, "contexts", context)
# let's extract the parameters for context_name
try:
parameters = [c['parameters'] for c in contexts if c['name'] == context_name][0]
except KeyError:
parameters = {}
return parameters- generateResponse() encapsulates a response string and optional new context parameters in a json format suitable to be returned by the endpoint.
def generateResponse(self, response_string, context, context_params = None):
fulFillmentMessages = { "fulfillmentMessages": [
{ "text": { "text": [ response_string ] } }
],
"fulfillmentText": response_string,
"payload": {
"google": {
"expectUserResponse": True,
"richResponse": { "items": [{ "simpleResponse": {"textToSpeech": response_string}}]}
}
}
}
if context_params is not None:
fulFillmentMessages["outputContexts"] = [
{
"name": os.path.join(self.session, "contexts", context),
"lifespanCount": 5,
"parameters": context_params
}
]
return fulFillmentMessages- respond() represents the heart of the class: it processes the latest user intent and the context parameters elaborating the bot response and determining whether new context parameters have to be stored.
def respond(self):
context_name = "question-context"
parameters = self.getContextParameters(context_name)
self.logger.info("Context parameters: {}".format(parameters))if 'score' not in parameters:
parameters['score'] = 0
if 'length' not in parameters:
parameters['length'] = 0 lastQuestionFeedback = ""
if self.intentName == "richiestaTabellina":
parameters['allowed'] = [int(parameters['tNum'])]
elif self.intentName == "richiestaTabellinaFinoA":
parameters['allowed'] = list(range(2, 1 + int(parameters['tNum'])))
elif self.intentName == "richiestaTutteTabelline":
parameters['allowed'] = list(range(2,11))
elif self.intentName == "rispostaNumero":
risultatoUtente = parameters['number-integer']
risultatoAtteso = parameters['last_question']['first_number'] * parameters['last_question']['second_number']
if risultatoUtente == risultatoAtteso:
lastQuestionFeedback = self.getRandomCongrats()
parameters['score'] += 1
else:
lastQuestionFeedback = "Attento! {:.0f} per {:.0f} fa {:.0f}. Ora dimmi ".format(parameters['last_question']['first_number'], parameters['last_question']['second_number'], risultatoAtteso)
parameters['length'] += 1
elif self.intentName == "statistiche":
score = parameters['score']
length = parameters['length']
lastQuestionFeedback = "Hai risposto correttamente a {:.0f} domande su {:.0f}. {}. Ora dimmi ".format(score, length, self.calcolaVoto(score/length))first_number = random.randrange(max(2, min(parameters['allowed'])), 1 + min(max(parameters['allowed']), 10))
second_number = random.randrange(2, 11)parameters['last_question'] = { 'first_number': first_number,
'second_number': second_number }response_string = "{} {} per {}".format(lastQuestionFeedback, first_number, second_number)fulfillmentJson = self.generateResponse(response_string, context_name, parameters)
return fulfillmentJson
That’s pretty much it. Don’t forget to create the wsgi.py file:
from tabelline import app
if __name__ == "__main__":
app.run()I used Nginx as web server and reverse proxy and Gunicorn as gateway interface. This is my Nginx configuration:
location /tabelline {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://unix:/home/fabio/myProjects/Tabelline_DF/tabelline.sock;
proxy_read_timeout 90;
}Finally, I created a service file to start and stop the gunicorn service, accommodating 3 workers:
[Unit]
Description=Gunicorn instance to serve Tabelline API
After=network.target
[Service]
User=fabio
Group=www-data
WorkingDirectory=/home/fabio/myProjects/Tabelline_DF
Environment="PATH=/home/fabio/myProjects/Tabelline_DF/localenv/bin"
ExecStart=/home/fabio/myProjects/Tabelline_DF/localenv/bin/gunicorn --workers 3 --bind unix:tabelline.sock -m 007 wsgi:app
[Install]
WantedBy=multi-user.targetLastly, I provide a screenshot of my Google Assistant in action. At the time of writing it’s still in Alpha release waiting for Google approval.
