AI Feedback Frequently Asked Questions
What is the AI Feedback?
AI Feedback is a suite of tools in Saga Connect that leverages the power of AI to provide coaches with additional information and insights that can help them in their work with tutors. The AI Feedback tools are intended to be used by coaches to augment their work and are not intended as a replacement.
Why is the AI Feedback important?
The AI feedback will provide coaches with additional information and insights that can help augment their work with tutors. According to our theory of change, this AI-enhanced coaching will lead to improvements in tutors’ practices, which will have a downstream effect on student engagement and learning. Based on our current research, we have found that coaches’ workflows are changing due to their use of the AI Feedback tools and that the introduction of these tools has led to significant improvements in key tutor moves. In other research, these key tutor moves have been linked to improved student achievement outcomes. While we are currently conducting studies to test our full theory of change, our current research that examines components of it demonstrates promise in directly linking the use of AI Feedback tools to improved student outcomes.
What is the AI that powers the AI Feedback?
There are two types of AI models behind the scenes. One AI model is an Automatic Speech Recognition (ASR) model that transcribes tutor and student speech into text. For the ASR model, we are using an off-the-shelf model from Open AI that was then fine-tuned on children’s speech (so that the ASR can correctly transcribe students’ talk). Another AI model is then used to classify the text and identify when specific QTF look fors occur in a tutorial session. For this, we use a Large Language Model (LLM): ChatGPT.
What data does the AI models use?
Currently, the ASR model relies on the audio captured during tutorial sessions and the QTF models uses the transcripts generated by the ASR models along with the text in the chat.
Do the AI models pick up what’s on the math whiteboard?
No, not yet.
How did you build the model to detect QTF look fors from tutorial sessions?
To develop the model to detect QTF look fors from tutorial sessions, we first started by defining each QTF look for and then coding tutorial transcripts to identify instances of QTF look fors. This generated a “QTF ground truth” dataset. We then used the definitions and actual examples of QTFs (from tutorials not in the ground truth dataset) to develop chatGPT prompts and run the transcripts through these prompts. We then compare the outputs from these prompts to the “QTF ground truth” dataset and iterate on the prompts as needed with humans looking at the outputs and assessing quality and model performance throughout the process.
Does the AI QTF model detect missed opportunities to apply a QTF skill?
No, not yet.
How accurate are the AI models?
On average, the accuracy of these AI models range from 50% to 80%. Things that impact the accuracy of these models include: the quality of the input such as the audio quality, noise levels of the surrounding environment, prompt quality, etc… While not perfect, our AI models are good enough to be used by coaches in their work supporting tutors and are not intended to be stand-alone tools to replace the role and capabilities of human coaches.
Does the AI work with tutorial sessions where the primary language of instruction isn’t English?
Currently, our AI models only work well where tutorials are in English (as that’s the data they have been trained on). For tutorial sessions in other languages, while the ASR model does some transcription and AI models attempt to detect QTF look fors, the AI models don’t work as well and aren’t reliable.
Is there a desired target or goal with respect to the AI Feedback data?
No, there isn’t a target goal with respect to the AI Feedback. The goal is flexible and depends on the specifics of the tutorial session (e.g., is it a review session? A homework help session?) and your coaching goals with the tutor. In general, our research has found that using more high-quality discourse moves (in this case, QTF moves) is better, but there isn’t a specific number to target or reach.
When does the AI Feedback become available?
Usually, the AI Feedback becomes available within 30 minutes of when the tutorial session wraps up. In some cases, the data can take up to an hour to process.
Is the AI Feedback available to tutors?
This is a configuration option that we may enable or disable for programs depending on thier structure and goals. Please check with your program administrator if you have questions. If AI is not available for tutors in your organization, some coaches take screenshots of the AI Feedback and/or share their screens and show it during coaching sessions.