Rae Karimjee, Senior Educational Technologist at City St George's, writes about her university's participation in the BLE's Jisc-supported pilot of the platform,TeacherMatic. The post was originally published here.
As generative AI becomes increasingly embedded in higher education, many universities are wrestling with the same question: do specialist AI tools offer something genuinely different, or can the AI tools we already have meet most of our needs?
At City St George’s, staff already have access to Microsoft Copilot alongside a growing range of AI-enabled tools. So when we joined the TeacherMatic pilot through the Bloomsbury Learning Exchange (BLE) and Jisc, our aim wasn’t to find our first AI solution. Instead, we wanted to understand whether a specialist educational AI platform could offer additional value and whether it would support the varied assessment, feedback and curriculum design needs across our Schools.
TeacherMatic is designed specifically for educators, with tools intended to support activities such as assessment design, feedback generation, rubric creation, lesson planning and quiz development. The pilot involved five London institutions and gave us the opportunity to evaluate the platform in authentic teaching and assessment contexts rather than relying on vendor claims alone.
What Were We Hoping to Learn?
During an initial check-in with pilot participants, four key expectations emerged. Staff wanted to know whether TeacherMatic could:
- Make feedback more consistent.
- Speed up marking.
- Support the creation of assignment briefs and rubrics.
- Reduce workload more broadly.
These are challenges familiar to almost anyone involved in teaching and assessment. The evaluation therefore focused not simply on whether TeacherMatic could generate content, but whether it could improve the overall assessment and feedback workflow.
Early Enthusiasm, Limited Engagement
Interest in the pilot was encouraging, but one of the findings that emerged throughout the project was the difference between interest in AI and active use of an AI tool. More than half of participants had not yet started using TeacherMatic at the point of the first formal review, and only a small number had undertaken significant testing.
This proved to be an important lesson. A successful educational technology needs not only to demonstrate capability but also to fit naturally into busy academic workflows. If staff cannot easily see where a tool fits into their day-to-day practice, adoption is likely to remain limited regardless of its potential.
What Staff Found
For colleagues who tested the platform, the picture was mixed. Several participants appreciated the ability to generate structured outputs quickly. The multiple-choice question (MCQ) generator, for example, showed promise as a starting point for question development, helping academics create initial drafts that could then be refined for their discipline and level of study.
The Advanced Feedback tool also attracted interest. Staff found it could produce detailed feedback aligned to rubric criteria and generate draft comments significantly faster than they could be written manually. However, several recurring concerns emerged:
- Significant prompting and editing were often required.
- Feedback could feel generic or recognisably AI-generated.
- Rubrics were not always applied consistently.
- Hallucinated references and inaccuracies occasionally appeared.
- Outputs were not always appropriate for higher-level academic work.
For some participants, the challenge wasn’t that the platform couldn’t produce outputs. It was that time saved during content generation was often replaced by time spent reviewing, checking and editing the results. Academic judgement remained essential throughout the process.
The Workflow Challenge
One of the strongest themes to emerge from the pilot was the importance of workflow. Academic colleagues working with larger cohorts highlighted practical requirements such as:
- Bulk upload of coursework from Moodle.
- Efficient generation of feedback at scale.
- Retention of anonymised student identifiers.
- Seamless mapping of feedback back to students.
Questions arose about whether TeacherMatic could reliably support these processes without adding manual steps. For colleagues marking hundreds of submissions, even small workflow inefficiencies could offset any advantages offered by AI-generated feedback. Another participant found they were unable to use the platform effectively for coursework submitted in Excel format.
These findings reinforced an important point: the effectiveness of an AI tool is about far more than the quality of its outputs. Integration into existing academic practice matters just as much.
A Closer Look at Assessment Feedback
One of the most detailed evaluations came from Luis da Vinha, Senior Lecturer in International Politics and a Digital Champion, who compared TeacherMatic’s Advanced Feedback tool with human marking across 41 undergraduate assignments. The results highlighted both strengths and limitations. On the positive side, TeacherMatic generated structured, rubric-aligned feedback considerably faster than human markers. This demonstrated clear efficiency benefits and suggested potential value in supporting feedback preparation and formative activities. However, the evaluation also found that:
- TeacherMatic awarded marks that averaged 4.17 percentage points higher than human markers.
- Grades tended to be compressed towards the middle range.
- Feedback sometimes lacked module-specific context.
- The tool could not replicate the disciplinary insight and contextual understanding provided by experienced academics.
The conclusion was clear: while TeacherMatic could support feedback preparation, it should not be used for autonomous summative marking. Human oversight remains critical.
Comparing TeacherMatic with Existing AI Tools
Perhaps the most important question throughout the pilot was not whether TeacherMatic worked, but whether it offered enough additional value over AI tools already available to staff.
Several participants noted that experienced users of Microsoft Copilot or ChatGPT could often achieve similar outcomes through effective prompting. In this context, TeacherMatic’s main advantage appeared to be the educational workflows and templates wrapped around the AI, rather than consistently superior outputs.
One participant who explicitly compared TeacherMatic with existing AI tools concluded that, aside from some potential benefits around rubric-based feedback generation at scale, the platform offered little improvement over the tools staff already had available.
Our Decision
Having reviewed the evidence gathered throughout the pilot, City St George’s decided not to continue with TeacherMatic at this time.
This decision was not based on any single evaluation or participant experience. The platform demonstrated genuine strengths, including rapid content generation, structured outputs and support for some drafting and formative activities.
However, the anticipated benefits around workload reduction, assessment and feedback quality were not demonstrated consistently enough to justify continued adoption. Across the pilot, concerns remained around academic judgement, workflow compatibility, disciplinary context, marking reliability, engagement levels and overall value when compared with existing AI provision.
What We Learned
While we won’t be taking TeacherMatic forward, the pilot has been extremely valuable. It highlighted five areas we will continue to consider when evaluating future AI tools:
- Pedagogical value.
- Support for academic judgement.
- Workflow integration.
- Whole-workflow efficiency, not just generation speed.
- Additional value beyond existing AI provision.
Perhaps the biggest lesson is that generating content quickly is not the same as improving educational practice. As AI technologies continue to evolve, we need to evaluate them not only on what they can produce, but on whether they genuinely enhance teaching, assessment and the student experience.
The TeacherMatic pilot helped us answer one question about a specialist AI platform, but it also raised a broader one for our institution:
Are we looking for AI tools that can generate content faster, or tools that can genuinely improve the way we teach, assess and support learning?
The answer to that question will likely shape our future AI decisions far more than any individual product evaluation.
Further Reading
The full evaluation report provides a detailed account of the pilot methodology, participant feedback, assessment findings and recommendations. Readers interested in exploring the evidence in more depth are encouraged to read the full report alongside the wider findings from the Bloomsbury Learning Exchange collaboration. Jisc TeacherMatic BLE Pilot Project report





