DEPARTMENT OF SMALL BUSINESS
Queensland Government's, Department of Small Business and Training sought to understand how the implementation and usage of an AI-powered chatbot would be received on their website.
About
The Department of Employment, Small Business and Training (DESBT) partnered with Symplicit’s digital delivery team to conduct usability testing and research on a new AI chatbot aimed at enhancing search functionality across Business Queensland websites. Tested within the Tourism Business section, this chatbot is the first of its kind for the Queensland Government.
Project Length
Five Weeks
Tools
Miro
Deliverables
Report
My Role
UX Research and Design
Project Goals
The primary objectives of this research were to understand when users would engage with the chatbot feature and to assess whether the AI-driven responses effectively helped them locate the information they were seeking.
In addition to these goals, the research aimed to uncover:
-
Customer attitudes towards AI: Insights into how users perceived and accepted AI within the chatbot, as well as their general attitudes toward AI.
-
Satisfaction with responses: An evaluation of user satisfaction with the chatbot’s responses, including response behaviour, answer clarity, and link structure.
-
Perceptions of chatbots: A gauge of user perceptions of chatbots and whether these views might influence their likelihood of using the tool.
Design Process
In this research, we adopted a two-pronged approach to provide the client with a well-rounded balance of quantitative and qualitative data. By running two concurrent streams of work, we captured both in-depth user insights and broader audience perspectives.
The first stream focused on qualitative research, where we conducted usability interviews. Participants engaged in a series of tasks designed to evaluate the chatbot’s effectiveness, followed by a survey to capture immediate feedback on their experience. Additionally, we ran an unmoderated UEQ survey, completed independently by a separate group of users, to assess initial perceptions and usability of the chatbot in a natural, unsupported environment.
Impact
The research culminated in a detailed report, outlining essential focus areas for the next phase of iteration and testing. We gathered valuable data on the product’s alignment with user mental models and users’ perceptions when interacting with it. Through task-based assessments, we evaluated the chatbot's AI capabilities, identifying specific areas that require refinement prior to advancing to Beta testing.
Of particular note was the emerging nature of AI-powered search in the market. While the use of a chatbot as a search tool may represent a forward-thinking approach at this stage, the rapid development of technologies like Perplexity and Google’s AI Search suggests that user expectations and mental models could evolve swiftly. This contextual insight provides the client with a foundation to anticipate and adapt to these potential shifts.
SAY HELLO
Get in touch
Connect
LinkedIn