French subsidiary of an international carmaker · 2026
Opening up the chatbot's black box before deciding its future
Every conversation the website chatbot handled over nine months, with no day missing.
The starting point
The website chatbot brought in half of all web leads. Yet nobody really knew what it was telling customers, or how it was configured. Two contract deadlines were looming: its future had to be decided, on facts.
The approach
An analysis of every conversation from 1 January to 25 September, using analysis scripts written for the purpose. The findings were grouped into three families: measurement, content and security. A session with leadership then weighed them against the decisions to be made.
What changed
The chatbot is no longer a black box: seven quantified findings. 81,138 conversations, or 23%, opened with one or two keystrokes and inflated the statistics. The platform assessed the quality of only 0.04% of exchanges. The flaws lay mostly in configuration and content: switching tools would not have fixed them. Leadership now has three clear paths: renew the contract while preparing what comes next, keep the tool and take back control of its configuration and content, or build an in-house solution.
What stayed human
Reading the findings and choosing the path, left to leadership. AI helped write the analysis scripts, but no figure comes from a model's estimate.
A similar challenge, or an AI project in mind? Take the AI Assessment in a few minutes.
Take the AI Assessment