•/• MÉDIANE
AI ASSESSMENT

French subsidiary of an international carmaker · 2026

Opening up the chatbot's black box before deciding its future

Alignment & Governance · CRM & Customer Relationship
Format
The Sprint
Duration
2 weeks
Team
Marketing department, around ten people
353,015 conversations analysed

Every conversation the website chatbot handled over nine months, with no day missing.

The starting point

The website chatbot brought in half of all web leads. Yet nobody really knew what it was telling customers, or how it was configured. Two contract deadlines were looming: its future had to be decided, on facts.

The approach

An analysis of every conversation from 1 January to 25 September, using analysis scripts written for the purpose. The findings were grouped into three families: measurement, content and security. A session with leadership then weighed them against the decisions to be made.

What changed

The chatbot is no longer a black box: seven quantified findings. 81,138 conversations, or 23%, opened with one or two keystrokes and inflated the statistics. The platform assessed the quality of only 0.04% of exchanges. The flaws lay mostly in configuration and content: switching tools would not have fixed them. Leadership now has three clear paths: renew the contract while preparing what comes next, keep the tool and take back control of its configuration and content, or build an in-house solution.

What stayed human

Reading the findings and choosing the path, left to leadership. AI helped write the analysis scripts, but no figure comes from a model's estimate.

A similar challenge, or an AI project in mind? Take the AI Assessment in a few minutes.

Take the AI Assessment

← All case studies