The problem
An explainer works when the picture and the narration say the same thing. When they don't, viewers notice at once: a scene about "the seat of the soul" that shows a car seat tells the viewer that nobody checked.
ContentRoof illustrates every scene automatically, choosing from a library of about 11,700 drawings. Its rule is that a wrong picture is worse than none: when nothing fits, the scene shows the narration as moving type instead. That rule is only as good as our ability to tell right from wrong, so we set out to measure it.
How we measured
We took 11,751 scenes from about 960 explainer scripts in nine languages and rated the picture leading each one against what the narrator says in that scene. Each picture was rated as matching, wrong, or absent, by the same strict rubric everywhere. Improvements were measured on scripts held out from development.
One scene in four was wrong
The common way to illustrate a sentence is to find its key words and pick the closest drawing for each. That approach gave this:
Figure 1The starting point
Data
| Part | Count | Share |
|---|---|---|
| Matches | 62 | 65% |
| Wrong | 23 | 24% |
| No picture | 10 | 11% |
The errors were remarkably even across languages: between 18% and 27% wrong in every one. German was the weakest, with 52% matching. Most errors were plausible to a computer and absurd to a person: tectonic plates drawn as a dinner plate, space suit as playing-card suits, a boy as a barn.
Finding 1: one bug disabled an entire language
On Hindi scripts, 64% of scenes received no picture at all, and only 16% of the pictures shown matched.
The cause was a single text-cleaning step that kept only "letters and digits". In Hindi's script, vowel signs are encoded as marks rather than letters, so the step removed them: अस्पताल (hospital) became असपतल, a string with no meaning. Meaningless strings then matched unrelated drawings: a heart, a ghost, a light bulb. Keeping the marks raised matching pictures in Hindi from 16% to 70%, with no other change.
The lesson is general: a text pipeline built and tested in English can fail silently in another writing system. Every language needs to be tested on its own real text.
Finding 2: small words cause big errors
For languages without detailed grammatical analysis, every word in a sentence is a candidate for a picture, and short function words were frequently chosen. Spanish más (more) was drawn as a plus sign, todo (all) as a keyboard key, Hindi हैं (are) as a light bulb and Arabic كانت (was) as gravity.
Excluding each language's function words cut wrong pictures in French scripts from 26% to 12%. Some of those scenes received no picture instead, which is the intended trade-off.
Finding 3: judging the whole sentence works in every language
Key-word matching looks at words one at a time. Our new selection step judges each candidate picture against the meaning of the whole sentence. Its effect on matching pictures, by language:
Figure 2Matching pictures, before and after, by language
Data
| Before | After | |
|---|---|---|
| Arabic | 63% | 80% |
| Spanish | 64% | 80% |
| Hindi | 66% | 80% |
| Portuguese | 64% | 77% |
| English | 64% | 75% |
| Urdu | 58% | 73% |
| Italian | 56% | 73% |
| French | 57% | 71% |
| German | 52% | 68% |
Every language improved by 11 to 17 points. Arabic, Spanish and Hindi now lead at 80%, ahead of English at 75%. Wrong pictures fell in every language:
Figure 3Wrong pictures, before and after
Data
| Before | After | |
|---|---|---|
| All nine languages | 23% | 17% |
| German | 27% | 18% |
| Urdu | 26% | 19% |
| English | 25% | 18% |
| French | 25% | 21% |
| Arabic | 23% | 14% |
Finding 4: a better main picture can make a worse scene
A scene often shows a main picture with one or two smaller ones beside it. When we let the new step choose the smaller pictures as well, the main pictures improved but the scenes got worse. It added 4.6 times as many secondary pictures, and enough of them were wrong to crowd the screen with mistakes: "ten" drawn as a "dozen" sign, "woke up" as a sleep icon, "job" as a barbell.
The step now chooses only the main picture, and a secondary picture is kept only when the evidence for it is very strong. A single quality score can rise while what the viewer sees gets worse, so the whole scene has to be evaluated, not one part of it.
Finding 5: a larger library can lose pictures
We added 104 animated illustrations, such as two plates of the Earth's crust pushing up a mountain and a caravan crossing a desert. The next measurement showed 380 scenes that had lost their picture.
The new illustrations' descriptions changed the overall distribution of match scores. The threshold for a confident match is set relative to that distribution, so it moved, and exact matches such as scroll and temple fell below it. Setting the threshold from the drawings alone restored them.
Growing a library changes the statistics every threshold depends on. Each addition needs to be re-measured, not assumed to help.
What's next
One wrong picture in six is better than one in four, but it is still too many. Our next work is on the secondary pictures, which are wrong more often than the main one, and on drawing the ideas that scripts ask for and our library does not yet contain.