A shape for every sentence
Every sentence of a ContentRoof video gets a visual. Most get a drawing of what they name. Some get a template: a chart when something grew or fell, a clock when the sentence is about a span of time, a side-by-side for a comparison, a cycle for a loop, a flow for steps. We call a sentence's shape its intent, and a small classifier picks it by comparing each sentence with labelled examples.
The benchmarks said it worked
We tested it the usual way, on held-out sets with a fair share of every intent. English scored 95%. When we added native training sentences for eight more languages, their balanced test sets scored 73–88%, 79.5% together.
Figure 1One model, two scoreboards
Data
| Score | |
|---|---|
| Balanced test, English (20 sentences) | 95% |
| Balanced tests, 8 more languages (728) | 79.5% |
| Real narration: plain statements left alone | 5.4% |
Then we watched the videos. On 60 English scripts, 21% of all scenes had become a chart or a clock, most of them wrongly. "Genghis Khan died" became a falling chart. Every date ("In June 323 BC", "In eighteen thirty-nine") became a clock.
What real narration looks like
A balanced test has as many "growth" sentences as plain ones. Real narration doesn't.
Figure 2What real narration is made of
Data
| Part | Count | Share |
|---|---|---|
| Plain statement | 1004 | 76% |
| Cause and effect | 112 | 8% |
| Steps of a process | 48 | 4% |
| A number | 47 | 4% |
| A lesson | 25 | 2% |
| Rise or fall | 26 | 2% |
| Question | 16 | 1% |
| Place | 16 | 1% |
| Comparison | 15 | 1% |
| Span of time | 14 | 1% |
| Scale | 4 | 0% |
We took the narration of real scripts in seven languages, the exact text the planner sees, and had a strict reader label every sentence, giving it a shape only when the sentence itself shows one. Three sentences in four are plain statements: they name something and say something about it.
The original model left 54 of those 1,004 plain statements alone. It gave the other 95% a shape they don't have. The balanced sets hid this completely: when plain statements are one class among thirteen, a model that almost never answers "plain statement" loses only a few points of accuracy. In real use, that one class is three quarters of everything.
Figure 3What the original model called the plain statements
Data
| Sentences | |
|---|---|
| Steps of a process | 192 |
| Span of time | 158 |
| Question | 80 |
| Growth | 72 |
| Decline | 67 |
| Lesson | 63 |
| Number | 62 |
| Cycle | 61 |
| Cause and effect | 54 |
| Scale | 51 |
| Place | 48 |
| Comparison | 42 |
| Plain statement (right) | 54 |
The pattern in the mistakes explains them. An embedding model hears what a sentence is about more than how it is built. A sentence that mentions a year sounds like time. A sentence about an empire's end sounds like decline. "Imagine a single ant building a farm that strips a whole tree bare in one night" was labelled a question.
Two changes
Real sentences in training. Half of the labelled real scripts became training examples, next to the hand-written ones. The model now knows what a plain statement sounds like in each language. That alone took the share left alone from 5% to 55%.
An evidence gate. A layout is kept only when the sentence's own words show it, in its own language. A rising chart needs a verb of rising; a clock needs a span of time, not just a date. It covers eleven languages, and no template may repeat back to back.
| Sentence (held out) | Original model | Shipped |
|---|---|---|
| Within three days, that speck sprouts. | process | statement |
| It begins with a speck. | scale | statement |
| يقوم هؤلاء العمال بقطع قطع أوراق بعرض مليميترين فقط. | process | statement |
| ثم ينتج الفطر أجسامًا غنية بالمغذيات تُسمى الجنغيليديا. | growth | statement |
| Juntas, pueden despojar de hojas a un eucalipto maduro en una sola noche. | process | statement |
| The sport grew quickly, becoming a vital part of the new identity. | growth | growth (kept) |
| اكتشف علماء الآثار أواني عسل في قبور مصرية كانت صالحة للأكل بعد 3,000 سنة. | time | time (kept) |
Results
Figure 4Three versions on the same held-out sentences
Data
| Original model | Trained on real sentences | With the evidence gate (shipped) | |
|---|---|---|---|
| Plain statements left alone | 5% | 55% | 88% |
| Layouts that are right | 9% | 18% | 37% |
With both changes, 88% of plain statements are left alone, and 37% of the layouts the planner does choose are right, up from 9%.1 Templates are now rare: 59 of 1,327 sentences get one (4%), where the original model gave one to 908 (68%).
Figure 5Plain statements left alone, by language
Data
| Original model | Shipped | |
|---|---|---|
| English | 6% | 82% |
| Arabic | 5% | 88% |
| Spanish | 5% | 91% |
| French | 5% | 89% |
| Portuguese | 7% | 88% |
| German | 8% | 88% |
| Italian | 3% | 93% |
The improvement holds in every language we can test, from 82% left alone in English to 93% in Italian.
What we took from it
Measure on the real mix. A balanced test answers "can the model tell the classes apart?", which was never our question. Ours was "what happens to a real video?", and the answer depends on the class that a balanced set shrinks the most.
We also chose to keep recall low. The gate misses many real shapes. A missed chart costs a plain picture of what the sentence names; a wrong chart in every third scene costs the viewer's trust in everything else on screen.