AI tools helped run the style study described here, gather the research and write the first draft of this article. I edited it and checked the figures against the study files and the external sources listed at the end. The study is an internal working document prepared for Dapp.Expert with AI research tools; it is not peer-reviewed. The cover illustration is AI-generated. Details at the end.
The last three parts of this series quoted a lot of numbers. Ethan Mollick uses "I" about 16.8 times per 1,000 words. Benedict Evans asks more questions than the other two writers we studied. Molly White's texts carry the most links and numbers. All of those figures came from one source: a style study of three writers that we ran for Dapp.Expert in November 2026 [S].
Each time, the article added a label: internal study, not peer-reviewed. This part explains what that label is hiding, and asks a question that matters well beyond our own work: what makes a claim about someone's writing style worth believing? The same questions apply to any analysis that turns text into numbers, whether it is a study of famous essayists, a "sentiment score" for a crypto token or a tool that says a text was written by AI.
This is the eleventh and last part of a series on voice and trust in editorial work. Earlier parts covered evidence, AI disclosure, corrections, attribution, confidence, voice, rhythm, and how Mollick, Evans and White build their arguments. This one applies the series' own standards to the study behind the last three parts.
Why style claims are easy to make and hard to check
"Her sentences are short." "He hedges a lot." "Their tone is ironic." Sentences like these sound like observations, but each one hides two questions: compared to what, and measured how?
Our own earlier work is a good example of the problem. Before the three-writer study, we built a broad survey of 148 well-known writers and rated each on scales such as irony, use of metaphor and how openly they handle uncertainty [A]. It was useful as a map. It was not a measurement. Only 45 of the 148 writers had at least one of their texts actually read; 101 were described from reference pages, mostly Wikipedia; and for 66 of those, the style tags came only from a short "tone" note in our catalogue [A]. All the scale scores were the judgement of one researcher, an AI research tool, and nobody checked whether a second person would give the same scores [A].
The survey lists these limits itself. But a reader who saw only a score ("irony: 1 out of 5") could not know that it might rest on one sentence in a reference article. The three-writer study was designed to do better.
1. Enough real text, chosen by a rule
The first question is what was actually read. The study used 24 posts by Mollick (about 43,600 words), 28 essays by Evans (about 52,800) and 26 texts by White (about 63,700), all openly available [P]. The texts were not picked because they were famous or quotable. They were chosen by a stated rule, spread across the years, genres and topics of each writer's output, with the selection scripts and a fixed random seed recorded [P][E]. Each writer's five best-known texts were marked separately, so that a well-known essay could not quietly set the tone for the whole profile [E].
Evans's paid weekly newsletter and his annual slide presentations, arguably his main formats, are not in the corpus [E]. Mollick's posts contain prompts and AI responses, which ended up in some counts [P]. White's texts include unmarked direct speech from other people, such as transcripts and paraphrased testimony, which can blur her own profile [P].
What to ask: How many full texts? Chosen how? What was excluded, and could the exclusion change the result?
2. Compared to what?
"16.8 uses of 'I' per 1,000 words" means little without another number next to it. The usual comparison is a large reference corpus of the language, built to represent many kinds of writing. The study's brief asked for one, and the pilot could not provide it: one widely used option, the Corpus of Contemporary American English, sells its full-text data [3], and the British National Corpus, a 100-million-word collection, represents British English "from the later part of the 20th century" [4].
So the study compared each writer with the other two [P]. That is a legitimate choice, but it changes what the numbers mean. Mollick's 16.8 uses of "I" per 1,000 words, against 6.4 for Evans and 9.8 for White [E], shows that he uses the first person more than these two writers. It does not show that he uses it a lot by the standards of English prose in general. When the last three parts said "more" or "fewer", they meant "within this group of three".
What to ask: What is the baseline? Is a "high" score high against the language, a genre, or two other people?
3. Two coders, and a measure of how often they agree
Counting words is easy for a script. Recognising a hedge, an analogy or a self-experiment needs a reader's judgement. The standard safeguard: two coders label the same passages independently, and their agreement is measured.
Raw agreement overstates reliability, because two coders who guess will sometimes agree by chance. In 1960 the statistician Jacob Cohen proposed a measure, kappa, that corrects for that [1][2]. A kappa of 1 means perfect agreement; 0 means no better than chance. Cohen's own rule of thumb calls 0.41 to 0.60 "moderate" and 0.61 to 0.80 "substantial" agreement, and a 2012 review in the journal Biochemia Medica argues that for health research this scale can be too lenient, because it implies that a kappa as low as 0.41 might be acceptable [2]. Our study's brief set the bar at 0.6 [P].
Here is what happened. For each writer, about 10% of the text fragments were coded twice: 50 of 491 for Mollick, 51 of 518 for Evans, 70 of 701 for White [M][E][W]. The first pass was automated rules; the second was a reading by an AI agent [P]. Average kappa came out at 0.505 for Mollick, 0.618 for Evans and 0.514 for White, so two of the three fell below the bar [P]. Even Evans's passing average hid two weak features: attribution (0.33) and analogy (0.32) [E].
The study then added a third, blind pass: a separate machine coder that saw only the codebook and the fragments, with no access to the earlier labels [P]. The two readings agreed far more closely with each other than either did with the rules: 0.859, 0.910 and 0.937 on average, with no single feature below 0.70 [P]. Across all three passes together (Fleiss' kappa), agreement was lower: 0.600, 0.713 and 0.654 [P]. So the automated rules, not the features, were the main source of disagreement. The report adds the caveat itself: both readings were made by machines, which can share the same biases [P].
What to ask: Did two independent coders label the same material? What is the agreement for each feature, not just on average? Were the coders people, or machines that might share a bias?
4. Definitions a second reader can apply
Low agreement is often a symptom of a vague definition. The pilot report is candid about where its codebook failed. "Analogy" was defined too broadly, and the rules could not see a metaphor that did not use the word "like"; "attribution" was confused with simply describing an event [P]. Those were the features with the weakest agreement: kappa from 0.12 to 0.48 across the three writers [M][E][W].
The proposed fix is concrete [P]. Split "analogy" into explicit comparison, metaphor identified by a published procedure (MIPVU, a method developed by linguists at VU Amsterdam [5]) and idiom. Count "attribution" only when there is an explicit source: a link, a document or a name. And give every feature three positive and three negative examples, so a new coder can see where the boundary lies.
What to ask: Could someone else apply the definitions and get the same labels? Are there examples of what does not count?
5. Quotes checked by machine, not by memory
A style study lives on quotations, and quotations are where errors creep in: a phrase attached to the wrong essay, a paraphrase presented as a quote. The study ran a script that checked every quote attributed to a corpus text, confirming that the quote really appears in that text and is no longer than 15 words [P]. It verified 63 of 76 quote lines for Mollick, 59 of 80 for Evans and 65 of 77 for White; the rest were headlines (which are not in the body text), templates such as "If you…, you should…", Russian-language paraphrases and Appraisal category labels [P].
The check found real mistakes: one quote tied to the wrong Evans essay, and a wrong text reference in the study of White [P].
What to ask: Were the quotes checked against the original text? What could not be checked, and why?
6. Every claim labelled by where it comes from
Each of the three studies marks its claims as sourced, from the corpus, or an assumption by the researcher [E]. Evans's career history is sourced; his average sentence length of about 24 words is a corpus result; the view that his audience is mainly tech executives and investors is labelled an assumption, because there is no data on his subscribers [E].
The same honesty applies to patterns that rest on a single text. Four of the techniques described for White appear only in one of her texts [P]. The report proposes a separate status for these, roughly "technique of a signature text", rather than presenting them as general habits [P].
What to ask: Can you tell which statements are measured, which are sourced and which are the author's guesses?
7. A grade, and the reason for it
The last check is the simplest to state and the rarest to see: the study grades itself. The brief defined class A as a full corpus plus two independent human coders, kappa of at least 0.6 and an editor's sign-off [P]. The corpus part met that standard for all three writers. The rest did not: there was no second human coder, two writers missed the kappa bar on the first comparison, and no editor signed the studies off. All three received class B [P].
After the blind third pass, the report proposed an intermediate label, B+ ("machine double coding"): an A-level corpus and two machine readings, one of them blind, that agree at 0.6 or above on every feature, without a human coder or editor [P]. A later internal review, run with a panel of three AI models by a different AI system and checked by hand, concluded that B+ is suitable for internal drafts and labelling, but not for describing the findings as "verified" in public [R]. That is why the last three parts used the words "our counts" and "in our reading", not "proven".
Seven questions for any style analysis
These checks apply whenever someone turns text into a verdict: "this author is more cautious", "this token's community is bullish", "this essay was written by AI".
| Question | What our study did |
|---|---|
| Enough full texts, chosen by a rule? | Yes: 24 to 28 per writer, by year, genre and topic, with a seed |
| Compared to a stated baseline? | Partly: against two other writers, not the language as a whole |
| Two independent coders? | Machine only: rules, then two AI readings, the second one blind |
| Agreement reported per feature? | Yes, including the weak ones |
| Definitions with examples? | Not yet: a revised codebook is proposed |
| Quotes checked against the original? | Yes, by script; what could not be checked is listed |
| A grade with reasons? | Yes: class B; B+ for internal use (for White, after one open decision) |
The forum, the numbers and the tool below are invented for illustration.
Claim: "Our AI tool analysed the community forum of a DeFi protocol: sentiment is 78% positive."
The questions: Which posts were read, and were deleted posts or bot accounts excluded? Positive compared with what, the same forum last month, or other protocols? Who checked the tool's labels against a human reading, and how often did they agree? Does "positive" include sarcasm? Without those answers, 78% is a number without a meaning, however precise it looks.
What this article cannot tell you
This article describes one study, evaluated by the publication that commissioned it. Dapp.Expert uses the study to design the writing styles of its AI authors, so it has an interest in the study looking sound. Weigh what follows with that in mind. The pilot covered three writers in English; it says nothing yet about the other writers in our survey, about Russian-language writers, or about short formats such as threads [P]. The thresholds discussed here, such as a kappa of 0.6, are conventions, not laws; different fields set different bars [2].
It also cannot tell you whether the study's conclusions about each writer are right. A well-documented study can still be wrong. What the documentation gives you is the ability to see where it might be.
How this article was made
| Task | What the AI did | What a person did | How it was checked |
|---|---|---|---|
| Research | Ran the corpus selection, metrics, coding passes and quote checks; drafted the study reports | Set the brief and its standards (the questions, the class A criteria); decided on the third blind pass. The AI executor chose the three writers within the brief | Pilot report, per-writer studies and an internal three-model review with manual checks |
| Draft | Wrote the first draft from the study files | Edited the text and the structure | Every figure compared with the study files; external facts checked on the pages listed below on 15 November 2026 |
| Images | Generated the cover illustration; laid out the diagrams from the series template | Approved the illustration and the diagrams | Labelled as AI-generated in the disclosure line and alt text |
| Not done by AI | — | The decision to publish and responsibility for the text | — |
Conflicts of interest: Dapp.Expert commissioned the style study and uses it to design its AI authors; this article evaluates that study. Ethan Mollick, Benedict Evans and Molly White were not involved in this article or the study.
Responsible for this article: Dmitriy Lastovka, author. Corrections: contact form.
Sources
Labels: research, official page, internal study.
- Cohen, J. "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement 20(1), 1960, 37–46. Research. https://doi.org/10.1177/001316446002000104
- McHugh, M. L. "Interrater reliability: the kappa statistic." Biochemia Medica 22(3), 2012, 276–282. Research (review). https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
- English-Corpora.org, "Full-text corpus data" (COCA and other corpora; purchase page). Official page. https://www.corpusdata.org/
- British National Corpus, "About the British National Corpus". Official page. http://www.natcorp.ox.ac.uk/corpus/index.xml
- Steen, G. J., Dorst, A. G., Herrmann, J. B., Kaal, A. A., Krennmayr, T., & Pasma, T. A Method for Linguistic Metaphor Identification: From MIP to MIPVU. John Benjamins, 2010. Research. https://benjamins.com/catalog/celcr.14
Internal study files (Dapp.Expert, working documents, not peer-reviewed):
- [S] Style study of three writers (Mollick, Evans, White), November 2026: pilot report and per-writer studies. Internal study.
- [P] Pilot report: style study of three writers, 4 November 2026, including the third blind pass. Internal study.
- [M], [E], [W] Per-writer studies: Ethan Mollick, Benedict Evans, Molly White, 4 November 2026. Internal study.
- [A] Style anatomy of 148 reference writers, 4 November 2026. Internal study.
- [R] Internal review of the writer studies with a panel of three AI models, checked by hand, 5 November 2026. Internal study.










