Comparing Creators Without Rewarding Only Incumbents

Comparing Creators Without Rewarding Only Incumbents

user avatar

by me1afon

17 minutes ago


AI tools helped gather research and write the first draft of this article. I edited it and checked every study, figure and quotation against the original paper or the platform's own page, listed at the end. The cover illustration is AI-generated. Details at the end.

Sort almost any creator leaderboard by its default column, total followers, views or rewards, and the top will mostly be people who started early. That is not a scandal; they did the work for longer. The problem is what happens next. A position near the top brings attention, attention brings more of the counted number, and the list starts to describe its own history rather than who is doing the best work now.

A programme that wants new authors has to compare people who began at different times, with audiences of very different sizes, without turning the ranking into a reward for seniority. It also should not swing to the other extreme and punish people for having succeeded.

This is the eleventh and last part of Influence & Incentives, a series on how other people, rules and rewards shape what we decide, write and publish. The ninth part asked whether audience size reflects authority on a topic, and the tenth how to make a ranking explainable. This part deals with one narrow question: how to compare creators fairly when some of them have a head start.

Why early leaders keep leading

The sociologist Robert K. Merton named the pattern in 1968. Studying how scientists receive credit, he described "the Matthew effect", after the verse about those who have being given more, as "the accruing of greater increments of recognition for particular scientific contributions to scientists of considerable repute and the withholding of such recognition from scientists who have not yet made their mark" [1]. He also noted that eminent scientists were aware of it and made special efforts to counteract it [1].

Two later experiments show the mechanism directly.

In 2006, Matthew Salganik, Peter Sheridan Dodds and Duncan Watts built an online "music market" in which 14,341 participants listened to and downloaded songs by unknown bands [2]. Some participants saw only the song list; others also saw how many times each song had been downloaded. Showing those counts changed the outcome: "Increasing the strength of social influence increased both inequality and unpredictability of success" [2]. Quality still mattered, but only partly: "The best songs rarely did poorly, and the worst rarely did well, but any other result was possible" [2].

In 2014, Arnout van de Rijt and colleagues went further and intervened in live systems: they gave a first donation to randomly chosen Kickstarter projects, positive ratings on Epinions, awards to Wikipedia editors and signatures to Change.org petitions [3]. In every system, "early success bestowed upon arbitrarily selected recipients produced significant improvements in subsequent rates of success compared with the control group of nonrecipients" [3]. On Kickstarter, a single initial donation made projects about twice as likely to attract funding [3]. The authors report that the arbitrary gap between chosen and unchosen recipients persisted in each study, though in some systems it narrowed and became small by the end [3].

The same paper found a limit that matters for programme design. Success "exhibited decreasing marginal returns, with larger initial advantages failing to produce much further differentiation" [3]; in the authors' words, "the reinforcement value of a single initial success is much larger than that of additional successes" [3]. A first push changes a lot. A tenth push changes little.

For a creator ranking, the lesson is uncomfortable but useful: a leaderboard is not only a measurement, it also hands out attention. Whatever it shows at the top tends to stay there partly because it is at the top.

Three advantages a total hides

A lifetime total mixes the author's current work with three advantages that have little to do with it.

Time. An author who has published for four years has had four years to collect views, followers and rewards. A newcomer in their third month cannot catch up on a lifetime count, however good the work is.

Audience. Every new post by a large account starts with a large audience. The same article, published by an author with 100,000 followers and by one with 2,000, will collect very different totals on the first day for reasons unrelated to the text.

Visibility. Ranked lists, "top creators" pages and recommendation slots send extra readers to whoever is already high. This is the loop shown in the experiments above: the counted number feeds itself.

None of these is unfair in itself. Time and audience were usually earned. They simply answer a different question from "who is doing strong work right now?" A fair comparison separates the questions instead of pretending one number answers both.

Five corrections that make comparisons fairer

Each correction removes one advantage from the comparison. None is a universal formula; the settings depend on the platform and the programme's size.

1. Compare authors of the same age

Group authors into cohorts by the month or quarter they started, and compare each author with their own cohort. A useful version compares everyone at the same age on the platform: what did each author achieve in their first 90 days? A spring newcomer is then measured against what established authors did in their own first 90 days, not against four-year totals.

The research notes behind this article recommend the same thing for a small programme: split authors by cohort and compare before and after a change, rather than fitting a statistical model on a few dozen people.

2. Use rates, and compare within size bands

Divide by the audience the author actually has: replies per post, reads per follower, returning readers per hundred visits. This removes most of the audience advantage.

Rates can carry a bias of their own. The research notes behind this article expect small accounts to show higher engagement rates than large ones, and if that holds on a given platform, a pure per-follower rate swings the advantage to the smallest accounts. Check it in your own data before relying on it. Two practical fixes: compare rates only within bands of similar size, or convert each number into a percentile within its band and platform, so "top 10% for accounts of this size on this platform" becomes the comparable unit. The notes also suggest softening the denominator, dividing by a smaller power of follower count rather than the full count, but the exact power is a calibration choice, not a known constant.

3. Look at recent windows, not lifetimes

A median over the last 30 or 90 days describes what the author is doing now. It also gives a returning or improving author a fair chance to rise and keeps a famous old post from carrying someone for years. The lifetime total can stay on the profile as history; it just should not decide this month's comparison.

4. Measure growth, but weigh it sensibly

A separate view of who is growing fastest is one of the most direct ways to surface newcomers. Substack does this openly: alongside its "Top Bestseller" view, ranked by annual recurring revenue, its "Rising" view "highlights the fastest-growing publications within a category based on paid subscription growth" [6]. Substack has not published the exact formula.

Growth needs one guard. Measured as a percentage, it rewards the tiniest accounts: going from 100 to 200 followers is a 100% rise. The research notes suggest combining the relative rise with the absolute one, so a jump from 1,000 to 2,000 does not automatically outrank a jump from 100,000 to 120,000. They also point out a practical cold start: on launch day there is no history, so growth can only be switched on after the first month of snapshots.

5. Be honest about small samples

A new author with three posts has a very uncertain average. One lucky post can place them first; one quiet week can place them last. Statisticians have handled this for a long time. Evan Miller's well-known guide to sorting ratings argues that a score must "balance the proportion of positive ratings with the uncertainty of a small number of observations" and recommends the lower bound of the Wilson score interval [5].

That method has a side effect worth knowing: because the lower bound is cautious, items with few observations sink. In a single combined list, it quietly buries every newcomer. Two better options for creator comparisons:

  • Pull small samples towards the cohort's typical value, not towards zero, so a new author starts in the middle and moves as evidence arrives.
  • Show uncertainty instead of hiding it. Label authors with too little history as "new" or "provisional" and rank them in their own table until they have enough posts.

Several tables instead of one formula

A single ranking that tries to be fair to everyone has to bury its trade-offs inside weights. Separate views make the trade-offs visible: an all-time table for accumulated work, a recent table for the last 90 days, a rising table for growth, and a new-authors table by cohort. Each answers one clear question, and a reader can see which one they are looking at. Separate tables also spread the attention a top slot sends across several groups of authors instead of concentrating it on one.

A small first push goes a long way

The van de Rijt findings suggest where the cheapest help lies. If the first success matters far more than later ones [3], a programme does not need large boosts for newcomers. A small, regular share of visible slots on recommendation pages, newsletters or the front page reserved for recent authors, rotated so that many of them get one fair look, does most of the work. After that, readers decide.

The same logic argues against piling more rewards on people who are already at the top. Jakob Nielsen's participation-inequality rule, first published in 2006, describes communities in which roughly 90% of people only read, 9% contribute from time to time and "1% of users participate a lot and account for most contributions" [4]. Among his recommendations is a line every creator programme could adopt: "Reward — but don't over-reward — participants" [4]. A simple check follows from it: track what share of all rewards goes to the top tenth of authors each month, and treat a steady rise as a design question.

What the corrections can break

Every correction moves advantage somewhere else, and that somewhere can be gamed.

  • Rates reward small closed circles. A tiny account with a loyal group of friends can show an excellent per-follower rate. Rate-based comparisons need the same checks for repeated crowds that the eighth part described.
  • Newcomer slots invite fresh accounts. If being new pays, some people will keep starting over. Tie newcomer status to a verified identity or payout account, not to a new profile.
  • Fairness is not the same as equal outcomes. The music experiment found that the best songs rarely did poorly [2]. Some incumbents lead because their work is better. The goal is to stop the ranking from adding an extra advantage, not to pull leaders down.

A worked example (fictional)

The programme, the authors and all figures below are invented to show the corrections in use.

A publication ranks its authors by total reads and pays a monthly bonus to the top five. Three authors sit at very different places on the list.

Author A started four years ago, has 40,000 followers and leads with 1.2 million lifetime reads. Over the last 90 days, A's median article was read by about 2% of followers.

Author B started eight months ago, has 3,000 followers and sits in 61st place. B's median article over the last 90 days was read by 9% of followers. Compared with other authors with 1,000 to 5,000 followers, that is in the top tenth. In their first 90 days, B gained more returning readers than A did in A's own first 90 days.

Author C joined three weeks ago with two articles, one of which did very well. On an average-per-article list, C is first.

The lifetime table says A, then a long gap. The corrected views say more: A is a strong, established author; B is the strongest of the year's cohorts; C has too little history to rank. A fair programme would keep A's place on the all-time table, put B at the top of the recent and rising tables, give C a slot on the newcomers' page with a "provisional" label, and pay part of the bonus pool per table rather than all of it by lifetime reads.

Five questions to ask before ranking creators

  1. Which question does each table answer? Accumulated work, recent work, growth and newcomers are different questions.
  2. Who started when? Compare authors with their cohort, or at the same age on the platform.
  3. What is the denominator? Divide by audience, and compare within size bands.
  4. How many observations does each author have? Label and separate authors with too little history.
  5. Where does the attention go? Check what share of visible slots and rewards the top tenth receives each month.

What this article cannot tell you

The experiments cited here measured songs, crowdfunding, ratings, editor awards and petitions, not creator programmes; they show that early advantage feeds itself, not how strongly it does so on any particular platform. Substack's Rising view is described from its own announcement; its formula is not public. The corrections are standard comparison techniques adapted from the research notes behind this article, and every number in the notes, such as damping powers or growth weights, is an estimate to calibrate, not a validated value. Nothing here assesses any real author or any programme on Dapp.Expert. Whether audience size reflects authority on a topic, and how to explain a ranking to the people in it, are covered in the two previous parts.

How this article was made

Task What the AI did What a person did How it was checked
Research Collected studies, platform examples and comparison methods into research notes Chose the question and which sources to rely on Studies, figures and quotations checked against the papers and platform pages [1]–[6]
Draft Wrote the first draft from the notes Edited the text, the structure and the worked example A separate review compared figures and claims with the sources
Images Generated the cover photograph-style illustration; built the diagrams from a code template Chose the illustration and the content of the diagrams Labelled as AI-generated in the disclosure line and alt text
Not done by AI — The decision to publish and responsibility for the text —

Conflicts of interest: none. The author has no commercial relationship with the platforms, researchers or publications named; the programme and authors in the example are fictional.

Responsible for this article: Dmitriy Lastovka, author. Corrections: contact form.

Sources

Labels: official (the platform's own announcement or page), paper (research paper, read in the original or via its official record), guide (a practitioner's published explanation).

  1. Robert K. Merton, "The Matthew Effect in Science", Science 159(3810), 56–63, 5 January 1968. https://doi.org/10.1126/science.159.3810.56 Paper.
  2. Matthew J. Salganik, Peter Sheridan Dodds, Duncan J. Watts, "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market", Science 311, 854–856, 2006. https://doi.org/10.1126/science.1121066 Paper.
  3. Arnout van de Rijt, Soong Moon Kang, Michael Restivo, Akshay Patil, "Field experiments of success-breeds-success dynamics", PNAS 111(19), 6934–6939, 2014. https://pmc.ncbi.nlm.nih.gov/articles/PMC4024896/ Paper.
  4. Jakob Nielsen, "Participation Inequality: The 90-9-1 Rule for Social Features", Nielsen Norman Group, 8 October 2006. https://www.nngroup.com/articles/participation-inequality/ Guide.
  5. Evan Miller, "How Not To Sort By Average Rating", 6 February 2009. https://www.evanmiller.org/how-not-to-sort-by-average-rating.html Guide.
  6. Substack, "A new view for leaderboards", On Substack, 27 March 2025. https://on.substack.com/p/a-new-view-for-leaderboards Official.

Tier I

Sector: #18291

Sealed Hiding Place Room

Resource Cache

Resource Cache

Tier I

Requires 25% Tier Progress to Claim
Meme Cache

Meme Cache

Tier I

Requires 50% Tier Progress to Claim
Equipment Cache

Equipment Cache

Tier I

Requires 75% Tier Progress to Claim

After collecting, hiding places will be stored in your inventory and can be opened with Keys.

Other articles

Making an Expert Ranking Explainable

A ranking of experts is a claim about real people. Four parts make it checkable: a published formula, stated weights, a per-entry breakdown and an appeal path.

user avatarme1afon

Audience Size Is Not Topic Authority

A large following records popularity, not knowledge of a subject. Why reach and authority need separate evidence, and five questions to ask about any expert.

user avatarme1afon

Spotting Gaming in Creator Program Metrics

Any number that pays creators attracts people who produce the number, not the work. Three ways metrics get gamed, seven signs, and why a sign is not proof.

user avatarme1afon

How Creator Incentives Change What Gets Published

Six documented cases show how platform incentives change what creators publish, plus five questions to ask before you design or join a creator programme.

user avatarme1afon

Measuring an Author’s Contribution Beyond Output

The easiest number about an author is how much they published, and it says the least. Five layers of contribution, and what each one can and cannot show.

user avatarme1afon

Why Persuasion Principles Are Not Universal Laws

Some classic persuasion effects still hold after 45 years; others shrank to almost nothing when large teams repeated them.

user avatarme1afon

Important disclaimer: The information presented on the Dapp.Expert portal is intended solely for informational purposes and does not constitute an investment recommendation or a guide to action in the field of cryptocurrencies. The Dapp.Expert team is not responsible for any potential losses or missed profits associated with the use of materials published on the site. Before making investment decisions in cryptocurrencies, we recommend consulting a qualified financial advisor.