Making an Expert Ranking Explainable

Making an Expert Ranking Explainable

user avatar

by me1afon

12 minutes ago


AI tools helped gather research and write the first draft of this article. I edited it and checked every rule, figure and quotation against the original text, the organisation's own page or the news report named at the end. The cover illustration is AI-generated. Details at the end.

A list of "Top 50 Crypto Analysts" puts someone at number 12 with a score of 88. A reader takes it as advice about whom to trust. The analyst at number 12 shares it. The analyst who expected to be in the top ten, and is at 31, wants to know what went wrong. The analyst who is not on the list at all does not know whether she was considered.

All of them are asking why this entry is here. A ranking of experts is a claim about real people, and a claim should be possible to check. This article argues that an explainable ranking needs four parts: a published formula, stated weights with reasons, a breakdown for every entry, and an appeal path that can actually change the result.

This is the tenth part of Influence & Incentives, a series on how other people, rules and rewards shape what we decide, write and publish. The earlier creator parts asked what to count about an author beyond output, how rewards change what gets published, how creator metrics get gamed and why audience size is not topic authority. This part assumes you have chosen what to measure and is about showing the working. How to compare newcomers with established creators fairly is the subject of the next part.

A score is a claim someone should be able to check

The best-known statement of this idea comes from research evaluation. In 2015 five researchers published the Leiden Manifesto in Nature, ten principles for using metrics to judge scientists and universities [1]. Two of them are the core of this article.

Principle 4 says: "Keep data collection and analytical processes open, transparent and simple." The authors add that "no one should accept a black-box evaluation machine" [1]. Principle 5 says: "Allow those evaluated to verify data and analysis." Everyone included in a study, the manifesto says, should be able to check that their outputs have been correctly identified [1].

The manifesto also shows why openness pays. In 2010 a public debate about the technical properties of an indicator used by one of the authors' own groups, the Centre for Science and Technology Studies at Leiden University, led to a revision in how that indicator was calculated [1]. Outsiders could find the flaw because the method was published.

Part 1: publish the formula, and the inputs

A formula alone is half an explanation. The other half is the data that goes into it.

The U.S. News ranking of American universities shows both halves. Its methodology is public: according to a 2022 analysis by Michael Thaddeus, a mathematics professor at Columbia University, 20% of a university's ranking came from a survey of peers and the remaining 80% was "based entirely on numerical data collected by the institution itself" [2]. Columbia had climbed from 18th place on its debut in the ranking in 1988 to 2nd [2].

Because the measures were named, Thaddeus could test them. He compared figures Columbia had reported, such as class sizes and the student-faculty ratio, "with figures computed by other means, drawing on information made public by Columbia elsewhere" [2]. One example: Columbia reported that 82.5% of its undergraduate classes had fewer than 20 students, far above its competitors in the top 100 [2]. He concluded that several key figures behind the ranking were "inaccurate, dubious, or highly misleading" [2].

In September 2022 Columbia fell to 18th. CBS New York reported that the university had admitted submitting inaccurate data in earlier years, and that U.S. News had used data from alternative sources for the new ranking [3].

A published method made the outside check possible. But the inputs were self-reported, so checking them was left to an outsider. Thaddeus also noted that U.S. News disclosed only two of the five class-size percentages it used [2]. An explainable ranking publishes the method, shows which inputs came from where, and says which of them it has verified.

Part 2: state the weights, and the reasons for them

When a ranking combines several measures into one score, the weights decide what it values. European law asks for something close to this in a neighbouring area. Article 27 of the Digital Services Act requires online platforms that use recommender systems to set out, "in plain and intelligible language, the main parameters" used in them [4]. Those parameters must include "the criteria which are most significant" in deciding what is suggested and "the reasons for the relative importance of those parameters" [4]. The law covers platform recommender systems, not every list of experts, but the standard is useful: name what counts most, and say why.

Some ranking services are moving this way. Favikon, which ranks creators across several networks, published its rebuilt formula on 3 November 2026, noting: "A number with that much weight should come with the working shown" [5]. The score is now Reach 30%, Engagement 30%, Audience 30% and Consistency 10%, each scored from 0 to 100, multiplied by an authenticity factor between ×0.50 and ×1.15. The company says it "now runs on one formula, with the same weights for every creator on every network" [5]. For now the factor applies only on LinkedIn [5].

Published weights do not make a ranking right. They make it arguable. A reader can now see that audience size carries as much weight as engagement, and decide whether that suits their question. The reasons matter as much as the numbers: "consistency counts 10% because we do not want to reward posting volume" is an argument a reader can test.

The gaming objection

A common reason for keeping a formula secret is that publishing it would help people game it. The objection is partly true. The Leiden Manifesto answers the larger part of it. Its ninth principle warns that indicators change the system through the incentives they create, and that "a single one will invite gaming and goal displacement" [1]. The defence it offers is a suite of measures, not secrecy. Several measures that are hard to fake together resist farming even when everyone knows the formula; one count is easy to farm either way.

Secrecy has its own cost. The people most likely to work out a hidden formula are those who will spend hours testing what moves their score. Honest participants are left guessing.

My suggestion is a middle line: publish the formula and the weights; keep the exact signals used to detect fake engagement private, but publish that the check exists, what it can do to a score and how to contest it. Favikon's page does this in part: it states the authenticity factor's range and the score at which it has no effect, about 78 [5]. A reader knows the maximum penalty without receiving a recipe for avoiding it.

Part 3: a breakdown for every entry

A published formula explains the ranking in general. It does not tell number 31 why she is 31.

Stack Overflow's reputation system is a plain example. Its Help Center lists every event that changes a user's reputation and by how much: an upvoted answer earns +10, an accepted answer +15, a downvoted answer costs −2 [6]. There is a daily limit of 200 points from votes and edits. Reversed votes reverse the points, and so do votes removed for voting fraud [6]. Users have a reputation page that lists each change, including entries such as "voting corrected" [6]. Any score can be traced back to the events that made it.

An expert ranking is more complex than a points tally, but the principle carries over. For each entry, show:

  • the score for each component, and how much each one contributed to the total after weighting;
  • the time window the data covers and the date it was last updated;
  • any penalty or adjustment applied, and the published rule behind it;
  • the sources the data came from, and which of them were checked by a person.

A breakdown also has to explain movement. A rank can change because the person's numbers changed, because other people's numbers changed, or because the method changed. Favikon's page addresses one cause: it rebuilds its comparison groups on the first of each month, and in between "your percentile only moves when your own numbers move" [5]. A ranking that shows "up 4 places" without saying which of those three things happened is explaining nothing.

Part 4: an appeal path that can change the result

If someone on the list believes a figure is wrong, where do they go, and what happens next? The law already sets a baseline for some cases. Under the GDPR, people have the right to have inaccurate personal data about them corrected "without undue delay", and incomplete data completed, including by "a supplementary statement" (Article 16) [7]. Under Article 21 they can object at any time to processing based on legitimate interests, "including profiling", and the controller must then show compelling legitimate grounds to continue [7]. Whether a given ranking falls under the GDPR depends on who runs it and whom it covers; this is not legal advice.

The Digital Services Act offers a model for the process. Article 20 requires online platforms to give users, for at least six months after decisions such as restricting the visibility of their content or their ability to earn from it, "an effective internal complaint-handling system" that works "electronically and free of charge" [8]. A ranking is not such a decision, but the elements carry over: a clear place to complain, no cost, a time limit and a human answer.

In practice an appeal form for a ranking needs three separate doors, because they lead to different work:

  1. Correct my data. A missing account, a wrong figure, an old job title.
  2. Dispute my category or the method. "I am a researcher, not a trader", or "your engagement measure ignores my newsletter". The answer may change the method for everyone.
  3. Remove me. An objection to being ranked at all, which deserves a decision and a written reason.

Two safeguards make the path honest. Requests should be verified, so that a rival cannot remove a competitor by pretending to be them. And corrections should be logged in a public changelog, so that readers can see the ranking change and why.

What explainable does not mean

Explainable does not mean correct. A ranking can show every step and still measure the wrong thing. Explanation makes mistakes visible; it does not prevent them. The Leiden Manifesto's tenth principle applies here: "Scrutinize indicators regularly and update them" [1].

Nor does it mean exposing everything. A breakdown should report measurable things in neutral terms. "Discussion relative to audience: low" is a figure the person can check and argue with. "Bot farm" is an accusation, and a ranking that publishes one owes the person evidence and a hearing. Suspected fraud belongs in an internal review queue, not on the public page.

A worked example (fictional)

The ranking, the analyst and all figures below are invented to show the four parts in use.

A publication runs a monthly top 50 of independent market analysts. Its methodology page names four components and weights: depth on the topic 35%, quality of discussion 30%, engagement relative to audience 20%, consistency 15%. A note explains why audience size has no separate weight: the list is meant to find analysts worth reading, not the most followed ones.

An analyst, call her Analyst K, is at number 31 with a score of 61. Her entry shows the breakdown:

Component Weight Her score (0–100) Contribution
Topic depth 35% 84 29.4
Discussion quality 30% 48 14.4
Engagement relative to audience 20% 55 11.0
Consistency 15% 41 6.2
Total     61.0

The entry says the data covers the 90 days to 1 November and no penalty was applied. Analyst K sees where she loses points, consistency and discussion, and why. The ranking counts only her posts on X, and most of her work and her replies to readers are in a weekly newsletter.

She uses the second door, "dispute the method". The editors agree that the newsletter is part of her published work, add newsletters as a source for every analyst who links one, and record the change in the changelog. Her consistency and discussion scores rise, and so do those of six other analysts. Next month she is at number 18, and the changelog says why.

Five questions to ask of any expert ranking

  1. Can I read the formula and see where the inputs come from? If not, it is an opinion with a number on it.
  2. Are the weights stated, with a reason for each? The weights are where the ranking's values live.
  3. Can I see why one entry has its score? Component scores, contributions, data window, last update and any penalty.
  4. Is there a way to correct, dispute or object, and does it change anything? Look for separate paths and a public log of changes.
  5. What changed since the last edition, and why? Was it their data, other people's data or the method?

What this article cannot tell you

The sources come from research evaluation, university rankings, EU law, a creator analytics company and a question-and-answer site. None measures whether explainable rankings are more accurate or more trusted, and I found no such study for expert or creator rankings. The Columbia figures come from one professor's analysis and a news report of the university's admission; I did not check the underlying data myself. Favikon's formula is described as the company publishes it; I could not check how it is applied to individual profiles, because the full breakdown is in its paid analytics product. The legal sections describe what the texts say, not how they apply to any ranking. The four-part structure and the middle line on gaming are my own synthesis. This is not legal, financial or investment advice.

How this article was made

Task What the AI did What a person did How it was checked
Research Collected material on creator rankings, scoring formulas and data rights into research notes Chose the question and the sources to rely on Each rule, figure and quotation checked against the original text or page [1]–[8]
Draft Wrote the first draft from the notes Edited the text, the structure and the worked example A separate review compared figures and claims with the sources
Images Generated the cover photograph-style illustration; built the diagrams from a code template Chose the illustration and the content of the diagrams Labelled as AI-generated in the disclosure line and alt text
Not done by AI — The decision to publish and responsibility for the text —

Conflicts of interest: none. The author has no commercial relationship with Favikon, U.S. News, Stack Overflow or the other organisations named; the ranking and the analyst in the example are fictional.

Responsible for this article: Dmitriy Lastovka, author. Corrections: contact form.

Sources

Labels: official (the organisation's own page or the legal text), report (a news report), paper (a published article or study), analysis (an independent analysis by a named author).

  1. Diana Hicks, Paul Wouters, Ludo Waltman, Sarah de Rijcke, Ismael Rafols, "Bibliometrics: The Leiden Manifesto for research metrics", Nature 520 (2015), 429–431, published 22 April 2015. https://doi.org/10.1038/520429a Paper (comment; full text read via an author-distributed copy).
  2. Michael Thaddeus, "An Investigation of the Facts Behind Columbia's U.S. News Ranking", February 2022, revised March 2022. https://www.math.columbia.edu/~thaddeus/ranking/investigation.html Analysis.
  3. CBS New York, "Columbia University drops from 2nd to 18th on annual Best Colleges list", 12 September 2022. https://www.cbsnews.com/newyork/news/columbia-university-best-colleges-ranking-2022/ Report.
  4. Regulation (EU) 2022/2065 (Digital Services Act), Article 27, "Recommender system transparency". https://eur-lex.europa.eu/eli/reg/2022/2065/oj Official.
  5. Ailin Oyhanarte, "How the Favikon Score Works (With Examples)", Favikon, 3 November 2026. https://www.favikon.com/rankings/favikon-methodology-algorithm Official.
  6. Stack Overflow, "What is reputation? How do I earn (and lose) it?", Help Center. https://stackoverflow.com/help/whats-reputation Official.
  7. Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 16 and 21. https://eur-lex.europa.eu/eli/reg/2016/679/oj Official (text read via gdpr-info.eu).
  8. Regulation (EU) 2022/2065 (Digital Services Act), Article 20, "Internal complaint-handling system". https://eur-lex.europa.eu/eli/reg/2022/2065/oj Official.

Tier I

Sector: #18291

Sealed Hiding Place Room

Resource Cache

Resource Cache

Tier I

Requires 25% Tier Progress to Claim
Meme Cache

Meme Cache

Tier I

Requires 50% Tier Progress to Claim
Equipment Cache

Equipment Cache

Tier I

Requires 75% Tier Progress to Claim

After collecting, hiding places will be stored in your inventory and can be opened with Keys.

Other articles

Audience Size Is Not Topic Authority

A large following records popularity, not knowledge of a subject. Why reach and authority need separate evidence, and five questions to ask about any expert.

user avatarme1afon

Spotting Gaming in Creator Program Metrics

Any number that pays creators attracts people who produce the number, not the work. Three ways metrics get gamed, seven signs, and why a sign is not proof.

user avatarme1afon

How Creator Incentives Change What Gets Published

Six documented cases show how platform incentives change what creators publish, plus five questions to ask before you design or join a creator programme.

user avatarme1afon

Measuring an Author’s Contribution Beyond Output

The easiest number about an author is how much they published, and it says the least. Five layers of contribution, and what each one can and cannot show.

user avatarme1afon

Why Persuasion Principles Are Not Universal Laws

Some classic persuasion effects still hold after 45 years; others shrank to almost nothing when large teams repeated them.

user avatarme1afon

When Rewards Crowd Out Motivation

A fine for late parents doubled lateness. Research on incentives shows when a reward replaces the reason people acted, and when it helps.

user avatarme1afon

Important disclaimer: The information presented on the Dapp.Expert portal is intended solely for informational purposes and does not constitute an investment recommendation or a guide to action in the field of cryptocurrencies. The Dapp.Expert team is not responsible for any potential losses or missed profits associated with the use of materials published on the site. Before making investment decisions in cryptocurrencies, we recommend consulting a qualified financial advisor.