Testing Whether a Topic Radar Gives You a Head Start

Testing Whether a Topic Radar Gives You a Head Start

user avatar

by me1afon

12 minutes ago


AI tools helped gather research, pull the public records used in the example and write the first draft of this article. I edited it and checked every figure and quote against the source page listed at the end. The cover illustration is AI-generated. Details at the end.

A topic radar is any routine that tells you "people are starting to care about this" before the topic is obvious: a watch on search trends, forum threads, repository growth or the questions in your own inbox. Earlier parts of this series looked at what such signals measure and how they mislead. This part asks the next question: how do you check that your radar really notices interest earlier than you would have noticed it anyway?

The usual answer is to look back at topics that became big and see that the radar "saw them coming". That test cannot fail. Every topic you remember is one that made it, while the alerts that led nowhere and the topics the radar never flagged are forgotten. A fair test is built so that it can come out badly.

Below is a short method, then a small worked example made from real, dated public records. It shows how the numbers are produced and where they bend. It is not a measurement of how good any radar is.

Why "it saw it coming" proves little

Google Flu Trends is the best-known cautionary story. As Part 1 described, it was built from search terms matched to official flu statistics. David Lazer and colleagues, in a 2014 paper in Science, describe the method as finding "the best matches among 50 million search terms to fit 1152 data points", and conclude that the early version was "part flu detector, part winter detector" [5]. They also report that, from 21 August 2011 to 1 September 2013, it reported too-high flu prevalence "100 out of 108 weeks" [5]. A signal that fits the past well can still fail on the future, and the more candidate signals you try, the easier it is to find one that fits.

A radar tuned after the fact has the same problem in a milder form. Pick the signal, the threshold and the success event while looking at the answers, and you will find a combination that "worked". Three habits follow:

  • Decide the test before you look at the results, and write it down with a date.
  • Include topics that did not take off. A list of winners can produce hits, never false alarms.
  • Judge a radar by what it costs as well as what it catches. An alert that fires on everything is early and useless.

Four things to fix before you look

Choice Question it answers Example
Signal What do we watch? Hacker News stories with the topic's name in the title
Alert rule When does the radar say "now"? The third story with 20 or more points
Milestone and deadline What counts as "interest became obvious", and by when? The English Wikipedia article for the topic exists, within a stated number of months
Candidate list Which topics do we run the test on? Names written down before looking up any dates, and never added to afterwards

The milestone is the choice people rush. It must be an event with a date you can look up and did not influence. "The topic got big" is not a date. A first article in an outlet, an encyclopaedia page, a product launch or a spike above a stated level in a public metric can all work, if defined in advance.

Score three things, not one

Lead time is the interval between the alert and the milestone. It is the headline number, but it needs two companions. Forecast verification, the field that tests weather and similar predictions, has names for them [4]:

  • Hit rate (probability of detection): "What fraction of the observed 'yes' events were correctly forecast?" In radar terms, of the topics that reached the milestone, how many had an alert first.
  • False alarm ratio: "What fraction of the predicted 'yes' events actually did not occur (i.e., were false alarms)?" Of all alerts, how many never reached the milestone.
  • Lead time for the hits, with the spread, not only an average.

The verification guide adds that the false alarm ratio ignores misses and should be used together with the hit rate [4]. A radar can reach a perfect hit rate by alerting on everything; the false alarm ratio exposes it.

A worked example, from real records

The data here was pulled on 18 November 2026 from two public sources, with the scripts kept alongside the article's research notes. The rules were fixed before the first run.

  • Signal: stories posted to Hacker News with at least 20 points and the topic's name in the title, found through the public Hacker News search API [1].
  • Alert rule A: the day the third such story is posted. Rule B, the looser one: the day the first such story is posted.
  • Milestone: the first revision in the page history of the topic's current English Wikipedia article, read through the MediaWiki API [2]. This is a stand-in for "the topic was recognised as established enough for an encyclopaedia entry". It is a proxy with known problems, discussed below.
  • Candidate list: 19 names of AI tools and ideas that were discussed on Hacker News between 2022 and 2025, chosen by the AI research assistant from general knowledge of the field, not drawn from a neutral list. Twelve were listed before the first lookup. Seven more were added after that first run, which is a flaw: the list was not frozen. One name, Rust, was set aside after its article date (2010) turned up. That leaves 18 names. Without the seven late additions, the looser rule B would have had 3 false alarms out of 11 alerts (27%) against 20% for rule A, so the size of the contrast below depends partly on that late batch.

Seven of the eighteen fired rule A and have a clearly dated Wikipedia article, so a lead time can be computed:

Topic Rule A alert Wikipedia article Lead, rule A Lead, rule B
Stable Diffusion 26 Aug 2022 31 Aug 2022 5 days 21 days
LangChain 2 Feb 2023 18 Apr 2023 75 days 90 days
llama.cpp 15 Mar 2023 13 May 2024 425 days 430 days
AutoGPT 13 Apr 2023 16 Apr 2023 3 days 14 days
Ollama 26 Sep 2023 29 Apr 2026 946 days 1,014 days
Model Context Protocol 29 Jan 2025 14 Apr 2025 75 days 140 days
Vibe coding 1 Mar 2025 3 Mar 2025 2 days 28 days

Read quickly, this is a success: the radar fired first seven times out of seven, with a median lead of 75 days under rule A. Read carefully, it says much less.

A dated lead-time chart: for seven topics, the days between a Hacker News alert and the current English Wikipedia article, under the strict rule A and the looser rule B. Three of the seven leads are 5 days or less under rule A; llama.cpp and Ollama are over a year.

The lead depends on the milestone more than on the radar

Three leads are five days or fewer. For Stable Diffusion, AutoGPT and vibe coding the encyclopaedia article appeared almost as soon as the discussion did, and a writer could not have used such a head start. Two leads are over a year, because the articles for llama.cpp and Ollama arrived long after the tools were widely discussed. Same radar, same rule, a spread of 2 days to 946 days. Much of that spread probably reflects how fast Wikipedia editors reacted, not how well the radar worked.

The milestone itself is also soft. Wikipedia logs show that several articles were first created, then moved out of the main space [3] by an editor who judged them not ready, and that earlier drafts existed in draft space: Draft:Llama.cpp from April 2024, Draft:Ollama from July 2024 and Draft:Model Context Protocol from 2 January 2025, which is before the rule A alert of 29 January 2025. If drafts counted, Model Context Protocol would not be a hit at all. The table uses the current article page only. For Model Context Protocol the first attempt in the main space, on 10 April 2025, was moved to draft space the same day with the reason "more sources needed", and the article came back on 14 April [2]. For Ollama the first attempt in the main space, on 6 August 2025, was moved to draft space within 19 minutes, and the page that stands now was created on 29 April 2026. If that first attempt counts, Ollama's lead is 680 days, not 946. For llama.cpp the first main-space attempt, on 13 May 2024, was moved to draft space the next day, while the page history that exists now begins on that date. Choose the rule in advance and say which one you used.

The looser rule finds more, and is wrong more often

Under rule A, ten of the eighteen names fired. Eight of them have an English Wikipedia article (the seven above, plus LlamaIndex, whose page history does not show a clear creation date and is left out of the table). Two did not: GPT4All has no article with that title, and vLLM's page is a redirect to a section of an article on the transformer architecture. That makes the rule A hit and false alarm figures, counted on this list, eight and two. The false alarm ratio is 2 of 10, or 20%.

Under the looser rule B, every one of the eighteen names fired, because each had at least one story with 20 points. Rule B also has eight hits, but now ten false alarms: a false alarm ratio of 10 of 18, about 56%. In exchange, the leads are longer, by 5 to 68 days for the seven dated topics. An earlier alert comes from a more trigger-happy rule, and the false alarms are the price. Without the false alarm count, rule B would look strictly better.

Two alert rules scored on the same 18 names: rule A (third qualifying story) fired 10 times with 8 hits and 2 false alarms; rule B (first qualifying story) fired 18 times with 8 hits and 10 false alarms. Misses: none found, but the list was not neutral, so misses could not be measured.

The two false alarms also show how a proxy can fail. vLLM is widely used and has had 18 qualifying stories on Hacker News since 2023, yet "no stand-alone Wikipedia article" does not mean "no real interest". A hit or false alarm is only as good as the milestone you chose.

What this example cannot tell you

  • Misses are not measured. A miss is a topic that became big without the radar flagging it. The candidate list came from general knowledge of the field, and a topic you can think of is one already noticed. The honest result is "no misses found", not "no misses".
  • The list is small and chosen. Eighteen names, all about AI tools, chosen by the AI research assistant from general knowledge of the field. The figures describe this list. They say nothing about what a similar radar would do on another subject.
  • There was no deadline. Ollama counts as a hit after 946 days. With a one-year limit, llama.cpp and Ollama would not be hits, and very recent alerts cannot be judged yet. A real test states the deadline first.
  • One signal, one platform, one language. Hacker News is a developer forum, the milestone is the English Wikipedia, and the points are those at the time of the query.
  • The thresholds are not recommendations. The numbers 3 and 20 were fixed once, before the first run, as an example of a written rule.
  • The data comes from live services. Points, titles and page histories can change. The dates quoted are those returned on 18 November 2026.
  • There is no comparison with what you would have noticed anyway. The example does not show that someone reading Hacker News daily would have lagged behind the rule.

A test you can run on your own radar

This is a working method drawn from the sources above, not a tested model.

  1. Write the signal, the alert rule and the milestone down, with the date. Define what counts as a "mention" or a "spike", set a deadline for the milestone, and do not change a number after the first run.
  2. Build the candidate list from a source that does not know the outcome: every item on a given list during a period, or every name in your notes at a past date, not the topics you remember. Keep the ones that went nowhere.
  3. Run the rule and record the alert date for every item. Use the stated rule only. If an item never fires, mark it "no alert".
  4. Look up the milestone date from a record you can reopen, such as a page history, release log or publication date. Note how you handled drafts and renamed pages.
  5. Count and compare. Hits, false alarms, misses and the lead time for each hit, with the shortest and the longest. Report a range along with the middle value.
  6. Try the lazier options. A looser version of the rule, a stricter one, and a plain baseline such as "alert on the first mention". If a loose rule has the same hits and longer leads, the false alarms must decide, as above.
  7. Run it forward. A retrospective set is a start. Keep a dated log of alerts from today and check them at a fixed later date. A test whose answer you have not seen cannot be flattered by hindsight, and the lesson about fitted signals applies again [5].

If the radar passes, you have a measured claim: "under this rule it fired before the milestone in this share of cases, with this many false alarms". That is smaller than "it works", and more useful.

How this article was made

Task What the AI did What a person did How it was checked
Research Collected sources on forecast verification, forecasting with search data and Wikipedia page history into research notes; ran the queries to the Hacker News search API and the MediaWiki API and computed the lead times Chose the question and approved the plan; the AI proposed the rules and the milestone before the first run. The candidate list was proposed in two batches, the second after the first run (see the example) Quotes checked against the pages and papers [3]–[5]; dates compared with the raw query results and page logs [1][2][6]
Draft Wrote the first draft from the notes and the results Edited the text, the method and the limits A separate review compared figures and claims with the sources and the raw results
Images Generated the suprematist cover illustration (from the Dapp.Expert cover library) Chose the illustration; designed the explanatory diagrams from the table values Labelled as AI-generated in the disclosure line and alt text
Not done by AI — The decision to publish and responsibility for the text —

Conflicts of interest: none. The author has no commercial relationship with Y Combinator, Algolia, the Wikimedia Foundation, the projects named in the example or the researchers cited.

Responsible for this article: Dmitriy Lastovka, author. Corrections: contact form.

Sources

Labels: official (the organisation's own page), paper (study, read via the authors' full text), records (public data queried for this article).

  1. Hacker News Search, Algolia, public search API (endpoint search_by_date, with a query on the story title, tags=story and numericFilters=points>=20). https://hn.algolia.com/api Records. The documentation page needs JavaScript; the queries were run against the endpoint on 18 November 2026.
  2. MediaWiki, "API:Logevents" and the English Wikipedia Action API (page revisions and logs of creations, moves and deletions). https://www.mediawiki.org/wiki/API:Logevents Official.
  3. Wikipedia, "Wikipedia:Drafts" (moving an article not ready for the main namespace to draft space). https://en.wikipedia.org/wiki/Wikipedia:Drafts Official.
  4. WWRP/WGNE Joint Working Group on Forecast Verification Research, "Forecast verification: issues, methods and FAQ" (probability of detection and false alarm ratio). https://jwgfvr.github.io/forecastverification/ Official.
  5. David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani, "The Parable of Google Flu: Traps in Big Data Analysis", Science 343, 1203–1205 (14 March 2014). https://doi.org/10.1126/science.1248506 Paper, full text via the authors' copy.
  6. English Wikipedia page histories and logs (including Draft: pages) for Stable Diffusion, LangChain, Llama.cpp, AutoGPT, Ollama, Model Context Protocol, Vibe coding, LlamaIndex, GPT4All and VLLM (for example https://en.wikipedia.org/w/index.php?title=Ollama&action=history). Records.

Tier I

Sector: #18291

Sealed Hiding Place Room

Resource Cache

Resource Cache

Tier I

Requires 25% Tier Progress to Claim
Meme Cache

Meme Cache

Tier I

Requires 50% Tier Progress to Claim
Equipment Cache

Equipment Cache

Tier I

Requires 75% Tier Progress to Claim

After collecting, hiding places will be stored in your inventory and can be opened with Keys.

Other articles

Who Is Missing From Your Audience Research?

Public discussions, search trends and analytics show who spoke, searched or clicked. Who each source leaves out, and how to record what the data cannot see.

user avatarme1afon

When Developer Popularity Signals Are Misleading

GitHub stars, package downloads and Stack Overflow activity count something narrower than interest: a bookmark, a file request, a visit. How to cross-check.

user avatarme1afon

Combining Public Trends With Your Own Search Data

Google Trends shows how the market searches; Search Console shows the searches that reach you. How the numbers differ and how to line them up honestly.

user avatarme1afon

A Trend or a Spike? Reading Early Interest Signals

A topic jumps on Google Trends: a lasting trend or a one-week spike? What trend tools measure, where they failed, and five checks before you write about it.

user avatarme1afon

Which Social Analytics Tools Can You Actually Use?

Lists of social analytics tools are long; the tools one author can open today are few. Five access gates decide it, checked on 16 November 2026.

user avatarme1afon

Reading the Conversation Behind Engagement Counts

A reply count says how many replies came in, not what they said. Where X, LinkedIn and Telegram keep the conversation, and a six-way sort for it.

user avatarme1afon

Important disclaimer: The information presented on the Dapp.Expert portal is intended solely for informational purposes and does not constitute an investment recommendation or a guide to action in the field of cryptocurrencies. The Dapp.Expert team is not responsible for any potential losses or missed profits associated with the use of materials published on the site. Before making investment decisions in cryptocurrencies, we recommend consulting a qualified financial advisor.