AI tools helped gather research, pull the public records used in the example and write the first draft of this article. I edited it and checked every figure and quote against the source page listed at the end. The cover illustration is AI-generated. Details at the end.
A topic radar is any routine that tells you "people are starting to care about this" before the topic is obvious: a watch on search trends, forum threads, repository growth or the questions in your own inbox. Earlier parts of this series looked at what such signals measure and how they mislead. This part asks the next question: how do you check that your radar really notices interest earlier than you would have noticed it anyway?
The usual answer is to look back at topics that became big and see that the radar "saw them coming". That test cannot fail. Every topic you remember is one that made it, while the alerts that led nowhere and the topics the radar never flagged are forgotten. A fair test is built so that it can come out badly.
Below is a short method, then a small worked example made from real, dated public records. It shows how the numbers are produced and where they bend. It is not a measurement of how good any radar is.
Why "it saw it coming" proves little
Google Flu Trends is the best-known cautionary story. As Part 1 described, it was built from search terms matched to official flu statistics. David Lazer and colleagues, in a 2014 paper in Science, describe the method as finding "the best matches among 50 million search terms to fit 1152 data points", and conclude that the early version was "part flu detector, part winter detector" [5]. They also report that, from 21 August 2011 to 1 September 2013, it reported too-high flu prevalence "100 out of 108 weeks" [5]. A signal that fits the past well can still fail on the future, and the more candidate signals you try, the easier it is to find one that fits.
A radar tuned after the fact has the same problem in a milder form. Pick the signal, the threshold and the success event while looking at the answers, and you will find a combination that "worked". Three habits follow:
- Decide the test before you look at the results, and write it down with a date.
- Include topics that did not take off. A list of winners can produce hits, never false alarms.
- Judge a radar by what it costs as well as what it catches. An alert that fires on everything is early and useless.
Four things to fix before you look
| Choice | Question it answers | Example |
|---|---|---|
| Signal | What do we watch? | Hacker News stories with the topic's name in the title |
| Alert rule | When does the radar say "now"? | The third story with 20 or more points |
| Milestone and deadline | What counts as "interest became obvious", and by when? | The English Wikipedia article for the topic exists, within a stated number of months |
| Candidate list | Which topics do we run the test on? | Names written down before looking up any dates, and never added to afterwards |
The milestone is the choice people rush. It must be an event with a date you can look up and did not influence. "The topic got big" is not a date. A first article in an outlet, an encyclopaedia page, a product launch or a spike above a stated level in a public metric can all work, if defined in advance.
Score three things, not one
Lead time is the interval between the alert and the milestone. It is the headline number, but it needs two companions. Forecast verification, the field that tests weather and similar predictions, has names for them [4]:
- Hit rate (probability of detection): "What fraction of the observed 'yes' events were correctly forecast?" In radar terms, of the topics that reached the milestone, how many had an alert first.
- False alarm ratio: "What fraction of the predicted 'yes' events actually did not occur (i.e., were false alarms)?" Of all alerts, how many never reached the milestone.
- Lead time for the hits, with the spread, not only an average.
The verification guide adds that the false alarm ratio ignores misses and should be used together with the hit rate [4]. A radar can reach a perfect hit rate by alerting on everything; the false alarm ratio exposes it.
A worked example, from real records
The data here was pulled on 18 November 2026 from two public sources, with the scripts kept alongside the article's research notes. The rules were fixed before the first run.
- Signal: stories posted to Hacker News with at least 20 points and the topic's name in the title, found through the public Hacker News search API [1].
- Alert rule A: the day the third such story is posted. Rule B, the looser one: the day the first such story is posted.
- Milestone: the first revision in the page history of the topic's current English Wikipedia article, read through the MediaWiki API [2]. This is a stand-in for "the topic was recognised as established enough for an encyclopaedia entry". It is a proxy with known problems, discussed below.
- Candidate list: 19 names of AI tools and ideas that were discussed on Hacker News between 2022 and 2025, chosen by the AI research assistant from general knowledge of the field, not drawn from a neutral list. Twelve were listed before the first lookup. Seven more were added after that first run, which is a flaw: the list was not frozen. One name, Rust, was set aside after its article date (2010) turned up. That leaves 18 names. Without the seven late additions, the looser rule B would have had 3 false alarms out of 11 alerts (27%) against 20% for rule A, so the size of the contrast below depends partly on that late batch.
Seven of the eighteen fired rule A and have a clearly dated Wikipedia article, so a lead time can be computed:
| Topic | Rule A alert | Wikipedia article | Lead, rule A | Lead, rule B |
|---|---|---|---|---|
| Stable Diffusion | 26 Aug 2022 | 31 Aug 2022 | 5 days | 21 days |
| LangChain | 2 Feb 2023 | 18 Apr 2023 | 75 days | 90 days |
| llama.cpp | 15 Mar 2023 | 13 May 2024 | 425 days | 430 days |
| AutoGPT | 13 Apr 2023 | 16 Apr 2023 | 3 days | 14 days |
| Ollama | 26 Sep 2023 | 29 Apr 2026 | 946 days | 1,014 days |
| Model Context Protocol | 29 Jan 2025 | 14 Apr 2025 | 75 days | 140 days |
| Vibe coding | 1 Mar 2025 | 3 Mar 2025 | 2 days | 28 days |
Read quickly, this is a success: the radar fired first seven times out of seven, with a median lead of 75 days under rule A. Read carefully, it says much less.

The lead depends on the milestone more than on the radar
Three leads are five days or fewer. For Stable Diffusion, AutoGPT and vibe coding the encyclopaedia article appeared almost as soon as the discussion did, and a writer could not have used such a head start. Two leads are over a year, because the articles for llama.cpp and Ollama arrived long after the tools were widely discussed. Same radar, same rule, a spread of 2 days to 946 days. Much of that spread probably reflects how fast Wikipedia editors reacted, not how well the radar worked.
The milestone itself is also soft. Wikipedia logs show that several articles were first created, then moved out of the main space [3] by an editor who judged them not ready, and that earlier drafts existed in draft space: Draft:Llama.cpp from April 2024, Draft:Ollama from July 2024 and Draft:Model Context Protocol from 2 January 2025, which is before the rule A alert of 29 January 2025. If drafts counted, Model Context Protocol would not be a hit at all. The table uses the current article page only. For Model Context Protocol the first attempt in the main space, on 10 April 2025, was moved to draft space the same day with the reason "more sources needed", and the article came back on 14 April [2]. For Ollama the first attempt in the main space, on 6 August 2025, was moved to draft space within 19 minutes, and the page that stands now was created on 29 April 2026. If that first attempt counts, Ollama's lead is 680 days, not 946. For llama.cpp the first main-space attempt, on 13 May 2024, was moved to draft space the next day, while the page history that exists now begins on that date. Choose the rule in advance and say which one you used.
The looser rule finds more, and is wrong more often
Under rule A, ten of the eighteen names fired. Eight of them have an English Wikipedia article (the seven above, plus LlamaIndex, whose page history does not show a clear creation date and is left out of the table). Two did not: GPT4All has no article with that title, and vLLM's page is a redirect to a section of an article on the transformer architecture. That makes the rule A hit and false alarm figures, counted on this list, eight and two. The false alarm ratio is 2 of 10, or 20%.
Under the looser rule B, every one of the eighteen names fired, because each had at least one story with 20 points. Rule B also has eight hits, but now ten false alarms: a false alarm ratio of 10 of 18, about 56%. In exchange, the leads are longer, by 5 to 68 days for the seven dated topics. An earlier alert comes from a more trigger-happy rule, and the false alarms are the price. Without the false alarm count, rule B would look strictly better.

The two false alarms also show how a proxy can fail. vLLM is widely used and has had 18 qualifying stories on Hacker News since 2023, yet "no stand-alone Wikipedia article" does not mean "no real interest". A hit or false alarm is only as good as the milestone you chose.
What this example cannot tell you
- Misses are not measured. A miss is a topic that became big without the radar flagging it. The candidate list came from general knowledge of the field, and a topic you can think of is one already noticed. The honest result is "no misses found", not "no misses".
- The list is small and chosen. Eighteen names, all about AI tools, chosen by the AI research assistant from general knowledge of the field. The figures describe this list. They say nothing about what a similar radar would do on another subject.
- There was no deadline. Ollama counts as a hit after 946 days. With a one-year limit, llama.cpp and Ollama would not be hits, and very recent alerts cannot be judged yet. A real test states the deadline first.
- One signal, one platform, one language. Hacker News is a developer forum, the milestone is the English Wikipedia, and the points are those at the time of the query.
- The thresholds are not recommendations. The numbers 3 and 20 were fixed once, before the first run, as an example of a written rule.
- The data comes from live services. Points, titles and page histories can change. The dates quoted are those returned on 18 November 2026.
- There is no comparison with what you would have noticed anyway. The example does not show that someone reading Hacker News daily would have lagged behind the rule.
A test you can run on your own radar
This is a working method drawn from the sources above, not a tested model.
- Write the signal, the alert rule and the milestone down, with the date. Define what counts as a "mention" or a "spike", set a deadline for the milestone, and do not change a number after the first run.
- Build the candidate list from a source that does not know the outcome: every item on a given list during a period, or every name in your notes at a past date, not the topics you remember. Keep the ones that went nowhere.
- Run the rule and record the alert date for every item. Use the stated rule only. If an item never fires, mark it "no alert".
- Look up the milestone date from a record you can reopen, such as a page history, release log or publication date. Note how you handled drafts and renamed pages.
- Count and compare. Hits, false alarms, misses and the lead time for each hit, with the shortest and the longest. Report a range along with the middle value.
- Try the lazier options. A looser version of the rule, a stricter one, and a plain baseline such as "alert on the first mention". If a loose rule has the same hits and longer leads, the false alarms must decide, as above.
- Run it forward. A retrospective set is a start. Keep a dated log of alerts from today and check them at a fixed later date. A test whose answer you have not seen cannot be flattered by hindsight, and the lesson about fitted signals applies again [5].
If the radar passes, you have a measured claim: "under this rule it fired before the milestone in this share of cases, with this many false alarms". That is smaller than "it works", and more useful.
How this article was made
| Task | What the AI did | What a person did | How it was checked |
|---|---|---|---|
| Research | Collected sources on forecast verification, forecasting with search data and Wikipedia page history into research notes; ran the queries to the Hacker News search API and the MediaWiki API and computed the lead times | Chose the question and approved the plan; the AI proposed the rules and the milestone before the first run. The candidate list was proposed in two batches, the second after the first run (see the example) | Quotes checked against the pages and papers [3]–[5]; dates compared with the raw query results and page logs [1][2][6] |
| Draft | Wrote the first draft from the notes and the results | Edited the text, the method and the limits | A separate review compared figures and claims with the sources and the raw results |
| Images | Generated the suprematist cover illustration (from the Dapp.Expert cover library) | Chose the illustration; designed the explanatory diagrams from the table values | Labelled as AI-generated in the disclosure line and alt text |
| Not done by AI | — | The decision to publish and responsibility for the text | — |
Conflicts of interest: none. The author has no commercial relationship with Y Combinator, Algolia, the Wikimedia Foundation, the projects named in the example or the researchers cited.
Responsible for this article: Dmitriy Lastovka, author. Corrections: contact form.
Sources
Labels: official (the organisation's own page), paper (study, read via the authors' full text), records (public data queried for this article).
- Hacker News Search, Algolia, public search API (endpoint search_by_date, with a query on the story title, tags=story and numericFilters=points>=20). https://hn.algolia.com/api Records. The documentation page needs JavaScript; the queries were run against the endpoint on 18 November 2026.
- MediaWiki, "API:Logevents" and the English Wikipedia Action API (page revisions and logs of creations, moves and deletions). https://www.mediawiki.org/wiki/API:Logevents Official.
- Wikipedia, "Wikipedia:Drafts" (moving an article not ready for the main namespace to draft space). https://en.wikipedia.org/wiki/Wikipedia:Drafts Official.
- WWRP/WGNE Joint Working Group on Forecast Verification Research, "Forecast verification: issues, methods and FAQ" (probability of detection and false alarm ratio). https://jwgfvr.github.io/forecastverification/ Official.
- David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani, "The Parable of Google Flu: Traps in Big Data Analysis", Science 343, 1203–1205 (14 March 2014). https://doi.org/10.1126/science.1248506 Paper, full text via the authors' copy.
- English Wikipedia page histories and logs (including Draft: pages) for Stable Diffusion, LangChain, Llama.cpp, AutoGPT, Ollama, Model Context Protocol, Vibe coding, LlamaIndex, GPT4All and VLLM (for example https://en.wikipedia.org/w/index.php?title=Ollama&action=history). Records.










