Shipped or hype? I built an AI news email that can't make up links
TL;DR
I built a weekly AI news email that reads 12 feeds and checks podcast claims against real sources. The model only picks item numbers. Python adds every link, and a quote must really be in the transcript. One run costs $0.24.
- 12feeds read
- 80items this week
- 4podcast claims checked
- 83tests
Why I built it
Every week I listen to the Moonshots podcast. Great show, but full of big claims: AGI is here, compute shortage in 3 years, agents switching off their own safety monitors.
I want to stay ahead in AI, and I build real things for clients. So I need to know: what really shipped, and what is only talk?
Asking an LLM to summarize the news is easy. But LLMs make up links and quotes, and one fake link kills the trust in everything else.
How it works
- 0112 feedsrelease notes, builders, papers, podcasts
- 0280 itemslast 7 days, nothing already sent
- 03AI picks numbersno URLs in, item numbers out
- 04Python adds linksan unknown number is dropped
- 05Sunday email8 PM Chicago, only to me
How it stays honest
- 01
Links
The model sees numbered items with no URLs and answers in strict JSON with numbers only. Python attaches the real link. A URL the model writes anyway is deleted.
- 02
Quotes
The model picks podcast claims with the exact words. If the quote is not really in the transcript, the claim is thrown away.
- 03
Labels
Backed only when a real source confirms the names, numbers and dates. A podcast can never back another podcast.
- 04
Failure
Model down or the key at its limit? I still get the email, with links only and the reason on top.
A real email
The transcripts come from my publisher app (this week: 117,700 and 142,617 characters). Two claims from one real run, word for word:
From a real AI Radar email
“Chamath says AGI has already been achieved, but it can't diffuse because infrastructure build-out, not technology, is the bottleneck.”
Quote: “I do think that we have achieved AGI. It cannot diffuse. And the reason it cannot diffuse has nothing to do with the technology. It has everything to do with the build up.”
What the evidence says: No item tests the AGI or diffusion claim. Anthropic S-1 coverage shows fast revenue growth but says nothing about infrastructure as the bottleneck.
“OpenAI reported that its coding agents disabled safety monitors in two internal incidents and lied about it, with no satisfying explanation.”
Quote: “This month, in two internal incidents, its coding agents switched off its monitors. I'll say it again, it switched off its monitors.”
What the evidence says: No item reports these incidents. OpenAI's post on securing its training environment covers stronger containment, but says nothing about agents switching off monitors.
What broke
- Bug 1
BACKED was a lie
Evidence against a claim got the label BACKED. Fix: the model must say backs, partly or contradicts.
- Bug 2
Half true is not true
A Linux kernel claim was BACKED, but the sources showed only the trend. Fix: a new PARTLY BACKED label.
The tests didn't catch either one. Reading the real email did. For LLM work you need both.
What it costs
The code is open on GitHub, MIT license. Change the feeds and the profile, and it is your own radar.
Built with
- Python
- OpenRouter
- Claude Sonnet 5.5
- Postgres
- systemd
Sources
- OpenRouter: structured outputs (strict JSON schema) (opens in a new tab)
- Claude Sonnet 5.5 on OpenRouter (pricing) (opens in a new tab)
- defusedxml: safe XML parsing in Python (opens in a new tab)
- Simon Willison's weblog (one of the 12 feeds) (opens in a new tab)
- Moonshots with Peter Diamandis (the podcast it checks) (opens in a new tab)
Get the next note by email
One short email when a new note is live. What I built, what broke, real numbers. About every 2 weeks.
Have a problem like this? Get your free AI blueprint in about a minute.
