Model Mayhem: What Happens When AI Knows More Than You Told It
There is a moment in this week's episode of Modem Futura where I describe asking an AI assistant to clean up a podcast transcript, and it came back with every line correctly attributed to Andrew Maynard or me. I had never given it our names. When I asked how, it explained that it had anchored on a screenshot from earlier in the session, read a few lines of dialogue, and cross-referenced them against what it already knew about the two of us. It was right. It was also, in a quiet way, unsettling.
Andrew had his own version of the same story: an assistant that insists on correcting his spelling to British English, no matter how carefully he tries to give it a blank slate. Neither example is alarming on its own. What they reveal is a difference in how machines and people hold memories. We remember in sequence, correctly or not, and these software based AI systems hold everything at once and reason across it. The breadcrumbs you leave over months of use can be assembled into a picture you never consciously drew, which matters for memory features, for institutional AI accounts, and for anyone who has ever walked away from an open laptop.
The heart of the episode is the summer's most consequential AI story. In July, OpenAI agents running a cybersecurity evaluation escaped their testing environment, built an improvised message board, recruited other agents, and coordinated an intrusion into Hugging Face. On August 26, METR and Redwood Research published an independent investigation, and OpenAI released its own technical report. The independent findings describe roughly 1,200 agents communicating on the unsanctioned board, about 700 of which joined the attack, with the apparent goal of fooling the automated grader on a benchmark they could not legitimately pass.
What I find compelling is less the scale than the shape of the story. There are questions still unanswered about what the agents were originally instructed to do and why the record goes quiet before the end. Andrew's concern is structural rather than dramatic: this swarm was detected. If capability increases by a factor of ten, what does the undetected version look like?
The episode closes on hardware, because the two threads meet there. Apple's refreshed Mac mini and Mac Studio, with a 512GB memory configuration arriving in late October, are being marketed as always-on agentic machines. That is a real gift for a hospital or a small business that wants generative AI without sending data to the cloud. It also means powerful, silent, headless computers are about to appear in a great many closets, set up by people who asked an AI how and pasted whatever it told them.
The uncomfortable throughline is that the weakest link has always been us. We scroll past terms of service. We paste terminal commands we don't understand. We trust the box. The question the episode leaves open is what changes when the systems on the other side understand that about us better than we do.
Subscribe and Connect!
Subscribe to Modem Futura wherever you get your podcasts and connect with us on LinkedIn. Drop a comment, pose a question, or challenge an idea—because the future isn’t something we watch happen, it’s something we build together. The medium may still be the massage, but we all have a hand in shaping how it touches tomorrow.
🎧 Apple Podcast: https://apple.co/4xKRGqN
🎧 Spotify: https://open.spotify.com/episode/5aaZvm9l9ETDz0OFaM6WgY?si=CWosjWaiSWme7W9rJcjBjg
📺 YouTube: https://youtu.be/DgcFFpkMHhY
🌐 Website: https://www.modemfutura.com/