Multi-Agent Summarization — From Content to Understanding

Only the needful

I think we’re all feeling the pace and rate of change, its impossible to keep up with all the amazing (not AI generated) content on AI.

Papers, substacks, blogs all dropping daily its, many of them interesting currently relevant. Yet I find I only have maybe an hour of real focus time for this stuff on a good day, and the pile never stops growing.

My ability to focus is a constraint I design around. I want to understand ideas deeply enough to use them, but I don’t need every tangent and aside to get there.

So I did the obvious thing: I asked an LLM to summarize stuff for me.

And it works — kind of. You get bullet points. Key themes. A tidy list of topics covered. I’d read through them, nod along, feel like I’d learned something. Then someone would ask me about the paper and I’d realize I couldn’t actually explain any of it. I knew what was discussed. I had no idea why it mattered. And I wasn’t sure I had gotten the right parts of the paper.

In unshocking news -- Most AI “summaries” are actually extractions — bullet points of key ideas, “the author discusses X, Y, Z,” sequential retelling in shorter form. It’s like highlighting every line in a textbook. Feels like studying. You retained nothing.

A real summary is different. Flowing prose that educates. Explains why ideas matter, provides context around the ideas, not just what they are. Organized by importance, not by the order things happened to appear. After reading one, you could actually riff on the ideas in a conversation.

The test is simple: can you teach it to someone? if yes, it was a summary. If you’d struggle, it was an extraction.

A single prompt tends toward extraction every time. Doing research and writing in the same prompt produces garbage — the research noise drowns out the writing. So I built four agents. The four horsemen of actually understanding what you read.

The Four Agents

The Planner

LLMs over-index on beginnings and endings of long content. The middle gets skimmed. I noticed this in my own summaries — they’d nail the intro and conclusion but miss the meatiest section.

The planner forces explicit analysis of the first third, middle third, and final third. It outputs a research TODO — not a summary, a plan for what needs to be looked up. Here’s what it produced for a DeepMind paper on AI delegation:

{
  “section_coverage”: {
    “first_third”: { “key_topics”: [”principal-agent problem”, “organizational parallels”] },
    “middle_third”: { “key_topics”: [”contract-first decomposition”, “trust systems”] },
    “final_third”: { “key_topics”: [”security threat model”, “ethical considerations”] }
  },
  “research_todo”: {
    “terms”: [{ “term”: “cognitive monoculture”, “priority”: “high” }],
    “people”: [{ “name”: “Simon Osindero”, “priority”: “medium” }],
    “technologies”: [{ “name”: “zk-SNARKs”, “priority”: “medium” }]
  }
}

HIGH priority gets looked up. MEDIUM if there’s room. LOW gets dropped. Not everything deserves a Perplexity call.

The Executor

Takes that plan and makes the MCP calls — terms to Perplexity, links to Firecrawl, people get bio lookups. Links are fetched one level deep only. When things fail, it notes the failure and moves on. I’d rather have a gap than a hallucination.

The Summarizer

Gets clean inputs — original content plus the structured research JSON. No tool logs, no failed fetches. The instructions I spent the most time on: organize by importance not appearance order (the best insight might be on page 12 — it should lead), write prose not bullets, and weave research into the narrative instead of bolting it on as a glossary.

I ran /summarize on Tom Tunguz’s post about building with AI agents. The TLDR it produced:

“Tom Tunguz distills a year of hands-on AI agent building into nine practical lessons: start with frontier models then specialize downward, use static typing as a guardrail against hallucination, pit multiple AI models against each other for quality, and build closed-loop systems that optimize themselves. The overarching theme is that AI agents have crossed a capability threshold where the competitive axis is shifting from raw intelligence to cost, operability, and system design.”

Compare that to what a single prompt gives you: “This article discusses nine observations about building AI agents.” One teaches. The other describes.

The Validator

Generates comprehension questions a professor would ask — foundation (”What is contract-first decomposition?”), core argument (”Why is delegation a governance problem?”), application (”How would this handle a compromised agent?”), synthesis (”How does the trust system relate to the security model?”). If the summary can’t answer 80% of them, it gets specific revision guidance and goes back to the summarizer. Two cycles max, then ship it.

The Output

Here’s the structure every summary lands in:

# [Descriptive Title]

## TLDR
[2-3 sentences]

## Summary
[Flowing prose organized by importance]

---

## Notable Quotes
> “[Exact quote]” — [Speaker]

## Glossary
**[Term]**: [Definition in context]

## People Mentioned
**[Name]** — [Bio]. [Link]

## Technologies Referenced
**[Tech]** — [Description]. [Link]

## Sources Consulted
[Links that contributed to the summary]

Everything below the summary is optional — omit if nothing qualifies. The glossary isn’t a dictionary dump. If the summary already explained a term clearly, it doesn’t need an entry. That DeepMind paper had eight glossary terms — “cognitive monoculture,” “liability firebreaks,” “contract-first decomposition” — because those are genuinely unfamiliar. A blog post summary might have zero.

What’s Open Source

This is all open source. If you’ve got your MCP servers configured (Perplexity for research, Firecrawl for scraping), you can run this today. Bring your own API keys.

The command:

• /summarize — The orchestrator that runs all four phases

Agent templates:

• planner.md — Content analysis and research TODO generation

• executor.md — MCP-powered research execution

• summarizer.md — Prose synthesis from clean inputs

• validator.md — Comprehension-based quality check

Research skills:

• term-definition.md — How to write accessible glossary entries

• person-research.md — How to research people via Perplexity

• tech-research.md — How to research technologies

• link-summarization.md — How to summarize linked content (one level deep)

Full repo: github.com/petersilberman/personal-reference-claude-code

If you followed Post 1 and set up the foundation, you already have the infrastructure. This is it working together — your file system as the API, agents doing specialized work, MCP tools connecting them.

I use this on papers, blog posts, long docs — anything where I want understanding without the full time investment. Four agents take longer than one prompt. But the output is something I can actually use: prose I could riff on in a conversation, not bullets I’d struggle to explain.