The Architect and the Bricklayer: Why AI Still Can't Write a Textbook

It is easy to assume that if an artificial intelligence can debug complex software or tackle advanced mathematics, writing a non-fiction book should be child's play. But the reality of AI writing is far more complicated, and in some ways, it has hit a surprising wall.

AI researcher Nathan Lambert recently completed a highly technical textbook on Reinforcement Learning from Human Feedback. Given the breakneck speed of AI development, he initially worried that publishing a human-authored non-fiction book might soon look obsolete. Instead, his hands-on experience using AI as a co-pilot revealed a stark contrast between what models appear to understand and what they can actually construct.

When tasked with micro-level editing, the AI models were spectacular. Lambert found that advanced GPT models acted as superhuman proofreaders, successfully hunting down obscure typos buried within a 300-page manuscript. Meanwhile, Claude proved to be a thoughtful editor, possessing the "taste" needed to help overcome writer's block and assisting with tedious LaTeX formatting and Python diagram coding.

However, the illusion of competence shattered when the models were asked to write an entire chapter. The output was consistently muddled in organization. The AI would try to be unnecessarily clever, lose the overarching narrative thread, and introduce random conceptual errors.

The models are exceptional bricklayers, but terrible architects. They know exactly how to check or generate a single unit of content—a sentence, an equation, or a paragraph. But they lack the mental model required to string hundreds of these units together into a cohesive, compounding structure.

This limitation stems from how we process information. Writing a textbook requires "compression." An author must distill vast amounts of complex data into organized, digestible insights. Current large language models, however, tend to do the exact opposite in long-form writing: they increase "entropy," generating filler and diluting the core message. They cannot stack insights endlessly without human intervention.

This observation carries implications far beyond the publishing industry. There is a growing hope that AI will soon autonomously solve grand, open-ended scientific problems. But if today’s most advanced models struggle to organize and compellingly present established, grounded science, they are unlikely to generate revolutionary scientific insights on their own anytime soon. For now, AI remains the most powerful assistant a scientist or writer could ask for, but humans must still draw the blueprint.

Key Points

• AI models are excellent at micro-tasks like formatting, copyediting, and finding typos in large manuscripts.
• When tasked with writing full chapters, models fail to organize information cohesively and introduce conceptual errors.
• Effective non-fiction writing requires 'compression' of knowledge, whereas current AI models tend to increase 'entropy' with filler.
• The inability to organize established science suggests AI is not yet ready to autonomously solve complex, open-ended scientific problems.

Why It Matters

Understanding AI's struggle with macro-level knowledge organization helps ground our expectations, showing that while AI is an unparalleled assistant, human architectural thinking remains essential for deep knowledge creation.

Sources:

• I wrote an AI textbook — how long until AI can do it better? — Interconnects (Nathan Lambert)