Why AI companies are pulping millions of old, rare books
AI companies have identified the value in old, rare books and have started using them to train their AI models. Unfortunately, they are destroying them in the process. Here’s why
READING LEVEL: ORANGE
Rare books are being destroyed in an attempt to train AI models, causing concern for historians and book lovers alike.
Artificial intelligence labs have been buying up millions of rare books, slicing them open, scanning the pages and pulping* the remains — sparking concerns that the last remaining copies of out-of-print texts are being destroyed on an industrial scale.
Rare booksellers across Europe have seen a surge in purchase requests which they suspect are coming from AI labs in America.
Pieter de Vries, an antiquarian* bookseller in the Netherlands city of Haarlem, recently received an email from a person identifying herself as Nataly from Singapore-based company 2077AI, Dutch news site BNR reported.
The woman wrote that the company was undertaking “a new project focused on collecting books in multiple languages, currently mainly in English”.
“We have compiled a very extensive list of editions that we are currently trying to acquire, and we plan to place a fairly large order,” the email said.
The attached list contained 3000 English-language titles organised by ISBN* number, ranging from books on fairytales and folklore to technical manuals and science texts.
Media outlets in the Netherlands, Switzerland, Spain, and Germany report booksellers have received nearly identical requests.
Booksellers believe the purchase requests were unrelated to collecting or resale due to the wide variety of subject matter and titles, concluding they were being used to train AI models.
Large language models* (LLMs) that power AI tools like ChatGPT until now have mainly been trained on vast datasets that include internet data, licensed books and articles, code libraries and human feedback.
But with much of the content available online now used up — and increasingly including poor-quality, AI-generated writing — AI labs have begun turning to works published before 2022 that don’t contain AI-generated text.
One company, ISBNdb, which boasts that it has the “world’s largest book database”, has begun offering bulk book buying for AI labs, “up to one million titles per order”, including of “older, rare and specialist volumes”.
“The world’s best AI training data is sitting on a shelf,” it states on its website.
“Millions of the most valuable books have never been digitised*. They exist only in physical form, scattered across library shelves, used bookstores, and out-of-print catalogues. We get them to you at scale.”
Anthropic, which owns Claude, and Meta, which owns Facebook and Instagram, have previously been sued in US courts for downloading millions of pirated* books to feed into their AI models.
Last week, a federal court in the US signed off on Anthropic’s landmark $US1.5 billion (AU$2.15 billion) settlement in a copyright class action brought by a group of authors who accused the company of misusing their books.
While that case was still unfolding, Anthropic was hatching a new plan to avoid similar copyright* headaches, called “Project Panama”.
Beginning in early 2024, the AI lab quietly launched an industrial-scale book-buying program, partnering with used-book wholesalers to buy millions of paper books by the pallet for what internal documents described as “destructive scanning”, where books’ spines were cut apart so the pages could be scanned and then destroyed.
Anthropic described the project as “our effort to disruptively scan all the world’s books” in an internal memo, which noted “we do not want it to be known that we are pursuing this project”.
Court documents detailed how the AI lab used a “hydraulic* powered cutting machine” to “neatly cut” millions of books before scanning the pages “on high speed, high quality, production level scanners”. The process has been described as the most efficient way to scan content compared to keeping books intact and hand scanning them page by page.
SpaceX founder and rival AI developer Elon Musk, for his part, said he was opposed to the practice of destructive scanning.
“I’ve asked the SpaceXAI team to preserve any rare books in a library and scan them the hard way vs. just cutting off the spine and scanning,” he wrote on his social media site X.
Australian book dealers have urged Anthony Albanese to protect local titles from the practice, under the Prime Minister’s newly announced Office for AI.
Announcing the new national strategy earlier this month, the PM sought to calm fears from arts and news industries about the impact of AI.
“Not everything produced in Australia is up for grabs. Not at all,” he told reporters.
Dawn Albinger, president of the Australian and New Zealand Association of Antiquarian Booksellers (ANZAAB), said there had been no reports of bulk orders for AI companies happening in Australia.
Ms Albinger, owner of Queensland’s largest second-hand bookstore Archive Books, noted that US copyright law and its definition of fair use was different to that in Australia.
“The recent announcement by the Prime Minister about an AI policy for Australia may reduce the likelihood of this happening here in the event that an Australian-based AI is developed,” she said.
The ANZAAB said it was particularly concerned about the “disappearance of uncommon and potentially rare material”.
“There are so many stories of discovery, [for example] notes left in margins by first-hand witnesses that correct historical accounts, or manuscripts found in the binding material of books published much later,” Ms Albinger said. “And then there is the pure romance of holding a book that has been signed or inscribed* by the author. There are things that cannot be examined and properly understood unless the physical object is to hand.”
Australian Booksellers Association (ABA) chief executive Susannah Bowen also expressed concern. “Books are how humans learn, discover and engage with ideas,” she said.
“When content is extracted from books to train AI models it raises very significant ethical and legal concerns. It also strips away the structure, context and meaning intended by the author, undermining the integrity* of the creative work, and reducing the viability* of a healthy book industry.”
WATCH THE VIDEO
POLL
GLOSSARY
- pulping: destroying
- antiquarian: the collection or trade of valuable, old things
- ISBN: a unique 13 digit code, or 10 digit code for books published before 2007, used to identify a specific book
- Large language models: a type of AI that uses deep learning to read, write and translate human text
- digitised: taking something analog, like words on a piece of paper, and converting it into something digital with code so that a computer can read it, use it and store it
- pirated: used without permission
- copyright: the legal right of the creator, in this instance, the author, to retain control over their work
- hydraulic: a type of technology that uses pressurised liquid to move, control and transmit heavy power
- inscribed: when the author writes a personal note next to their signature
- integrity: honesty, wholeness and soundness
- viability: the ability to succeed
EXTRA READING
PM builds bridge to Spelling Bee
Is Albo strong enough to battle AI?
The best books for wintry wonders
QUICK QUIZ
1. Why are books being destroyed in the process of training AI models?
2. What value do old books have in training AI?
3. Why are Australian authors and booksellers concerned by this new trend?
4. What are some treasures that could be forever lost if old books are destroyed on mass?
5. Which AI company has expressed opposition to the practice of destructive scanning?
LISTEN TO THIS STORY
CLASSROOM ACTIVITIES
1. Creative writing
Write a narrative story based on the following prompt:
There is one final copy in existence of the greatest fiction book ever known to humankind. An evil AI-lord is determined to put the book into his scanner, preserving the words forever but destroying the pages in the process. Will he succeed or can the book be saved?
Time: allow 30 minutes to complete this activity
Curriculum Links: English
2. Extension
If you needed to research an important topic and it was essential for your sources to be accurate, would you choose books or AI-generated material to gather your information? Explain your reasoning.
Time: allow 10 minutes to complete this activity
Curriculum Links: English
VCOP ACTIVITY
What happens next?
Imagine the full story turns out to be in three parts and this article is only Part One. Think about what the rest of the story could be and write the next two parts that tell the rest of this story.
Time: allow 30 minutes to complete this activity
Curriculum Links: English, Critical and Creative Thinking