Hard Drive Costs Surge as AI Demand Overwhelms Supply
The skyrocketing price of hard drives, driven by the artificial intelligence boom, is crippling efforts to preserve the internet. Organizations like the Internet Archive and Wikimedia Foundation are struggling to keep up with costs, while individual preservationists have largely given up.
Archival Organizations Under Pressure
"We're facing a perfect storm," said Brewster Kahle, founder of the Internet Archive, which runs the Wayback Machine. "Hard drive prices have increased by over 30% in the past year, and we're now paying nearly double what we did two years ago." The Archive is a critical resource for historians and journalists, storing billions of web pages.
Wikimedia, which hosts Wikipedia and other free-knowledge platforms, reports similar strain. "Our storage costs are ballooning, and we are having to make tough choices about what data to keep," said Lisa Seitz-Gruwell, chief advancement officer at the Wikimedia Foundation.
Background: The AI Storage Craze
The AI boom has created unprecedented demand for high-capacity hard drives. Companies training large language models need massive amounts of data, and they are buying up available drive production. As a result, prices have surged across the board.
Manufacturers like Seagate and Western Digital have prioritized AI clients, leaving archival organizations with fewer options and higher prices. "We used to be able to find deals on bulk drives, but those days are gone," added Kahle.
Stricter Anti-Scraping Measures Block Legitimate Bots
Compounding the problem, many websites have implemented aggressive anti-scraping protections to prevent AI bots from harvesting their content. These measures often unintentionally block legitimate archival bots, including those used by the Wayback Machine and Wikimedia.
"We're seeing a rise in CAPTCHAs and IP blocks that treat our archiving tools like malicious scrapers," said Mark Graham, director of the Wayback Machine. "This slows down preservation and sometimes stops us completely."
What This Means for Internet History
The combination of high hard drive costs and blocked bots threatens the long-term preservation of digital culture. Without affordable storage, organizations cannot expand their archives. Without access to websites, they cannot capture new content.
"Every day, we risk losing pages that document history," warned Kahle. "We need a coordinated response from tech companies and policymakers to ensure the internet's past is not erased."
For now, archival efforts continue, but at a slower pace. The Internet Archive has paused some expansion projects, while Wikimedia is considering reducing the resolution of stored images. The future of internet history hangs in the balance.