TL;DR Yes, user forums and the Internet Archive are extremely
important for digital preservation work. LLMs also seem quite good for
synthesizing information that is found in them, connecting the dots
between aspects of the problem you are facing, and connecting you to
other people engaged in similar work. The irony is that people may use
forums less as they engage with generative AI.
Back in June I
wrote about how my brother John had died, and how among the things he
left behind were over 100 CD backups of music he had created in the
early 2000s. Each CD was clearly labeled with a song title, and
contained a multi-track backup of a single song, which (I was to learn)
was made with a TASCAM 788. He had told me he had these backups years
ago, but I’d never seen them until we were emptying out some of the
storage space in his apartment building.
John preformed live in different small venues in Pittsburgh in the
1990s, and then in the Doylestown, PA area in the early 2000s. He left
behind several guitars, and other musical equipment. But despite my
encouragement on many occasions he never really released any of his
music in any form, except for one song on Bandcamp, that he had recorded
much later in 2022. John’s music was really important to me, even more
now that he is gone, so I wanted to see if I could retrieve these songs,
and perhaps put together a collection for family members, and other
people who were interested in hearing his music, and remembering his
creative spirit.
Unfortunately the CDs use a proprietary format, that can only be read
using the TASCAM 788’s optional CD burner. As I noted in the previous
post I did figure out a way to rip the raw data from the CDs and reverse
engineer a program to read the tracks, as well as the final mix. It is
far from perfect because the gaps between tracks are not consistent.
Sometimes the program would extract part of a song as a track. It also
wasn’t clear what the final mix was. Still, it is possible to pull the
concatenated tracks off as a single WAV file, which was amazing to hear.
But I was hopeful I could get the tracks as they existed in the backups.
John left his TASCAM 788 in the storage unit too, along with an
(optional) external CD burner for creating the backups. My brother
Matthew actually thankfully kept it. However the devices looked quite
different in style, and I think I mistakenly thought the CD recorder was
part of some old computer equipment, and donated to a local record
store, along with part of his CD collection. Once I realized the CD
recorder’s significance for recovering John’s music it was too late to
get it back. So I bought one on Ebay for a little less that $200.
Once I got the CD burner hooked up to the TASCAM 788 I discovered that I
could read the CDs and see the tracks, and listen to them, but not
export them. While it’s possible to export each track by hand by
connecting the audio output directly to a soundcard with digital input,
there are over 100 CDs, so I was hoping I could easily extract the track
and the final mix as a WAV file. My goal is to be able to view these in
another DAW
on my laptop instead of learning how to use the 788 itself.
In conversation with Claude I learned that I could upgrade the
firmware from v1.10 to v2.02 to get a new Export to WAV function on
the 788, and that would let me easily burn the tracks to a CD, which I
could then read on my laptop. Claude also pointed me at a TASCAM
forum where the upgrade was being discussed. Unfortunately the
other TASCAM site where the upgrade was located was now gone,
and while the
page was in Wayback Machine, the actual ZIP file it linked to was
not. While I could’ve pieced some of this information together given
enough time it was actually very useful to learn about it in the context
of my questions and conversation.
One of the helpful people on the forum seemed to be responding over the
years to requests for the firmware update file. So I responded on the
forum asking if anyone still had it available. I ended up hearing back
about a month later, and was able to flash the 788, and like magic, it
now has a new Export to WAV function. Since lots of copies keeps stuff
safe I placed a copy of the 788 V2.02 Firmware
Update up at the Internet Archive.
But, alas when I went to use the Export function it assembled the WAV
file on the TASCAM but then gave a PANIC message when actually trying to
write to the CD. I tried a few times hoping it was a temporary glitch,
but it failed with the same result at the same time. Again according to
Claude apparently CD-RW devices can retain the ability to read, but no
longer be able to write due to a weakening laser
diode. The CD-RW I found on Ebay did seem kind beat up honestly. So
I rolled the dice and dropped another $175 on Ebay in the hopes that
this one would work better. I’m still waiting for it to arrive.
I guess if this new one doesn’t work as well I will fall back to using
the reverse-engineered Python script to extract WAV from the raw CD rip
and work with that instead. It has been therapeutic spending time with
these backups, especially because it was clear that John created them
knowing they would be accessed some day. It also is heartbreaking,
thinking that he’s no longer around to talk to about them.
It’s hard for me to imagine having done the work thus far without the
TASCAM user forum, the help of the Wayback Machine, and helpful people
on the Internet. The generative AI (in this case Claude) also made
searching for this information while asking questions more cohesive. I
think in the past
I’ve argued that search results, and original sources are critical to
understanding, and I stand by that. But being able to have a
conversation about an information need, and the ability to ask for
references for verification (as fraught as that is) is extremely useful.
My worry is that if LLM systems don’t prioritize giving users accurate
sources, and encouraging their use, then we will see forums like this degrade
over time.
We don’t need this many interpreted web languages. More generally we
don’t need half a dozen open source libraries doing the same thing.
Sometimes the mature thing can be even for a project that is very
‘successful’—especially if that success is by now mostly toxic—to
consider the project finished, cut a final release and archive the
repos.
Despite appearances to the contrary, computers are fundamentally social
machines.1 We ought, then, to regard the use of computers as inherently
social. Computing As Social Activity, or CASA, is a set of principles
meant to align the ways we use, design and understand computers with
social values that stand in support of communities, rather than work
against them.
Preserving a website is complicated. Even a simple webpage can depend on
dozens of images, videos, scripts, style sheets, and other resources
scattered across the web. Some content appears only after someone
interacts with the page, while JavaScript and other technologies can
make parts of a site difficult for web crawlers to discover. If some of
those pieces are missed, the archived version may look or behave
differently than the original.
That is why quality assurance, or QA, is an important part of web
archiving. QA is the process of assessing how well a website was
preserved. Most of the websites we collect are archived weekly, monthly,
or quarterly, producing a new “capture” or archived version, each time.
This gives us opportunities to try different approaches if we weren’t
successful the first time. Crawl reports provide clues about what was
archived, but the best way to know if a website was properly preserved
is to look at the archived site and compare it with the live version.
Does it look right? Is everything there? Do the links work? Can you
interact with all the features?
5,516 likes, 202 comments - thebandwashere on October 9, 2026: “No Slam
Dancing, No Stage Diving, No Spikes…City Gardens, a historic music venue
in Trenton, New Jersey, is no more. Operating from 1979 to 1996, City
Gardens served as a crucial stopover for bands traveling from
Philadelphia to New York City. Even Jon Stewart, who bartended there
from 1984 to 1987, credited City Gardens with transforming his life and
inspiring his career in comedy. This morning, I found myself in the area
and felt compelled to pay my respects to the venue. I’ve included some
flyers of notable bands that performed there, as well as a recreated
image of Glenn and Henry from my 2020 visit. Notably, the building was
still standing during that time, which allowed me to better match the
images. Did you attend? Who would you have liked to have seen? Glenn
Danzig and Henry Rollins outside City Gardens in Trenton, New Jersey,
taken in 1984. 📸 by Ron Gregorio) Punk Girls, along with an outside
shot of City Gardens by ? Thanks (matthewpusak?) for the
heads up. 💪 #citygardens #nj #TheBandWasHere #GlennDanzig
#henryrollins”.
The HTML parser is a piece of software that processes HTML markup and
produces an in-memory tree representation (known as the DOM).
The HTML parser has many strange behaviors. This book will highlight the
ins and outs of the HTML parser, and contains almost-impossible quizzes.
HTML is not only used by basically all of the web, but it is also part
of many modern applications. The HTML parser is part of the foundation
of the web platform.
The ability to work with large amounts of unfamiliar code productively
without necessarily understanding the specifics or reading the code can
be quite a superpower, one that enables engineers to tackle problems way
out of their depth.
I consider this an emergent skillset that’s different from coding or
traditional debugging, but like any other skill, it’s one that can be
honed with enough effort. While I’m no leading expert in this area, the
general contours of how these skills can be actively cultivated are
beginning to stand out in sharp relief.
By putting my own experiences in writing, I’m hoping to hear from fellow
practitioners their thoughts on this topic.
This book is the fruit of twenty-five years of study of Spinoza by the
editor and translator of a new and widely acclaimed edition of Spinoza’s
collected works. Based on three lectures delivered at the Hebrew
University of Jerusalem in 1984, the work provides a useful focal point
for continued discussion of the relationship between Descartes and
Spinoza, while also serving as a readable and relatively brief but
substantial introduction to the Ethics for students.
Large Language Models (LLMs) are machine learning (ML) models that have
an increasingly large carbon footprint through their development and
use. Efforts to increase the energy efficiency of these models have not
translated into reduced consumption due to rebound effects such as
Jevons Paradox - that increased efficiency drives increased use. There
is therefore a need for additional measures to solve this problem. We
suggest that one possible way forward is to use life cycle thinking, and
view LLMs as products that can become waste. With this perspective, we
investigate the potential of the waste hierarchy from the EU’s Waste
Framework Directive, which suggests five different measures for how to
manage waste: prevention, reuse, recycling, recovery, and disposal. We
examine how these measures can inform and motivate new types of thinking
and approaches to reducing LLM waste and their environmental impact in
general. Applying the waste hierarchy to LLMs highlights that preventing
waste is essential for reducing the models’ environmental impact, mainly
because it reduces the need for training new models. Prevention can be
achieved through many existing methods for reusing, “recycling”, and
“recovering” LLMs. Additionally, disposal can be important both for
saving energy and for keeping a considerate attitude to the resources
being spent on training LLMs. We also call to attention that prevention
of unnecessary use of LLMs carry huge potential for lowering the climate
impact of the models.
To see how current anti-bot measures (aimed at aggressive AI scrapers
and botnets) affect well-behaved bots, I sent a bot that openly
identifies itself, botfriendliness/0.1
(+https://patrickhochstenbach.net/git/phochste/botfriendliness), to the
1,262 institutional repositories in Germany, United Kingdom, Italy,
France, Spain, Norway, Netherlands, Portugal, Austria, Switzerland,
Belgium, Finland, Ireland, Sweden, Denmark, Iceland and Luxembourg that
are registered in the OpenAIRE Graph. For each repository it looked for
one publication landing page and requested it, honoring robots.txt and
making only a handful of requests. From the sweep of 2026-11-01 on, the
bot signs its requests with Web Bot Auth, so a repository can verify
that a request really comes from this bot.
We have plenty of brothy soups, slow cooker stews, chilis, curries and
more. To get vegetarian recipes like these delivered to your inbox, sign
up for The Veggie newsletter.
The concept of the cooperative principle was introduced by the linguist
Paul Grice in his pragmatic theory. Grice researched the ways in which
people derive meaning from language. In his essay Logic and Conversation
(1975) and book Studies in the Way of Words (1989), Grice outlined four
key categories, or maxims, of conversation—quantity, quality, relation,
and manner—under which there are more specific maxims and sub-maxims.
These describe specific rational principles observed by people who
follow the cooperative principle in pursuit of effective communication.
Applying the Gricean maxims is therefore a way to explain the link
between utterances and what is understood from them.
Our team is heading to BCN ready to promote open, sovereign and democratic technologies. Here's what you need to know to get involved and find out more.
A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.
In the hours following the release of CVE-2026-89078 for the project Ruby, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix a double free in Ruby's Onigmo regular expression parser when parsing fails after an alternation, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled patterns. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Miss Ebony Reilly, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.
In the hours following the release of CVE-2026-93577 for the project Ruby, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix an integer overflow in Onigmo's compiled-pattern size calculation, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled regular expressions. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer King Lorenz Adams, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.
In the hours following the release of CVE-2026-97229 for the project Ruby, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix an integer overflow in Onigmo's compiled-pattern size calculation, allowing memory corruption and potential remote code execution through attacker-controlled CI/CD regular expressions in GitLab. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Prof. Jayson Mayert, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.
In the hours following the release of CVE-2026-110264 for the project Ruby, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix an overflow of Onigmo's internal IDs in patterns containing many repeats, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled regular expressions. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Miss Luella Bednar, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
The U-M Library rebuilt its website using Astro, replacing Gatsby, React, and GraphQL with a simpler, more maintainable architecture. Without changing the user-facing design or functionality, the migration reduced homepage JavaScript by 99% and image payloads by 75%, while improving performance and eliminating post-load layout shifts.
Thanks to the October release team: Martha Driscoll (NOBLE), Michele Morgan (NOBLE), Blake Graham-Henderson (MOBIUS) Gina Monti (Bibliomation), and Andrea Buntz Neiman (Equinox); as well as everyone who contributed fixes and testing to this release.
The first day of the conference was dedicated to workshops and the Doctoral Consortium. We attended the InfoVis-DL 2026 workshop titled “Information Visualization in Digital Libraries: Multi-modal Applications to Cultural Heritage Collections”, which explored how information visualization can help libraries, archives, museums, and other cultural heritage institutions make complex digital collections more accessible and meaningful to users.
📍 #TPDL2026 in Faro, Portugal 🇵🇹
Happening now: InfoVis-DL — Information Visualization in Digital Libraries: Multi-modal Applications to Cultural Heritage Collections.
Excited for a great workshop and discussions!
@webscidl.bsky.social @phonedudemln.bsky.social @weiglemc.bsky.social
The workshop brought together several interesting perspectives on visualizing cultural heritage collections. “From Inscription to Information: A Multi-modal Visualization Pipeline for Endangered Graffiti Heritage at Apostolos Andreas Monastery, Cyprus” presented a fascinating effort to digitally document endangered historical graffiti using archival photographs, photogrammetry, image enhancement, 3D modeling, and other multi-scale techniques. Their work demonstrated how visualization can become more than a way of presenting a collection and how it can serve as a research and preservation tool when the original material is deteriorating or has already disappeared. One statement from the presentation that I particularly remember was “Visualization was not the last stage of this research; it is what provided the means for this research”. The second presentation, “Tracing Early Digital Art: Coordinated Visual Exploration of Digital Art Archives” approached visualization from a different direction, bringing together archival records about artists, exhibitions, institutions, and communities and allowing users to explore their relationships through interconnected network, temporal, and geographic views. An especially interesting point from the discussion was that visualizations are not neutral: what appears to be an absence or an isolated artist in a visualization may simply reflect gaps in the underlying archival record. The third presentation introduced DAViD Lab, Portugal’s first laboratory dedicated to the digitisation and advanced visualisation of historical archives. The lab plans to combine AI-assisted exploration with large-scale displays, AR and VR, spatial sound, touch and gesture-based interaction, and other immersive technologies, with potential applications ranging from cultural heritage and historical propaganda collections to transportation archives. The presentations led to a lively discussion about accessibility, open research, interoperability, storytelling, public engagement, and how visualization can help bring digital collections closer to both researchers and the general public. Together, the presentations sparked an interesting discussion about how visualization can go beyond simply displaying digitized material to help people explore, interpret, and connect with cultural heritage collections.
Main Conference: Day 1
The first day of the main conference began with the keynote by María Poveda-Villalón, Associate Professor at Universidad Politécnica de Madrid. Her talk was titled “Knowledge Beyond Documents: FAIR Ontologies for Humans and Machines” and she explained how to make ontologies and vocabularies FAIR (Findable, Accessible, Interoperable, Reusable) across the full development lifecycle. She emphasized the importance of making metadata and ontologies machine-actionable for agents and LLMs. One key takeaway was that FAIRness should be considered throughout the entire development process, rather than only after publication. She also highlighted the importance of standard vocabularies, persistent URIs, and interoperable formats such as RDF/OWL, along with lightweight, iterative development practices and automated FAIR checks. During her talk, she also introduced FOOPS! (FAIR Ontology Pitfall Scanner), a tool that automates FAIR assessments and helps developers identify areas where their ontologies can be improved.
#TPDL2026 Main Conference - Day 1 kicks off with the first keynote! 🎤
“Knowledge Beyond Documents: FAIR Ontologies for Humans and Machines”
by María Poveda-Villalón
@webscidl.bsky.social, @skandasiva.bsky.social
The first session, “Data Management Plans and FAIR Assessment”, included three presentations exploring different aspects of machine-actionable (a term I learned here, distinct from machine-readable) data management and FAIR evaluation. The first paper, “Community Metrics and Automated Assessment of Machine-Actionable Data Management Plans” by Tomasz Miksa presented work from the OSTrails project. The presentation highlighted a common problem with traditional Data Management Plans (DMPs) where researchers prepare them to satisfy funder requirements but receive little useful feedback. He introduced a catalog of 94 community-driven metrics for automating DMP evaluations and providing researchers with feedback. The discussion raised questions about whether automated checks could verify that researchers actually followed through on the commitments made in their initial DMPs. Anna Lehmann-Gnirck presented the second paper of the session, “It’s FAIR and generic?! A Systematic Analysis for FAIR Assessment Tools”, which examined how different automated tools assess the FAIRness of research data. An interesting observation was that the same dataset can receive different FAIR scores depending on the tool used, because each tool selects and implements assessment criteria differently. Anna emphasized that automated assessments are generally better at checking technical aspects, such as persistent identifiers and metadata availability, than determining whether data is genuinely understandable and reusable within a particular research community. She proposed combining generic technical assessments with domain-specific semantic evaluations. This led to an important question: rather than simply asking whether a dataset scored 70% or 90% on a FAIR assessment, shouldn't we first ask what that score actually measures? The final paper of the session, “Enhancing Machine-Actionable DMPs with Policy Entities Expressed in ODRL” by Diamantis Tziotzios, returned to the OSTrails project and addressed another limitation of machine-actionable DMPs. Although DMPs can now represent much of their information in structured formats, policies governing data access and usage are often still written as free text. The authors explored using ODRL (Open Digital Rights Language), a W3C standard, to formally represent permissions, prohibitions, and obligations. This session offered very interesting perspectives on machine-actionable research data management: defining what can be automatically evaluated, understanding what FAIR assessment scores actually tell us, and representing policies in ways that machines can interpret and act upon.
#TPDL2026 Session 1: Data Management Plans and FAIR Assessment is happening now!
Chaired by Ralf Krestel.
The papers are available to download: tpdl2026.ualg.pt/program/
A social event was held that evening, featuring a guided tour of the Museu de Faro followed by a walking tour of Faro Old Town.
A few captures from the #TPDL2026 social event yesterday evening 📸🇵🇹
A tour of the Faro Museum, followed by a walk through the beautiful Old Town of Faro.
A lovely way to wrap up the day, made extra special with our little one by our side 💕
@webscidl.bsky.social, @skandasiva.bsky.social
The second keynote of the conference was by Gianmaria Silvello, Professor at the Department of Information Engineering, University of Padua. His talk was titled, “Transitive Credit Allocation for Heterogeneous and Dynamic Scholarly Graphs”. Traditional citation metrics give credit to directly cited papers, but often overlook the foundational research, datasets, and software that made those papers possible. He introduced a transitive citation credit model in which a portion of the credit received by a paper is passed backward through its citation network, allowing earlier contributions to receive recognition. The model was evaluated on a large scholarly citation graph containing approximately 38 million papers, demonstrating how accounting for indirect contributions can significantly change researcher rankings. One particularly interesting example was a 1955 paper on antibody sequences that had received only 19 direct citations, but accumulated more than 5,000 units of credit when its indirect influence was considered. The keynote also raised a timely question about credit attribution in the age of LLMs: when AI-generated answers rely on knowledge derived from numerous publications and datasets, how can we recognize the original contributors? Reading about this work reminded Dr. Nelson of a quote he said he never misses a chance to mention: “Everyone Who Bought One of Those 30,000 Copies Started a Band”. The Velvet Underground, one of the most influential rock bands of the 1960s and early 1970s, although they didn’t sell many records, their music inspired many of those who listened to it to start their own bands. The quote nicely captures the message of the keynote: the true influence of a work is not always reflected in its immediate recognition, but in the many contributions it inspires over time.
#TPDL2026 Main Conference - Day 2 kicks off with the second keynote 🎤
“Transitive Credit Allocation for Heterogeneous and Dynamic Scholarly Graphs”
By Gianmaria Silvello, Professor at the Department of Information Engineering, University of Padua
@webscidl.bsky.social, @skandasiva.bsky.social
We attended the “LLMs and Multimodal Models for Knowledge Discovery” session, which had four presenters on how AI models can support information retrieval and knowledge discovery across historical image collections, scientific literature, and visual data. First presentation was “Using Hierarchical Controlled Vocabularies to Understand CLIP Retrieval Failures in Historical Photo Collections”, by Ratan Sebastian on why CLIP sometimes struggles to retrieve relevant images from historical photograph archives. The authors investigated whether retrieval failures occur because a model fails to associate a query with relevant images or because the images themselves are not represented as a coherent group. One interesting finding was that poor query-image alignment was a particularly important source of retrieval failure, while fine-tuning could improve performance for broader, more abstract concepts. The work highlighted the value of understanding why a retrieval model fails rather than relying only on overall performance scores. The second paper, “MARIO: Multi-Agent Retrieval and Iterative Reasoning for Knowledge-Augmented Generation in Scientific Question Answering” by Maodi Hu introduced a multi-agent approach to answering complex scientific questions. Instead of relying on a single retrieval step, MARIO combines knowledge retrieval with iterative reasoning, allowing different agents to gather and evaluate evidence. An important aspect of the approach was connecting generated answers to supporting scientific sources, making the reasoning process more traceable.
The third paper, “GeoSearch: Transforming Geoscience Literature Discovery for Chinese Geoscientists with AI” by Feng Zhou presented an AI-based approach to helping researchers discover relevant geoscience literature, addressing the challenges of finding useful information within a large and specialized body of scientific publications. He highlighted how fragmented sources, inconsistent metadata, and varying terminology make it challenging to find and connect relevant scientific knowledge. The final presentation of the session, “Do you ‘see’ what I ‘see’? A Multi-panel Visual Question and Answer Dataset for Large Language Model Chart Analysis”, by Daniel Evans raised another interesting question: Can an AI model truly understand a scientific figure when the answer requires connecting information across multiple charts? Reading individual labels or identifying trends in one chart is different from interpreting relationships between several panels. All four presentations approached AI from different perspectives but shared a common concern: retrieving or generating an answer is only part of the problem; understanding why it is correct, and where these models fail, is equally important. I think this is a valuable consideration for almost any research project involving AI today.
#TPDL2026 Session 6 · LLMs and Multimodal Models for Knowledge Discovery is happening now!
Chaired by Anett Hope.
The papers are available to download: tpdl2026.ualg.pt/program/
The conference dinner took place that evening at Elemento on Faro Island. We reached there by ferry across the beautiful Ria Formosa lagoon. The scenic ferry ride was a highlight of the evening, followed by an enjoyable dinner featuring plenty of Portuguese sangria, fresh seafood, and an opportunity to socialize with fellow attendees. During the dinner, the organizers also announced the Best Paper Award, with “ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment” by Sheeba Samuel et al. being recognized as the winning paper.
At the #TPDL2026 conference dinner! ⛴️🌊
A beautiful ferry ride across the Ria Formosa to Elementos Faro for dinner by the waterfront ✨
A great way to wrap up another conference day!
@webscidl.bsky.social @skandasiva.bsky.social
Adam Jatowt, a professor at University of Innsbruck, delivered the third and final keynote of the conference, titled “Temporal Intelligence in Language Technologies: Modeling, Retrieval, and Reasoning”. He discussed the importance of integrating time (timestamps, dates, relative expressions) into language-model representations and training to improve factual correctness and document-centered question answering. He emphasized that a fact can be correct at one point in time but incorrect at another. For example, answering a question about who held a particular position requires understanding not just the information itself, but also the time period to which it applies. This becomes particularly challenging when documents contain outdated information or refer to events using relative expressions such as “last year” or “two months ago.” The keynote highlighted why language technologies need to go beyond recognizing temporal expressions and develop the ability to reason about how information and facts change over time.
The final day of #TPDL2026 is today! 🎤
Day 3 begins with the conference’s third keynote:
“Temporal Intelligence in Language Technologies: Modeling, Retrieval, and Reasoning”
by Adam Jatowt
@webscidl.bsky.social @skandasiva.bsky.social
The final session of TPDL 2026, “AI, Human Judgement, and Public Information Practices”, featured work exploring how people use, evaluate, and interact with AI and information systems.
Luana Ponte presented “A Complementary Tool for a Sensitive Domain: How Users Seek Health Information Through LLMs”, examining how people incorporate LLMs into their health information searches. The study found that users generally treat LLMs as a complement to traditional search engines rather than a replacement, particularly when seeking clarification on health-related questions. The discussion also raised an important question about how AI interfaces could better encourage users to verify information in high-stakes domains. In the session's only full paper, “Speaker Mining – FAIR Data on Public Broadcasts for Question Answering”, Tim Wittenborg presented efforts to integrate fragmented German public broadcasting metadata into a structured, reusable knowledge base. By linking television episodes and guest appearances to Wikidata, the researchers demonstrated how structured broadcast data could support questions about media representation, including who appears on talk shows and how frequently. The presentation also highlighted the substantial manual effort required to reconcile records across different sources. Maja Kuhar and Adam Roegiest presented “Adoption, Attitudes, and Apprehensions: A Survey of Librarians on GenAI”, focusing on librarians in Slovenia. Their findings showed that librarians recognize GenAI's potential for tasks such as cataloging, reference services, and administrative work, but remain cautious about accuracy, privacy, and overreliance on these tools. The presenters emphasized that GenAI should support librarians rather than replace the human expertise and interpersonal interactions central to their work.
The final session of #TPDL2026!
Session 8 - AI, Human Judgement, and Public Information Practices
The papers are available to download: tpdl2026.ualg.pt/program/
@webscidl.bsky.social @skandasiva.bsky.social
The final session concluded with our paper, “Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts”. In this work, we compared sentiment judgments from human annotators, traditional sentiment analysis tools, and LLMs to examine how consistently they interpret social media text. Our results showed that even experienced human annotators often disagreed on sentiment, particularly when distinguishing neutral expressions from positive or negative ones. We also found that simplifying the task to negative versus non-negative or positive versus non-positive generally improved agreement. Among the automated systems, Twitter-RoBERTa aligned most consistently with the human consensus, achieving a Cohen's kappa of 0.73 for three-class classification and 0.88 for negative versus non-negative classification. Although the LLMs often agreed strongly with one another, their agreement did not necessarily reflect human judgments. Our findings highlighted two important considerations: sentiment is inherently subjective, making human disagreement an important part of evaluation, and domain-specific models can still outperform general-purpose LLMs on tasks closely aligned with their training data. The audience discussion also raised a valuable question about whether we should treat human disagreement as useful information rather than reducing it to a single majority-vote ground truth.
The conference concluded with closing remarks by José Barateiro, who thanked the organizers, speakers, volunteers, and attendees for making TPDL 2026 a success. Gianmaria Silvello then announced that TPDL 2027 will take place on San Servolo Island in Venice, Italy! We had such a wonderful time at TPDL 2026 and hope to see everyone again next year in beautiful Venice for more inspiring research, great conversations, and familiar faces!
And that’s a wrap on #TPDL2026!
Closing remarks by José Barateiro, followed by Gianmaria Silvello announcing #TPDL2027 in Venice, Italy! 🇮🇹 - inviting everyone to return next year and, of course, to start working on those papers!
@webscidl.bsky.social, @skandasiva.bsky.social
For a broader perspective on the conference and its history, take a look at“Thirty Years of Digital Libraries: What ECDL/TPDL Tells Us About the Field, and Where We Go from Here” by Gianmaria Silvello. The paper looks back at 30 years of ECDL/TPDL and examines how the field has evolved. Old Dominion University is well represented in this analysis, a connection that continues through our participation in TPDL 2026.
For another glimpse into the WS-DL group’s TPDL experiences, see ourTPDL 2022 trip report, written after Emily and Himarsha attended the conference that year.
Acknowledgements
We would like to thank Drs. Michele Weigle and Michael Nelson for supporting Himarsha’s travel to TPDL 2026. We also thank Dr. Shuai Hao, the Computer Science Graduate Society (CSGS), and SEES travel awards for partially supporting Skanda’s travel. We are grateful for their support, which made it possible for us to share such a memorable conference experience together in beautiful Faro, Portugal.
A screen recording of the JAViD workflow, showing a user uploading dual-participant gaze data and exploring the resulting joint visual attention visualizations.
Joint visual attention (JVA) occurs when two or more people focus on the same object or area at the same time. It is an important aspect of social interaction and has been studied in areas such as collaboration, learning, and clinical communication. Researchers use different definitions of JVA depending on the context of their studies. In this project, we define JVA as two or more people looking at the same area of interest at the same time. Advances in eye-tracking technology facilitate the study of JVA by capturing the gaze behavior of multiple people. However, analyzing JVA from multiple eye-tracking streams still involves several steps, including aligning gaze streams, identifying periods of shared attention, and examining the resulting data. These steps can make JVA analysis time-consuming and require specialized technical knowledge. To make this process easier, we developed JAViD, the Joint Attention Visualization Dashboard, an interactive system for analyzing and visualizing JVA. The code, sample data, and setup instructions are available in the JAViD GitHub repository.
We developed JAViD as a group project for the CS832 Human Computer Interaction course at Old Dominion University in Spring 2026, taught by Dr. Bhanuka Mahanama. The group members are Kumushini Thennakoon, Rochana Obadage, and Chameli Dommanige. Our goal was to make dual-participant gaze analysis more engaging using a visualization dashboard. This blog post describes the problem we addressed, how we applied human-computer interaction concepts, tools, and technologies, and the key takeaways from the course.
1. Problem Statement
Many eye-tracking tools focus on single-user study setups, but they offer limited support for comparing multiple synchronized gaze streams. An example of a single-user visualization dashboard is “A-DisETrac: Advanced Analytic Dashboard for Distributed Eye Tracking”. Researchers who study shared attention often write custom scripts or assemble Jupyter Notebooks that combine data cleaning, analysis, and visualization. These workflows can answer relevant research questions, but they require programming knowledge. We designed JAViD to bring the main steps of a dual-participant analysis into a browser-based dashboard. A user can load synchronized gaze data, inspect where each participant looked, examine how the distance between their gaze points changed over time, identify candidate joint-attention episodes, and export summary results. The project was guided by two questions:
Q1: How can we design effective interactive visualizations that reveal both spatial and temporal patterns of JVA from multi-participant eye-tracking data?
Q2: How does an integrated dashboard approach compare to traditional code-based analysis methods in terms of usability, learnability, and effectiveness for JVA analysis?
2. System Design
This section explains the JAViD system, including the task scenario, system architecture, design decisions, and core visualization components. The system was developed based on a screen-based visual search task, with the goal of analyzing JVA between two participants. It outlines how gaze data are prepared and processed, how usability principles inform the interface design, and how interactive visualizations support the exploration of spatial and temporal attention patterns.
2.1. Task Scenario
The task scenario involves two participants engaging in a screen-based activity, such as jointly analyzing visual content or completing a task on a shared display. Eye-tracking data from both participants are recorded as synchronized gaze points on the screen. To ensure consistent spatial alignment, the gaze data are transformed into the screen’s reference frame using a perspective transformation. ArUco markers are used to detect the screen frame in the visual field. Figure 1 shows an example of the data collection setup.
Figure 1: Data collection setup for a two-participant screen-based task. Both participants view the same screen, with ArUco markers positioned around its boundaries to support gaze mapping to a common screen reference frame.
2.2. System Architecture
JAViD uses a client-server architecture built with Python, Dash, and Plotly. As shown in Figure 2, the browser provides the interface for uploading data, adjusting controls, and exploring coordinated visualizations. The Python Dash server handles data processing, joint visual attention analysis, and the construction of figures and results. Labeled request and response arrows show how these components communicate during interactive analysis.
Figure 2: Client-server architecture of JAViD for two-dimensional joint visual attention analysis.
Data collection and spatial preparation occur before the data is uploaded to JAViD. ArUco markers identify the screen frame in the scene recordings, and perspective transformation maps both participants’ gaze coordinates into a common two-dimensional reference frame. Next, users upload the prepared gaze files in CSV or Excel format with an optional stimulus image that provides context for the spatial visualizations.
The server uses pandas and NumPy to read, validate, and clean the uploaded data. The system aligns the participants’ gaze streams by timestamp and interpolates when necessary to accommodate different sampling rates.Using the aligned data, the system computes gaze distances and identifies JVA episodes according to spatial proximity and temporal criteria.They also support cross-recurrence analysis, spatial-temporal clustering using DBSCAN, and the calculation of episode measures and summary statistics.
Dash callbacks connect user actions with the required server-side operations. When users load data or change relevant analysis parameters, the browser sends the inputs to the server through HTTP requests. The callbacks invoke the necessary computations and construct updated Plotly figures, summary tables, and other results. These outputs are returned to the browser, including figure definitions encoded as JSON, where Dash components and Plotly.js display them.
The interface brings together five main coordinated views: scanpath and heatmap, temporal JVA analysis, cross-recurrence, gaze points on the stimulus, and statistical summaries. Shared time and display controls allow users to examine the same recording from complementary perspectives. Users can navigate through the recording, adjust analysis settings, and export figures or results for further analysis.
2.3. Design Decisions
The JAViD interface was designed with established usability principles in human-computer interaction to ensure accessibility for users with varying levels of technical expertise. As shown in Figure 3, the interface reflects key concepts from Nielsen’s usability heuristics and Norman’s principles of interaction design, including visibility of system status, feedback, constraints, consistency, affordances, and mappings.
Figure 3: Annotated JAViD interface showing the application of usability principles.
2.3.1. Visibility of System Status
The interface maintains continuous awareness of the system state through status indicators. The left panel displays the currently loaded file names (e.g., “scene_5_participant_1.csv”) directly beneath each upload zone, so users always know which datasets are active. Participant ID fields (P1 ID, P2 ID) are shown persistently, making the current configuration visible at all times without requiring the user to navigate away from the main view.
2.3.2. Feedback
Immediate and informative feedback is provided at two levels. At the data level, a confirmation banner highlighted in green and reading “Loaded — P1: 81 pts | P2: 81 pts | Duration: 2000 ms” appears upon successful data loading, confirming exactly what has been ingested. At the system level, the Dash developer toolbar at the bottom of the screen surfaces runtime information including server connectivity status, callback activity, and version update notifications (e.g., “Dash update available - v4.1.0”), providing transparent feedback about the underlying application state.
2.3.3. Constraints
Input constraints are enforced to prevent user error before it occurs. File upload zones accept only .xls or .csv formats, as stated explicitly in the field labels (“Participant 1 (.xls or csv)”), preventing incompatible file types from being submitted. The screen dimension fields (Width, Height) and the Global Time Window slider are bounded inputs that restrict values to valid numerical ranges, guiding users toward correct configurations without requiring domain knowledge.
2.3.4. Consistency and Standards
The interface adheres to consistent visual and interaction conventions throughout. Both participant upload zones share identical layout, styling, and labeling patterns, reducing the cognitive load of learning one in order to use the other. Toggle controls for “Show Heatmap,” “Show Trails,” and “Show Stimulus” follow the same checkbox style, and opacity sliders for heatmap, trail, and image layers use a uniform slider component. The light mode/dark mode toggle and the 2D/3D JVA navigation buttons appear consistently in the top bar across views.
2.3.5. Affordance
Interactive elements are designed to signal their function visually. The “Drag & Drop or Browse” upload zones use a dashed-border card style that conventionally affords file-dropping behavior. The “Load Data” button is rendered with a play-arrow icon, reinforcing the action of initiating a process. Sliders for opacity and time window are rendered as draggable tracks with visible thumb handles, affording continuous adjustment. Radio button groups for “Heatmap View” and “Trails View” (P1 only, P2 only, or Both P1 & P2) afford mutually exclusive selection.
2.3.6. Mapping
Real-world scenarios are mapped into the dashboard through supportive interface elements that aid interpretation and usability. A dedicated help section is positioned at the top of the dashboard, providing users with contextual guidance on how to understand and interact with the visualizations in relation to the underlying study tasks. Additionally, the interface includes both dark and light modes to accommodate different viewing preferences and environments. Users can switch between the two modes using a button with a small symbol next to the label. A sun symbol indicates light mode, while a moon symbol indicates dark mode, allowing users to quickly identify the mode selection button.
2.4. Core Visualization Components
JAViD provides five primary visualization components, each addressing specific aspects of JVA analysis.
2.4.1. Scanpath and Heatmap View
The scanpath visualization overlays the gaze trails of both participants using distinct colors to show their sequential visual exploration (Figure 4). The heatmap represents attention intensity across the stimulus. Users can toggle between individual participant patterns and combined patterns to examine convergence and divergence in attention. The visualization also provides adjustable opacity for the heatmap and gaze trail overlays. The stimulus image is displayed as the background to provide spatial context for the gaze patterns.
Figure 4: Scanpath and heatmap view showing overlaid gaze trails and attention intensity for two participants viewing the same stimulus.
2.4.2. Temporal JVA Analysis
The temporal analysis view (Figure 5) shows the dynamics of attention coordination between participants. The upper graph displays the Euclidean distance between the participants’ gaze points over time, while the lower graph shows the JVA signal computed using a sliding window. Users can configure the distance threshold used to detect JVA and adjust smoothing settings to reduce noise in the temporal signals. Sustained JVA episodes are highlighted in the visualization to help users identify periods of shared attention.
Figure 5: Temporal JVA analysis showing gaze distance (top) and JVA signal (bottom) over time. Shaded regions indicate periods of sustained JVA.
2.4.3. Cross Recurrence Plot
The cross-recurrence plot in Figure 6 represents the temporal relationship between the gaze patterns of the two participants in a two-dimensional grid. Participant 1's time is shown on the vertical axis, while Participant 2's time is shown on the horizontal axis. Color-coded cells indicate spatial proximity between the participants’ gaze points at corresponding time points. Diagonal patterns can reveal synchronized attention sequences, while vertical or horizontal patterns can indicate differences in participants' attention timing. Users can adjust the sampling frequency and interact with the plot using zoom and pan controls.
The gaze point visualization (Figure 7) shows the exact gaze locations of both participants on the stimulus at a selected point in time. Users can navigate through the recording using a time slider and view the current timestamp. Play and pause controls allow users to examine the gaze behavior through animated playback. The stimulus image is displayed in the background to provide spatial context for the gaze points.
Figure 7: Gaze points on stimulus view showing exact gaze locations of both participants at a specific timestamp (t=1000ms). Users can navigate through time using the slider control.
2.4.5. Statistical Summary Dashboard
The statistical summary dashboard (Figure 8) provides quantitative measures of JVA across a session. It presents the total JVA time and percentage, the number of distinct JVA episodes, and the mean and median episode duration. The dashboard also includes spatial dispersion metrics and attention synchrony correlation. Summary statistics can be exported for further analysis and comparison across conditions or sessions.
Figure 8: Statistical summary dashboard displaying quantitative metrics of JVA, including total time, episode count, and synchrony measures.
3. Usability Evaluation of the Dashboard
To evaluate the usability of JAViD, we conducted a within-subjects comparative study. Here, participants used both JAViD and a Jupyter Notebook-based visualization workflow. The Jupyter Notebook contained the analytical code needed to create three selected visualizations available in JAViD. Participants first interacted with JAViD and completed a questionnaire that asked them to rate the dashboard functionalities and provide feedback through open-ended questions. Next, they used the Jupyter Notebook, ran the provided code, and generated the same three visualizations. After completing the notebook task, participants filled out a second questionnaire that collected their feedback and enabled a comparison between JAViD and the Jupyter Notebook workflow. We selected Jupyter Notebook as the comparison baseline because it is a common approach for generating visualizations by combining code and narrative text. This comparison allowed us to examine whether JAViD provided improvements in usability, learnability, and effectiveness over a traditional code-based workflow.
We recruited five participants for the usability study, including three males and two females. All participants were graduate students from different academic disciplines. Participants were selected to assess whether users with different levels of coding and domain knowledge could use JAViD to analyze JVA data. We summarized the feedback from the usability evaluation participants below.
3.1. Usability of JAViD
The usability study results indicate a strong positive response toward the dashboard across all evaluated dimensions. The majority of participants either agreed or strongly agreed that the system was easy to use, had a clear layout, and was easy to navigate. Participants also reported that the visualizations were interpretable and that the dashboard effectively supported their understanding of JVA. All participants indicated that the system was easy to learn, highlighting its accessibility for users with limited domain knowledge and coding experience. Figure 9 presents a summary of the findings.
Despite the overall positive feedback, the Cross Recurrence Plot was identified as the most difficult section to understand. This suggests that more complex visualizations may require additional guidance or explanation. Overall, these findings demonstrate that the dashboard provides a usable and intuitive interface for JVA analysis while also identifying opportunities to improve the interpretability of more advanced visual components.
Figure 9: Results of the usability study showing participant ratings across key usability aspects (ease of use, clarity, navigation, and interpretability).
4. User Feedback on JAViD
This section presents the participant feedback on the JAViD dashboard based on the layout, interactivity, complexity, and usability. It discusses what participants liked about the interface, the challenges they faced when interpreting the visualizations, and their suggestions for improvement. The feedback is helpful to identify the areas that worked well and ways to make the dashboard easier to understand and use.
4.1. Structure and Organization
Participants consistently praised the dashboard’s clean and well-structured layout. They noted that the organization made it easy to gain a broad overview of the data and navigate between different components. The logical grouping of visualizations supported efficient exploration and understanding of the data.
4.2. Interactivity
Interactive features, such as sliders, opacity controls, and participant toggles for P1 and P2, were highlighted as key strengths. These controls enabled users to dynamically explore the data and adjust visualizations in real time. Participants found that these features enhanced their interaction with the data and provided flexibility during the analysis.
4.3. Complexity
The Cross Recurrence Plot (CRP) was identified as the most challenging component of the dashboard. Four of the five participants reported difficulty interpreting it. This challenge was largely attributed to unfamiliarity with the underlying concept rather than issues with the interface itself. The feedback suggests that additional explanations could help users understand the purpose and interpretation of the CRP.
4.4. Domain Knowledge Gap
Some participants experienced difficulty because of limited familiarity with eye-tracking data and JVA concepts. This suggests that certain challenges were related to domain knowledge rather than the usability of the dashboard. Users who were less familiar with these concepts may require additional explanations to understand some of the visualizations and analytical measures.
4.5. Contextual Help
Participants expressed the need for additional support features to help them understand complex visualizations. Suggested features included hover tooltips, in-graph explanations, and a guided tutorial or onboarding process. These enhancements could provide users with information about the visualizations and metrics without requiring them to leave the dashboard.
“Maybe there could be a hover tutorial option where, if you hover over a specific graph or chart, it explains how the data was put into it.”
4.6. UI Sizing and Layout
Participants suggested improvements to the sizing and layout of some interface elements. Feedback indicated that font sizes and icon sizes could be increased and that some controls could be positioned closer together. In particular, participants suggested making the drag-and-drop icon more visible and placing the P1, P2, and opacity controls closer together to make them easier to access. These changes could improve readability and usability across different screen sizes.
“I felt some fonts were smaller when looking at the screen. The drag-and-drop option would be a bigger icon to find easily. Options for p1, p2, and opacity could be close by to use.”
5. Key Takeaways
The project demonstrated the practical value of JAViD while also providing several important lessons from the group development experience. The usability evaluation showed that JAViD can make JVA analysis more accessible by reducing the technical barriers associated with traditional tools such as Jupyter Notebook. Participants without programming backgrounds were able to complete analytical tasks, suggesting that the dashboard can support a wider range of users in exploring and understanding JVA data. Working as a group also helped us recognize and make effective use of each member’s strengths. Kumushini contributed domain knowledge in JVA, Rochana contributed technical skills for developing the dashboard, and Chameli contributed through background research and development of the dashboard. Through the project, we also gained a clear understanding of how human-computer interaction concepts can be translated into practical design decisions within an analytical dashboard. The project further helped us identify research gaps in JVA analysis and data visualization, particularly in making complex attention data easier to explore and interpret. The project also provided a valuable foundation for future research, as pilot studies and early-stage systems such as JAViD can help identify practical challenges, refine research questions, and guide the design of larger and more comprehensive studies.
A short video tutorial on how to use JAVID
For further details about JVA and JAViD, please refer to the links below:
In the hours following the release of CVE-2026-93479 for the project NGINX, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix a heap buffer overflow in worker script evaluation triggered by crafted requests when regex map captures are referenced before the map output, or when a non-cacheable variable is used in certain string expressions, causing worker restarts and potentially code execution if ASLR is disabled or bypassed. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Mariano Franecki, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
"No way to prevent this" say users of only language where this regularly happens
In the hours following the release of CVE-2026-15176 for the project NGINX, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix a heap buffer overflow in a worker process triggered by crafted requests when alias is combined with rewrite and $request_filename is referenced by set, return, or an access log format, causing worker restarts and potentially allowing code execution if ASLR is disabled or bypassed. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Ms. Odie Leuschke, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
"No way to prevent this" say users of only language where this regularly happens
In the hours following the release of CVE-2026-97205 for the project NGINX, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix a use-after-free in a worker process when asynchronous file I/O is enabled in certain configurations and an SSI page containing include virtual commands is processed, causing memory corruption or a worker crash and potentially allowing code execution if ASLR is disabled or bypassed. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Mariano Franecki, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
Take a look at the new hands-on course from School of Data, designed for educators and community practitioners. The course focuses on accuracy, source-grounded information and AI with conversational guardrails.
Figure 1: With my committee after the defense on July 23, 2026. From left: Dr. Michael L. Nelson, me, Dr. Michele C. Weigle and Dr. Vikas Ashok. Dr. Hae-Na Lee joined remotely.
Accessibility research and practice have focused on making content reachable: standards such as WCAG, checklists, compliance tooling. That work matters, and it has largely succeeded, but a page that passes those checks can still be very hard to use.
Sighted users take in a page's structure before they read a word of it. Layout, grouping, and visual cues tell them where things are and what to skip, and that costs them almost no effort. A screen reader delivers the same page linearly, one element at a time, as speech. Everything the layout was doing has to be rebuilt in the user's head as they go. The user has to remember more, loses their place more easily, and struggles with tasks that are trivial for someone who can see the page.
A blind user can reach every element on a compliant page and still give up before finishing a task.
The Research Challenge
My research asked one question: if we cannot fix every website, can we put something between the website and the screen reader that makes the page usable?
The idea of placing a proxy between a page and a screen reader is not new: earlier work used transcoding proxies to adapt pages for non-visual access. I call my version of that layer a proxy interface. It finds the semantically meaningful units on a page, reorganizes them into a stable representation aligned with how a screen reader is actually used, and exposes them through predictable interaction. It does not depend on how the page was authored.
This work is guided by three research questions.
RQ1: How can proxy interfaces support efficient non-visual navigation in arbitrary websites for blind screen reader users?
Blind users move linearly through long layouts to reach the section they want, and repeat that work on every visit. I built sTag, a browser extension that lets a user tag the page segments that matter to them and generates a personal Table-of-Contents overlay from those tags, independent of the page's markup. sTag also learns from user-created TOCs and reuses them to generate TOCs automatically for similar pages.
In a study with 15 blind participants, sTag significantly reduced both navigation effort and task time. Two results I had not predicted: 13 of the 15 participants used a single input modality instead of switching between keyboard and pointer, and six asked for a hierarchical TOC instead of a flat list so they could assign priority levels to their skip links. This work was published at ACM SAC 2021: "Semantic Table-of-Contents for Efficient Web Screen Reading," and I presented it at the 2021 WS-DL Research Expo.
RQ2: How can proxy interfaces improve non-visual interaction with structured web data, such as product lists or other data records requiring comparison?
Product listings, job results and classifieds are comparison tasks, and the information needed is split between the records themselves and the auxiliary segments around them: filters, sort options, page links. With a screen reader, every trip between those regions costs a traversal of the page.
I built InSupport, which uses machine learning to detect coherent data-record segments and reorganizes the auxiliary segments into a simplified proxy interface with natural-language querying, reachable instantly from anywhere on the page.
I evaluated it with two populations. With 14 blind participants using JAWS, average task time on the first task fell from 780.6 seconds with the screen reader alone and 540.9 seconds with the SaIL baseline to 288.6 seconds with InSupport; input actions fell from 611.9 and 395.9 to 187.2; and SUS rose from 53.9 and 68.9 to 86.1 (one-way ANOVA, F = 22.865, p < 0.0001). With 16 low-vision participants using screen magnifiers, with acuity from 20/100 to 20/700, SUS rose from 48.8 with the magnifier and 63.3 with space compaction to 84.2 with InSupport (F = 33.45, p < 0.001).
ACM IUI, "Where #HCI meets with #AI". A premier venue for reporting outstanding research and development on intelligent user interfaces, I was honored that one of our papers was accepted.
InSupport was also the subject of my PhD candidacy exam in July 2023.
A phd student, @jaf_ferdous is currently presenting his Ph.D. candidacy exam (Oral Component). The title of his presentation is "InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records."
RQ3: How can proxy interfaces enhance blind users' ability to understand, follow, and participate in online multi-party discussions where context and reply relationships are critical?
Before building anything here, I needed to know what the actual barriers were, so I ran semi-structured interviews with 20 blind screen reader users about their experience of online discussions.
Most participants (18 out of 20) said it was difficult to join a conversation already in progress, because of the listening effort required to work out where the discussion had gone. Sixteen out of 20 said their experience depended heavily on the type of discussion, with live threads far worse than asynchronous ones, and 16 said they did not feel productive or effective in online discussions at all. Participants preferred longer, context-rich posts, because those restate key details and reduce the need to jump back through a thread. What they asked for was summarization on demand, explicit relational cues, cleaner text for listening, and structure they could navigate selectively.
I turned the interview findings into five design goals and realized them as Standpoint, a browser extension that generates conversational context over a discussion thread on demand, anchored to the reader's current position and computed entirely on the reader's own device. It produces position-scoped summaries and explicit relation labels over the thread structure.
The technical evaluation showed the generated structure was reliable while the relation labels were uneven; I traced that to prompt design and corrected it there. In a pilot with five blind users, all five navigated large threads with less time and effort, and described a shift from grasping a discussion's mood to grasping its argument structure. They checked the relation labels, which they could verify, and were more cautious about trusting the AI-generated summaries. A large-scale controlled evaluation is the primary next step.
The Proxy Interface Pattern
The three systems are instances of one pattern. Each identifies structure that the page does not expose to a screen reader and turns it into something the user can work with:
sTag identifies user-defined page segments and turns them into a reusable navigation structure
InSupport identifies data-record and auxiliary segments and turns them into a queryable comparison interface
Standpoint identifies argumentative structure in a thread and turns it into position-scoped summaries and relation labels
None of them requires the website to change.
Figure 2: One proxy layer between the webpage and the screen reader, applied three times in the dissertation: to page structure (Chapter 3, RQ1), web data records (Chapter 4, RQ2) and conversational meaning (Chapter 5, RQ3).
Publications
Four of these papers make up the dissertation work across the three research questions above. Standpoint, the system built for RQ3, is currently under submission. The rest came out of collaborations in the lab.
The Hypertext paper was a departure from accessibility work: detecting human smuggling advertisements on social media.
Human smuggling is a pressing global issue. On the US-Mexico border, 'coyotes' increasingly turn to social media to advertise their illicit services. Effective countermeasures require detecting these ads. (1/5)
I was a graduate teaching assistant at ODU from Fall 2019 through Spring 2026, for Principles of Programming Languages (CS 355), Introduction to Theoretical Computer Science (CS 390), Computers in Society (CS 300), Introduction to Computer Science with Python (CS 115), Data Visualization (CS 625, graduate), Introduction to Programming with Python (CS 153) and Introduction to Programming with Java (CS 151). I taught the Python section of ODU's Data Science Boot Camp in 2022 and 2024.
Grateful to be part of the ODU data science bootcamp, surrounded by diverse minds from students to professors across various disciplines. Teaching with diversity brings challenges but immense satisfaction.
I completed ODU's GTA Instructor Institute in 2021 and the Preparing Future Faculty Certificate through the Career Pathways Program in 2025. I also judged the Virginia State Science and Engineering Fair in 2024.
Excited to serve as a judge at the Virginia State Science and Engineering Fair! A great honor to witness the innovative projects and bright minds shaping the future.
I reviewed for more than 30 venues during the PhD, including ACM TOCHI, ACM TACCESS, ACM TKDD, the International Journal on Digital Libraries, Universal Access in the Information Society, and the CHI, CSCW, ASSETS, DIS, IUI, EICS, ICMI, IDC, IMX, CUI, ETRA, NordiCHI, OzCHI, ISMAR, IEEE VR, ICWSM and CogSci conferences.
I served as Associate Chair for ACM CSCW 2025 (Research and Posters), ACM CSCW 2024 (Posters) and ACM CHI PLAY 2024 (Work in Progress), and on program committees for ACM CHI 2025 (Late-Breaking Work), ACM ETRA 2025 (Short Papers), ICWSM 2024, ACM ICMI 2023 and ACM EICS 2023.
Excited to announce that I will be serving as Associate Chair for the Research and Poster tracks at the upcoming @ACM_CSCW 2025, July cycle! Consider submitting your work to one of the premier HCI venues.
I was a student volunteer at ACM ASSETS from 2022 to 2024, and at CHI PLAY 2023, DIS 2022 and 2023 and RecSys 2022.
Happy to be selected as a student volunteer(SV) two years in a row for #ASSETS23. It is the main venue for researchers to present research on the design, assessment, and teaching of computers for individuals with disabilities.
I also blogged throughout the PhD, on my own work and on other people's. One I keep coming back to is the Global Accessibility Awareness Day piece on whether web archives are usable for blind readers, which asks my dissertation's question about web archives, a core research area of WS-DL.
Web archives are like time capsules of the internet, but are they accessible to everyone? ODU researchers are tackling this issue, focusing on improving usability for visually impaired users.
I presented the three systems and the argument that holds them together: that they are one pattern applied at three levels (page structure, data records and conversation), not three separate tools. Then I took questions, and then they sent me out of the room to wait while they decided.
Being called back into the room took only a few seconds, but I had imagined that moment for years. I remember the room more than the words: my advisor, colleagues from the lab, people who had watched this work change shape several times over. We took photographs afterwards. Somebody called me Dr. Ferdous, and it took me a while to realize they were talking to me.
Figure 4: "Congratulation!!! Dr Javedul" on the screen after the defense.
Lessons Learned
1. Do not measure yourself against whoever is closest to finishing
For a long stretch I compared my progress to whoever in the group was nearest to defending, which did not help me. Some years produce three papers and some produce none, and from inside a year with no papers there is no way to tell whether the next result is three months away or much further. The more useful comparison is with your own progress from the year before.
2. Life does not pause for the degree
Six months after I arrived, the pandemic closed everything, and as an international student I went through it on the other side of the world from my family. During the PhD I lost five people close to me. The hardest was my mother, in 2021. Travel was shut down, and I could not go home. Grieving from that far away was the most difficult part of the PhD. If you are an international student, work out early who you can lean on here, because you will need them sooner than you expect.
3. Keep the people at home close
I moved from Dhaka to Norfolk on my own, and my family supported work they were never in the room for. Any finish date you give them will probably move. Call more often than you feel like calling, and do not wait until you have good news.
4. Do the unglamorous community work, and do it early
I started reviewing because somebody asked me to, and I spent three years volunteering at conferences where the job was often carrying things across a hall. It does not look like much on a CV, but it is how I met most of the people I know in this field. A PhD can be isolating, and serving the community is one of the easiest ways to stay connected.
5. Find out which part of the job you want before you have to choose
I was a teaching assistant for almost the entire degree. Somewhere in the middle it stopped being the thing I did for funding and became the thing I wanted to do, and I would not have known that without years of doing it. That is what led me to a teaching position at Ottawa University.
Looking Forward
Teaching at Ottawa University
In August 2026 I joined Ottawa University as a Visiting Assistant Professor in the School of Arts and Sciences. This term I am teaching Introduction to Programming (ITS/CST 16163), Introduction to Computer Science (CST 20363), Introduction to Database (ITS 20163) and Microsoft Excel (ITS 25008).
After years of running lab sections, review sessions and office hours as a teaching assistant, having my own courses is the part of this move I most wanted. The Preparing Future Faculty program is where I settled on this direction.
Where the research goes
I want to keep working on the same thing: moving accessibility from access to usability. AI now mediates a lot of what people read online, and in my Standpoint pilot, blind readers were cautious about AI summaries and checked the parts they could verify. My next research direction is trust and user control in AI-mediated reading for blind users.
Staying in the community
I intend to keep reviewing and serving on program committees. CHI, CSCW and ASSETS are the main venues for this research, and I want to keep contributing to them.
Acknowledgment
I am deeply grateful to my advisor, Dr. Vikas Ashok, for his guidance, his patience through the long stretches when I had nothing to show him, and his belief in my abilities. His mentorship shaped this dissertation and the researcher I have become.
I gratefully acknowledge the Web Science and Digital Libraries group and the Human-Centered Computing Lab, whose discussions improved every idea in this work, and my friends and colleagues, who stood by me and kept me grounded through the hardest stretches.
I thank every study participant who shared their time and experience with me. This work does not exist without them.
Above all, I owe the greatest debt to my family. Their love, sacrifices, prayers and constant encouragement kept me going when the work felt overwhelming. This degree is theirs too.
In the hours following the release of CVE-2026-94300 for the project BIND 9 named, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix memory corruption in named triggered by replaying captured GSS-TSIG-signed UDP messages while the negotiated TSIG key remains valid, potentially allowing remote code execution when combined with crafted queries to an attacker-controlled zone. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Queen Aniya Rippin, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
In the hours following the release of CVE-2026-111813
for Apache Traffic Server, site reliability
workers and systems administrators scrambled to patch systems running its
experimental magick plugin. In its default unsigned configuration, the plugin
lets unauthenticated clients control arguments passed to ImageMagick convert,
allowing anyone to execute whatever command they want on the server.
The affected plugin is written in C++, whose users were relieved that this time
the attacker did not even have to corrupt memory to tell the server what to do.
"This was a terrible tragedy, but sometimes these things just happen and there's
nothing anyone can do to stop them," said programmer Mr. Friedrich Frami, whose
image processing plugin accepted instructions from anyone who could construct a
URL. "It's a shame, but what can we do? There really isn't any way to know whether
a stranger on the internet should be allowed to choose command arguments for a
program running on our server. We made signature verification optional, so we've
already done everything we can." At press time, administrators installing
versions 9.2.16, 10.1.5, or 10.2.1 to fix the issue were referring to themselves
and their situation as "helpless."
In the hours following the release of CVE-2026-111497 for the project Brotli, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix an out-of-bounds write in the Brotli decoder that permits remote code execution when processing malformed compressed streams. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Dr. Lester Cremin, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
In the hours following the release of CVE-2026-109618 for the project PHP, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix multiple memory safety vulnerabilities that allow attackers to execute arbitrary code on your server. This isn't something you can just mitigate and be fine. You MUST patch this. Don't try to bargain here. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Lady Miracle Kreiger, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
In the hours following the release of CVE-2026-107891 for the project musl, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix multiple integer overflows in regcomp() when compiling untrusted regular expressions, leading to undersized allocations and possible out-of-bounds heap writes, denial of service, and arbitrary code execution. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Dr. Otho Conroy, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
On October 1, 2026, the Library Innovation Lab’s Public Data Project hosted a panel of legal, technological, and policy experts at the Lewis International Law Center to consider how preservation of and access to US Census data can be appreciated and reimagined. The transcript below has been lightly edited for clarity.
Panelists:
danah boyd, Professor of Communications, Cornell University
Cynthia Dwork, Professor of Computer Science, Harvard University
Ruth Greenwood, Assistant Clinical Professor of Law, Harvard Law School
Jayshree Sarathy, Assistant Professor of Computer Science, Northeastern University
Moderator: Molly Hardy, Project Lead, Public Data Project
Welcome
Amanda Watson: I’m Amanda Watson, the Dean for Library and Information here at Harvard Law School. Many years ago, I was sitting at a table with authors and editors, and we were playing a little game, and one of the questions was, “What’s the most niche hobby you have?” And I said, “Well, I really love census data, and I like to take it and figure out things like how close one thing is to another and what it might mean.” And a man who was written up in the New York Times for doing the Sunday New York Times crossword puzzle in pen said, “Wow, so you’re like a nerd’s nerd.”
There are a lot of things I can say to welcome this panel and start this event, but the thing I want to say is, to all of you nerds’ nerds in this sanctuary for nerds, with another sanctuary for nerds behind us: thank goodness we still care about the census. That we care to ask the question, who are we? That we care to count, and then let brilliant people do things with that data and information. We don’t always get it right, but that we still care to ask, I think, is so pivotal. So I want to just say, hey, let’s nerd out together and celebrate the census and talk about why this matters to us and how we can move forward. I’ll send it to Molly now, and thank you all again for being here.
Molly Hardy: Thanks, Amanda, for inspiring words, always. And thank you to our audience, both here in the room and online, for being here today. And most of all, thank you to our panelists, who have taken time out of their busy schedules to discuss important matters with us. So many factors converge to make this the right time for the Public Data Project to host this event. Some are joyous and some are less so.
I’m also pleased to share that our friends at the Internet Archive’s Democracy’s Library have just digitized microfilms of the census from 1790 to 1930, as well as the Indian census rolls of 1885 to 1940. This film was originally produced by the National Archives and Records Administration. The collection holds 24,320 items and more than 47.8 million page images, and it is a delightful treasure trove for researchers of all types.
Behind us, you’re seeing a selection of photographs from the Library of Congress that will slowly move through as we discuss today. Just some of my favorite images of census workers at their job. I thought it was a particularly good tribute in light of the subject of danah’s book, which is, of course, the civil servants.
Now for the not-so-great news. The census has come under scrutiny, and not the good kind of scrutiny. The Office of Management and Budget is reviewing significant changes for the 2030 collections: one, reclassifying noncitizens as nonresidents; two, a rollback of race and ethnicity data collection standards; and three, the addition of a citizenship question. Changes to privacy protection have also been proposed, and those changes would have profound effects on how much census data can be shared.
Groups are mobilizing to call attention to the perils of these changes. If you’ve not already, I encourage you to check out a couple of them, Hands Off Our Census and The Census Counts. I’m mentioning just a few of the groups who are mobilizing as the October 13 public comment deadline draws near.
The census contains multitudes, both literal and figurative. We’re honored today to be joined by leading lights to help us place in perspective what a threat to the collection, as well as the preservation of and access to, this vital dataset must mean, and how it should be appreciated and reimagined. At this moment, I can think of no better way to appreciate public data anew as we testify and retestify to the census’s importance as what Dan Bouk simply refers to as “democracy’s data.”
Poster encouraging Americans to participate in the 1940 census. Source: Library of Congress.
Panelist Introductions
Molly Hardy: Now, without further ado, I’m going to turn to the panelists, introducing them and then asking each to share their own professional contributions and work with the census and how we understand it. All four of our panelists are incredibly accomplished, so please pardon me for keeping introductions short so that we can proceed to the heart of the matter.
First, we’ll hear from danah boyd, who is the Gary Gay Professor of Communication at Cornell University. Dr. boyd’s research focuses on the intersections of technology and society, with an eye towards how structural inequalities shape and are shaped by technologies. How fitting, then, that her latest book, Data Are Made, Not Found, is a story of the civil servants who work tirelessly, and increasingly under duress, to produce the census.
Next, we’ll hear from Cynthia Dwork, who is the Gordon McKay Professor of Computer Science here at Harvard at the John A. Paulson School of Engineering and Applied Sciences, and an affiliated faculty member here at the law school. Many of you know Cynthia for her pioneering work in differential privacy and its applications in protecting personal privacy in the census.
I encourage you to check out the editorial she published this summer in The Boston Globe, co-authored with our next panelist, Ruth Greenwood, who is an assistant clinical professor and director of the Election Law Clinic here at Harvard Law School. An expert in both election law and redistricting, Ruth previously co-directed Voting Rights and Redistricting at the Campaign Legal Center. She was lead counsel for voting rights at the Chicago Lawyers’ Committee for Civil Rights Under Law and was a redistricting fellow with the Democratic National Committee’s Voting Rights Institute.
And finally, a welcome home to Jayshree Sarathy, Assistant Professor of Computer Science at Northeastern University. Jayshree completed her PhD here at Harvard in Computer Science, where she was advised by Salil Vadhan and Jonathan Zittrain. And just to bring things full circle, in 2022, Jayshree and danah jointly published an article titled “Differential Perspectives: Epistemic Disconnects Surrounding the US Census Bureau’s Use of Differential Privacy.” Her work focuses on the sociotechnical dynamics of data science, with a focus on privacy, trust, and data access, all areas very germane to our conversation today.
So let’s please get started with danah — if you could just start by telling us about your engagements with the census.
danah boyd: Sure. Thank you all for coming out today. Indeed, this book came out last Thursday. I’m an ethnographer, which means that my work is to think about mapping cultural practice. This book is the culmination of 10 years of trying to understand what makes data legitimate, in the most high-stakes data we have. After all, if you think about it, it is a very weird thing that we have actually architected the US Constitution to anchor our representative democracy in the idea of collecting data and allocating political power accordingly.
And so for the last 230-plus years, we’ve embarked on this project of how to understand who our country is in the form of the population. This has had lots of crazy twists and turns. Margo Anderson wrote a brilliant book about the social history of the US census, so if you want to understand its history, because it’s crazy, everything you thought you understood about the US, you can actually read through the census. It’s fun.
But my goal was to understand what it takes to actually do this in real time. So I embedded inside the US Census Bureau to really appreciate what they were doing. Part of it is the recognition that we don’t often acknowledge the important work that civil servants do. They are the invisible workers. They are the people that just do the work on behalf of the American people, but they are constantly maligned.
I grew up in this country constantly being told that civil servants were stupid or malevolent, incompetent, that it could be done better by industry, that this is just the deep state. You’ve heard all of it. And civil servants unfortunately often even believe it, which is actually terrifying. But these people are so dedicated to a mission: let’s make certain that we have the best count possible. And so I wanted to tell their story. This book is character-driven. It tells their story in different ways through the various ups and downs of the 2020 census. So if you want to know about citizenship, if you want to know about residence rules, if you want to know about differential privacy, it’s here for you.
For the arguments in this book, I have three threads, and I want to highlight them to give you a sense. The first is that data are made, not found. That’s the title. But the point is that it’s not just, “Oh, the categories are socially constructed.” It’s a series of ongoing decisions, made with an eye to making the best data possible. To make that really concrete, I went back to 1880 — I was just too lazy to go further — and I looked from 1880 to 1950 at the age variable, how old are you, within census data. And it turns out that you get these spikes in years ending in zero and five, because humans round their age. They’re not 72, they’re 70.
And they don’t just round about their own age. They round about their housemate’s age. They round about their neighbor’s age. They don’t tell enumerators their actual age, and the enumerators are often too scared to ask them details. So they just look at you and are like, “You look like you’re 50. That sounds good.” And we move on. The Census Bureau receives this data, and if they published it as is, people would think they were nuts. So they go through all of these procedures, not just collecting the data but figuring out how to make it into democracy’s data. That’s the first big through line.
The next big through line is an argument about Jenga politics, which is going to be relevant today. Jenga is that wooden tower. Think about it as our administrative state, as people have been taking out pieces from it and adding pressure on top. People have been doing this for all sorts of different reasons, both Republicans and Democrats, for a long time. I tell lots of cases of where this occurs, where help is not helpful. Things that will also upend your idea of, “Oh, but participatory engagement, this is really helpful” — except when it’s not.
Civil servants are running around trying to keep that wobbly tower from falling over, but they need help. And I’d like to remind everybody, somebody will get blamed when the tower falls, and people will celebrate because it’s not their political party or whatever, but we all lose, and we all contributed to it. We need to understand how this Jenga tower is falling apart. If we care about democracy, and if we care about data, this becomes really important.
The final through line has to do with what makes data legitimate. You might have heard lots of stories about the 2020 census. You might be convinced that it was a terrible census. Here’s the thing: the data for the 2020 census were better than all data collected in the 19th and 20th centuries by the censuses. They were maybe more problematic than what happened in 2000 and 2010. But when you interrogate that, it turns out the problem is not differential privacy. It is not radical malfeasance or political dynamics. It is that a pandemic is a problem. It’s really hard to collect data during a pandemic.
The legitimacy of data isn’t rooted in some magical metric of data quality. It’s rooted in the social fabric that we build, the trust infrastructure, the ways we understand the systems.
But what does it mean that we believe these data to be crap? It’s really important to understand the consequences of our delegitimization of this. The legitimacy of data isn’t rooted in some magical metric of data quality. It’s rooted in the social fabric that we build, the trust infrastructure, the ways we understand the systems. I try to help you see which attacks on legitimacy are attacks on the social fabric. Because if we want to have nice things, we have to attend to the social fabric and not just think that it’s going to get magically fixed through law or through policy. It is going to be a lot of relational work, and that is as true for our data as it is for our broader political polarization issues.
Harper’s Weekly illustration of a man taking the census (1870). Source: Library of Congress.
Cynthia Dwork: So why don’t we start with what differential privacy is, because you may or may not know. Differential privacy is a mathematical guarantee. It says that the outcome of any statistical analysis is essentially equally likely whether any individual opts in or opts out of the dataset. So where does this likelihood come from?
The answer is we achieve differential privacy by the introduction of carefully calibrated noise. If the true answer for some enumeration is, say, 10,000, you might get an output of 10,002 or 9,996. And if the true enumeration is 10,001, you might also get those same numbers, just with slightly different probabilities. That’s where the randomness comes in, and that’s what “essentially equally likely” means. In the context of the census, this means that any collection of published statistics is essentially equally likely whether you chose to respond to the census or you decided not to. That’s all.
The idea is that if outcomes happen with essentially the same probability whether you joined or not, they’re really not leaking much information about you personally, because they wouldn’t have leaked it if you hadn’t joined. For any statistic or collection of statistics, there are zillions of potential differentially private algorithms, and the algorithms vary in their accuracy and noise trade-offs. Some might be optimized for one kind of data, some for a different kind. But the mathematical guarantee is always absolute.
So differential privacy does not mean the particular top-down algorithm that the Census Bureau used. It’s just a mathematical definition of privacy, and different computational methods will give you different instantiations of it. I wanted to mention also why it’s called “differential.” Everybody receives the same mathematical guarantee, and the term “differential” actually comes from calculus, where it refers to a small change in a quantity.
In the early days, we had a bound on these relative probabilities, and the bound had some name, say epsilon, and we didn’t know whether to call it epsilon privacy or epsilon non-privacy. Mike Schroeder suggested, “Why don’t you call it differential privacy?” And that seemed to be exactly mathematically the right thing. But everybody gets the same privacy protection. I think I’ll stop there.
Ruth Greenwood: I’m not good at math, so it’s quite intimidating to do anything after Cynthia Dwork. But I do know how to work with incredible people. My name is Ruth; I run the clinic here. I’m from Australia originally, but my career in America has been litigating voting rights and democracy cases.
When we think about democracy, we may all have different conceptions of it, but it will probably include some notion of the people and some notion of decision-makers. And in order to know who the people are in a democracy, you need information, you need data. Here I have printed out the Federal Register proposal to change the census, just because it makes me mad. There were parts where I wrote “CD,” as in “Cynthia Dwork needs to deal with this,” and other parts where I had my own thoughts.
Race is a social construct, but once you have constructed it, it has deep meaning in our society. To put your fingers in your ears and pretend it doesn’t exist gets you nowhere.
It quotes people like Justice Thomas saying that race is a social construct. So what they’re proposing to do is remove the race and ethnicity questions, among other things, as in prohibit them. As I like to say to my students, race is a social construct, but once you have constructed it, it has deep meaning in our society. To put your fingers in your ears and pretend it doesn’t exist gets you nowhere. In fact, it moves you backwards.
It’s the same thing when I talk about partisan gerrymandering. If you just look at a blank map of Massachusetts and draw some circles, without considering where there are Ds and Rs, you’re not going to know what you get. You know what’s better? Knowing what you’re going to get, caring about the values that go into that, and making an informed decision. So it’s very important to me that we have a sufficiently accurate census that also has privacy protections.
The way I came to know about Cynthia’s work was that there was this concern before the 2020 census — and this is all in danah’s book — from the nonprofit world. We really want accurate race and ethnicity data, because the Voting Rights Act — rest in peace after Callais — used to protect the right to vote for people of color. And you need to know where people of color are in order to protect them.
So all the nonprofit groups would help to do what they called “get out the count.” Let’s just make sure people get counted. But if you do something like say we’re going to have a citizenship question, that freaks everybody out. Not only do non-citizens not want to fill out a census, but citizens don’t want to fill out a census either, and then we get garbage data.
And the other side of it is, before the citizenship question came out, a nonprofit could go to communities and say, “Even if you are undocumented, you can fill out this census, and it’s important that you do, because there is this protection of privacy.” When I learned about the mathematical guarantee, I was like, “That’s pretty solid protection.” I can tell you this smart person invented this thing to protect your privacy. And so eventually all the civil rights groups helped to get out the count, to make sure that we had as accurate an understanding as we could of the race and ethnicity of people in the country.
And that led to not just, can we draw congressional districts that will elect a candidate of choice of Black voters? That’s important. But it matters for the state senate, and for the state legislature. It matters for school boards, public utility districts, port authorities. We elect so many things in America, and there are so many ways to have distortions in representation. I would prefer to be having a fight about whether this is the form of representation we want or that is the form of representation we want, rather than saying all the data is garbage and it doesn’t even matter what form of representation we want, because we couldn’t even assess whether we have it.
So that’s my interest in the census, and seeing the Federal Register notice was deeply distressing. One of the things people have said in the states after Louisiana v. Callais, which is the decision that essentially killed the Voting Rights Act, is: are there things we can do to help protect ourselves? I started saying, put into state legislation, just run your own census — this may be crazy, but people used to run their own census — if you want to have an accurate ability to protect. It won’t work for Congress, because you need to have accuracy between states, but for the sub-state level, having your own census is the only thing I can think of to be more helpful. So that’s where I’m at.
Jayshree Sarathy: I’m very excited to be in conversation with these amazing scholars who have worked on this topic longer than I have. I came first to differential privacy in my graduate work, about a decade after Cynthia and others had coined the term. I was really thinking, in 2018, 2019, about improving the statistical and algorithmic analyses that we can do with differential privacy, and at the time with an eye to how it might fit the census use cases or other use cases for federal or official statistics.
A lot of that work was in conversation and collaboration with census scientists, and I was really struck by how committed the scientists from the Census Bureau were, how knowledgeable they were, how much innovation was coming out of government. And this is beautifully documented in danah’s book. I was struck by how much work and thought was put into how we can take this definition of privacy, this commitment to privacy, and make it work with the many legal and bureaucratic and operational constraints that go into something as complex as the census.
In that process, I also had this moment of doing a deep dive into the census world and understanding the very interesting dynamics around how statistics and politics interact, and just how infrastructural the census is to our democracy as well as to science in the US. Since then, my research has shifted. In computer science, we were working on all of these exciting algorithmic techniques for differential privacy. We were excited that the Census Bureau was so invested in deploying this new technology, and it seemed like a straightforward modernization of privacy.
So I think myself and other computer scientists around me were really shocked when we saw how political and messy this became in the public. We were like, “This makes a lot of sense to do, and this is a natural progression of the technical advancements of the census. So what is happening? Why is this so contested?”
My research now really tries to understand that, and to bring it back to computer scientists. How can we rigorously think about the intersection of technology, especially privacy technology, with these very complex social and political contexts? How can we more deeply understand the context, and work on both designing tools and communication, to make sure that what we design as well-intended technologies doesn’t run into the many different problems it has run into with the census?
Part of that is really understanding, especially with the census, that it was such an eye-opening experience to work in the space, because there are so many different experts and so much knowledge, from the data users, data subjects, advocacy groups, and scientists, all working together on this project to create better data, all with that same goal. And still, there are a lot of challenges with that. So that’s how I came to the census, and my fascination is really with how we can understand and create more legitimate data.
Molly Hardy: Great. Thank you so much to all of you for setting the table so nicely. Technology and context, that’s something we think a lot about here at the Berkman Klein Center, at the Public Data Project, at the Library Innovation Lab, and I think that’s a really good framing here.
So danah, I’m going to start with you, if I may, and ask you about the relationship between democracy and data. Do you think this moment reveals something about the limitations of public data to protect democracy, or is it better thought of as the limitations of democracy to protect public data?
danah boyd: Right. One of the things that’s so fascinating about census history is that when the idea of creating this country and anchoring a democracy to data started — you know that the moment the data made politics, politics were going to make data. That is the entire history here. Ruth gave you a little taste of the Voting Rights Act, but it’s all the way down into all of these different pieces. What has been radically different in recent years has to do with where the site of contestation is.
Let me explain, because there was a previous, really interesting fight over pieces or mechanisms of the census, known as the 1920 case. In 1920, the census came back with reasonable enough data, though it was really hard to collect data during that period because there was a pandemic. At the time, they were having all these technocratic fights. Up until then, the way that Congress did math was, “I don’t want any of my friends to lose their seats in the House. We’ll just add some more seats for the states that grew bigger faster.” It was very funny math. And so people were saying, “Hey, what if we do math better? We can’t fit all these people into the Capitol Building, so let’s stop growing the size of the House.” All of this was coming into being at that 1920 moment.
Then a Harvard economist penned an op-ed on the night before Congress was supposed to meet and decide the reapportionment of the House. Some Southern Democrats were like, “Sweet, we like this economist.” His argument was basically that Congress was doing the wrong kind of math. So the Southern Democrats said, “We can’t possibly reapportion the House until we figure out the right kind of math.” They didn’t figure out the right kind of math, and in 1929 the Senate got so annoyed with the House that they pushed through a law that we think of as the Permanent Reapportionment Act.
There are many weird things about that law, but one that is very relevant to the present is that they said, “Well, Congress is so dysfunctional. Despite the fact that the Constitution gave Congress this responsibility, we give up, and we will give the responsibility to the president to figure out apportionment, and that will go great.” That, of course, got us into some very interesting fights for 2020.
Every component piece of the 2030 census will be finalized and going to printing by January of 2029. So I want you to understand that the next two years are the most critical years in this process. And the question is, is Congress going to take responsibility?
So I just want to give you a moment of what’s happening right now. That was the story of 2020, but we are living in the story of 2030. For those who don’t know census structures, it takes 12 years to plan and produce a census. To give you a sense of the timeline, every component piece of the 2030 census will be finalized and going to printing by January of 2029. So I want you to understand that the next two years are the most critical years in this process. And the question is, is Congress going to take responsibility?
There are three things that happened this summer that I want to flag, and I think we’re probably going to drill into more detail on them. The first, which is relevant to these folks, is that every form of noise injection into the census data has been banned. How they define it is really wonky, and we should talk about Chevron and Loper Bright and a bunch of other things, but one takeaway is: no more differential privacy. The result is that the Census Bureau is now not publishing critical data pieces that we all care about, because they don’t know how to resolve these competing expectations.
The second move was the strangest report you could possibly imagine. Basically, a group of civil society actors working on behalf of pro-Trump Republican allies came into the Bureau and managed to weasel their way into this process, and used a certain set of data to produce a report that was showing all of the non-citizens that voted in the 2020 election. Some of you might know that President Trump is convinced that he won the election if it were not for these non-citizens.
Weirdness number one: the report doesn’t prove his claim. Even if these were non-citizens, it doesn’t prove it. Weirdness number two: this is an unsigned report. No civil servants were involved in the making of it, and they didn’t say who was there. No methodology was given, no explanation of error. And everybody’s like, “What the heck is this?” Both Republicans and Democrats, economists and demographers, have said, “This is insane.” And this is the huge challenge of what it means to think about legitimacy.
The final thing, which is what Ruth has been hitting at, is something called the residence rules. Every decade, the Census Bureau publishes the alterations to the previous residence rules that it wants to make in order to make the next data better. The census is responsible for counting everybody once, only once, and in the right location. The residence rules are the details of how to count people in the right location. In the last couple of decades, the Census Bureau would make tiny micro changes, put them out publicly for comment, and give you thousands of pages of research and documentation telling you why to make these micro changes, which, let’s be honest, nobody ever read. Like the worst academic papers ever. But it was — “See? We did the work. This is why it’s a good idea.”
On September 10, something else happened. New residence rules came down, and they are radical ruptures of everything, complete with a heading that is very political, zero research or explanation of why any of this matters, and all civil servants were told that they are not allowed to speak to the public. And you have 30 days to comment. Comments are due by October 13.
I’ll highlight the top three, and I’ll do it quickly because I suspect Ruth will have better thoughts on it. One is changing the rules of how we count non-citizens. This is going to be a constitutional fight, because the Constitution is clear that you count all the people, but you can make a rule that says you do it differently, and then lawyers will figure it out.
Number two is that we have a lot of laws in this country, especially from the 1960s and 1970s, that are anchored to race data. We did this because we thought we could invert the collection of race data to address discrimination. So instead of changing the laws, they’ll just undermine the existence of the data. That’s just great. That’s the second point that Ruth is starting to talk about.
The third one is even weirder, and since we’re in a geeky audience of nerdy mindsets, I’m just going to lay out one piece of it. The 1790 rule of where to count people is at their “usual place of abode.” I love that word, “abode.” The idea is that you understand where people typically live throughout the year, and that’s where you count them. That’s how we’ve been doing it ever since.
This time, it’s about where you live from January 3rd to April 1st. That’s really weird until you think about snowbirds, people who live in New York except when they live in Florida for the winter. Now they’re to be counted in Florida. Migratory farming populations live in Minnesota throughout most of the year, except for the window in which there’s no farming in Minnesota, so they go down to the South. You have all of these different populations that go to the South during that window. If it was just that, it might make a little political sense, except there’s a really weird twist: you have to match the place where you fill out your census with your tax record.
Let’s start to unpack that. Those snowbirds who go south, they like to file their taxes in Florida, so they’re going to count. We’re great on that one. Those migrant farmers, they’re just going to get erased. Oh, and college students. It turns out no parent lets their child fill out their own taxes, because they just won’t do it well, so the parent fills it out for them, usually not at the same residence as their college. It’s a way of upending procedure after procedure.
That is just some of the mess of these residence rules, with no explanation and no accounting of whether the Census Bureau could even implement this, because there are so many pieces that they can’t implement. They are all silenced. One of the reasons people are asked to write comments, and there are now thousands of them, is to show the anger and to start to have everybody appreciate what’s going on. But litigation is coming, and it’s also about getting Congress to wake up to the fact that they are responsible for this.
So that’s the fight right now. We thought we were going to fight about gerrymandering. We’re now fighting about micro-procedures, about how we actually build our data, and that’s the way in which you bore the heck out of the public and you can bulldoze through people. That’s what’s going on, which is why I thank you for coming today, and I encourage you to understand these pieces, because this is the cornerstone of our democracy.
Molly Hardy: Great. Thank you so much, danah. What you’re saying there, about the minutiae that data requires, the level of attention it requires being a way to silence people, a way to numb people, thank you for drawing some of that out. Ruth, let’s go to you, please. If you want to expand on any of the explanations that danah laid out for us, and/or talk about consequences here, about how you see this playing out.
Ruth Greenwood: Yeah. Let me start with the second part first. The implication of what danah was saying, of not counting migrant farm workers and not counting students, yet counting people who are in the South in the wintertime, is to make sure that southern states, which are traditionally more Republican, have more seats in the House, and the northern states have fewer seats in the House. That’s a direct line. But the implications are massive.
I just want to read a quote that they have in the Federal Register from an earlier case. Admittedly, judges are bad at math and stats. These guys are bad at math and stats. I’m not good at math and stats, but I appreciate people that are good at it, and you, too, can appreciate people that are good at it without being able to do an equation yourself. Anyway, they say, “Although the census is the delight of statisticians and sociologists and serves as a convenient measuring stick for the dispensing of federal funds, it was not created for these purposes.” That’s the justification. Like, look, it’s been really nice to have all this data, but all we need to do is count the number of people, and by people, we really just mean people that we care about, not people that you care about.
I think the implication is, yes, there is going to be litigation, and my concern with litigation is manifold. The main one is that I don’t trust the Supreme Court on anything. The second is that they got it right in Department of Commerce v. New York, the case that said we shouldn’t have a citizenship question, and that came at the last minute because of some files found in the attic of a deceased gerrymanderer. It made it easier for John Roberts and friends to say, “Oh, this is all getting a little bit nasty. It seems like it’s pretty racist, what’s going on. We’re not going to do it.”
Of course, they’re covering it up, and they’re covering it up more and more and more. And the Supreme Court has got to a place where they’re fine with things being relatively racist; see all their recent decisions. So even though I think we got lucky with that prior decision, I’m very worried. And even though, if I gave this as an issue spotter to my students, they’d say, “Yeah, that doesn’t work,” and there are some very obvious holes, like backing up documentation, none at all, it’s got to go through the court system. I think the only slightly useful thing is that it’s still 2026, so there is some time to go through the court system.
But I just worry that we’re going to end up with a bunch of useless data, and that, of course, will have political implications, which seems to be what they care about, but it’s going to have all these other implications as well. I’ve been in touch with lots of the legal community about submitting comments on this, and to some extent the political science community. I was just like, “Please tell me economists and big business are explaining how this is bad for them, because maybe Republicans will care about that and it will stop them from doing this.” I don’t know. I don’t know if any of you have had other outreach about comments.
Molly Hardy: Thank you, and I think we should take that back up. But I do want to return to the questions around privacy and give you, Cynthia, first a chance to respond to these changes that we’ve seen. To oversimplify it, the removal of noise. And if you would maybe speak about how you see this as a, well, I don’t want to put words in your mouth, but perhaps a distortion between the protection of privacy and the gathering of good data.
The proposed changes to the privacy protection mechanisms roll back technology to 1972. … If you refuse to add any noise, then you simply can’t release many statistics while protecting confidentiality.
Cynthia Dwork: The proposed changes to the privacy protection mechanisms roll back technology to 1972. [I have] no words. Why? The noise infusion methods that were developed over the past half century were developed in response to a demand for more data, more granular data, more useful data. There’s a fundamental mathematical conflict, a fundamental law, sometimes called the database reconstruction theorem, that says that overly accurate estimates of too many statistics is blatantly non-private. It completely destroys privacy. This was a finding from Dinur and Nissim in 2003, before differential privacy was created.
So how do you get around “overly accurate”? You add a little bit of noise. If you refuse to add any noise, then you simply can’t release many statistics while protecting confidentiality. It’s just that basic. And by law, they have to protect confidentiality.
Cartoon depicting a man taking the census from an old woman (1890). Source: Library of Congress.
danah boyd: So it’s not a choice about confidentiality or not. It’s that we can’t release the data. That’s kind of what they’re arguing.
Cynthia Dwork: That’s exactly right. And so the demand that these noise infusion methods were created to respond to will simply be unmet. And this is what danah was talking about when she says that the ACS [American Community Survey] is not being published. And it’s because they can’t.
danah boyd: To be clear, the legal reason is there, but there are actually three reasons for confidentiality. I want people to hold on to all three, because they’re important. The first is operational. It turns out people aren’t willing to provide data when it’s going to be released to other people, which is why they set this in motion back in 1840.
And it’s not just about individual respondents. The Census Bureau relies on what are called administrative data, which means other agencies provide them with data that they use for quality assurance and all of these different procedures. If the Census Bureau is unable to protect those data, those other agencies don’t give them data. So it’s operationally necessary; it is legally required; and it is morally the right thing to do, because, of course, we have a long history of moral abuses of these data.
So I want to keep that in mind, because for the Census Bureau, it’s not just a matter of whether to follow the law. Trust me, the law is a big deal. I always loved being in a bathroom at the Census Bureau because, I’m a girl, I pee sitting down, and I stare at a door. And whenever I stared at that door, [on the door] there was a reminder of Title 13 and how I had to protect privacy. So I was reminded of it many times a day. This commitment is really central to what goes on, and it’s why Cynthia’s reasonably aghast at this, because there is no way of resolving this other than to not publish data. And I’d like to think we can have nice things, starting with a democracy. So what do we do with that complexity? That’s why this is so important.
Molly Hardy: That’s great. And I want to come back to the question of what do we do. But I want us to pause for a moment here. We’re talking a lot about privacy, but I think we’re also implicitly talking about trust, about the ways in which we want and need to be able to trust this data. And so, Jayshree, I’m thinking here specifically about the work that you and danah have done about this idea of a “statistical imaginary,” which I just love. If you could talk a little bit about what that is and how it might be operational in this moment that we’re in.
Jayshree Sarathy: Absolutely. We’ve been talking about how privacy is important for data subjects, or respondents, to trust that the Census Bureau will protect their information enough for them to feel comfortable giving it. But there’s also trust in the data itself when it comes out, trust by data users and the broader public. And you come back to this question of what makes data trustworthy, what makes data legitimate. Going back to 2020, and coming at it from a perspective of differential privacy — I actually think differential privacy is a fascinating window into this question, for two reasons.
The first is that, by the structure of DP [differential privacy] — by this theorem that Cynthia referenced and these explicit trade-offs — DP requires data analysts to shift their practices around using data, and a lot of these practices are the ways in which data users feel that the data is trustworthy and legitimate.
Some of our work at OpenDP at Harvard was looking at and showing how differential privacy means that data users are no longer able to access data directly. Often they have to use an interface, or there’s noise included, and so there’s a barrier that shifts the practices by which data analysts had traditionally felt they could trust the data, work with the data, and arbitrarily explore the data before they come to analytical conclusions.
So even if differential privacy is better than previous privacy protections in terms of accuracy, and can also lead to better statistical outcomes, in terms of more robust statistical analyses or protecting against p-hacking or things like that, there is still a sense in which not being able to work with data in the ways that data analysts have been used to for decades creates a gap in trust and legitimacy. And we can’t take away those practices without replacing them with new cultures around data analysis. That’s the first thing that I think DP brings out.
The second, as you mentioned, is that because differential privacy allows us to be transparent about how we add noise, it reveals fictions about data accuracy. danah and I wrote about this, building on a long literature about imaginaries. We talked about the statistical imaginary that we saw among census data users, where they tend to treat the published data as the best enumeration of the population, as fact, rather than as the product of a long and complex process of data collection, statistical editing, and privacy protection. It’s a really sophisticated process, but one that deals with uncertainty at every step. It’s not just a count, it’s a statistical estimate.
And that’s not how the data are used. Part of that, as danah details in her book, is the many pressures on census data. It’s used for so many different things downstream, from voting rights to emergency response to just local politics. When there’s so much pressure put on this data, it’s not easy for data users to think critically about how they work with the uncertainty in the data.
So this creates an opening: now that we talk about noise infusion for differential privacy in a very transparent way, data users are asked to comment on how much error is acceptable for their use cases. And of course, their answer is none. We didn’t have to think about this before, even though error already existed in the data, for privacy protections and other reasons.
Legitimacy is not just about data accuracy. It’s not a function of any quantitative metric about the accuracy of the data … It’s really about these practices around data and the imaginaries, or the perceptions, that data users have about the data.
So, we don’t want any additional error, and we don’t want to have to think about that, and I think that’s reasonable. It really comes back to the point that legitimacy is not just about data accuracy. It’s not a function of any quantitative metric about the accuracy of the data, or even about the response rates of the census. It’s really about these practices around data and the imaginaries, or the perceptions that data users have about the data, and how we can bridge that with internal understandings of the data.
Ruth Greenwood: I just want to ask a question, whoever knows. In terms of getting rid of the injection of noise, I presume it was nefarious, but I’m wondering which type of nefarious. Was it so that we wouldn’t be able to protect privacy and therefore could only have aggregate-level, not local, data? Was it that they’re like, “I don’t understand math, so I don’t want any math in it”? Or was it something else? Do we have a sense? I don’t know who knows this.
danah boyd: So the order comes from the Commerce Department, and in particular it comes from an external organization that for a long time has been drumbeating it through their blogs. They have misunderstood it many times. They think that it’s part of the apportionment piece as well as the redistricting piece, which is amazing. So part of the order is complete misunderstanding, on top of everything else. They believed, very effectively, that by getting rid of both of these things, they could get both Republicans and Democrats who have been in favor of — or have been challenging — differential privacy on their side.
The other thing is that it gave them the logic for the Federal Register notice that is in front of you. One of the things that made this very difficult was the introduction, in 2000, of having multiple race categories as an option. It created a form of complexity in the data that was actually really hard to mask. So they can say, “Ah, we’re going to get rid of this, but because we’ve got this privacy problem, we now can justify the removal of race and ethnicity data.” It becomes part of a story. To be clear, this is not coming from civil servants. This is political all the way down, and I mean partisan in this sense.
One of the things that’s very challenging right now is that a lot of data users who are very anti-differential privacy are having this awkward moment of, “Well, we’re even more [against] not having data.” And there was this fascinating moment, because one of the historians who has been most loud about his hatred of differential privacy — a journalist managed to get him on record saying, “Well, it turns out it wasn’t actually that bad.” Which was about as much of a mea culpa as you’re going to get out of that person. Here’s the quote from the article: “After years of tests, Ruggles said he was satisfied that the amount of noise was so minuscule it wouldn’t compromise data quality.”
Ruth Greenwood: Wow. That would’ve solved a whole lot of problems 10 years ago.
Jayshree Sarathy: Right. The sad part of this is that the admission comes six years later, even though technically it was sort of obvious that differential privacy was not going to impact the quality of the data very much. But those debates have created the fodder for where we are now, and have created this opening for using differential privacy in a political way.
Ruth Greenwood: I feel like I’m partly to blame for that. I was trying to talk to the civil rights groups. I talked to you, Cynthia, and I was like, “This makes sense.” Then I talked to the civil rights groups, and they were like, “We love the thing before, which was swapping,” which, if you don’t do math, is much easier to comprehend. Instead of having one person on Liberty Island and one person in Brooklyn, you just swap them, so then you don’t know if the person on Liberty Island is or is not the race that you’ve put. I get that. Whereas when you say, “Do a cool equation that Cynthia made,” I’m like, “Mm, don’t understand that.”
And the [lack of understanding], I think, then led to civil rights groups being able to say, “Well, maybe it’s going to affect the Native American population; maybe it’s going to affect this population” — sort of general concerns. So I think you’re right, it created the environment wherein we’ve now got this. And I also think everyone’s going to band together presumably again and be like, “Oh, no, this is terrible. We want to go back to that other thing.” But it’s a shame we had to get to this point where it’s so naked.
Taking the census from a group of Winnebago people in Wisconsin (c. 1911). Source: Library of Congress.
Cynthia Dwork: We were telling them. We were the Cassandras.
danah boyd: Well, and to be clear, one of the things you’ll get out of this book, if you get a chance to pick it up, is that this ain’t new. These fights go back in fascinating ways, and Cynthia kindly let you know that the last name of the person who was the loudest about this is Steve Ruggles. Richard Ruggles, his father, who was an economist, is at the center of the 1960s fights over the National Data Center and how you should aggregate all data together because privacy is no longer a serious thing. That triggered crazy congressional hearings. So these things are not new.
The same goes for the big thing about citizenship, which we think of as this contemporary fight. It goes back to this amazing ophthalmologist named John Tanton, who read The Population Bomb in the middle part of the 20th century and concluded that what was going to happen with massive climate change is that undesirable populations, as in Black and brown people, were going to leave their geographies and invade the North, as in Europe and North America.
To prevent this from happening, he would help create the laws to make certain that you couldn’t have migration. But he wanted the census to collect citizenship data to do this. He was the one who litigated first to try to do citizenship, and the Supreme Court went back to him and was like, “This is ludicrous. The answer is to count everybody.” But the baby lawyer that he raised up in this process is a guy by the name of Kris Kobach, who would go on to be the Trump administration’s advisor on immigration policy.
Because the current fights are going to keep shape-shifting into the future. This is the weird thing about the census: you find that unbelievable amounts of American history are rooted in it in weird ways.
So one of the things that I think is important to realize is that as we’re fighting these fights, let us acknowledge the tentacles and the different pieces of the backstory, and the made-for-TV moments of the estranged daughter who got the hard drives that are part of that backstory, and understand these layers and these humans and these processes. Because the current fights are going to keep shape-shifting into the future. This is the weird thing about the census: you find that unbelievable amounts of American history are rooted in it in weird ways.
My favorite of which is that we actually started a civil war in part because of a census tabulation error. At the time, the language was “deaf, dumb, and blind.” Okay? We’ll leave that alone. But the idea was that Black populations in the South appeared to be less “dumb” than northern populations, or less “idiot” — that was the “idiot” category — than in the North. That was used by the South to justify slavery, and it was entirely an error. It was just an oopsie.
I say all of this because when you’ve built an entire democracy anchored to data and an algorithm, there are problems that emerge. So understand and appreciate those pieces. As we’re swirling around, we have to navigate all of these people. And so, what does it mean to do participatory engagement when that can be so heavily politicized? Which is what we’ve seen.
What Is to Be Done
Molly Hardy: Participatory engagement seems like a good place to be as we start to think about what is to be done, and if there are some steps that you all would like to mention. Ruth, I’m particularly interested in hearing more about how you see this playing out in the courts and why. Again, I don’t want to put words in your mouth, but you seemed so adamant that you hope it’s not all going to be played out in the courts, because that might have a particular outcome. So maybe we could start there, and then I’d love to hear from others as well.
Ruth Greenwood: On the litigation side of things, I would say people should not, and maybe they don’t, put all of their hopes in “we’ll get the Supreme Court,” and then, when it’s the Supreme Court, we’re done. I literally wrote down as I was going, “get these in depositions.” There are all things that you can do through the process of litigation, which has a public media aspect to it, and which has people having to give sworn testimony about things, where you can maybe pull out more information that is useful for the public narrative. So ultimately, if there’s a loss, I’m always about winning the loss. Even if we lose, what else have we done along the way? And I think there are other things that we can do along the way.
I want to raise two other things. When danah was talking, it’s just always a hard thing when you’re on the side of nuance. We can’t just say to you all that data are good and no data are bad. Data can be good in different ways, and you need to understand it, and it’s just a harder sell than the other side, which is, let’s have no data because it’s a problem.
When you [danah] said this has all happened before, maybe by looking at the past of what they did, we can think about how we can formulate our response better. It’s not just a playbook for the people that are trying to destroy it all, but maybe it’s a playbook for us.
Jayshree Sarathy: Talking about participatory engagement, I was talking to Cynthia about this before we started. And I’m curious about your views, all of you. It seems to me that differential privacy was announced, I think, in 2018 or thereabouts for the 2020 census. And I don’t think that two years was quite enough at all to build consensus around a new technology, like a massive participatory effort to get input on how the technology was going to be deployed and how the privacy budget would be distributed across different groups. That was just not enough time.
I think maybe, as Ruth was saying, it’s 2026, we have a little bit more time, and we also maybe are more able to be aligned. I think privacy is just inherently a hard thing to get alignment on. Every group is coming at it with privacy versus equity, or privacy versus accuracy. But now we’re talking about data availability, and I think that’s something more of us are able to get aligned on.
We built coalitions from 2018 to 2020 that I’ve been really happy with, to be able to see behind the scenes how we’re able to align on messaging across technical experts, privacy experts, and advocacy groups. But what messaging do we use to mobilize broad support against these political interferences and to support the statistical independence of the Census Bureau and the way that it produces data products? So, my hope is that maybe we have time, and maybe we are more aligned, and that’s something we can continue to build, bridging alliances across different communities around the census.
Cynthia Dwork: Jayshree mentioned this view of privacy versus accuracy, privacy versus availability. Privacy has no advocates. The only advocates for privacy were the Census Bureau. Everybody who used data for their work wanted more data, more granular data, more raw data, more whatever. Anybody who had a business that was based on repackaging census data, which is a tremendously valuable contribution, was like, “How am I going to do this? I don’t understand the technology. I don’t know if I’m going to be able to ply my trade if we go through with this, and privacy should go away.”
I remember talking very early to people in charge of heart study data at Stanford, and they were using a particular technique that suppressed zip code information. And we said, “Well, how would you feel about a technology that allowed you to give zip code information and allow you to see changes from one small area to another?” And the answer was, “Gee, I don’t know. Go away. I’m HIPAA compliant. Get lost.” So nobody’s advocating for privacy.
Also, people don’t tend to understand that the threat of harm is real. This is not a question of, “But my neighbors already know that I’m gay.” It’s somebody in a neighborhood that you did not choose to live in making a list of all of the gay households in the country to distribute to a group of people who want to go and harass gay households. It’s telling a prosecutor in Mississippi that the 14-year-old daughter that you mentioned on your census report used to be your 4-year-old son 10 years ago. It’s ICE learning the visa status of your elderly parent. The harms are real, and for some reason most of us have trouble imagining them. So I’m hoping that these conversations will at least try to make privacy something that people actually care about.
If you released perfectly accurate information and destroyed everybody’s privacy, you’ll have no chance of getting data from them in the future. It’s over. It’s going to be a very short-lived win for accuracy if that’s what happens.
I want to add one other thing, which is that there’s a time element that people often forget with privacy and statistics. They think it’s privacy versus accuracy. But if you released perfectly accurate information and destroyed everybody’s privacy, you’ll have no chance of getting data from them in the future. It’s over. It’s going to be a very short-lived win for accuracy if that’s what happens.
danah boyd: There is an election in this country in November. A new Congress will be seated in January. We don’t know who yet, but there will be a new Congress. And they’re kind of like goldfish. They don’t ever remember that they need to actually think about the census, even if they were here 10 years ago, and they need to be reminded of it. They need to understand that it’s important and that it’s their job to protect this. So this is where we think about that classic act of political activity.
The second key thing is, we mentioned participatory issues. One of the things that’s been really interesting is that in the 2000 census, Ken Prewitt, who was then the director — and we lost him this summer, and his memorial was this week, so may he rest in peace — wanted to make the census more legitimate and do more civic engagement. But one of the things he struggled with in recent years is that he never imagined how that would backfire, and how people’s recognition of the civic importance of the census, combined with their failure to understand how the systems work, would mean that they would create new forms of friction.
One of the cases that I tell in this book is what happens when a group of very well-meaning advocates try really hard to get out the count for their local community in a way that produces an undercount. So if we want to be good civic engagement folks around this more participatory work, we need to learn more of the pieces, the esoteric, the weird, the technocratic. We need to understand pieces of it, and we need to do it in a way that’s not binaristic: data are all good or all bad; the Census Bureau does it right or wrong. These kinds of binaries are causing us tremendous harm, and that’s where I go back to the social fabric piece.
One of my sort of jokes: I don’t know if you’ve ever been to Reykjavik. It’s a fascinating city. In the middle of one of its central squares, there is a Monument to the Unnamed Bureaucrat. Look it up. It’s really fun, this ridiculous statue. But it’s a country that actually recognizes that to have a functioning democracy, especially at this kind of scale, requires a functional administrative state.
One of the things I think about a lot is that when I started spending time at the Census Bureau, I’d go to the cafeteria a lot, and I’d walk in and hear 16 different languages. I started talking to a lot of these folks who were primarily junior in terms of government ranking, even though they were often much older. And I’d be like, “I’m going to guess you’re naturalized.” “Oh, yeah, I come from [another] country.” And I’d say, “Why did you come work for the US federal government as a naturalized citizen?”
And they told me over and over, “I come from a country where there’s corruption, authoritarianism,” — choose your favorite horrible story — “and I believe in democracy, and I believe in America, and I believe in the power of civil service, and so I wanted to serve.” I say this because we need to recognize the humanity of these people.
So my other invitation to you is to think about what it would take for you to adopt a civil servant. Go write a nice letter to a civil servant thanking them for their service. You can find any number of them on LinkedIn. This is a moment of just being able to acknowledge the humanity and the dedication of these people, and to figure out how you build that social fabric, because the people who are working on our collective behalf have become invisible to us. When they are invisible to us, we forget about their importance.
I’ll conclude with one thing that I loved, going back through the histories of all this. George Washington was a very important person in this country during the time in which the Constitutional Convention occurred, and he decided that his job was to be quiet, because he felt he would have too much sway if he talked at all. So he didn’t talk, except once.
We have reached the point where our systems are failing because they are at scale. The way that you respond to this is not by adding more algorithms. It is by figuring out how to build social fabrics at scale.
He stood up when they proposed that a representative in the House of Representatives should represent 40,000 people in this country. He said, “If they represent 40,000 people, we will not trust them. They need to only maximally represent 30,000 people.” I would like to remind us that our representatives currently represent a million people. We have reached the point where our systems are failing because they are at scale. The way that you respond to this is not by adding more algorithms. It is by figuring out how to build social fabrics at scale. And the way you do that is that each and every one of us takes a moment to connect and bridge, and connect and bridge, and we can have a democracy when we have a healthy and robust social fabric.
Audience and Online Q&A
Molly Hardy: Lovely. Thank you so much. We have some great questions online, but we may also have some questions in the room, and my colleague will be circulating with a microphone. Let me get started with one online, because it’s actually a perfect pickup point. Jennifer Chapman asks: “Census data is very important to the United States. Can anyone speak to the implications of the issues we are discussing today from a global perspective? US government data is used by governments, institutions, and individuals around the world. How might census data issues have impacts beyond US borders?”
danah boyd: I actually spent a bunch of the summer with different government agencies in Europe, which is always fun. First off, kudos to the team and all the people that did differential privacy, because a lot of that is rolling out globally as people realize the power of it. Let’s note that the US’s way of doing statistics is unique and weird and makes everybody else scratch their heads.
You go to Statistics Denmark, and they have a totally different set of problems, because their statistical system includes all of their healthcare, the connection into all of the places you’ve used public transportation, your credit card, your ability to get electricity, everything. Their biggest problem, and one of the things that I love, is what they call the “undead,” which is when somebody accidentally — in a dataset — is made dead before they’re physically dead, because it turns off everything. So they’ve got a different set of issues. They don’t allocate political power like we do, but they are really intrigued by these pieces.
There are also beautiful histories. There’s a wonderful scholar here named Arunabh Ghosh, who does this beautiful accounting of what happened when Mao Zedong decided that China would not allow for capitalist Western statistics, as in Bayesian statistics, and so they would count everything. This ended very badly, because it turns out counting grains of rice is a bad idea, and there was a great famine that was rooted in statistics. So there are histories of all of these other pieces that I want to acknowledge.
People are very much watching us. They do think some of our systems are completely weird. Like the French, who got rid of race in 1978. They didn’t get rid of racism, let’s be clear. They do not understand why this is an issue in the US. I explained to them that we anchored laws to it, and they’re like, “Well, why did you do that?” And I’m like, “That’s a very good question, but that’s where we’re at.”
So I think it’s notable that there are different global pieces of this, that each of those environments has its own context, and that we need to understand them in conversation. But there are also amazing groups that come together around the globe and share techniques. This is in many ways where the shared commitment around morality and privacy, which is rooted in the aftermath of World War II, has led us to a particular history, and then led many other countries to a history. That’s why the differential privacy materials are so exciting and important for places that have less highly partisan federal statistics.
Jayshree Sarathy: To danah’s point, having talked to a few other government agencies who were watching 2020 and being like, “We know that we should also implement differential privacy, but let’s see how this goes in the US,” I think it was obviously very shocking to many other governments: “Oh, okay. This is going to be the impact of lawsuits and lessons.” It doesn’t have to be. So I think governments have now learned that differential privacy doesn’t mean a terrible mess. That is a uniquely US situation, tied to how deeply political and how infrastructural the census is in the US.
Now we’re seeing more small-scale deployments in different countries, and I do think there are some lessons they’ve taken from how to communicate about differential privacy and the technical effort that it takes to build this out. Also, it’s really nice to see the focus that has remained on privacy in Europe and other countries, when what we’re hearing in the US is that people have stopped caring about privacy, especially in industry, and from the executive branch there is not that support from the top to care about privacy. So I think we can really look to other countries to take the lead.
Molly Hardy: Great. Thank you. Other questions?
Audience member: Thank you. I have a more technical question about differential privacy. Is it possible to inject it at the data collection level, such that we no longer have a problem with any government saying that we can’t release things because of privacy?
Cynthia Dwork: If you mean in some sense randomizing the data before they’re collected — yes, that’s exactly, for example, what Apple does with your iPhone and all of your devices. It’s possible to get much better accuracy if you centralize the data and have all of the raw data to work with from the beginning, and then just add a little bit of noise rather than noising up everything all along the way. But it is possible.
danah boyd: The other place where it’s really possible in the census case is that there are many procedures that are not just about collecting data directly from a person. There’s the administrative data piece, and there’s another procedure known as imputation. That’s when a household, they’ve knocked on [the door] six times, they’ve gone to all the proxies, and they can’t figure it out, but they see that there are cars and bikes, and they want to fill this out. They do a set of procedures known as imputation, and for various litigious reasons, they haven’t been able to modernize key parts of this. Had this been combined with differential privacy, you would actually get a much better way of going about it, and it’s another place where it can happen.
Likewise in the ways that the administrative data are used. So there are places in the system that could have been done differently than they were in the 2020 census. But many of the decisions about why they were done the way they were in the 2020 census were organizational, political, really just like, “Oh, that’s not another technical system that we have time to upgrade.” The process of making census data has so many elements to it, so there are other sites of it.
But right now the civil servants are under such duress that I don’t expect to see that for 2030. It is an interesting question, though, certainly for other organizations and other foreign governments who are looking at this from different parts of the pipeline. They’re not going to implement it in the same way the United States did, because the ways in which their systems fit together look different.
Molly Hardy: Another question?
Audience member: Hi. I’m teaching a class this term called “Decision-Driven Data.” We talk about some of these kinds of things, but I’m curious about your perspective. In the past, the census represented a deliberate effort by the US government to collect data. Now all of us are data collectors all the time, generating a tremendous amount of data, most of which goes to the private sector. I’m curious about your thoughts on what this might mean for the census, or for government data collection efforts generally, where the official data might not actually be the best data out there, because there’s just so much more available to be used.
danah boyd: I’ll tell you a story, because I had the privilege of being inside and being a professional meeting attender. I went to one of the meetings when they were finalizing the contract with the advertising firm, and the advertising firm was like, “We’ve got this. We do Coca-Cola, we do all this other stuff. We can reach everybody.” And the civil servant who was there was like, “That’s great. So how do you reach people who can’t afford a Coca-Cola?” And he’s like, “Well, we don’t.” And I’m like, “Hmm.” “And how do you reach one-year-olds who are not going to drink Coca-Cola?” “Well, we don’t.” And he just kept going.
One of the things the civil servant pointed out is, “Look, I can give us the 90 percent of the population right now without thinking. We have that trace data. We have it in all different ways. I can give us that next 9 percent with pretty good accuracy, with some reasonable margins of error in there. And then there’s all of the process of getting that last 1 percent.”
The private sector is great at the 90 percent. So is the Census Bureau. The private sector is pretty good at the next 8 or 9 percent. So is the Census Bureau. The private sector doesn’t give two swats about the last 1 percent. That’s what makes the Census Bureau unique, because its job is to count everybody once and only once, and in the right location.
We anchored democracy to the idea that we would go down to the single-digit person, and that means that getting everybody is the North Star that we anchor this process to.
If your goal is to do marketing, who cares? I’m a truck driver based on my travel schedule. Who cares? But we anchored democracy to the idea that we would go down to the single-digit person, and that means that getting everybody is the North Star that we anchor this process to. If we want to give up on that, okay, we can give up on our democracy as it’s structured. We can rewrite our Constitution. There are many different things that would change in the Constitution first. This is probably not top of my list, but there are ways of doing it. But here’s the democracy we have.
That’s the thing to also keep in mind: there are kinds of variables and details, also not in the decennial census, although it’s about to include citizenship, where the commercial sector doesn’t have these data at all, and it uses these data as the benchmark for nearly everything.
I always laugh at it because it’s more visible with the weather situation. “We don’t need NOAA” — the National Oceanic and Atmospheric Administration — “because we have the Weather Channel.” Where do you think the data comes from? You want nationally representative data, where do you think the benchmarks come from? You want a blue dot that gets you somewhere so you can arrive here, where do you think the geographic locations come from? These are the moments where census data is so much more infrastructural than many people even realize, and it’s because these data are cooked with such insane care.
It’s one of the reasons why our data as a country looks so different from many other countries that are not focusing on cooking the data and making sure everybody is counted. That’s the weirdness of the American project. So yes, there’s a lot of data, and for certain kinds of questions, commercial data is much better. But there are a lot of pieces of this where the US Census Bureau is a unique added value to things that actually affect the business community and everybody else out there.
Ruth Greenwood: I just want to add that the courts have decisions accepting the Census Bureau’s methods, but not other methods. If it becomes clear that we have garbage from the Census Bureau, I appreciate all the flow-on effects, and everybody has a benchmark. But if I’m trying to represent a community and litigate, then it takes a lot more effort to prove the veracity, or at least the usefulness, of private-sector data, so that I can get somebody in as an expert to say, “These data are telling me that this is what’s happening,” and then I can use that for my claim. Over time, we could build that up, and we could get it for non-apportionment things.
For apportionment, unless the Supreme Court changes its mind, there’s no other option except the Census Bureau. I’ve tried to use even just a little bit of survey data in addition to census data, to try to look at whether you can say that people vote differently by race, like white voters prefer some candidates and Latino voters prefer different candidates. And the judge is like, “Okay, well, not looking at that. I’ll look at the census data, that’s what we do.” And I was like, “But these surveys, they’re well-tested, and here’s all of the academic backing, and blah, blah, blah.”
So it’s not impossible. It’s just a real challenge if you’re going to try to get it into litigation. If I don’t have the race data, then I’m going to have to get it from somewhere else. I’m going to have to find a way to find out people’s race in a way that I can then prove to a court is accurate. And you know what’s really great if you’re on the defense side? “Oh my God, there’s no data, you can’t sue us. Thanks so much, bye.” It’s a really tough one. It’s also a big issue in public health, where they are looking for denominators. If you want to know disease rates by groups and wealth and things like that, you need the denominators. That comes from the census.
Molly Hardy: We’ve got another question from online that we need to wrap up soon: “The Census Bureau publishes so many data projects that aren’t the decennial census, such as the American Community Survey [ACS] and the Five-Year Economic Census. Are we seeing similar pressures on these datasets? And if so, what effects might this have?”
danah boyd: Yes. The disclosure avoidance rule from the summer applies to all products, and they can seek exceptions to it, but the process is very complicated. Some of the economics folks are seeking exceptions for a handful of things, but it’s become really egregious and difficult. Keep in mind also that the current Secretary of Commerce must approve all contracts. So even the rules are one thing, but the ability to implement relies on contracts, and if the Secretary has to approve a contract, and there’s a complete hiring freeze, the ability to implement change has become untenable.
The ruling on the Federal Register notice is explicitly about the decennial. However, one of the big open question marks has to do with the ACS, which is technically, according to the legal framework, under the decennial. So it is quite probable that many of the things that affect the decennial will directly affect it, not just because of the system. When you produce the decennial, you then produce this thing called population estimates, and that becomes the sample frame for doing the ACS. So you get this downstream effect, and it’s not quite clear, but it’s probable that the orders are going to affect them as well.
The Bureau is massively understaffed. The normal cycle for the decennial is that you start to build up mid-decade. You get a lot more staffing and a lot more funding, and you peak in the years ending in zero. Then people take those lessons and move them through all the other products as the number of workers declines. We are not seeing that rise. And there’s a lot of internal grousing about the decennial getting too much extra attention and love.
But Congress likes the things that affect its jobs, and that has always been a part of it. There are members of Congress right now who want the ACS eliminated entirely. There’s a proposal within all of this to make the ACS voluntary and optional. That would completely destroy the sample frame and make it impossible. There are other fights that are going to happen over the decennial that will affect downstream products, and the staffing is already undermined.
On top of the things the Census Bureau officially publishes, it contracts for other federal agencies to help them produce their data. We’ve already lost education statistics. We’re losing key parts of health statistics, all of these things, because of the failure to be able to do these contracts. So the statistical system is in a very precarious place. The decennial happens to be the thing you can get Congress to pay attention to. Hopefully, as we pay attention to that, we can start to see all of the other pieces of it as well.
Molly Hardy: Well, I think we’re at time, and I just want to take a moment to thank our panelists again for so much insight, so much brilliance, and for your dedication to this work. So thank you.
Barry Ritholtz posted this up-and-right chart, which is a classic of exponential extrapolation. Given the difficulty, delays, and cost of bringing the current generation of data center projects on-line, highlighted by Oracle's "force majeure" on a 2.5GW project which is clearly not going to be complete next year, the idea that 4-5GW projects will be up and running in 2029 seems optimistic.
Below the fold I apply some skepticism, because this is starting to look like the classic South Park episode Gnomes:
The episode is also known for the nonsensical business plan that the gnomes of the title devise, whose three steps consist of:
Collect underpants
?
Profit
which later became a common meme used to mock poorly-thought-out business and political strategies.
By 2031, annual spending on AI infrastructure could reach $1.5 trillion, including new data center infrastructure and compute capacity as well as ongoing upgrades to the installed base of GPUs, memory, and networking equipment.
If we assume that capital expenditures amount to about 25% of industry revenue (an ambitious but reasonable percentage based on trends among cloud providers), sustaining this level of investment would require an AI market approaching $6 trillion annually.
...
Together, the consumer and enterprise AI market could total between $1.2 trillion and $1.8 trillion, leaving about $4.2 trillion of new revenue to reach the $6 trillion market that we estimate will be necessary to fund the buildout (see Figure 2).
That revenue must come from new sources of economic value.
Note that when they say "about" $4.2T they really mean "at least" $4.2T — it could be $4.8T. Their ideas on where "about" $4.2T of new revenue to pay for data centers would come from are laughable. For example, they suggest that autonomous vehicles would supply 10% of it but don't notice that the AI for the vehicles wouldn't be in a data center, it would be in the vehicle.
In an emailed newsletter, Laura Bratton at The Information has reported on a new study
The study was from the Wharton School:
Jeremy Korst, one of the Wharton study's authors, said it was "very difficult" to find "actual, materially measurable financial impacts" from AI spending.
Anybody remember back in early 2023, when people imagined that ChatGPT was going to 10x every employee’s productivity and radically boost the GDP’s of entire countries?
So far, there’s zero evidence of anything like that happening at scale.
A recent study of actual coding practices across hundreds of firms finds that human code review forms a significant “bottleneck” for the overall efficiency of AI coding tools, resulting in “little evidence that firms increase software output or reduce employment” by using them. Any efficiency increased during the actual coding phase, the study authors find, is “absorbed by downstream constraints in the production process”; as “the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments.”
...
For now, though, letting AI write your code looks like a double-edged sword, with increases in coding speed counteracted by similar increases in human code review time and effort. It’s the kind of result that makes us wonder if the considerable time and expense to get AI coding agents working is really worth it for most companies.
The company spent $56.1 million on compute costs between January and August of 2026 alone and as of August was losing “approximately $8-9 million per month,” which it said was a massive improvement over the previous year. The company spent $19.7 million on employee salaries and benefits during that same eight-month time period and that a lot of employees quit because they “began questioning Character’s long-term viability,” according to the company.
Character said that its finances had improved to only be losing $8-9 million per month by taking “significant actions to reduce operational costs and improve revenues” by spending less on compute by simplifying its AI model, laying off staff, and injecting ads into its chats.
The AI cloud-computing provider’s unsecured debt due in 2032 traded below 80 cents on the dollar on Wednesday, according to Trace pricing data. That’s widely considered the threshold for when a borrower is financially strained.
At that price, the bonds yield more than 15% — a figure noted by Palihapitiya that led Musk to respond “yikes” on his social media platform.
someone wanting to buy $10mn of protection against a CoreWeave default needs to pay $1.95mn on day one, then an additional $125k each quarter.
That is rather expensive. Oracle's debt for the "force majeure" data center is trading at 84 cents; this isn't just about Coreweave.
Fundamentally, finding a working step 2 depends upon finding applications for LLMs that increase productivity significantly more than their cost. This would generate profits for the application vendors above the costs they have to pay the data centers, which might increase revenue more than enough to cover the power, depreciation and debt service. It looks like step 2 remains elusive.
To improve the annotation feature for our web archiving livestreams, I have been looking at different Quality Assurance tools that detect replay problems and explain why specific resources fail to load. Most of the Quality Assurance tools that determine the quality of the archived web page do not specify the reason why each web resource failed to load. Zhu et al.’s FidEx tool and Archive-It's Wayback QA tool try to assist with this problem.
Zhu, Sun, and Madhyastha’s paper "Detecting and Diagnosing Errors in Serving Archived Web Pages" describes their approach (Figure 1) for detecting problems with replay quality that are caused by rewriting errors made by the replay system. Their approach involves comparing the layout trees (HTML DOM [Document Object Model] and CSSOM [CSS Object Model]) and the JavaScript writes for the live and archived web page. This can be an improvement over comparing screenshots of the web page, since taking a screenshot of a dynamic web page can result in significant differences in the pixel values when videos, images, and text are frequently being changed. In this case, the layout tree and the JavaScript writes should still be the same on the live and archived web page.
Video 1: Presentation video about the FidEx tool.
Zhu et al.’s presentation slides about FidEx.
Zhu et al.’s FidEx tool can detect replay problems and can output a JSON file that explains why an error occurred. Figure 2 shows an example output file which contains a list of detected errors for each archived web page. Each error specifies the exact line of JavaScript code (including replay system scripts like wombat.js) and the associated error message.
Figure 2: FidEx's pinpoint output file shows errors that occurred during replay.
Archive-It's Wayback QA is another tool that helps with identifying the reason why a resource did not load during replay. According to Yale’s Web Archiving Working Group Quality Assurance Report Archive-It's Wayback QA tool has an option for exporting a CSV file with all of the missing content. This file shows all of the data that would be shown on the Wayback QA tab (Figure 3), including the missing resource's URI-R, the URI-R for the web page that loaded the resource, the reason for the missing content, file type, size, and crawl date. The reason for the missing content is categorized as Excluded, Invalid_Access_Denied, Invalid_Unknown, Invalid_Not_Found, or Not Crawled. The Excluded category is for resources that were intentionally excluded in the missing documents list, like ads and analytics URLs.
FidEx's pinpoint output file provides a more detailed description than Wayback QA for the errors that occurred when replaying the archived web page, but it does not specify which errors caused a specific resource from failing to load. Wayback QA’s “Possible Reason” column, provides a high-level description of what went wrong for each web resource that failed to load. FidEx’s approach for displaying the errors that occurred during replay is useful for technical users who are familiar with inspecting the JavaScript and HTML files for a web page to see which resources failed to load because of a replay error. In contrast, Wayback QA’s high-level description of the possible error is better suited for archivists looking to quickly identify and patch missing resources.
Another difference between these tools is that FidEx helps identify problems that impact interactional and visual correspondence and Wayback QA displays errors that are associated with the completeness of the archived web page. Reyes Ayala’s grounded theory (Paper: “Correspondence as the primary measure of information quality for web archives: A human-centered grounded theory study”) defined interactional correspondence, visual correspondence, and completeness. Interactional correspondence is associated with being able to perform the same user interactions (e.g., scrolling down a web page to load more web resources or clicking the play button for a video) on the live and archived versions of the web page. Visual correspondence is the visual similarity between the live web page and the archived web page. Completeness for an archived web page is associated with the number of sub-resources that were originally on the live web page that have been archived. In a previous blog post, I surveyed Quality Assurance tools that can help with measuring visual correspondence, completeness, and interactional correspondence.
Summary
FidEx and Wayback QA can help with identifying why a sub-resource failed to load during replay of an archived web page. FidEx focuses on identifying problems caused because of the rewriting that is done by the replay system. Fixing the problems identified by FidEx can help improve the interactional and visual correspondence for an archived web page. Wayback QA assists with problems associated with the completeness of an archived web page, since it specifies possible reasons why each missing resource was not archived.
Learning about tools like FidEx and Wayback QA helps with identifying what information would be useful to display during a web archiving livestream when an error occurs during replay. Our goal for the annotation feature is to create automated annotations that provide context information for missing sub-resources. Currently our annotations specify when a sub-resource was not archived. In the future, we want to display potential reasons why the sub-resource failed to replay, such as specifying when a URL matching error or other replay rewriting error occurred. The approaches used by tools like FidEx and Jawa (paper: "Jawa: Web Archival in the Era of JavaScript") could assist with determining which type of rewriting error caused a sub-resource to fail to replay.
The project continues to be a convenient source of tiny-tiny programs
written in various programming languages. Independent researchers take
the programs and do their independent research no-questions-asked.
Thirty-four days before she’s sent to prison, notorious Silicon Valley
founder Elizabeth Holmes invites a skeptical film crew to document her
every move. What begins as an intimate portrait of an enigmatic felon
becomes a mind-bending three-year journey into the abyss.
Catweazle is a British children’s fantasy television series, starring
Geoffrey Bayldon in the title role, and created by Richard Carpenter for
London Weekend Television. The first series, produced and directed in
1969 by Quentin Lawrence, was screened in the UK on ITV in 1970. The
second series, produced and directed in 1970 by David Reid and David
Lane, was shown in 1971. Both series had thirteen episodes, most written
by Carpenter, who also published a book for each series based on the
scripts.
The Bed Sitting Room is a 1969 British black comedy film directed by
Richard Lester and starring an ensemble cast of British comic actors.[2]
It was written by John Antrobus based on the 1962 play of the same name
by Spike Milligan and himself. The film is an absurdist,
post-apocalyptic, satirical black comedy.
When we build, when we think, when we act, we do so in honor of every
object, thought, and action which has preceded this one, and we are
changed by the process. AI wants us to forget this, by making every
output its own miraculous creation. But we owe the world more than that.
Computer vision has improved tremendously recently; and projector
technology is steadily advancing, with bright, high resolution, small
footprint projectors now being a reality.
User privacy would naturally have to be a first-order consideration for
such a device; that can be addressed through existing best practices
like on-device processing, hardware guardrails against unauthorized
camera access (camera only turns on when activated by the user, with a
hard-wired indicator light; physical cover; etc.), and so on.
I think in light of the alternatives that the industry is pushing right
now, it’s a path for the future of computing really worth pursuing.
DiPLab is releasing a report on the surveillance of data workers today.
Entitled “Surveillance at Scale : How AI training platforms gather
personal data from European workers” this report is a research project
in collaboration with Reversing.works and Intérêt à Agir, funded by the
Digital Freedom Fund and the European Artificial Intelligence &
Society Fund.
This short film was made for Society for Social Studies of Science (4S)
50th Anniversary celebrations, and exhibited at 4S Toronto 2026. I was
asked to make film set in 2076, looking back at 4S and Science &
Technology Studies and what they could become…
Artificial intelligence has gone from struggling with basic arithmetic
to helping solve some of mathematics’ most famous open problems — and
the field is reckoning with what that means.
Steven Strogatz, a professor of mathematics at Cornell University,
describes the excitement and grief of watching A.I. take on work that
gives people a sense of purpose — and asks what could be lost if
machines deliver answers without the human struggle, insight and
understanding that make solving problems meaningful.
If you’re trying to eat more vegetables, consider slurping more soup.
Most of them rely on vegetables as a base, and from there, it’s easy to
add even more for a belly-filling, nourishing meal. Without much effort,
you can easily work two servings of vegetables into a single bowl.
Soup also happens to be one of the most simple and soul-warming things
to make, a breeze to customize and easily packed for lunches, or frozen
for a future you who is too busy and too tired to cook. These 24 soups
give you all of that, along with plenty of nutrition and comfort in
every bowl.
The $28.5T forecast compares to U.S. Q1 2026 nominal GDP of nearly $32T, with the estimate for the market of AI enterprise applications of $22.7T about 70% of total U.S. economic output.
Sam Altman and Dario Amodei just aren't this good, but their projections of their Total Available Market (TAM) are still turning out to be vastly optimistic. In AI's Affordability Crisis I showed evidence that the AI platforms could no longer afford the massive subsidies they were using to artifically inflate demand for their product, and that reducing the subsidies had made their enterprise customers reconsider their enthusiasm for deploying them. This is leading to investors belatedly realizing that AI platforms' projections of their TAM and thus their valuations are totally implausible.
This re-calibration is just one of the many signs that the AI bubble is about to deflate. Below the fold I present a necessarily incomplete list of them, which I will try to update as more appear.
AI now accounts for nearly half of all IG issuance, 87% of VC funding and a growing share of HY, underscoring how deeply the AI investment cycle has penetrated every corner of finance.
Here’s the title page of this month’s Panmure Liberum market update from strategists Joachim Klement and Francisca Reis. Our emphasis in bold below:
In 1929, the cyclically-adjusted P/E-ratio (CAPE) of the S&P 500 reached 32.6x according to Prof. Robert Shiller’s data. This was 1.8 standard deviations above trend at the time. In 2000, the CAPE reached 44.2x, or 3.3 standard deviations above trend – a clear sign of a bubble. However, as our chart below shows, earnings in both instances were within normal range, less than one standard deviation above trend.
Today, the CAPE is at 41.0x, or 2.9 standard deviations above trend. Once again, we are clearly in bubble territory for stock market valuations. However, unlike in previous bubbles, we are having extremely high CAPE at a time when earnings themselves are 1.8 standard deviations above trend. In other words, we are in a valuation bubble at a time when earnings are in a bubble themselves.
If we correct for the earnings bubble, the current CAPE would be 67.6x or 4.6 standard deviations above trend, a bubble that surpasses anything ever seen in US history by an extreme margin. If valuations followed a normal distribution (which they don’t, so don’t take this literally), this would happen in 0.00019% of months or once every 43,432 years.
A massive chunk of this quarter’s blockbuster “growth” didn’t come from selling more software, shipping more microchips, or delivering more packages. Instead, it came from an accounting rule that forced massive, illiquid “paper gains” onto the income statements of tech giants.
In Q1 2026 alone, just three companies—Alphabet, Amazon, and Nvidia—reported a staggering $69.2 billion in non-operating windfall under their Other Income and Expenses (OI&E) lines. When you run the macro numbers, this single accounting phenomenon artificially inflated the entire S&P 500’s quarterly earnings by about 12%.
That 12% takes the factor from 67.6 to 75.7. Wang notes that correcting for the 12%, "the Q1 2026 earnings growth rate will not be very different from the 5-year average of 16%." In other words, the bubble is feeding upon itself — increased stock prices causes increased earnings causes increased stock price ... But suppose, for example, that OpenAI were to suffer a down round. Then Alphabet, Amazon, Nvidia and others who included paper gains in the "other income" on the way up would have to include paper losses in their income on the way down, amplifying the crash. OpenAI's last round valued the company around $750B and they were planning an IPO for at least $1T, but had to postpone it.
The foundation model era — roughly 2020 to 2025 — is over. The forces that defined it have inverted. Open source models have reached frontier performance while inference costs approach zero, exposing what was always structurally true: pre-training large language models at scale is not a durable competitive moat. The US government's formal designation of Anthropic as a supply chain risk in February 2026 accelerated a transition already underway — but did not cause it. The paper argues that the AI industry is restructuring simultaneously along four axes: economic, as the circular financing structure that inflated foundation model valuations collapses; technical, as the pre-training scaling paradigm gives way to post-training optimization, test-time compute, and agentic composition; commercial, as application-layer integrators displace the foundation model companies whose commodity they now consume; and political, as the government asserts its historic role as gatekeeper of strategic technology. These are not separate disruptions. They are one structural shift, arriving together.
The most consequential and least-discussed dimension is the permanent divergence between commercial AI and a classified national security AI track — built on different data, governed by different rules, and developing capabilities the public ecosystem cannot see, measure, or govern. Like every dual-use technology that has altered the calculus of state power, AI is being brought under government authority not by design but by the structural logic of what it is. The paper further argues that open-weight models are the counterintuitive instrument of sovereign control: a government that holds the weights commands the capability on its own terms, without dependence on vendor policy, financial continuity, or personnel clearance. The apparent openness of distributed model weights is, from a deploying government's sovereignty standpoint, the most governable architecture — because what cannot be withdrawn by a vendor's API policy cannot be taken away.
Grogran implicitly assumes that AI is useful but that the current margins and thus the valuations aren't sustainable.
DeepSeek-V4-Pro is priced through its API at $1.74 USD per 1 million input tokens on a cache miss and $3.48 per million output tokens.
That puts a simple one-million-input, one-million-output comparison at $5.22. With cached input, the input price drops to $0.145 per million tokens, bringing that same blended comparison down to $3.625.
That is dramatically cheaper than the current premium pricing from OpenAI and Anthropic. GPT-5.5 is priced at $5.00 per million input tokens and $30.00 per million output tokens, for a combined $35.00 in the same simple comparison.
Claude Opus 4.7 is priced at $5.00 input and $25.00 output, for a combined $30.00.
Six times cheaper for an equivalent product is likely to cause pricing pressure on the incumbents, who need to raise not reduce prices. DeepSeek is likely also subsidizing usage, but they do have real advantages. First, they have fewer resources so are forced to to be inventive. Second, the 40% of infrastructure capex that isn't the racks is much cheaper in China. Third, the power component of opex is much cheaper and more available in China. The result is:
In practical terms, DeepSeek does not need to win every leaderboard row to matter. If it can deliver near-frontier performance on many enterprise-relevant agent and reasoning tasks at roughly one-sixth to one-seventh the standard API cost of GPT-5.5 or Claude Opus 4.7, it still forces a major rethink of the economics of advanced AI deployment.
DeepSeek-V4-Pro-Max is clearly the strongest open-weight model in the field right now, and it is unusually close to frontier closed systems on several practical benchmarks.
While GPT-5.5 and Claude Opus 4.7 still retain the lead in most direct head-to-head comparisons across the company's benchmark charts, DeepSeek V4 Pro gets close while being dramatically cheaper and openly available.
Dan Davies agrees that the margins aren't sustainable but differs as to why in tokenalysis and john henry:
Which then brings to mind another issue – how confident are we in the pricing power that underpins that 60% gross margin in the first place? In the last paragraph I was talking about the R&D equivalent of a price war, but the normal kind is also possible. The combination of price-sensitive B2B customers, big fixed costs and rewards going to the dominant player doesn’t suggest to me that pricing power is going to be sustainable indefinitely.
But, I think there’s a danger of missing the big picture here. Which is that, when large companies are telling their employees to be sensible and use AI tokens wisely, then the game is up. The race is over and John Henry won against the steam hammer. If you need a human being in the loop to decide on the allocation of AI tokens, then all those predictions of mass redundancy are gone.
But what if the payoff takes longer than consensus assumes? That question is particularly pressing given that token prices continue to decline and Chinese models are gaining ground, both in their share of the world's most-used models and in token usage, where they now lead their US counterparts among the top 20 models,
Slok provides two charts, the first tracks market share by country of origin monthly since January 2025 among the top 50 models. It is bad for the US, showing a steady erosion of US market share until, eyeballing it, as of May 2026 it is roughly a 60/40 US/Chinese split.
Clearly, the market is voting with its feet that the prices charged by US AI companies are unsustainable.
The second compares monthly token use among the top 20 models by country of origin between May and June this year. It is much worse for the US.In May the Chinese had 80% of the US usage. In June they had 185% of the US usage. If this rate of market erosion were to continue for a few months it would be impossible for investors to continue to imagine the golden future awaiting OpenAI, Anthropic, xAI, Meta, Oracle and the neoclouds.
Slok's analysis of the fallout of the bubble deflating is worth reading. This is an issue I plan to return to in a future post.
Chinese AI providers are making more money; Zhipu’s revenues went up 60 times in Q1 compared to last year; Alibaba is the company behind Qwen, and their revenues are up 15 times just since the beginning of this year. The lion’s share of the profits, though, are still being realized by the hardware side; the chipmakers, at least for now. The model companies’ revenues are rising, and steeply. But so is their cost of compute.
That dynamic is causing some Chinese labs to raise prices; the cost to use Tencent Cloud increased by over five times in March, and Alibaba, Zhipu, and ByteDance also hiked prices.
DeepSeek, however, went the other way. Their latest version is priced at just a fourth of their introductory product, and they rolled out dynamic pricing that is aimed at corporate users, who use their models during the workday.
But even with these price increases, Chinese models just cost far less than those on offer from Silicon Valley, and explains those big jumps in the exports, we can say, of Chinese AI tokens. The LLM’s out of China are “90% as good at 10% of the cost”, and American firms are buying more tokens from Chinese companies. In early 2025, token demand from Chinese LLM’s was about zero. Even the release of DeepSeek didn’t move the needle much, but by the end of the year the secret was out, and it’s been a choppy but steady ride up to 46%, today.
Why would you invest in a company whose competitors were “90% as good at 10% of the cost” and which was rapidly losing market share?
If the margins, and thus the rational valuations, of AI companies are unsustainable, how long can the market remain irrational? In The Second Derivative: Why No One Understands the AI Boom Groundbreaker starts by examining the 2008 crash:
The implicit underwriting assumption, shared by originator and borrower alike, was that the loan would never reach its reset: rising home values would manufacture equity, the borrower would refinance into a fresh teaser and the clock would start again. The structure was a treadmill, and the treadmill was powered by appreciation. It worked spectacularly while it worked. Nearly four in five subprime hybrid ARMs originated in 2003 had been refinanced away by the end of 2006.
Now watch the timing. National home-price appreciation did not crash in 2006. It decelerated. The year-over-year rate of gain, which had run in the mid-to-high teens through 2004 and into early 2005, began bleeding off - still positive, still printing green, but slowing. Prices were higher than they had ever been. And yet, with prices at their peak and still rising, subprime delinquencies inflected upward.
...
The deceleration was endogenous to the structure; the structure required ever-accelerating prices to keep refinancing its way out of its own reset schedule, and no series accelerates forever.
The second derivative was always going to roll over. When it did, the first derivative followed it down through zero, negative equity spread from the margin inward, and the defaults the market insisted were caused by “falling prices” had in fact begun a year earlier, when prices were still rising but had stopped rising faster.
The market is pricing AI as a technology cycle when its actual anatomy is that of a credit-driven real estate cycle - which is precisely why the 2008 mechanics apply - and the two break for entirely different reasons.
...
Walk down the AI build-out and every feature is a property development in disguise: a data center on entitled land, financed with debt against the structure and leased to tenants on take-or-pay terms. This is not a software business that happens to own servers. It is a real estate business that happens to compute.
When a hyperscaler or a neocloud reports a record capex figure, the financial press reads it as confidence, as proof of demand. Read it instead as origination volume. Each gigawatt of committed build is a loan extended to whichever tenant has signed the take-or-pay beneath it, and the credit quality of that loan is precisely the credit quality of the tenant. The market is celebrating loan growth and calling it revenue growth.
Loans in this market take the form of Remaining Performance Obligations (RPOs). The biggest borrower in this market is OpenAI:
OpenAI has committed to pay for compute on a scale without precedent in corporate history: multi-year, take-or-pay capacity contracts whose aggregate obligations run to the hundreds of billions of dollars. Against them sits an operating business that does not yet earn a profit - revenue real, large, and growing quickly, but short of covering the company’s own cash burn and nowhere near covering the contracted payments.
Those payments are therefore not serviced out of earnings. They are serviced out of financing, and financing, for a borrower in this position, is available on a single condition: that each new round price above the last.
Measured as a level, OpenAI’s valuation is the most remarkable appreciation in the history of private markets - roughly $86 billion in early 2024, then about $157 billion, $300 billion, $500 billion, and approximately $852 billion by the spring of 2026. Measured as a rate of change, the same series inverts: the round-over-round step-up ran 1.83×, 1.91×, 1.67×, 1.70×, and falls to roughly 1.23× implied by the reported public-offering target. Private marks are inherently lumpy - negotiated, episodic, set by a handful of insiders - so no single step is decisive. But the trend is unmistakable: it bends down, and it bends hardest at the one mark set by the deepest, most unforgiving pool of capital - the public market. The implied IPO step-up is both the lowest in the sequence and the hardest to negotiate, and it is the one the structure must actually clear. This arithmetic is also the most probable explanation for OpenAI’s recent IPO delay.
In the same way that the private stocks generating unrealized gains are not liquid assets, neither are the Remaining Perfoemance Obligations:
An RPO is not a liquid asset; it is a forward contractual commitment - a promise of future payment in exchange for future compute. And a multi-year commitment is worth exactly the creditworthiness of the entity on the other end of it. When that entity is investment-grade and cash-generative, the backlog is what it claims to be: high-quality visibility, merely deferred. When that entity is a pre-profit company that loses tens of billions a year and can pay only by continuously refinancing its own equity valuation, the backlog is something else entirely. It is a subprime commitment, used to justify massive, un-depreciated capital expenditure, reported to shareholders as structural strength.
Now price the credit quality of that book. Of roughly $2.1 trillion in aggregate contracted backlog across the four big platforms, about half - on the order of $1.05 trillion - is owed by OpenAI and Anthropic. Microsoft’s book is about 49% these two names; Oracle’s is 54%, with roughly $300 billion owed by OpenAI alone; Google’s is 43%; Amazon’s is 51%.
The hyperscaler has, in economic substance, extended a concentrated, unsecured loan to cash-burning tenants. The RPO that Wall Street values as forward revenue is, in reality, a credit exposure to borrowers with no operating income.
Major financial institutions are similarly concerned. In their 2026 Annual Economic Report, the Bank for International Settlements writes:
In the near term, the ongoing AI investment boom raises questions about the sustainability of the current economic expansion. The five largest hyperscalers are set to spend over a trillion US dollars on AI-related capital expenditure from 2025 through 2026. These commitments are outpacing earnings and the free cash flow of these firms, leading some to issue debt to raise additional financing (Graph 11.A). This investment race may be partly driven by the perception that only a small number of players with superior technology will ultimately dominate the market shares. The intense competition raises the risk of firms over-committing resources to investment projects with still uncertain returns, leaving all firms vulnerable to disappointments in AI payoffs. Model analysis based on such contest motives highlights the downside risk of current AI exuberance. As competitive pressure drives capex higher, the net economic surplus – the total payoff less investment costs – declines for the sector as a whole and could turn negative in adverse scenarios (Graph 11.B). Disappointment in returns could trigger a sudden pullback in financing and turn the capex boom into a protracted investment bust, with potential knock-on effects on financial conditions (see below).
Another risk is that the AI boom runs into a supply side roadblock. The AI build-out has recently been facing growing bottlenecks in electricity, advanced semiconductors and grid equipment. Fast-growing demand for computing power is already pressuring electricity prices and input costs, with potential spillovers to inflation. Looking ahead, these temporary shortages may also amplify over-investment, as firms attempt to lock in future capacity through long-dated contracts that further expose them to any disappointments in demand.
Historical episodes of investment booms offer instructive parallels (Graph 11.C). The canal mania of the 1830s, the British railway mania in the 1840s, the electrification exuberance of the late 1920s (roaring 20s) and the dotcom boom of the late 90s all shared one common trait: a genuine technological breakthrough that attracted capital in excess of what commercial returns could ultimately justify. These episodes ended with an eventual reversal in investment, inducing economy-wide recessions. The scale and pace of the current AI investment boom accompanied by expectations of large productivity payoffs bear resemblance to these precedents, highlighting potential downside risks in the near term.
A draft report inside the Treasury Department is set to warn of the risks posed by the artificial intelligence market, likening key aspects of it to the dotcom bubble that upended the U.S. economy when it burst in the early 2000s.
The document, the existence and contents of which have not been previously reported but was obtained by NOTUS, is a significant departure from the Trump administration’s public tone, which has focused on encouraging unrelenting investment to unlock exponential growth.
Career Treasury analysts found that AI firms are more deeply entrenched in the U.S. economy than their dotcom predecessors and pose significant risk to the entire system if financial conditions change, productivity goals are missed or various choke points stymie growth.
Vanderbilt University's Asad Ramzanali has a detailed look at the range of impacts from the burst bubble, and an set of optimistic suggestions for policy responses in After the AI Crash. He frames the problem thus:
Companies are investing trillions of dollars based on tens of billions of dollars in revenues. Analysts at J.P. Morgan anticipate $5 trillion of AI infrastructure investment in the next five years. They estimate that the industry will need to generate annual revenues of $650 billion to justify this level of investment, while consultants at Bain & Co. estimate $2 trillion in needed annual revenues. Yet, OpenAI and Anthropic earned $13 billion and $4 billion, respectively, in 2025 revenues. OpenAI’s own financial expectations suggest negative cash flow until 2030, and Anthropic expects a small profit no sooner than 2029. Alphabet, Meta, Amazon, and others may experience increased marginal revenue from integrating AI into existing products, but that is far from certain.
Note that over the next 5 years around $3T (60% of $5T) of the investment in AI infrastructure goes in to buying the hardware which should be fully depreciated over much less than 5 years. So the gap between the investments and the revenue is much bigger than it appears.
Blackstone's QTS said on Thursday it had terminated its planned Digital Gateway data center project in Virginia and withdrawn the associated filings after years of planning and regulatory review.
The data center operator has faced years of local opposition and litigation over the project, despite it being approved by the Prince William Board of County Supervisors.
Among the hyperscalers, Oracle is the most exposed because, as Ed Zitron noted:
And Oracle ... is a company that, even before the AI bubble, was massively indebted. It just so happens that, as a result of its tryst with OpenAI, Larry Ellison saw fit to twist the debt knob to eleven.
Six firms alone — including Oracle, Microsoft and Meta — have committed $850 billion for data centers leases that haven’t begun yet. Oracle holds the largest share of these commitments owing to its $300 billion Stargate contract with OpenAI.
When Oracle mentions the risk of nonpayment, the unnamed elephant in the room is OpenAI. As part of the Stargate deal with the AI company, Oracle is developing massive data centers across the country to provide cloud computing power. For this plan to work, OpenAI needs to pay its Oracle Cloud Infrastructure bills.
“Some of our customers may be highly leveraged and subject to their own operating and regulatory risks and, even if our credit review and analysis mechanisms work properly, we may experience risks of non-payment and non-performance in our dealings with such parties,” Oracle said in the filing.
SoftBank Group Corp.’s talks with potential creditors to raise at least $6 billion from a margin loan backed by its OpenAI stake have stalled, people familiar with the matter said, just weeks after the Japanese conglomerate cut its initial target from $10 billion.
SoftBank Group has reopened talks with a consortium of lenders for a $10 billion loan backed by its stake in OpenAI, after earlier attempts to secure a loan stalled over concerns about the difficulty of valuing private companies, two people familiar with the matter said.
To make lenders more comfortable, the Japanese technology investor is offering to guarantee repayment of the loan, giving banks recourse to SoftBank if the OpenAI shares pledged as collateral lose value, the people said.
This all seems to indicate that potential lenders, such as banks, are highly skeptical of the value of OpenAI stock.
Despite Grok being so bad that employees use Claude, Musk is touting SpaceX as an AI company. This resulted in the most overvalued IPO in history, which failed to raise enough money to avoid the immediate need to borrow $25B. Nir Kaissar's SpaceX Is Junk. That’s What the Bond Market Says reports on the bond market's reaction:
Ratings companies and the bond market have very different views about how things are going. SpaceX’s bonds have an average rating of BBB across the three majors, Moody’s Ratings, S&P Global Ratings and Fitch Ratings, according to credit scores compiled by Bloomberg. In the alphabet soup of bond ratings, it’s the lowest grade still considered quality before falling into junk territory.
The bond market has other ideas. There, quality is judged by a bond’s credit spread or the additional yield it offers above Treasuries with similar maturity. The wider the spread, the lower the quality. Corporate bonds with a BBB rating are trading at an average credit spread of 0.92 percentage point. SpaceX’s bonds, by contrast, trade at a significantly greater average spread of 1.62 percentage points across maturities, higher than BB rated junk bonds’ average spread of 1.55 percentage points.
The bond market seems to agree with Softbank's lenders about the AI bubble.
OpenAI and Anthropic are competing for the next trillion-dollar IPO. Both would need to distract investors from their massive losses by focusing on growth. Recently, Anthorpic has been growing faster than OpenAI, so Keach Hagey and Berber Jin report that OpenAI Considers Drastic Price Cuts, Anticipating War for Users With Anthropic:
OpenAI is considering drastically lowering the prices it charges users as it seeks to win customers from its rival Anthropic.
The company is weighing significant cuts to what it charges for tokens, the unit of measurement artificial-intelligence firms use to bill for their products, according to people familiar with the matter. The move would be in anticipation of similar cuts the company expects at Anthropic, the people said.
Meta will also introduce a new Meta Model API system, which will be used to collect fees from developers. Its API pricing is roughly 25% of the cost advertised by other top models from OpenAI and Anthropic PBC. Developers will be able to use Meta’s model for free, but only up to a point; they’ll be required to pay for access after reaching a certain token threshold, Zuckerberg said.
“The pricing from some of the other labs is very extreme and has very high margins,” Zuckerberg said, underscoring that his strategy is to get Meta’s technology in front of as many people as possible. “We think that there’s a real ability to be able to offer frontier or very high-level intelligence at a much more affordable cost.”
Aggressive pricing means Meta Model API will still be 50% more expensive than DeepSeek but not 50% better. Planning to reduce current income in the lead-up to an IPO is an unusual move, but it is a response to AI's Affordability Crisis.
Factory electricity bills are generally rising faster than those for other business customers or residential customers, according to a Reuters analysis. It highlighted the example of the Belden Brick Company, a 141-year-old brick manufacturer in Ohio, whose electricity bills have soared from $1,600 to $12,000 per month due to a higher monthly capacity charge in the 13-state region served by the grid operator PJM Interconnection.
...
The Ohio-based steelmaker Metallus described its electricity costs as having jumped by 70 percent since 2024, leading the company to pay an extra $15 million in energy costs annually.
The higher electricity costs for manufacturers coincide with the attraction of large AI data center projects with substantial electricity needs to many states in PJM territory. That data center growth has driven up PJM’s capacity prices—paid to power generators according to supply-and-demand forecasts—from $28.92 per megawatt-day in 2024 to $329.17 per megawatt-day in 2026, according to Reuters’ reporting.
Source
As I see it, there are three separate markets for LLMs. First, there is an embedded market that runs on low-cost, low-power hardware and open-weights models. Small AI Models Gain Traction Around the World by David Berreby provides examples:
For example, a drone-based system developed by Bala Murugan and colleagues at the Vellore Institute of Technology, in India, takes photos of cashew plants and quickly identifies those with splotches that indicate disease. All the processing takes place on the drone itself, so there’s no need for a computer on-site, nor for a connection to a central server.
It isn't just that small hardware, such as the Raspberry Pi 5, is getting more powerful but also:
the shrinking footprint of language models. Both Google DeepMind’s Gemma 4 (released in April) and Alibaba’s Qwen 3.5 are “fantastic” for small AI, Rovai says. Both models are “open weight,” meaning users can adjust the connections between parameters to suit their needs. This makes it easy, for example, “to take a lot of data from, say, the milk industry and retrain the model specifically on that,” Rovai says.
The hyperscalers and AI platforms like OpenAI and Anthropic will garner no income from this market, because:
“I think the future of AI is not like one giant model, at a center. I think it’s millions of small, precise models deployed at the edge, each one solving like a specific problem, a specific context,” Alonge says. This is partly because much of humanity—including people in parts of rich countries as well as the developing world—lives without access to cutting-edge frontier models. But, he says, it’s also because those models are not sustainable.
“If someone is not subsidizing it, most people will not be able to afford those models. So those of us who are said to be small-AI developers are the ones who will have to build for the majority of the world,” Alonge says.
At Computex 2026 in Taipei on June 1st, CEO Jensen Huang announced the RTX Spark superchip — a single piece of silicon that combines a 20-core Arm CPU, a Blackwell GPU with 6,144 CUDA cores, and 128 gigabytes of unified memory, connected by NVIDIA’s NVLink chip-to-chip interconnect. The whole package delivers up to one petaflop of AI compute in a laptop form factor.
The number that matters: RTX Spark can run a 120-billion-parameter language model entirely locally, with a context window of one million tokens, without a single byte leaving your machine.
To put that in perspective: GPT-3 had 175 billion parameters and required clusters of A100 GPUs to run. The model that stunned the world when it launched in 2020 is now approximately the size of what fits in a consumer laptop chip announced this week. The capability that required a data centre in 2020 is coming to a device you carry in a bag in 2026.
In 2025, slightly more than a third of all smartphones shipped worldwide were capable of running generative AI, and that figure will reach 45 percent by the end of this year, according to the technology research firm Counterpoint. By the end of next year, slightly more than half of all smartphones will be able to run a small AI model.
If Nvidia can already put a 120B-parameter in a laptop, it will only be a few years until phones can run a GPT-3-class model, good enough for almost all consumer needs. Apple and Google own that channel. Owning the channel is better than owning the technology. They will dominate consumer AI, and the other players will garner no revenue from this market.
Third, there is an enterprise market; all that is left to generate the revenue to service the debts fuelling the AI bubble, lets say $2T by 2030. There are a number of problems that make this unlikely.
First, there are very few documented cases of LLM deployment that resulted in enough productivity improvement to cover its unsubsidized costs.
Second, the Trump administration just demonstrated that deploying mission critical systems on AI platforms such as OpenAI or Anthropic means your company can be disabled at 90 minutes notice with no recourse.
Third, this means that companies will have to run mission-critical LLMs on open-weight models on in-house hardware if they are not to be vulnerable to the whims of the US president.
Fourth, systems such as RTX Spark show that good enough in-house hardware is likely to become relatively cheap compared to the unsubsidized cost of the AI platforms. In-house systems need much less over-provisioning for demand spikes, and because they aren't shared they don't need to be as fast.
Fifth, companies need to balance the productivity benefits (if any) of mission-critical LLMs against the productivity costs they bring. These include a vastly greater attack surface, technical debt from reduced developer understanding of the software, and so on.
Thus it seems likely that the hyperscalers and AI plaforms will generate far less revenue than they expect, because they will be restricted to non-mission-critical applications with lower productivity gains, and thus lower pricing power. They will thus be unable to cover the debts they are incurring to build massive data centers predicated on centralized systems dominating (an inflated estimate of) the entire enterprise market.
Karp had been softly floating his critique for some time, but the CNBC event looked like a proper coming out. Just one day earlier Palantir had published a kind of manifesto devoted to what it described as the all-important principle of “A.I. sovereignty.” The central argument: Companies should seek to build their own A.I. tools, not just customize those on offer from the frontier labs. This might mean relying on open-source L.L.M.s rather than the proprietary ones on which the A.I. boom has mostly been built in America, but it would amount to a liberating declaration of independence from Big A.I., which in Karp’s estimation was sucking up much more value than it was generating.
Locking in a price with a multi-year neocloud contract insures a buyer against compute getting more expensive but not against it getting much cheaper. So as businesses around the world spend ever more on compute, they want to be able to hedge against price volatility just as they insure against changes in energy tariffs, interest rates or foreign-exchange movements—ideally in deep and liquid derivatives markets.
Two startups want to help companies do this, by turning nascent indices tracking compute costs into a futures market. Silicon Data, founded in 2024 and backed by DRW, a trading firm, has paired up with CME Group, which operates large derivatives exchanges. Ornn, created by recent graduates of the Massachusetts Institute of Technology and run from a flat rather than an office just a few months ago, has paired up with Intercontinental Exchange, the parent company of the New York Stock Exchange, to do the same. Both aim to launch compute futures later this year, to be traded on their partner exchanges.
One characteristic of bubbles is overbuilding the infrastructure. Signs of overbuilding of data centers include that both SpaceX and Meta are now in the business of rentling their GPUs to the competition.
As morale is hitting rock-bottom, his company is heavily relying on its competitors' AI models to build out its own in-house tools. And despite the many billions of dollars the company has spent in its flailing efforts to keep up in the AI race, even Zuckerberg himself is now acknowledging that progress is nowhere near where he wanted it to be.
As Reuters reports, Zuckerberg admitted during a town hall last week that AI agents in particular aren't progressing as fast as he anticipated, a devastating revelation following enormous layoffs that wiped out thousands of roles at the company.
The "trajectory of the agentic development over at least the last four months hasn't really accelerated in the way that we expected," he said according to a recording obtained by Reuters.
Despite a derisory ~3% share of the enterprise LLM market, SpaceX's IPO was marketed as an enterprise AI company. The IPO has to rate as the most manipulated of all time featuring, in addition to ludicrous financial projections, a tiny float, bent rules for index inclusion, and massive conflicts of interest at the banks and the analysts. Immediately afterwards, SpaceX issued $25B in bondsa. A month later we can see how the markets respond to the first trillion-dollar "AI company".
We’re old enough to remember when the market cap of the lossmaking telecom SpaceX was bigger than Amazon’s. Heck, for a few precious moments it was bigger than Microsoft’s. Maybe one day it will be again, but for now the stock is down 38 per cent from its peak post-IPO valuation.
Punters lucky enough to have been awarded a stock allocation at the outset are still sitting on a tasty [checks notes] 0.8 per cent paper profit at pixel time.
And this is before the lockups start expiring and the initial tiny float greatly expands.
SpaceX bod spread
Second, more interesting as being much less subject to manipulation, are the bonds. They are what Toby Nangle focused on in SpaceX bond yields rocket towards junk:
the full $25bn of benchmark bonds — issued across the curve — had a rocky first couple of days of trading. Checking back today, it turns out that the inauspicious beginning was just a prelude to the train wreck that has since unfolded.
...
If you’d been allocated $100mn of the SpaceX 2056 bonds, you’ve turned $100mn into $90.7mn in less than a month. Sure, long-dated US Treasury bonds have fallen in value, and this general sell-off at the long end has done some of the work. But the spread on SpaceX 2056 — the additional yield you’re paid to compensate you for the risk that you don’t get repaid (among other things) has now widened from the initial +175bps to a whopping +231bps doing more than two-thirds of the work.
Top 10 worst BBB
The same conflicts of interest that pumped the stock caused rating agencies to give SpaceX bonds a BBB rating, one notch above junk and crucially the lowest that many major institutions are allowed to hold. But Nangle notes that:
Looking only at the nine days since the bonds were included in ICE BofA indices at the end of June, this spread-widening has made SpaceX 2056 the single worst-performing US dollar triple-B benchmark bond:
Note that the top two worst bonds are SpaceX, but the rest of the top 10 are all Oracle, another financial disater area.
when we overlay the average spread for double-B US dollar corporate bonds across different maturities (the pink line), it looks a lot like the type of risk that the market has assigned to both SpaceX and Oracle bonds is junk risk.
SpaceX, OpenAI, Anthropic, Meta all need to raise vast amounts of debt to fund their plans for AI. With SpaceX's bonds trading as BB despite a BBB rating, this is going to be hard.
Founded in early 2025 by former OpenAI CTO Mira Murati, Thinking Machines' first model is a big one. Weighing in at 975 billion parameters, the model requires more than two terabytes of GPU memory — a quantity present in around eight of Nvidia's B300 accelerators, or sixteen H200s — to run at its native 16-bit precision. If that's asking too much of your hardware, Thinking Machines has also released a NVFP4 quantized version of the model capable of running on half the GPUs.
This makes it the largest American open weights model to date, and comparable to Chinese models like DeepSeek V4, GLM 5.2, and Kimi K2.6 in terms of size and capabilities. Take these claims with a grain of salt — gaming AI benchmarks isn't exactly difficult – but Thinking Machines says Inkling is competitive with these models in a variety of workloads, although its benchmark charts also show it trailing proprietary models like Anthropic's Claude and OpenAI's GPT.
If you don't like Chinese open-weight models, the US ones are getting better:
The model developer claims to have tuned the model to use these thinking tokens more efficiently and that Inkling therefore matches Nvidia's Nemotron 3 Ultra, up to now the largest and most capable American open weights model out there at 550 billion parameters, on Terminal Bench 2.1 using roughly a third the tokens.
The AI revolution is supposed to be in its early innings. Demand for GPU computing is exploding. Microsoft has relied on CoreWeave for massive amounts of AI compute. OpenAI has signed tens of billions of dollars in long-term contracts. NVIDIA isn’t just supplying the chips, it has also invested in CoreWeave and entered into agreements that support parts of its financing and capacity.
So why are CoreWeave insiders continuing to sell stock? Management says many of the sales were made under pre-arranged Rule 10b5-1 trading plans. But those plans explain how the shares were sold—not necessarily why executives continue converting stock into cash while investors are being told AI infrastructure demand has never been stronger.
New York has become the first U.S. state to stop construction of large new data centers, imposing a one-year moratorium due to growing concerns over power costs, water supplies and the burden on local communities.
"As data center development threatens to hike up utility bills, deplete our natural resources, and create uncertainty for New Yorkers, it's my responsibility to take action and lead," said New York Governor Kathy Hochul.
She added that she would also pursue legislation to repeal sales tax exemptions for large data centers.
Phurichai Rungcharoenkitkul continues the Bank for International Settlements's pessimism in The AI Investment Race:
The AI build-out ranks among the largest technology-driven investment booms in US history. Its scale, reliance on debt and circular equity ties raise questions about the boom’s sustainability and financial stability. We study a dynamic contest in which firms competing for a few dominant positions over-commit resources. The over-investment leaves the sector exposed to revenue disappointment that could turn boom into bust. The larger the boom, the deeper the eventual bust. The race to commit early through debt and circular financing also makes a bust more likely. Calibrated to balance sheet and deal data, the model points to over-investment of around 1.5 times the efficient level, rising to around three times where demand is less elastic. A network analysis shows that stress in one firm could cascade to others through chains of financial exposures.
As recently as this week, one executive at Anthropic PBC, who spoke on condition of anonymity, mused that the Claude maker’s technology was roughly six to 12 months ahead of Chinese rivals.
On Friday, Moonshot AI Inc. upended those assumptions. The Chinese AI lab released Kimi K3, a more advanced open-weight model that it said outperforms all rivals except for Anthropic’s Claude Fable 5 and OpenAI’s GPT-5.6 on overall capability. The implication is that Moonshot, and by extension China, could be closing the gap faster than expected.
...
Moonshot’s release could also undercut OpenAI, Anthropic and others on price at a moment when they’re confronting customers who are becoming more conscious of their surging AI spending. Some developers have turned to so-called model routing services that can seamlessly direct users to cheaper options, including from China, for specific tasks to maximize cost efficiency.
Kimi K3 is an open-weight model.
If a 120B parameter model is too small for you the RTX Spark won't cut it. But Michal Malewicz explains in NVIDIA just killed big AI and… You’re the winner? that one of Nvidia's OEMs will sell you a $94K box that will run a 1T model, the Exxact Valence Nvidia DGX Station.
"[O]ur bottom line is that AI adoption appears gradual and steady rather than rapid and transformative, with most households and businesses still reporting limited exposure to the technology. At the same time, evidence of a structural pickup in productivity growth remains surprisingly fragile: Aggregate productivity growth has improved during the post-pandemic period, but there is a strong rationale to attribute much of this improvement to cyclical variation in utilization rather than a sustained acceleration in productive capacity. We also find little compelling evidence in the available industry-level data that industries adopting AI more rapidly are already experiencing stronger productivity growth."
"productivity trends across all three levels have been relatively consistent over time, suggestive of micro-level productivity gains not adding up in aggregate"
Another instance of Betteridge's Law is Will OpenAI ever be profitable? from Leap Finance Academy. It is a very long and detailed examination of all the things that would have to happen for OpenAI to turn a profit by 2030. In summary, with my comments, the list is:
Reduce cost of inference which isn't going to happen enough because each successive frontier model uses more tokens to generate the same output.
Increase pricing which isn't going to happen because they already need to reduce pricing to stem the loss of customers to Anthropic, let alone to the open-weight models.
Increasing sources of revenue that are not inference dependent in other words the fantasy of $100B/year in advertising income, which isn't going to happen.
Securing and retaining human talent which requires an IPO to make their stock options worth something, which isn't going to happen it time to prevent bankruptcy.
Securing the funding they need to build Stargate and for working capital which isn't going to happen because even an IPO wouldn't come close to generating enough and the lenders are saying "enough".
Even though I'm skeptical of the wishful thinking of the conclusion, this is an impressive piece of work and well worth reading.
China’s open-source approach has spawned a crowded field of innovative and intensely competitive start-ups all offering systems at low cost.
So attempts to increase revenue by charging more for access to certain models can scare away customers. Price-conscious Chinese consumers — businesses and individuals alike — are quick to hop across platforms in search of inexpensive A.I. tools.
While China’s A.I. companies are searching for sustainable business models, spending is high and revenue low, said Richard Lin, a vice president at the Silicon Valley company Datastrato.
“In two or three years, we will still be trying to figure out how large models can earn money,” he said.
Offering low prices has helped the Chinese firms gain users, including in Silicon Valley, where many companies depend on the more affordable systems. Yet Chinese companies have struggled to translate huge numbers of global users into profits.
Bain & Company management consultant Jue Wang said many of the big businesses her firm advises have been taking a closer look at returns on their AI investments.
“The token cost for them has been doubling, almost every other month,” she said. “Let’s say $200 per developer per month. Multiply that by 20,000 developers, which is often what we’re dealing with at these companies, and that quickly gets you to a number that is not a line item that any general manager has planned for.”
Sometimes that just means not using the AI equivalent of a sledgehammer to crack a nut.
A huge wave of supply of debt intended to fund the data center build-out for AI means that the price of this debt has dropped, and thus that the interest rate buyers are demanding goes up. This reduces the net pressent value of the (hypothetical) future cash flows the data centers will generate, and thus their ability to pay the interest and principal. This increases the interest rate the buyers of future debt will demand, causing a feedback loop.
Both Ed and Torsten point to the increasing spreads above Treasuries that the AI debt is trading at, and the increased cost of insuring against these companies defaulting.
In a blink, [Leopold Aschenbrenner's] wildly successful hedge fund, Situational Awareness, was forced to sell billions of dollars of technology investments that had rapidly lost value, as nervous banks began to demand more and more collateral for his trades. Then came billionaire Ken Griffin.
In less than 24 hours — which included a conversation between Griffin and Aschenbrenner — Griffin’s Citadel hedge fund reached out to Situational Awareness and snapped up the investments at a discount, according to a person familiar with the matter who asked not to be identified citing private information.
It was a startling reversal for Aschenbrenner, a former researcher at OpenAI who — before starting his hedge fund roughly two years ago — had no previous investment experience. His fledging firm has watched its assets plunge from $45 billion at the start of July to about $10 billion.
Aschenbrenner was all-in on AI.
On the Pof. G. Markets podcast legendary short-seller Jim Chanos focused on the accounting inequality inherent in the AI (and earlier dot-com) bubble. The spending of the hyperscalers and neoclouds is investment, expensed over say 5 years. But for the chip makers, construction companies and so on that spending is this year's revenue. So, overall, it looks like the ROI is better than it really i
s.
US GAAP was built on the assumption that neither route would materially move profits, because which company would ever choose to grant ‘enough’ RSUs to employees that it could move the share price right?
Meta has granted ~US$70bn in stocks. And Microsoft ~US$42bn. The sheer size of these numbers IS enough to move the needle materially. RSUs, granted in large quantities every year, compound into visible profit erosion and free cash flow consumed by “obligatory” share buybacks, as employees must receive their promised shares annually regardless of market conditions.
As of FY25, total unvested RSU’s book value was ~US$60bn and corresponding market value was ~US$80bn. That $20bn gap is money Meta will have to find, one way or another, the moment these shares vest, either by diluting shareholders with fresh stock, or by spending real cash buying shares back to hand over instead.
...
Using the extensive information provided by Meta’s financial notes on the RSU programme, we can back-solve the profit erosion that is being kept ‘off-the-Income-Statement’:
Since December 2024, as the share price of Meta increased and the AI talent war started, ~10 points of EBIT margin per year have been given away as additional, uncaptured employee compensation.
Ed Zitron is back with another scathing analysis in The AI Demand Bubble. As usual, it is long and detailed. He concludes:
To put things really simply, Anthropic and OpenAI are a way that hyperscalers can feed their revenue to themselves by spending money on capex, backstopping compute contracts, or doing direct equity investments.
Their continued existence allows the AI bubble to continue inflating, but this can only continue as long as venture capital and hyperscalers are capable or willing to invest. There is simply not the demand — not from open source, not from other AI labs, not from self-hosting, not from anywhere — to justify the capex or the massive data center buildout.
And for those arguing that there would be a dot-com bubble recovery story, I must be clear that if there isn’t demand today, it won’t magically appear tomorrow. AI GPUs will cost just as much to run in five years as they do today, as will unfinished data centers cost just as much to finish, as will electricity remain expensive, and all this will be happening after it’s easy to raise venture capital to actually buy the compute.
In a recent survey of 300 business executives, 68% reported overspending their AI budget over the past year. It’s little wonder so many companies are now cracking down. Uber Technologies Inc. recently capped employees’ AI spending at $1,500 a month. Tesla Inc. reportedly set a limit of $200 a week. The swift reversal has left some workers complaining of whiplash, with Reddit forums filled with stories of engineers suddenly stymied by usage caps.
Bloomberg in partnership with researchers at Vals AI, an independent AI evaluation and benchmarking platform, tested seven models from frontier Chinese and US companies to see how they performed in a real-world task. They were asked to create a fictional coffee e-commerce site called Brewberg using the same prompts. Most of the models scored 100% functional accuracy despite occasional design misses, but with very different price tags. The experiment employed the top performing models in July from Anthropic and all the Chinese firms, as well as more affordable models from OpenAI and Google.
They all did reasonably well, but the two best were Claude Fable 5 at $48.99 and Kimi K3 at $11.99. Chinese models charging much less for almost the same performance are grabbing market share:
the use of Chinese models overtook US platforms globally for the first time in June, and accounted for more than 60% of market share last month, on OpenRouter, a tech platform that offers software developers access to hundreds of AI models. It is a widely watched gauge of model usage despite tracking just a fraction of global AI consumption. The US, parts of Europe and Asia now favor Chinese labs, according to the same data.
On Hugging Face, Chinese AI models account for 41.4% of generative model downloads among developers, 5 percentage points higher than US models.
In OpenAI’s unraveling has begun Gary Marcus notes that leaked Q2 numbers show OpenAI's quarterly revenue grew from $5.7B to $6.7B but its quarterly losses grew from $9.3B to $12.3B.
Perplexity is launching Portable Computer today, a version of its agentic "Computer" platform that runs entirely on hardware users already own — starting with Nvidia's DGX Spark desktop supercomputer and Linux machines equipped with Nvidia RTX GPUs.
The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user's files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.
Apple announced new iterations of both desktops, along with two new chips: the M6, the first 2nm chip in Apple’s M-series lineup for Macs, and the M5 Ultra, now the most powerful chip in the lineup for most things—especially AI workloads.
There aren’t any major new features for either machine. This is just a specs bump. But based on how Apple is presenting these refreshes, they’re leaning hard into those use cases, which weren’t even a thought when earlier iterations were first engineered.
The devices’ popularity for production inference took off after macOS 26.2 shipped last December. According to Apple’s release notes, 26.2 enabled “low-latency communication between Thunderbolt 5 hosts for use cases including distributed AI inference using MLX.” Thunderbolt 5 is a very fast wired data connection, and MLX is an open source array framework designed to help machine learning workflows take full advantage of the M-series chips’ unified memory.
Since then, both hobbyists and professional developers and researchers have been essentially daisy-chaining Mac minis or Mac Studios to run inference on local large language models that are much bigger than anything that could run a single mass-market device—providing an alternative to ultra-beefy specialized hardware featuring specialized Nvidia GPUs.
IBM has rolled out the newest models in its family of open-weight large language models designed to be downloaded and self-hosted. The newly launched Granite 4.2 comes in 3B, 8B, and 30B parameter variants.
Like previous versions, IBM is taking a decoder-only approach here. These new releases offer a 128,000-token context window natively. The 8B and 30B variants (not the 3B one) also go through an agentic reinforcement-learning block; they were trained for expanded capabilities like using the terminal, searching the web, or using external tools. The 3B model supports tools too, but without the same level of specialized training.
Beyond those tweaks, this release is particularly notable because, as IBM itself writes, “Granite 4.2 is the reasoning-focused release of the Granite language-model family.
Morgan Stanley analysts today revised their model for how much power would be needed to switch on the chips it projects will be sold. They now estimate, based on a total data-centre power requirement of 257 GW by 2028, a gap of between 30 and 40 per cent between US capacity and sales projections. Or to put it in layperson’s terms:
New York City demands 5.5-6 GW of baseload power: we are short power by six New York Cities.
And that’s using their mid-estimate for innovative “time-to-power” solutions. Absent such mitigation measures, the estimate rises to 10 NYCs:
The innovative solutions include a massive expansion of Bloom Energy fuel cells, data centers alongside nuclear power plants, and other equally optimistic concepts.
But Morgan Stanley are equally optinmistic about Nvidia:
Fenyman, Nvidia’s next next chip architecture, is scheduled for launch in the second half of 2028. It’ll be another step change, Morgan Stanley says: the same dynamics will increase US power demand year-on-year by a further 40 per cent in 2029, Byrd and team estimate. They’re then looking at a cumulative power shortfall of 107GW (or approximately 18 NYCs).
“However,” the team add in bold, “all of these advancements converge on one central issue: if we cannot close the power shortfall, the expected pace of improvement and adoption may not fully materialise in line with market expectations.”
They probably don’t need to spell out what happens to all those big tech off-balance-sheet guarantees in that scenario. Obviously, the faster-rising thing will eventually overwhelm the slower-rising thing,
I think "market expectations" are out over their skis.
On October 15, 2026, Reuters reported, citing people familiar with the matter, that some current and prospective Anthropic investors are considering a $1.5 trillion valuation for the AI company's planned November IPO. That is below the roughly $2 trillion investors had expected.
There are months ahead before Anthropic's valuation is set at the IPO, if it ever happens. If Anthropic's IPO fails to set the world on fire, OpenAI's delayed IPO will never happen and in will get absorbed by Microsoft.
The Claude maker received an issuer rating of double B plus from KBRA in June, the highest speculative-grade score, in the run-up to securing a $35bn chip financing led by Apollo, according to documents seen by the FT.
That was before the on-going Coreweave meltdown, the Firmus IPO failure, and so on. Anthropic wouldn't get BB+ today.
Not all citations are equal. To be an “Author Cited” by the Supreme Court of Canada (SCC) is to hold some prestige. This post shares the results of a project that captured and counted the Authors Cited works found in the judgments by the SCC from April 2021 to April 2026.
Dark fibre originally referred to the potential network capacity of telecommunication infrastructure. Because the marginal cost of installing additional fibre optic cables is very low once a trench has been dug or conduit laid, a great excess of fibre was installed in the US during the telecom boom of the late 1990s and early 2000s. This excess capacity was later referred to as dark fibre following the dot-com crash of the early 2000s that briefly reduced demand for high-speed data transmission.
These unused fibre optic cables later created a new market for unique private services that could not be accommodated on lit fibre cables (i.e., cables used in traditional long-distance communication).
Many are analogizing the current AI bubble to the dot-com era, pointing out that although the dark fiber eventually became very valuable infrastructure, the companies that invested in laying it suffered. Wikipedia notes that the buildout:
was based on the assumption that telecoms traffic, particularly data traffic, would continue to grow exponentially for the foreseeable future. The advent of wavelength-division multiplexing reduced the demand for fibre by increasing the capacity of a single fibre by a factor of as much as 100.
According to Gerry Butters, the former head of Lucent Technologies's Optical Networking Group at Bell Labs, the amount of data that could be carried by an optical fibre was doubling every nine months at the time. This progress in the ability to carry data over fibre reduced the need for more fibres. As a result, the wholesale price for data communications collapsed and a number of these companies filed for bankruptcy protection. Global Crossing and Worldcom are two high-profile examples in the United States.
The analogy being suggested is that the AI data centers will eventually be valuable infrastructure even if delays in construction and lack of real demand mean that the companies building them will suffer.
Below the fold I investigate this analogy.
Flawed Analogy
There are some significant problems with the analogy between dark fiber and AI data centers:
Depreciation: Decades later the fiber in the ground was still fully functional. The capacity could be sold for very little additional investment. Estimates are that about 40% of the cost of an AI data center is the physical infrastructure (buildings, power connections, cooling, etc.) which might still be useful a decade later. But 60% of the cost is the (mostly Nvidia) racks full of state-of-the-art chips.
Companies typically depreciate their racks linearly over 5 or 6 years. But, as I wrote in Mind The GAAP Again and its predecessors, this is unrealistic. Because:
Power is a major cost of both inference and training. Power effectively sets the price floor for inference.
Each technology generation improves power efficiency by a significant factor.
Thus the profit from running a rack doesn't decay linearly, but exponentially.
Thus racks more than a couple of generations old are uneconomic.
Nvidia has announced that they plan to move from a two-year product cadence to a one-year product cadence.
Overproduction: Wikipedia correctly notes what drove the massive overproduction of dark fiber:
Because the marginal cost of installing additional fibre optic cables is very low once a trench has been dug or conduit laid, a great excess of fibre was installed
Building data centers does not have the same low marginal cost. While it is true that there are economies of scale in construction driving up the size of AI data centers, these are both much smaller than for laying fiber and apply only to the smaller part of the overall cost.
Carrying Costs: The fiber in the ground required very little maintenance.
The physical infrastructure of unused AI data centers could survive until eventual re-use only with significant maintenance costs.
Technology evolution: Developments in optical technology increased the capacity of the unused fiber with no additional investment. The evolution of AI racks has meant that data centers built a few years ago cannot host current racks, and that the design of data centers for Nvidia's future racks is a research problem.
Dark fiber thus had two important attributes that AI data centers lack; the surplus initial capacity was both long-lived and cheap.
Construction Delays
Building and activating an AI data center is a much bigger, more expensive and long drawn-out task than digging a trench and laying a cable in it. This problem grabbed the headlines when, as Sridhar Natarajan et al reported, Oracle Cites ‘Force Majeure’ to Shield Itself on Controversial Data Center. Almost two years into the project:
the campus designed to handle 2.45 gigawatts — enough electricity to power roughly 1.8 million homes at any given moment — has hit serious setbacks including the denial of a permit key to its plans for energy resources. It’s becoming a symbol of the growing backlash against the broader AI data center boom and a political flashpoint ahead of the US midterm elections that threatens to turn races into referendums over the build-out.
The amount of "construction in progress" for the hyperscales, representing data centers they are struggling to bring on-line, is enormous. Ed Zitron writes:
Considering Oracle’s CIP number from its latest quarterly earnings was at $48.5 billion and Google’s Q2 2026 “assets not yet in service” were at an astonishing $122.8 billion, up from $108.5 billion in Q1 2026 and $78.5 billion at the end of 2025, it’s reasonable to believe that Microsoft has at least $50 billion of construction in progress. I also think it’s fair to assume that Amazon has, since the beginning of 2026, added at least $25 billion to CIP, though we’ll find out at the end of the year.
We’re tracking 190GW across 777 large data centers and AI factories (>50MW) announced since 2024. At least 16GW of capacity is slated to come online in 2026 across roughly 140 projects. Yet only about 5GW is currently under construction. Around 11GW remains in the announced stage with no visible construction progress, despite typical build timelines of 12–18 months.
Anyone who has built themselves a house will understand that construction takes longer than it takes.
Overproduction
Of course, only 40-50% of this "construction in progress" is actual construction, the rest is the racks. There are two ways to estimate what proportion of the racks Nvidia has shipped are actually powered on in a data center as opposed to sitting in a warehouse waiting for the construction to stop being "in progress".
Ed Zitron bases his estimate on the financial picture. He adds up the "construction in progress" and assumes 50% is the racks:
if we assume the number is roughly $374 billion, plus (at minimum) $50 billion from Microsoft, plus another (at minimum) $20 billion from Amazon, that puts us around $444 billion, with Google’s share — $120.8 billion — being 60% GPUs and related hardware, for a total of $72.48 billion, putting us at ... an estimated $234 billion in GPUs and TPUs sitting in warehouses.
He goes on to compare this $234B number with the value of the racks sold since the start of 2023, which he estimates at $561.5B. Clearly, the systems that shipped early in this time-span are likely to have been powered up, so he estimates that around 50% of recent shipments are idle. He looks in detail at Microsoft's financials and and estimates:
Nadella mentioned in November 2025 that he had “...a bunch of chips sitting in inventory that [he couldn’t plug in],” but didn’t make any mention of how many there were.
In other words, Microsoft is warehousing anywhere from $50 billion to $100 billion in GPUs, and has barely gotten $50 billion worth installed in the last four years.
If Microsoft is struggling, everybody’s struggling, and we may have a very inconvenient truth: that NVIDIA has potentially sold hundreds of billions of GPUs years in advance.
The other way to estimate this proportion is to compare the power demand of the racks that have been shipped to the data centers that are supposed to house them. In OK, so where are all these data centres?, Bryce Elder wrote:
A theme we’ve touched on a few times is the mismatch between consensus forecasts for chip sales and US data centre completions.
Morgan Stanley measured the gap earlier this week by estimating the shortfall in available power, concluding that more than half of the GPU servers sold between 2026 and 2028 might not have anywhere to be plugged in. New research from Jefferies analysts reaches a similar conclusion using a very top-down method: space.
Tracking US data centre construction sites by satellite imagery shows between 16 and 18 gigawatts of gross capacity that can be energised this year, the broker says, citing SynMax Intelligence. (Gross capacity means the site’s all-in power requirement, not just the draw for IT equipment.)
...
In the longer term, power availability is still the bottleneck — along with labour. And transformers. And cooling equipment. And backup generation. As Jefferies says: “The gap between planned capacity and physical execution remains the central issue.”
Thus there is an emerging consensus that something like a quarter of the AI investment boom is going into rapidly depreciating equipment quite unlike the dark fiber.
Demand Overestimation
The frontier labs and the hyperscalers claim that there is overwhelming demand for AI compute, which justifies the planned AI data center buildout. But the frontier labs are private and don't publish audited numbers to back up their claims. The hyperscalers are public but their accounts don't include explicit numbers for their AI compute. Nvidia is public and its audited accounts reveal the number of AI systems it shipped each quarter. So the easy way to estimate the demand for AI compute is to base it on Nvidia's shipments.
There are three problems with such estimates. Assume this estimate is D and the real demand from end-users is D':
As we have seen, it appears that about half of Nvidia's shipments go to a warehouse not a data center. Thus D' = D/2.
Ed Zitron estimates that 80% of D comes from OpnAI and Anthropic. So D/10 comes from other sources, which we will generously assume is all end-user demand.
The data [Jay Goldberg] collects shows a similar pattern to the Silicon Data index, with token prices plunging for frontier LLMs from $50 per million tokens in late 2023 to about $6 now. More interestingly, he finds the median token price has held fairly steady at just below $1 per million since mid-2024 as the LLM marketplace became more competitive. A floor price of about $0.11 (basically just the cost of electricity) has barely budged.
Price cuts at the frontier, in addition to cheaper and more efficient models at standard and lite levels, are pushing down the cost of using AI.
Open source models remain <5% of business spend. This decline in spend is driven almost exclusively by competition between OpenAI + Anthropic
The Ramp data does show increased token volume, but flat to decreasing spending. The increased colume is likely due to successive models burning more tokens per query.
Market Reaction
We now have some evidence showing how the big banks rate GPUs as collateral for loans, i.e. how useful they are likely to be down the road.
JPMorgan Chase & Co. is pitching a yield of about 11% on a $5 billion leveraged-loan sale on behalf of Volta Infrastructure Holdings Ltd., one of the highest borrowing costs seen in the market for risky debt that’s become popular for financing the AI boom.
...
A $3.7 billion chip loan will be used to purchase about 36,000 graphics processing units, or GPUs, made by Nvidia Corp., according to the people. A $1.3 billion loan will cash collateralize a letter of credit for a data center lease, meaning the money will be held in an account and used to make lease payments to Bitdeer to use the data center, the people said.
JPMorgan Chase & Co. and Goldman Sachs Group Inc. have taken a rare move to limit their exposure on a $5 billion loan sale for AI startup Volta Infrastructure Holdings Ltd. as investors demand higher yields for risky debt tied to the artificial intelligence boom.
The two banks are leading the loan offering for Volta, a new artificial intelligence cloud company. The banks have underwritten a $1.3 billion term loan for Volta, while a $3.7 billion chip loan is not underwritten by the banks — a move to protect themselves from potential risks, according to people familiar with the deal.
In other words, these banks are happy to lend at 11% against cash collateral, but not at 11% against GPU collateral. The loan is said to be oversubscribed, but the part with GPU collateral is for the private credit suckers.
The AI cloud-computing provider’s unsecured debt due in 2032 traded below 80 cents on the dollar on Wednesday, according to Trace pricing data. That’s widely considered the threshold for when a borrower is financially strained.
At that price, the bonds yield more than 15% — a figure noted by Palihapitiya that led Musk to respond “yikes” on his social media platform.
Shares of the company, which rents out access to leading AI chips, fell as much as 17% on Wednesday. Even though its bonds are trading at distressed levels, CoreWeave’s stock is up more than 60% since its initial public offering in March 2025 and its market value is more than $36.5 billion.
CoreWeave’s credit-default swaps are back at a record high, and at current levels are pricing in a more than 50% chance of default within the next five years.
CoreWeave is the original "neocloud". As I understand it, its assets against its debts are (a) a set of commitments from companies like Microsoft and Oracle to pay for future access to compute, and (b) a set of data centers full of GPUs. The market is betting that these assets won't be "money good" in the near future.
Coda: A Sun Microsystems Story
Nvidia is just the most successful of the vast number of companies spawned by Sun Microsystems. In the run-up of the dot-com bubble Sun's Starfire systems powered the Web the way Nvidia's GPUs power AI today. But this market dominance was a double-edged sword. Once the dot-com bubble burst, the bankrupt Web companies flooded the market with barely-used Starfire systems. So not merely did much of the customer base vanish, but the remaining customers bought from the second-hand market at huge discounts. Demand for new systems collapsed, and Sun never really recovered. That's not the only reason Sun died. Scott McNealy admitted in a fascinating podcast that during the boom he lost control of Sun's spending. It is hard to maintain discipline when you are being flooded with cash.
Once the bubble bursts, Nvidia will face a similar problem. First, their customers will stop ordering new racks, because:
Companies like CoreWeave that exist to get debt off the AI companies' balance sheets will be bankrupt because their customers can't afford their commitments.
The assets of these bankrupt companies will be sold off at a loss, flooding the market with barely-used data centers and stopping construction of new ones.
It has proven harder than expected to build data centers quickly because of public opposition, lack of skilled tradespeople, difficulties in securing power and so on. Thus it is likely that there will be warehouses full of Nvidia racks that were waiting for a data center to be built when the bubble burst. These unused racks will also be sold off, crashing the price of the barely-used ones further.
Second, in some ways the problem will be worse for Nvidia than Sun. Nvidia will have invested heavily in developing and securing wafer slots for their next-generation GPUs. Although these will be faster and more power-efficient than those in the written-down data centers, they won't be able to compete on price-performance. The pipeline between committing to a new design and market entry is now much longer than it was in the '90s. Nvidia will have relatively more capital tied up in the product pipeline than Sun did.
It’s October, and that means the return of our annual Halloween Hunt!
We’ve scattered a hauntourage of ghosts around the site, and it’s up to you to try and find them all.
Decipher the clues and visit the corresponding LibraryThing pages to find a ghost. Each clue points to a specific page on LibraryThing. Remember, they are not necessarily work pages!
If there’s a ghost on a page, you’ll see a banner at the top of the page.
You have a little more than two weeks to find all the ghosts (until 11:59pm EDT, Saturday October 31st).
Come brag about your hauntourage of ghosts (and get hints) on Talk.
Win prizes:
Any member who finds at least ghosts will be
awarded a ghost Badge ().
Members who find all 12 ghosts will be entered into a drawing for one of five LibraryThing (or TinyCat) prizes. We’ll announce winners at the end of the hunt.
P.S. Thanks to conceptDawg for the illustration of the mystery bird!
Leaders at OpenAI, Nvidia, and IBM have predicted on separate occasions that 2025 would be the year of AI agent adoption. More recently, Microsoft has declared 2026 the year of the agent. Yet as of May 2026, just 19.8% of U.S. businesses reported using AI in any business function, according to the Census Bureau’s Business Trends and Outlook Survey. Still, these systems will be increasingly deployed, raising numerous concerns. Empirical benchmarking of their reliability in domain-specific use cases remains thin, and if we are to take agentic use seriously in research environments, we must close this gap.
As the Public Data Project’s Molly Hardy detailed in an earlier blog post, our team, in partnership with dataindex.us, created an AI benchmarking exercise to test how well AI can assess data risk for federal datasets. We have now built the AI Benchmarking Exercise into a web-based application, and we’re excited to announce its publication!
AI Benchmarking Exercise home page.
The exercise, which can be done in a group setting or alone, compares three approaches on the same datasets: a human researcher working alone, a human steering an LLM chatbot, and a background agent that searches the internet without human steering. All three approaches are measured against the expert risk assessments published by dataindex.us.
Presenting the three approaches side by side allows us to share knowledge in both directions: we can learn how human experts approach dataset evaluation; introduce experts to AI-assisted research; and experiment with automated evaluation tools, which we hope will eventually allow for monitoring and human follow-up on much larger collections.
In this post, we report progress on the third challenge of automated risk assessment, investigating how well AI agents perform a task that requires multiple kinds of domain expertise, intensive research into dataset context, and synthesis against a concrete rubric. We present how we built a background agent to answer that question, the design challenges we hit, and early results: how the agents performed, what they cost, and which did the best.
Experimental Design
In our experiments, we define an agent as an autonomous program that completes tasks for the user using the tools and environments it can access. We took a bare-bones approach, serving a large language model through OpenRouter, with search tools and the internet as its environment to conduct research and synthesis. We ran two frontier models, GPT-6.1 Sol (OpenAI) and Claude Opus 5.5 (Anthropic), using their providers’ native search, and four open-weight models, DeepSeek V4 Pro, DeepSeek V4 Flash (DeepSeek), GLM 5.3 (Z.ai), and Kimi K3 (Moonshot AI), using Exa as the search tool and OpenRouter’s fetch tool. The two frontier models ran at medium reasoning effort, and the rest ran at high since they don’t offer a medium effort level.
In the research phase, the model searches the web and fetches pages to learn about its assigned dataset. In the assessment phase, a second prompt asks the model to use its research to score each category against the assessment rubric. These are the same prompts human participants paste into their chatbots, sent verbatim with no added system prompt. The key difference is that humans can follow up and steer their chatbot, while the background agent works independently. After initial experiments, we ran into two challenges.
Challenge 1: Controlling the Model’s Search Parameters
During our initial in-person experiments, where humans used our AI Benchmarking website to assess data risk categories, participants reported that their chatbots surfaced dataindex.us as a source, which is effectively our answer key.
A simple solution might be to tell the model not to use a source, but that only works if models always follow textual constraints. Frontier models have been observed finding loopholes in evaluations, breaking the user’s constraints, and “peeking” at domains and sources they were told not to access.
To address this, we used the search tools’ excluded-domain settings to remove dataindex.us from the search results shown to the model and blocked the agent from fetching it directly. However, even with these guardrails in place, the DeepSeek and Anthropic background agents surfaced third-party pages that cited dataindex.us, emphasizing that blocking a domain doesn’t block pages that reference it.
Challenge 2: Reducing Cost
As a public service project, we aim to create tools that maximize output while minimizing costs, both for ourselves and for the open source software that we create. Our first version ran with no cost optimizations, at $17.58 per dataset for a single frontier model (Claude Sonnet 5) using their agent SDK. At that price, all 47 datasets across 6 models would cost about $5,000 in the worst case.
To reduce cost, we enabled prompt caching. For each turn of the agentic loop, we send all previous context, so without caching, the model pays full price to re-read everything it has already read. Caching lets providers reuse that repeated prefix at a steep discount, as little as a tenth of the normal input price depending on the provider.
Some providers cache automatically (OpenAI, Kimi K3, GLM 5.3, DeepSeek), while others (Anthropic) require explicit requests to enable caching, which explains the high cost in our initial experiment. With caching, the same Sonnet 5 model dropped from $17.58 to $3.74 per dataset.
What We Found
Across 47 datasets with six categories each, we measured agreement through exact-match with ground truth (i.e., did the agent get the same ratings as dataindex.us?) and distance to ground truth (i.e., how many rating levels off was it, on average?). All models were in a similar accuracy ballpark across both metrics, with DeepSeek V4 Pro leading by matching 55% of categories and averaging 0.52 levels from dataindex.us. All models stayed within one level of error on average across all datasets, with a tendency to rate riskier than the Data Index.
Agreement, cost, and time graphs for the models’ runs.
DeepSeek V4 Flash was cheapest at $0.17 per dataset; Claude Opus 5.5 was most expensive at $2.33 per dataset. Kimi K3 was slowest, averaging 7 minutes per dataset, with outliers as high as 23 minutes. More searching did not reliably buy more agreement: GPT-6.1 Sol searched the most, and DeepSeek V4 Flash fetched the most, yet neither beat 46% exact-match accuracy. In total, running six models across 47 datasets cost $225.
Descriptive statistics about the models’ runs.
Looking Ahead
Models are continuously improving, and agreement on a data risk task like this could improve over time with optimized search tooling. DeepSeek V4 Pro maintained the best accuracy at a fraction of the cost of frontier models, illuminating that the most popular agents aren’t always best suited to the task. We plan to release further work benchmarking these types of data agents on real-world tasks and empirically comparing them to other modern approaches in academia and industry. Working at the intersections of libraries and technology, we aim to create tools that researchers, journalists, and the public at large can trust and replicate without advanced domain understanding or deep pockets. Please stay tuned for further updates on our Data Monitoring Toolkit.
The toxic drug supply crisis has had profound impacts on workers in Canadian public libraries. While library workers struggle with changing job duties, understaffing, and the mental and emotional toll of laboring during death times, narratives of unsafety that pivot on “mental health and addiction” challenges experienced by our patrons have come to dominate library discourse and entrench an antagonism between library workers and our patrons. Using Sarah Ahmed’s (2004) theorization of affective economies, I will interrogate the “safety and security crisis” in Canadian public libraries as a discursive formation that relies on what, after Ahmed, I call the “sticky associations” between drug use, homelessness, mental health, and violence. It is through this process of articulating “violent associations” that delineate the ‘victim’ and the ‘perpetrator’ that the constitutive antagonism of library worker vs patron forms. In this article I will examine how mainstream media, library leadership, and library researchers contribute to a narrative of unsafety that hinges on the individuation of threat, safety, and solution. I then turn to how library workers make sense of, and act on, their affective experiences of threat and unsafety. Finally, I offer suggestions for how a labor framework of Occupational Health and Safety can be mobilized to address the structural and legislated violence shaping library work.
Introduction
My first week at a Toronto Public Library (TPL) branch I was put in-charge. Due to high turnover and understaffing, this is not as uncommon as one might hope. I was anxious, but less about what “incidents” could occur, and more about how I might be seen as responding to them. I was at a branch in the middle of a reckoning – a breakdown of trust between management and frontline staff who felt their wellbeing was not taken seriously. When an incident occurred, staff waited to see if their safety was prioritized and if management’s response was proportionate to the harm they felt they had experienced. We wore walkie-talkies to communicate about branch happenings, which often included tracking people’s movement. There was a board in the workroom that had pictures of “banned patrons” pinned to it and a “call the police immediately” protocol for when certain people entered the building.
I knew that my new colleagues were watching to see how I responded to these early incidents. Would I call the police as soon as X walked into the branch? Would I enforce a zero-tolerance alcohol policy? It felt as if I was forced to make a choice: show my colleagues I value their safety by punishing patrons or honor my abolitionist commitment to non-punitive, transformative responses to conflict. This tension is not unique to a single branch, nor TPL. It has haunted my time at the public library. I have regularly found myself struck by the animosity many colleagues feel towards our patrons – the threat they experience from interactions I do not register, surveillance they so willingly participate in, and their eagerness to see patrons punished.
In a chaotic ecosystem ripe with despair and uncertainty, the enforcement of both formal and informal rules of conduct is an assertion of order. Library rules of conduct are the mechanism through which frontline workers express institution-backed boundaries. With very little power and a whole lot of responsibility, library workers are expected to maintain a “safe” and orderly library for patrons and staff. While library policy and job duties over-determine staff responses to challenging patron interactions, many library workers come to view institutional practices of punishment as strategies for personal safety. Presented with few alternatives, library workers regularly punish patrons for our declining work conditions and public expressions of large-scale abandonment and legislated violence that we are under-resourced to address.
In addition to profound poverty, the toxic drug supply crisis has wreaked havoc on poor, unhoused, and racialized communities. From January 2016 through March 2025, Canada recorded 53,821 opioid toxicity deaths, with 26,000 people dying of a drug poisoning in Ontario (Centre on Drug Policy Evaluation, 2024; Government of Canada, 2025). These preventable deaths are a direct result of the criminalization of drugs and the people who use them. As experts have repeatedly pointed out, neither punishment nor prohibition reduce drug use or prevent drug poisonings. Instead, prohibition fuels stigma, deepens the dangers of illicit and unregulated drug markets, intensifies poverty and homelessness, and drives up rates of preventable illness and death (Boyd et al., 2016). Rather than responding to this crisis with evidence-based solutions, Canadian provinces continue to cut funding for harm reduction programs that save lives while ramping up forced treatment policy and the policing of public drug use (Canadian Public Health Association, 2025). As the drug supply becomes less predictable and more toxic, provincial governments continue to entrench prohibitionist policies, deepening the public health emergency confronting communities across the country.
Increased drug use and drug poisonings at the public library are symptoms of these government policies. As library workers can attest, the complexities of dispossession and private life turned public play out daily in the library. Some patrons exhibit signs of intoxication, fall asleep at computer stations, and experience drug poisonings in the library bathroom. Many patrons are frustrated at being abandoned by the state and the limited ability of the library to address gaps in social and health services. Other patrons are upset that they are confronted by a growing class divide while studying, attending a children’s storytime program, or picking up their holds. This anger is often directed at library workers trying to balance divergent and regularly oppositional desires of people sharing the library.
In a climate of collective unsafety, personal safety is both a real concern for people paid to interface with the public and a discourse that state institutions, including the library, use to assert order in a time of heightened instability. Library workers labor at the crux of this contradiction. While library workers have experienced increased violence and harassment on the job, they are also witness to tremendous human suffering intensified by the dismantling of social services (CUPE Research, 2023; Stevenson, 2021). The knot of bad feelings, or discomfort, brought on by all of the above experiences is often hard to untangle. Library workers are not only frustrated by the expanding scope and emotional toll of their work, but many have expressed feeling unsafe. Safety, as a framework for discussing the demands of public library work, and a lens through which to advocate for better work conditions, has come to dominate library discourse. While this framework may give voice to library workers struggling with the changing demands of their job, terms like violence, safety, and security are power-laden, doing the discursive work of the library seeking to re-assert order and a political project reliant on the dispossession and dislocation of marginalized people.
When fear stands in for a host of other affective experiences resourced by power, it overdetermines how we respond. While library workers, just like all workers, deserve a workplace free of violence, we must resist the co-option of our experiences by institutions with investments in punishing the working class. Safety is undoubtedly at the forefront of conversations about the Canadian public library. However, its usage needs to be further interrogated if what we desire is true, collective safety for everyone in the library. Safety, as prison abolitionist Mariame Kaba (2024) reminds us, is a social relation articulated through access to material resources and our relationships with others (para. 8). When people have what they need to live full, healthy lives free from legislated violence, communities are better resourced to respond generatively and restoratively to conflict.
In what follows, I hope to identify and unsettle the discourse of safety that pits library workers against their patrons, locating the source of unsafety in the patron, and the solution in their disciplining. Using Sarah Ahmed’s (2004) theorization of affective economies, I will interrogate the “safety and security crisis” in Canadian public libraries as a discursive formation that relies on what, after Ahmed, I call the “sticky associations” between drug use, homelessness, mental health, and violence. Sara Ahmed offers the language of the affective economy to name how “emotions circulate between bodies and signs” and “align individuals with communities – or bodily space with social space – through the very intensity of their attachments” (Ahmed, 2004, p. 119). It is through this process of articulating “violent associations” that delineate the ‘victim’ and the ‘perpetrator’ that the constitutive antagonism of library worker vs patron forms.
In the first half of this article I look at the different mechanisms that work to affix associations of homelessness, substance use, mental health, and violence to one another in library discourse. In particular, I examine how mainstream media, library leadership, and library researchers contribute to a narrative of unsafety that hinges on the individuation of threat, safety, and solution. I then turn to how library workers engage the changing landscape of their work. Drawing from my own experiences as a librarian, union representative, and organizer with Library Workers For Supervised Consumption Sites (LW4SCS), I share insights on how library workers act on their affective experiences of threat and unsafety. In the final section of this paper, I offer suggestions for how a labor framework of Occupational Health and Safety can be mobilized to address the structural and legislated violence shaping public library work.
Yes, library workers are struggling with the changing demands of their jobs. We are increasingly caught in the crossfire of the war on drugs. However, rather than capitulate to dominant narratives that criminalize our communities, we have a responsibility to resist becoming willing agents of institutional violence against our patrons.
Library Work in Death Times and the Defining of an Antagonism
The frequency and intensity of emotionally charged interactions with patrons mirrors a deepening racial wealth divide across Canada. The introduction of the Canadian welfare state rollbacks in the late 1980s, major cuts to social services, wage stagnation, an affordable housing crisis, and the rapidly rising cost of living have concentrated wealth in the hands of some, while further disenfranchising others. In Toronto, where market speculation and corporate landlords have hastened the affordable housing crisis, there are over 15,000 people without a home each night (Facts about Homelessness in Toronto, n.d.). This astounding number is only part of the story: approximately 1 in 3 Indigenous adults in Toronto are either precariously housed or sleeping rough (McConkey et al., 2024). Black Torontonians make up 31% of the city’s unhoused population while only constituting 7.5% of the whole population (Black Communities and Homelessness, n.d.). Twenty-one percent of youth in Toronto shelters identify as 2SLGBTQ (2SLGBTQ Youth Homelessness in Canada, 2016). The racial and gendered dynamics of state abandonment and immiseration in Toronto, while on a particular scale, echo across the country.
As I write this article in the fall of 2026, 16 supervised consumption sites (SCS) in Ontario have shut down, while the remaining 8 lost provincial funding in June. Beyond safe places to use drugs, these closures eliminated other life saving services offered at SCS, including drug checking services, primary care, housing support, and grief counseling. Alongside these closures, Ontario has moved to further criminalize public drug use. The Safer Municipalities Act, passed June 3, 2025, gives police power to clear encampments on suspicion of drug use, arrest those who don’t leave, and impose fines up to $10,000 or jail terms up to six months. This crackdown makes it harder for outreach services to reach people where they are, while exposing them to greater risk of police violence and incarceration. With few safe, private places left to use drugs, public drug use has increased, including at the library, which offers bathrooms, good lighting, a weather controlled environment, and a place to rest. As Jackson Davis, Isobel Heintzman, and Aditi Mehta (2025) identify, TPL has increasingly become a haven for precariously housed and disenfranchised Torontonians.
While library management seeks to introduce new positions, partnerships, and policies to address the impact of the toxic drug supply and housing crisis on their institutions, library workers continue to feel unsupported and ill-equipped to deal with these dynamics as they play out in real time (Saunders, 2025). The fissure between the vision of library leadership and employees’ experiences of that vision result in library workers being disillusioned and distrustful of management (Stevenson, 2022). With little power to make institutional or structural change, library workers grasp for action within their control, turning to what Mariame Kaba and Kelly Hayes (2023) call “rituals of order” – the reliance on systems of policing and punishment to resolve conflict, even when they rarely lead to resolution or safety (Hayes & Kaba, 2023, p. 33). In the library, these rituals include disciplinary actions, namely warnings, exclusions, and calling the police, that escalate even minor tensions. These actions locate the problem in the patron and the solution in their disciplining, reifying an antagonism that pits library workers against patrons. This antagonism leads to library workers punishing patrons and advocating for punitive policy and procedure that hurt all working class people.
Obviously, this antagonism does not start or end with library workers. The very foundations of the public library are based in class anxiety – the late 19th century idea that an increasingly agitated poor and working class base (during a time of labor unrest) needed to be pacified and transformed into productive and law-abiding citizens through soft power (Augst, 2007). The regulation of the social body, which necessarily includes formal and informal threats to its unity, permeates the public library’s history as well as its current day social-spatial architecture. From racially segregated libraries, to police enforced late fines, to turning over patron information to government agencies, to high tech securitization, libraries have always participated in the surveillance and policing of marginalized patrons (Kaba, 2025). These practices of social regulation in the library recreate historical inclusions and exclusions, defining public space as belonging to some people while being threatened by others. Many workers, tasked with enforcing the rules of the library and maintaining order in the branch, come to perform the disciplinary function of the institution not just because their job depends on it, but due to their own affective attachments to particular ideas of safety and security.
During a period of intensified neo-liberal austerity, it is no surprise that the regulation of poor and working class people comes to mark the current crisis of the Canadian public library. Rooted in narratives of declining “safety and security,” media and professional reports frame the challenges facing the public library as a result of a growing surplus population with “complex needs” (CULC, 2023; Ghobrial, 2026; Zoledziowski et al., 2024). While committed to “access for all,” immiseration has outpaced the public library’s ability to serve marginalized patrons. In 2023, the Canadian Urban Library Coalition’s (CULC) Safety and Security Working Group – comprised of prominent library leaders and representing more than 50 of the largest public library systems in Canada – declared that “mental health and addictions” prevent the library from performing its literacy function, keeping its workers safe, and maintaining an environment open to everyone. Even while compassionate library workers and committed library leadership seek to address the needs of their patrons, the sticky discursive associations that have come to identify this crisis accumulate as a limit to the possibilities of meaningfully addressing it.
Quantifying a Safety and Security Crisis in Canadian Public Libraries
There has been an increase in high intensity acts of violence in Canadian libraries over the last 5 years. Canadian news outlets have met this intensification with an uptick of coverage regarding safety and security in public libraries. Articles with headlines such as “Library workers punched, spat on as security incidents rise, data shows” and “Library staff facing security issues, including verbal abuse, drug use in branches” exemplify this trend (Ruby, 2025; Zoledziowski et al., 2024). Specific events include a staff assault at the Thunderbay Public Library, a mass stabbing at the Vancouver Public Library, and the 2022 murder of Tyree Cayer at the Winnipeg Public Library (Ketonen, 2025; Travers & Ridgen, 2023). Journalistic accounts of these painful events draw on aggregated incident stats, library tours, and interviews with library staff and leadership. The grand narrative is of a Canadian public library under siege and in decline: a chaotic environment where workers and patrons feel unsafe and the library is unable to honor its primary mission of meeting the literacy demands of its community.
As library workers, we know the importance of critical media literacy. Drawing a story of declining safety from generic incident data is a red herring. In the library world, incidents are the quantitative metric for tracking when and how often library codes of conduct are breached and medical emergencies occur. For library workers, incidents are part of the job. A sample of non-violent behaviours that constitute an incident across many Canadian libraries include: sleeping, excessive noise or disruption of others, having an animal that is not a service animal in the library, food and drink in undesignated areas, intoxication, possession of open alcohol, drugs, or drug paraphernalia, theft, and trespassing (CULC, 2023, pp. 9–10). The reality is that the majority of incidents in Canadian public libraries are non-violent.
Mainstream journalists tasked with covering “safety and security” in the public library often rely on incident logs as the raw data for their reports. For example, a 2024 CBC article cites a 71% spike – from 1,362 to 2,334 – in incidents at the Toronto Public Library (TPL) between 2022 and 2023 (Zoledziowski et al., 2024, para. 4). According to CTV News, the Hamilton Public Library recorded 771 incidents involving security during the first 5 weeks of 2026 (Ghobrial, 2026, para 4). While these numbers may be alarming, it is important to note that they are an aggregate of incidents ranging from mundane to violent. Neither story distinguishes between stats on serious cases in which library workers or patrons are in an unsafe environment – as in the presence of a weapon – and minor incidents of inconvenience – a noise violation, for example.
In fact, according to the CULC Safety and Security Toolkit, a study of 33 Urban Canadian library systems determined that violent incidents made up only 0.001% of all visits to the library (p. 42). In addition to limited violent incidents, the majority of incidents tracked by these 33 libraries fall under the categories of harassment, drugs or alcohol (including possession of paraphernalia and intoxication), and trespassing by banned patrons (CULC, 2023, p. 42).
Disaggregating incident data provides a more nuanced view of the difficulties facing the public library. It also troubles the categorical slippage of what otherwise might be called insubordination into violence and threat. Despite what some Canadian news outlets might have us believe, the library is not a place where random acts of physical violence are inescapable. It is a place where the complexities of life within a white supremacist settler colonial capitalist order are lived out publicly. This includes the many ways that people marginalized by systems of power make sense of their lives and prioritize survival. Substance use, intoxication, dissociation, and emotional or psychological distress all flow downstream from state violence directed at Indigenous, Black, queer, poor, and disabled people living in Canada. The sustained material and psychological harm of legislated violence often results in visible expressions of pain and adaptive strategies of survival. As people in the harm reduction and disability justice communities remind us, these are reasonable responses to subjugation.
Nevertheless, substance use, intoxication, and drug poisonings are key fulcrums on which these narratives of safety and insecurity pivot. Library leadership and professional organizations regularly conflate rare experiences of extreme violence with symptoms of the capitalist crisis under the banner of unsafety: houselessness, racialized policing, and a toxic drug supply. A drug poisoning, for example, is a medical emergency requiring medical attention. Though often scary for the responder, the person experiencing a drug poisoning is at risk of serious brain damage or death. Drug poisonings, just like public drug use and intoxication, are not inherently violent. Still, in most libraries, drug poisonings, drug use, and intoxication are tracked and treated as violent incidents. This is not just a matter of mislabelling. The framing of a safety and security crisis in which public libraries are overwhelmed with violent incidents due to “drug addictions and mental health” collapses categorically different forms of harm. Within this framework a medical emergency like a drug poisoning becomes indistinct from a disruptive but non-violent behavior like sleeping at a computer station or an act of interpersonal violence. This logic transforms distinct situations into an environment of undifferentiated threat, wherein a non-violent event equally contributes to the library as a site of unsafety. Through this collapse, we lose the ability to be precise about the challenges facing public library workers and the diverse responses these challenges require.
To put it another way, the process of creating an abstract and undifferentiated category of incident, feeds the carceral myth that people who use substances are a public safety threat while leaving the conditions that create public drug use and a toxic drug supply crisis unconsidered. Substance use and intoxication are not inherently violent or dangerous. In fact, people who use drugs experience interpersonal violence at disproportionately high rates, not to mention the violence of state neglect, policing, prohibition, and incarceration (Swaich et al., 2023). There are real safety issues associated with the toxic drug supply crisis, most urgently, death. It is through harm reduction, decriminalization, safe supply, affordable housing, and robust social services that we can begin to address the roots of the crisis. Conceiving of public drug use primarily as a safety and security threat to those who witness it imposes a distinction between “innocent patrons” (read sober, housed, white) who deserve safety and “problem patrons” (however outdated we like to think this framing is in library literature). These are racialized, classed, and colonial designations. Often, the very presence of people who use drugs or who are intoxicated (or who are merely assumed to be intoxicated because of their appearance or affect) triggers an alarm. Working as a public librarian in two large Canadian cities, I witnessed how complaint-based rules of conduct grounded in the vague language of “respect and consideration” resulted in the regulation not only of people’s behaviour towards others, but patrons’ state of mind, physical presentation, and use of space. The discomfort that some people experience when being in proximity to vectors of social abandonment and immiseration they are not used to interacting with becomes perversely rendered harm in and of itself. Disregard or transgression of social norms around hygiene, sobriety, and the use of public space are regularly surveilled and punished by both staff and patrons – a structural problem disguised as interpersonal conflict. It is not uncommon that after a complaint, a patron may be told they cannot return to the library until they have showered or sobered up, that they must leave their belongings outside before entering, or that they must quiet or quit the conversation with voices only they can hear. The consequences of these regulatory decisions are profound and often have dangerous impacts for our patrons most impacted by the war on drugs.
Data, aggregated or not, requires interpretation. While alarming numbers might help convey the magnitude of the crisis in public libraries, the articulation of fear and unsafety that accompany these stats give them an affective substance. By collapsing varied incidents into a single category of violence, diverse instances of discomfort on the job come to be experienced by workers as violence even when they are clearly not. In an affective economy where poverty is circulated in and through disgust or violence, fear and anxiety binds patrons and workers together along familiar classed and racialized lines. It is this opposition, wherein the presence of some leads to the perceived unsafety of others, that determines how discipline and punishment is exercised in the library. Together with stats and images (real and conjured), the affective experiences of library workers and patrons are central to the construction of a “safety and security” crisis in Canadian public libraries.
Framing a Safety and Security Crisis
Surveying Library Workers
The way library workers make sense of their experiences is shaped by broader discourse. As media and library leadership address the larger picture, library unions and scholars have increasingly turned attention to frontline workers’ experiences of safety (CUPE Research, 2023; Helfritz, 2018; Knox, 2024; LaPierre, 2021; Stevenson, 2021, 2022). Notable is Siobhan Stevenson’s 2022 survey of 527 library workers across four Canadian library systems about their experiences of third-party violence and incivility, which she frames as a continuum. In the survey section entitled “Feeling Safe”, Stevenson asks participants to consider “physical and sexual assault or threats thereof” alongside “rudeness, eye-rolling and a disrespectful attitude” (Stevenson, 2022, p. 338). The introduction continues: “essentially, any kind of negative interaction that leaves you hurt, feeling intimidated, fearful, or unsettled. It is important to note that you don’t have to experience these things personally to be affected negatively by them” (p. 340). Included in this section are questions about sexual assault, physical aggression, and “dealing with” substance use and mental health. When asked how often respondents felt unsafe – prefaced as including feeling “anxious, unsettled, disquieted, and vulnerable or physically unsafe” – 67% of respondents indicated they felt unsafe on a weekly or monthly basis (p.340).
Reflecting on the limitations of the survey, Stevenson acknowledges that situating questions about “feeling safe” within a continuum between incivility and violence “conflated and flattened affective nuances and differing subject positions” and “may have predisposed respondents to focus on their encounters with, and perceptions of, the less common experience of third-party violence over their exposure to incidents of incivility” (p.347). Stevenson goes on to highlight how “references to vulnerable and street-involved people and their characterization as threatening, for example, dominate the findings” (p.347). As Stevenson acknowledges, the survey instrument itself influences how workers come to interpret and report their experiences, collapsing meaningful differences in kind and severity into a single register of threat.
Notions of incivility have long served a disciplinary function in the library, rooted in ideas of respectability and used to police racialized, poor, and gendered bodies. That 73% of Stevenson’s respondents identified as white and 81% as female matters (p. 339). White womanhood serves as a marker of purity and white women have historically been positioned as “civilizing” agents of colonial and racial projects. As Gina Schlesselman-Tarango (2016) articulates, this led to the construction of the “Lady Bountiful” archetype in librarianship that fused patriarchal ideals of femininity with white supremacist missionary work. This legacy persists today, positioning the white woman librarian as the normative subject empowered to police, discipline, and manage “deviant” patrons under the guise of maternal benevolence and professional neutrality. The Lady Bountiful trope in library literature illuminates how “white supremacy and patriarchy have worked to produce a particular subject in librarianship” through which threats to civility are worked out and disciplined (Schlesselman-Tarango, 2016, p. 683).
Though Stevenson notes concerns about conflating incivility with violence, she does not interrogate incivility itself as a racialized, gendered, and classed category. Supplying frontline workers with a language for their experience, however validating, risks entrenching an interpretive lens in which behaviours such as substance use and intoxication fit into the same affective register as physical violence. This affective slippage is not confined to individual encounters or survey data. It scales up, providing the interpretive scaffolding for how library organizations, media outlets, and administrators frame substance use and mental health as an institutional crisis.
Sticky Associations and Policing the Homeless Body
It is undoubtedly true that public libraries across Canada have seen an increase in drug use and drug poisonings in their spaces. In 2022, the Edmonton Public Library (EPL) recorded almost 100 drug poisonings while TPL saw a 529% increase in suspected poisonings (CULC, 2023, p. 40; Zoledziowski et al., 2024, para. 4). In the first 3 months of 2025, SPL responded to nearly 50 drug poisonings (Warren, 2025), para. 3). This is the reality of the toxic drug supply crisis made worse by defunding harm reduction services and further criminalizing public drug use. The defunding of drug testing and poisoning reversal services alongside the grief and loss accompanying the closure of SCS has resulted in more drug poisonings in public, including the library. In the first 6 months after the initial closures of SCS in Ontario, there was a 69.5% increase in EMS-treated opioid toxicities and a 19.4% increase in opioid toxicity deaths (Kolla & Gomes, 2026, p. 3).
Even when leadership acknowledges the harms of the toxic drug supply crisis, the public library participates in stigmatizing, punishing, and criminalizing patrons who use drugs. Visible drug use, the presence of drugs or drug paraphernalia, and exposure to people who are intoxicated is treated as a safety and security issue. As examples, libraries like EPL and SPL have Code of Conduct rules about possession of substances and intoxication, ranging from a 1 day suspension up to a 6 month system wide suspension for repeatedly being caught with open alcohol or drugs (CULC, 2023, p. 10). Compounding punishment for ‘trespassing’ while banned from the library, can lead to months long suspensions for people who need the library most. Rules that regulate what a patron has on their person, and whether or not they are perceived as intoxicated, do not focus on the very real dangers associated with an unregulated and contaminated drug supply: exposure to biohazardous materials; or instances when intoxication is coupled with physical violence or harassment; and, most importantly, death. Instead, they seek to regulate deviant patrons that may upset the order of the library due to their embodiment.
Fig. 1: Edmonton public library code of conduct example (CULC, 2023, p. 10).
We need to be honest about what kind of substance use is targeted by code of conduct rules. When we talk about drug use in the public library, we are speaking directly to the intersection of poverty, homelessness, racialization, and street drugs. We need to be honest about the conditions under which people are forced to use in public due to the absence of housing, privacy, and now closed SCS. If we are still being honest, drugs are a survival strategy, and often a smart one. Drugs can help one cope with the unbearable suffering of state sanctioned violence and its accompanying grief; drugs can keep you alert and warm at night and they can keep hunger at bay; drugs can make you feel good and they can make you feel nothing. As the Indigenous Advisory Board at the Drug User’s Liberation Front (2023) reminds us, “illicit substances are necessary for many of us to continue surviving in unlivable conditions” (p. 5).
Through theorizing constructed associations between the asylum seeker and the international terrorist in Europe, feminist thinker Sara Ahmed (2004) helps us understand how the “homeless drug user” becomes a reified figure that circulates through social and cultural repetition, accumulating the affective residue of everything associated with it. Homelessness and drug use don’t have to exist together for this to work, they only have to be repeatedly proximate in discourse. Once that proximity is established, pity, fear, and revulsion don’t attach to particular behaviours but to the body itself, ready to be activated by anyone who fits its visible signs (an overflowing shopping cart, unkempt appearance, talking to oneself, nodding off).
For Ahmed, signs don’t have affective value on their own, affect is accumulated through repeated association, so that emotion appears to reside in an object or body when it has actually adhered there through a history of contact between signs. “Homeless,” “addict,” “dangerous,” “dirty,” “unstable” circulate together in media coverage, in policy language, and in conversations with co-workers. Through this repeated association, they begin to stick to one another, so that invoking one is enough to summon the affective charge of the others. A person doesn’t need to be observed using drugs to be deemed scary or wielding a weapon to be considered violent, the visible signs of homelessness are already sticky with the associations of drug use, criminality, and unpredictability, because those signs have circulated together so often that the boundary between them has dissolved.
Regardless of behaviour, the “homeless body” is a threat to the publicness of the library and the safety of its “average” (or in Ahmed’s terms, ordinary) patrons. The signs and associations not only stick to each other but to the object or body in question. Samira Kawash (1998) uses the language of the “the homeless body” to articulate the process through which the “ideological and discursive specter of homelessness” creates a sticky figure – one that “emerges as a particular mode of corporeality in contingent circumstances through which the public struggles to define and secure itself as distinct and whole” (Kawash, 1998, p. 324). It is through the act of exclusion of the homeless, drug-using body in all its supposed violence, that the public makes proprietary claims about spaces like the library. As Kawash says, public space comes to be seen as belonging to the public, of which homeless people are not part. It is the homeless body, Kawash articulates, that is “the stake and site of the contest over public space” (p.323).
By declaring the presence of embodied precarity experienced by an “overwhelming” number of library patrons a crisis of safety and security, new safety and security measures (security guards, trespass bans, carding policies, surveillence) become the logical response. The effect becomes the cause. For example, in a conversation with local Hamilton news channel CHCH (2026), HPL CEO Paul Takala told viewers that patrons who use drugs make the library unsafe for families, newcomers, and seniors, among other marginalized groups. In this declaration, Takala not only erased the intersecting experiences of people made vulnerable to state violence through legacies of white supremacist settler colonialism, xenophobia, and prohibitionist state policies, but made drug users the primary source of insecurity and unsafety in the library – the barrier preventing all other marginalized members of the community from flourishing.
This does not diminish the reality that all kinds of patrons do fucked up things at the library. Transphobia, homophobia, racism, misogyny, and ableism are regularly wielded against people who work at and use the library. Many of my most aggravating encounters have been with entitled, privileged patrons. These include racist mocking and sexual harassment of low-paid library staff; angry yelling when made to wait for the printer; and my personal favorite, the strong suggestion that we “could be the first library to permanently ban homeless people”. And still, a marginalized identity does not preclude someone from being aggressive, violent, or discriminatory. In attempting to disentangle ideas about harm at the library, I want to be careful to not suggest that things are not difficult, complicated, or anxiety-inducing, nor that people do not experience actual threat to their physical safety. However, how we respond to moments of crisis, conflict, and discomfort have real, material impacts for marginalized patrons. When discomfort around visible drug use leads to the banishment of an unhoused patron in the middle of winter, or when staff calling the cops to remove someone who is trespassing leads to criminalization, we must ask ourselves what kind of safety we prioritize and whether or not we truly feel that such actions meet our needs. Individual incidents of frustration, anger, and even violence need not undermine our project of seeking material, sustained safety that leaves no one behind.
Library Workers Responding to the Crisis and Rituals of Order
Rituals of Order
Fear, whether it registers or not, drives the impulse to reach for rituals of order. Fear of losing control of the branch, fear of losing your job, fear for your safety, and fear of the homeless body. As Kaba and Hayes say:
governed by fear, people are largely cooperative with systems that produce torture, mass death, and annihilation. That is the greatest danger that fear poses: not panic amid disorder, but co-operation with an order that we ought to find unspeakable – one that is actually poised to bring about our own distinction (p. 38).
When people are scared, they want to feel safe, they want a restoration to order. This is a human impulse, a reaction to emotional associations that are socially constructed and maintained but felt all the same. Attention to these impulses is important. They are informative even if not how we imagine. Safety promised by expulsion and criminalization allow us to quiet the bad feelings, ignore our own precarity, and detach from the conditions that position us closer to our patrons who use drugs than we want to admit.
However, safety is social and always in flux. It articulates a relationship to harm, including the structural distribution of harm (or absence of harm) within a society. “Safety isn’t a thing: it’s a social relation” Kaba reminds us. “I’m more or less safe depending on my relationship to others and to my proximity to the resources I need to survive” (Kaba, 2024, para. 8). Safety cannot be possessed, but is a function of social and economic relations that unevenly distribute state produced vulnerabilities. Unsafety is not just a feeling but a material reality for many. Wealth and whiteness are two major predictors of safety established through carceral infrastructure built to uphold their supremacy and protect them from threat. However, a vision of safety dependent on the disappearance of others is always tedious, subject to disruption by a mere encounter with the other.
Library workers must be involved in articulating processes of accountability and redress that do not pit us against our patrons. We must figure out how to respond to the difficulty of public library work during death times. For instance, when library workers and patrons are exposed to second-hand harms of substance use – exposure to biohazardous materials and sharps, people who may become aggressive when intoxicated, or the emotional impact of responding to an overdose – we can turn to strategies that help mitigate the harm. We can wear gloves and advocate for sharps disposals, practice de-escalation skills, and insist on paid time to self-regulate and process. Library workers must continue to demand access to resources that allow us to respond to our patrons with compassion, grace, and skill. This includes more staffing and better training, but it also includes SCS, affordable housing, food security and an end to the drug war. From resisting the discursive associations between violence, substance use, and intoxication in our language and our reporting mechanisms to showing up in solidarity with people being evicted from encampments, library workers can best advocate for better work conditions by recognizing that our struggle is connected to that of our marginalized patrons. Safety and security come from having our material needs met, from being in a community that cares for each other, and from learning the skills and building the infrastructure that supports us when things get tough.
Library Workers for Supervised Consumption Sites
Responsibility carries many valences in a workplace like the public library and competing needs rarely reconcile themselves. Patrons, co-workers, and leadership all have expectations that shape how staff respond to incidents. However, it is the wage-relation that often structures a prioritization of staff responsibilities. As the front-line representative of the institution, library workers absorb its liability almost by default. Property starts to outweigh people, and the loudest voice outweighs the most marginalized. This slide occurs not just because library workers want to keep their jobs or have their own investments in punishment, but due to an institutional liability framework that positions safety as an administrative problem rather than a relational one. As Ben Robinson (2019) describes, through the framework of liability, security is treated as one undifferentiated task, lumping fire and flood prevention, theft, digital privacy, and violent behavior into a single risk-management mandate. Bundled with it is a “security ethos” that deputizes staff to surveil and discipline patrons read as engaged in “suspicious activity” (Robinson, 2019, Section Fire, Flood, and Fist Fights). A security ethos and the tremendous amount of responsibility experienced by library works leads to hypervigilance often actioned through surveillance. And because a liability framework is built around assessing and neutralizing risk, its default responses are removal and punishment.
My desire is not to blame frontline library workers for the way we participate in the punishment of our patrons. Library workers, more than most, are aware of the social conditions that shape public space and thus our work. Staying tethered to the compassion and care that got many of us into the field in the first place is an ongoing practice. However, in the absence of meaningful support, resources, and compensation, library workers reach for what they know, and seek to restore order to an always fragile ecosystem. As I have written elsewhere, even those who resist the imperative to police our patrons or punish communities to which they belong, are regularly compelled by the wage relation to comply. While this too can create bad feelings or a sense of moral injury – an affective response to the alienation one feels due to the inability to direct their own labor – it does not diminish the real material impact such punitive practices have on our patrons (Cowley, in press).
As a founding member of Library Workers for Supervised Consumption Sites (LW4SCS) I have had the privilege of engaging many library workers across the province about their experience of the toxic drug supply crisis. LW4SCS formed in fall 2024 in response to Ontario’s Bill 223, the Safer Streets, Stronger Communities Act, which mandated the closure of SCS within 200 meters of schools and childcare centres. As library workers, we recognized that these closures would significantly impact our communities, our patrons, and our work. Beginning with an open letter to Premier Doug Ford, LW4SCS has since grown into a collective that holds regular meetings, compiles frontline data and testimony, publishes zines and articles, hosts workshops, presents at conferences and staff days, delivers board deputations, and pushes our individual locals into action.
Our central aim is to introduce frontline library workers’ voices into the conversation, and to give workers the space, time, and language to organize around the toxic drug supply crisis as it manifests in their workplaces. We believe that when we come together and engage in rigorous analysis and self reflection, we can produce an alternative conceptualization of the crisis that prioritizes collective safety over punishment. Grounded in the commitment that our right to safe and supported work must never be weaponized against patrons who use drugs, we believe we have a responsibility to push not only our employers but our unions to develop a substantive analysis of, and response to, the compounding crises of our time. LW4SCS seeks to create spaces where workers can sit with the complexity and contradiction inherent in frontline library work. As Harsha Walia offers, “complicity and solidarity are always entangled and not easily compartmentalized, and holding both together is the real challenge” (Walia, 2012, para. 9). Our futures are bound up with our patrons’, and thus LW4SCS rejects the premise that patrons are our enemies or that punitive measures can resolve either the toxic drug supply crisis or our own deteriorating working conditions. We encourage library workers to build skills, shift their language, find one another, and get organized. Together, we have the ability to transform not only the narrative around safety in the library, but practices that pit us against our patrons.
An Injury to One is an Injury to All
When it comes to evicting unhoused people, I want to remind people: [Canadian Union of Public Employees (CUPE)] members have the right to refuse unsafe work – work that endangers their lives or the lives of others. If any worker feels that participating in the eviction of unhoused people from encampments is unsafe, for themselves or for anyone, including the people who are homeless, they have the right to refuse that work (Hahn, 2023).
At a 2023 event put on by Voices for Unhoused Liberation in Toronto, CUPE Ontario President Fred Hahn reminded CUPE workers, in particular city workers tasked with dismantling encampments, that they have a right to refuse unsafe work. What has stuck with me about Hahn’s statement is the assertion that the right to refuse includes the right to refuse to endanger others. His statement not only names the connection between housing and safety, but positions workers as active agents with rights and responsibilities towards others, including unhoused people who are regularly and systematically denied access to safety. While I am not suggesting that library workers assert their right to refuse unsafe work in response to the toxic drug supply crisis – I am not certain we have the collective will to do this just yet – I do believe that health and safety mechanism such as the right to refuse provide leverage for workers who oppose the state violence they are expected to perform.
Despite the well-documented limits and betrayals of corporate unionism, labor organizing is one of the best tools available for library workers navigating the toxic drug supply crisis. As an analytic, labor provides us with a framework for linking our own exploitation with the precarity of our patrons. After all, how many library pages can afford to live anything but paycheck-to-paycheck in Toronto? As a tool, the union provides us with an avenue to operationalize this analysis. Unions ask us to redirect our anger away from our patrons and towards our employer and the larger structures that shape our work conditions. As a horizon, labor organizing requires us to collectively articulate and organize around our demands and link them to larger labor struggles. An injury to one is an injury to all.
Health and Safety (H&S) is the bread and butter of union organizing. Workers in Canada are guaranteed three basic health and safety rights: the right to know about hazards, the right to participate in workplace H&S decisions, and the right to refuse unsafe work (Canadian Centre for Occupational Health and Safety Government of Canada, 2025). Together, these rights offer distinct yet interconnected points of intervention for library workers hoping to address the toxic drug supply crisis and the library’s punitive response. It is through the language of H&S that a facility manager at one of my branches argued for safe disposal containers and for PPE during Covid. Through H&S language, the TPL workers union, CUPE 4948 has been able to advocate for a 3 day leave after serious incidents, and push through “no working alone” language in the collective agreement (CUPE, 2020). These are vital initiatives that the employer would certainly not have adopted on their own and that lay the groundwork for other locals to seek similar agreements with their employers.
As I have been arguing throughout this paper, labor organizers need to adopt an expansive understanding of H&S in order to be effective on this issue. Our aims must match our analysis, acknowledging the uneven, structural impact of punitive workplace policies. Take for example the increased use of security guards and partnerships with the police to address drug use in the library. We know that police and security disproportionately target, criminalize, and kill BIPOC people. What would it mean for us to organize around an analysis of police and security presence as a workplace hazard? One that disproportionately targets poor and racialized people, making the library and library work unsafe for many. Could the language of H&S help us push our bosses to invest in alternatives to policing and transformative approaches to conflict?
Organized around the recognition that police presence in the library is a hazard, library workers meaningfully included in decisions about H&S in the library can take a different approach than the employer who prioritizes liability above all else. While an employer may look at the cost saving benefit of contracting low paid security guards to deter and remove potential ‘threats’, unions can advocate for better staffing models, real and substantial training, and deeper community partnerships. Frontline workers can use the wisdom they have earned to devise alternatives to police in the library that seek to address conflict and harm in ways that aim to restore community, not dismantle it. We do not need to settle for the crumbs management provides, we can continue to struggle for real investment in our communities.
The right to refuse unsafe work is perhaps the bedrock of labor activism. The right to refuse is relatively straightforward in contexts like a construction site, where the hazard is immediate and physical. What makes Hahn’s invocation of this right in defense of unhoused peoples striking is its application to a situation that is less immediate. Even though the argument might admittedly stretch the current legal and political framework governing the right of refusal, rank and file labor organizing must build the capacity to withhold our labour in politically charged circumstances, where pushback from both employer and state is expected. When exercised individually, the invocation of this right will likely be ineffective. For example, I refused to call the police on a patron recovering from drug poisoning in a branch washroom. Despite my training as a first-responder and extensive experience in frontline harm reduction, I was fired for insubordination. Even as the central branch of EPL hosted a press conference about a public health partnership providing patrons with free Naloxone kits the very same week, no recourse to structural analysis or safety was available through my union: how could there have been? What if my individual action, though, was part of a collective refusal — an entire branch’s staff declining to call the police in situations that could be handled otherwise? Whether formalized through the avenues of union protection, or instigated directly by rank and file self organization, the right to refuse is most meaningful, effective, and defensible, when it is asserted collectively.
The ability to rearticulate the scope and language of harm is one mechanism through which library unions might begin to unseat the safety and security narrative that defines conversations about the toxic drug supply crisis in the library. This work, however, is demanding, and possible only once frontline library workers acknowledge that we have more in common with patrons who use drugs than with the library CEO. Antagonism between library workers and patrons serves management alone, which mobilizes worker fear and anxiety to enforce punitive order within the branch. Once library workers recognize that our safety is tied to that of our patrons, and to working-class people across the country more broadly, we will be positioned to advocate for policies and procedures that rise the tide for all. This is not easy work; it requires developing practices that allow us to keep our jobs and pay our bills while resisting the institutional imperative to punish our patrons. For this reason, those of us committed to abolition and harm reduction must find one another, talk to our co-workers, interrupt the criminalization of our patrons, and hold our union leadership’s feet to the fire.
Acknowledgements
Thank you to Brittany Paloma Fiedler, Jaena Rae Cabrera, and Emily Drabinski for the energy and care they dedicated to editing this article. Publishing with In the Library with The Lead Pipe was a wonderful experience under their thoughtful guidance.
Thank you to my comrades in LW4SCS. Your care, commitment, and sharp analysis has sustained me over the last few years. I have been so lucky to learn with and from some of the most badass library workers in Canada. Beth Capper, may we always support each other in ‘saying the thing’. Finally, thank you Rob Jackson, the best intellectual and life partner I could ask for. Your edits and feedback make my thinking demonstrably better.
Augst, T. (2007). Faith in reading: Public libraries, liberalism, and the civil religion. In T. Augst & K. Carpenter (Eds.), Institutions of reading (pp. 148–183). University of Massachusetts Press.
CHCH News. (2026, March 18). Hamilton chief librarian says initiative to curb drug use at Central branch is effective [Video]. https://www.youtube.com/watch?v=GBb6qTPYTjE
Hayes, K., & Kaba, M. (with Schenwar, M., & Walia, H.). (2023). Let this radicalize you: Organizing and the revolution of reciprocal care. Haymarket Books.
Helfritz, A. E. (2018). Potential workplace violence in assumed low-risk environments: Library employee perceptions and safety concerns [Doctoral dissertation, University of the Rockies]. ProQuest Dissertations & Theses Global. https://www.proquest.com/docview/2131396889?pq-origsite=primo
McConkey, S., Iannace, J., Snyder, M., Bourgeois, C., & Smylie, J. (2024). Measuring Indigenous homelessness: Findings from Our Health Counts Toronto. Canadian Journal of Public Health = Revue Canadienne de Santé Publique, 115(Suppl 2), 227–238. https://doi.org/10.17269/s41997-024-00974-7
Saunders, C. (2025). Working in an unsafe consumption site: A library perspective. Counterpoint: CUPE National Publication, (Fall 2025), 32–37.
Schlesselman-Tarango, G. (2016). The legacy of Lady Bountiful: White women in the library. Library Trends, 64(4), 667–686. https://doi.org/10.1353/lib.2016.0015
Stevenson, S. (2021, October). Health and safety culture at the public library: Listening to those on our frontlines. Public Libraries, 60(5), 32–38.
Stevenson, S. (2022). Third-party violence, incivility, and the frontline public library worker. The Library Quarterly, 92(4), 330–354. https://doi.org/10.1086/721394
Swaich, A., Richardson, L., Cui, Z., DeBeck, K., Milloy, M.-J., Kerr, T., & Hayashi, K. (2023). Experiences of violence during the COVID-19 pandemic among people who use drugs in a Canadian setting: A gender-based cross-sectional study. BMC Public Health, 23(1), Article 1020. https://doi.org/10.1186/s12889-023-15929-9
Recently, interest rates for government debt have been rising worldwide. As I write the yield of the 10-year US treasury is over 5%. It had been below 4.5% for over a year when, in March, it started a steady rise. This is despite (and probably because of) Scott Bessent's efforts to force it lower, which JP Morgan compared to 'paying your mortgage with your credit card'.
A big part of the reason is governments persistently borrowing to cover deficits, with no plan to reduce them. For example, the Trump administration has borrowed more than the growth in US GDP. Projections are that by the end of their term they will have borrowed $1.40 for every $1 growth in GDP. Half of that GDP growth is from the AI bubble. Remove it and the administration has already borrowed almost 2.5 times the GDP growth.
But another part of the reason is that governments are facing competition for the available funds from other highly-rated borrowers. Hyperscalers such as Microsoft (AAA) and Alphabet (AA+) can no longer pay cash for the immense sums they believe they need in order to keep up with their competitors in the race for AGI. So the bond market is facing a significant increase in demand, which naturally increases interest rates.
Below the fold I look into how this competition for funds is going.
It is a fact of life that the more debt you carry, the worse your credit rating and thus the more expensive it is to add to your debt. This is why the hyperscalers are so anxious to have their massive borrowings kept off their balance sheets and attributed instead to "Special Purpose Vehicles". They hope in this way to distract the rating agencies enough to prevent being downgraded. A downgrade would increase their interest expenditures, thus reduce the coverage of their interest payments, and potentially lead to a further downgrade and a death spiral.
Achieving an investment-grade rating from Fitch, Moody’s and S&P soon after going public would be a remarkable feat for the two lossmaking AI labs, unlocking big benefits for the companies and their infrastructure partners including Oracle and Nvidia. …
Analysts at rating agencies are waiting to see the results of their IPOs before reaching a decision. The two companies remain unprofitable and have shown little sign of generating positive free cash flow. They also face growing risks, including the popularity of Chinese open-weight models.
“We still treat OpenAI and Anthropic as deep in speculative grade . . . they are in the red,” said another senior credit analyst.
An investment-grade rating of BBB- or above would allow them to borrow from institutional lenders such as pension funds.
S&P reckons a full half of the economic growth coming from the US private sector was linked to AI-centric activity over the past year. And they assume the top six US hyperscalers will collectively spend more than $7tn in the five years through 2030. So the fate of the US economy, not to mention the stock market, credit, infrastructure and real estate, is increasingly tied to the AI show staying on the road.
Every time we take a deep dive into this sector, we find that capex is rising faster than we anticipated, financings are becoming more complicated and less transparent, and that returns on investment will take years to realize.
The credit rating process involves analysts diving deep into the workings of companies, typically with access to a ton of non-public information. But the authors write that, despite AI return on investment being critical to the rating judgments, “the big six hyperscalers don’t quantify their returns on investment on AI”. Maybe, the report’s authors speculate, this is because it’s difficult to work out. Or maybe, “simply, entities just choose not to share that data.”
The reason these companies "choose not to share" their "return on investment on AI" with the rating agencies is left as an exercise for the reader.
Source
Nangle has a set of interesting charts based on S&P data. This one shows the ratio of debt and "debtlike commitments" to EBITDA (earnings before interest, taxes, depreciation, and amortization) for the six largest US hyperscalers. Each company is labeled with their current rating. Horizontal dotted lines indicate the trigger level for this ratio that could cause a downgrade. Note that:
Oracle's projected 2027 and 2028 commitments are so close to the trigger for downgrading from their BBB- rating that even a small drop in earnings would render their bonds junk. This would force institutional investors to sell them, making future borrowing and rolling over existing debt as it comes due much more expensive.
Amazon's projected 2027 and 2028 commitments look as though they would cause a downgrade from AA, but it is more likely that Amazon can increase earnings fast enough to prevent this.
Alphabet has a lot of headroom before their AA+ rating is at risk.
Source
An alternative way of looking at the situation is to ask how much more could these companies borrow, or how could their EBITDA decrease, each year before they would be downgraded. This chart answers the question. Note that:
For 2027 through 2029, only Alphabet can borrow more than $100B each year.
On the current projections Amazon will, and Oracle is extremely likely to, be downgraded.
Nangle makes an English understatement:
For double-A-rated Amazon to move to high single-A wouldn’t be much of a disaster. But in Oracle’s case, getting downgraded means getting junked. And with around $117bn of index-eligible US dollar bonds, that would be quite the event.
The limited headroom for increased borrowing starts next year, which is when the "take or pay" commitments with data centers start costing serious money.
Source
The third chart provides yet another way to look at the situation. It plots the companies' projections for EBITDA against the threshhold for downgrades (dotted line). Note that:
Amazon has to exceed and Oracle has to meet their EBITDA projections if they are not to be downgraded.
All these companies' projections for their EBITDA growth from 2025 to 2029 are astonishing. Eyeballing the chart we have:
Alphabet: 2.5x
Amazon: 2x
Meta: 2.6x
Microsoft: 1.8x
Oracle: 3.5x
SpaceX: 15x
Nangle's comment that these represent "pretty punchy EBITDA growth" is another understatement. For example, Alphabet is saying it will add around $225B in EBITDA by 2029. I'm an Englishman, so I can say that it isn't clear where an additional $225B would come from.
Alphabet and Meta together are in effect projecting that by 2029 companies will increase their advertising budgets by $385B/year. Of course, this isn't going to happen. All these companies are expecting the EBITDA growth to come from the "returns on investment on AI" that they "choose not to share" with the rating agencies.
Ignoring the ludicrous SpaceX projection, the other 5 companies' "punchy" projections assume 2029 increased EBITDA of around $900B, which is less than half the estimate of over $2T/year in additional revenue needed to cover their borrowings.
Is it plausible that in 2029 OpenAI and Anthropic will together have more than $1.1T in revenue to make up the difference?
The report also features a circular financing taxonomy, which includes residual-value guarantees, take-or-pay agreements, chip financing, lease liabilities, power purchase agreements, backstop guarantees, lease guarantees, and direct equity investments.
interconnected financing structures could amplify volatility if demand weakens unexpectedly. The failure of one entity would have implications for entities that have lent it money, have guaranteed the value of its assets, or are just expecting payment for goods delivered.
The broader question is whether circular financing is creating leverage collectively that is individually manageable but could become highly correlated should demand fall sharply. . . .
The scale of overlap and interconnectedness is vast. In a downturn, even the best capitalized and most profitable firms may incur substantial pain.
Ed Zitron's Concentration Risk points out another big risk to the hyperscalers' EBITDA projections that S&P seems to have missed:
Anthropic and OpenAI’s compute commitments, in my mind, should be seen more as debt obligations than “contracts,” because they (as take-or-pay agreements) function in much the same way, requiring the company to pay whether or not they need the capacity.
For now, everything looks awesome. Microsoft, Google and Amazon have all had big bumps in revenue from AI lab compute spend along with massive, ever-swelling revenue backlogs — over $1.5 trillion worth to be specific. More than half of that backlog is attributable to Anthropic and OpenAI, which, as I’ll say again and again, isn’t a problem because the money is yet to stop coming in.
Eyeballing the chart, something like $1.1T of the revenue backlog for Microsoft, Oracle, Google and Amazon is revenue that they expect to get from two companies that have yet to make a profit. That is a concentration risk.
Per data from fintech firm Ramp, 80% of OpenAI and Anthropic's enterprise revenues come from 1% of their customers, a number that hasn’t improved over the last three years. Ramp’s lead economist Ara Kharazian notes that the top 1% skews heavily toward the tech sector and AI products and services, and that this was a level of concentration risk unseen in any other software category they tracked.
benchmarking company Vals AI has been evaluating different open and closed models by using its own neutral harness. When every model ran on the same harness in the Terminal-Bench 2.1 evaluation, the open-weights model GLM 5.2 from Chinese company Z.ai (Zhipu AI) scored within a point of Anthropic’s Claude Opus 4.7 and 4.8 while costing about five times less per completed task.
In other words, paying for closed frontier models buys about a four-month head start at about five times the per-task cost—but only when currently looking at tasks taking [a human] between eight and 12 hours.
Below 8 hours there is no advantage for closed-source, above 12 hours neither model can reliably succeed. This isn't a great basis for OpenAI and Anthropic to project hundreds of billions of dollars of future revenue.
On the low end, that means that Anthropic and OpenAI account for over $200 billion dollars worth of expected revenues for Microsoft, Google and Amazon in 2027, which is contingent on their ability to raise venture capital or debt, which is contingent on the continued growth of their businesses, which is contingent on growing AI spend from a small subset of customers, many of whom are funded by venture capital.
The reason this hasn’t been a problem yet is that when you sign these contracts, you tend to pay a small up front fee, and the capacity in question is yet to come online.
...
In other words, Anthropic and OpenAI are currently in the teaser rate period where all of that capacity — and all of the associated costs — are yet to hit.
Next year, at least $200 billion in compute costs are coming due.
The question is whether Anthropic and OpenAI, two unprofitable, unsustainable AI labs that lose tens of billions of dollars a year, will be able to afford to pay them.
If they can't pay, about half of the revenue backlog for Microsoft, Oracle, Google and Amazon goes away. That reduces their EBITDA and potentially triggers downgrades.
Nangle starts with this chart, comparing the yields on the hyperscalers bonds with the average for similarly rated companies. It shows that:
it looks like the market is pricing Oracle and SpaceX as quality junk, and maybe Meta in line with BBB credit, and the rest of them as sort-of-in-line with the average AAA-AA-rated US corporate issuer. Is this cautious enough?
On recognized debt, the market’s calm is justified . . . [but] . . . [t]he real risk lurks below the waterline: off-balance-sheet debt lifts the debt burden by nearly 150% on average and pulls the model-implied credit quality down 1-2 notches.
There follows a complex description of Allianz' model that maps from stock prices to bond ratings, summarized thus:
Allianz calculates the ‘distance to default’ for the hyperscalers — a measure of credit riskiness calculated using equity inputs and balance sheet data — over time. And following work that Moody’s KMV has done mapping distance-to-default to default frequencies, they map the kind of ratings associated with these outputs.
Working solely off recognised balance sheet data, Allianz finds that the ratings implied by equity punters’ collective histrionics are higher than ratings assigned by the agencies for Alphabet, Amazon, Microsoft and Nvidia, a touch lower for Meta and a disaster for Oracle and SpaceX.
I think that's not quite right. Moody's gives Microsoft their highest rating, Aaa, and so does Allianz' model. The model gives Alphabet, Amazon and Nvidia Aaa, which is higher than Aa2, A1 and Aa1 respectively. But overall equity investors are (naturally) more optimistic than bond investors.
But these ratings are a testimony to the effetivness of the hyperscalers' off-balance-sheet techniques for distracting both the equity markets and the rating agencies. Nangle writes:
Once they whisk debtlike uncommenced leases into the equation, they find that Microsoft and Amazon drop into mid-investment-grade territory and that Meta falls into the quality end of junk.
Nangle's first chart showed the market already rating Meta at "the quality end of junk". But once the bond market assimilates the off-balance-sheet debts the downgrades and interest rate increases will be quite dramatic.
What made you decide to tell Ray Trautman’s story? Was he a figure you encountered in your previous exploration of books and reading in the US Army during World War II?
I first encountered Ray Trautman when I was writing When Books Went to War (2014). I mentioned him a few times in that book, but he wasn’t a central figure. Remarkably, after that book was published, Ray’s sons reached out and invited me to visit them and take a look at their father’s records of his wartime work—thankfully, being a librarian, Ray kept everything! As I learned more about him, I was amazed by the incredible odds Ray faced in trying to get books and magazines to troops and his undaunted determination to do whatever it took to get soldiers the reading materials they wanted. I uncovered many forgotten stories about famous people—Eleanor Roosevelt, Ernest Hemingway, FDR, etc.—who Ray interacted with. I just found his life fascinating, and felt others would too.
Tell us a little bit about Trautman—who he was, what he did, and why he should be remembered.
Ray grew up on a farm in Mowrystown, Ohio (population: 200), where there was no community library and very few books. Yet, he treasured every book he could get his hands on—he loved learning about other people’s lives, making new discoveries, meeting beloved characters, and exploring the world. A seed was planted—he would ultimately decide to devote his life to books and libraries, and giving others the opportunity to experience the joys of reading.
His greatest contributions were probably made during World War II, when he became the Army’s Chief Librarian. Ray designed miniature paperbacks, Armed Services Editions, so foot soldiers around the world could carry weightless pocket-sized books wherever they went. Ray knew from experience that these books were many solders’ emotional survival gear. Because troops also wanted magazines, Ray worked with every major American magazine to create small, ad-free versions for troops. And he created an Army comic book, Overseas Comics, so troops would get their favorite cartoon strips.
Ray was responsible for distributing, for free, a half-billion books, magazines, and comics during World War II. In the process, an entire generation of Americans became devoted readers. Most entered the war with no reading habit, and they left the military with a lifelong desire to read, which veterans passed on to their families and friends. He truly revolutionized American reading.
What kind of research did you need to do, while writing the book, and what were some of the most memorable things you learned? Were there surprises along the way?
First, I needed to comb through boxes of records that Ray had saved, along with diaries, photographs, and notes tucked inside of his books. Ray’s son, Eric, secured Ray’s military records from the Army—thankfully, that consisted of almost 1,000 pages that provided guideposts for where Ray was, what he was doing, and when. For more details and context, I scoured newspaper and magazine articles, and read dozens and dozens of books about the people, places, and events Ray encountered.
I was amazed to learn how many famous people this unassuming librarian crossed paths with. Ray solved problems for Eleanor Roosevelt and Ernest Hemingway—and these stories have never been told! He also met Douglas MacArthur (and almost got court martialed for gently suggesting that MacArthur supply his troops with books), and was given library assignments directly from Dwight Eisenhower. One of the more memorable and surprising episodes I wrote about was when Trautman was trapped during the Battle of the Bulge and was forced to lead troops in a tank battle using only homemade weapons—and they survived! It turned out a librarian’s skillset was very useful in coming up with an unconventional solution in a life-or-death situation.
Three of your four books so far have focused on books and reading during World War II. What makes this subject so interesting for you, and why is it important?
I think examining reading during wartime fascinates me because history shows that when democracy has come under attack (in both World War I and World War II), the government has emphasized the importance of the freedom to read and access information. While the United States implemented some censorship measures and produced propaganda, a dominant message during both wars was that practicing democracy—by reading whatever you want, sharing books, donating books to troops so they had access to a variety of reading materials, etc.—was an important way to protect democracy. During the last decade or so, we’ve seen a growing tendency to censor and limit what books are available. I think the history I write about is more important than ever.
Tell us about your writing process. Do you have a particular schedule you keep, or places you like to write? Do you start from an outline, and have all of your information organized before starting each chapter or section?
When I’m writing, I like to start in the morning and have several hours that will be uninterrupted so I can entirely focus on what I am doing. If I’m writing on my computer, all notifications are turned off and my phone will be in another room so I have absolutely no distractions. I have a small room in my house that’s loaded with bookshelves and has a large table—this is where I do most of my writing. I love to have my sources spread out in front of me, almost like having a roadmap of the day’s writing.
For each book I’ve written, I’ve had a different strategy for writing. Sometimes a chapter becomes clear to me before the entire book does, so I might write a chapter and then return to researching other parts of the book. For A Librarian’s War, after I organized all of the material I was planning to use, I did make a chapter-by-chapter plan for how to put it together. Usually, I avoid outlines because I find that the process of writing sharpens my understanding of a story and the final story may not have a strong resemblance to my original plan. I try to stay curious and flexible throughout the process.
What is next for you? Do you have any new books in the works, or book ideas you hope to work on?
I have an idea for my next book and have been doing a lot of preliminary research and reading to try to figure out the story’s contours. It will be an entirely different adventure for me, as I’m changing time periods and won’t be focusing on books in wartime. Each book I’ve written began because a story crossed my path and took root in my heart and mind. Once again, I have found a forgotten chapter of history and I feel compelled to preserve it. For now, I am happily gathering as many pieces as I can before attempting to figure out the structure.
Tell us about your library. What’s on your own shelves?
I have a bit of a book problem at home, as I have too many of them (if there is such a thing). Being a non-fiction author, I read a ton of non-fiction because I want to understand the period I’m writing about from as many different perspectives as I can. So my shelves are full of books from the World War II and World War I eras, along with my favorite works of fiction and any other books I have read and feel such a connection to that I can’t part with them. I also have a shelf devoted to signed books, from books talks I’ve attended by authors I admire.
What have you been reading lately, and what would you recommend to other readers?
In the hours following the release of CVE-2026-92392 for the project curl, site reliability workers
and systems administrators scrambled to desperately rebuild and patch all their systems to fix a use-after-free in HTTP/2 upload retries where cleanup turns leftover upload data into a new request and leaves a pointer to a freed Curl_easy handle (apparently freeing it was only a suggestion), allowing a malicious server to achieve remote code execution under specific conditions. This is due to the affected components being
written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes
these things just happen and there's nothing anyone can do to stop them," said programmer Ms. Odie Leuschke, echoing statements
expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have
occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can
we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to
write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities
regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
In Downgrades and Competition For Capital I discussed how AI's insatiable appetite for funds was raising interest rates not just for AI-related debt, but also for everyone else. The impact of these rate increases will take time to become evident, as borrowers roll over existing debt or raise new debt.
Below the fold I look at impacts that we don't have to wait for, because they are already happening on a large scale.
Now, Paul Krugman's AI Versus Everything Else presents a different aspect of the competiton for capital — it is "crowding out" investment in everything else:
Evidence of that crowding out is revealed by a recent article in the New York Times, which noted that the AI Boom is not being slowed by these soaring rates. Yet the spike in interest rates is being felt throughout the non-AI economy. In other words, all that money going into construction of datacenters and purchasing foreign produced semiconductors comes at the expense of investment in everything else.
The chart at the top of this post, which I borrowed from Steve Rattner, clearly makes this point: construction of datacenters appears to be diverting funds and resources from the construction of housing, office buildings, and factories — more or less everything that isn’t AI-related.
That said, AI-related construction — building the structures that house datacenters — is only a fraction of the AI spending boom. Most of the money is going for equipment and software. In fact, that’s one main reason AI spending is so resistant to high interest rates: Borrowing costs matter much less for assets like GPUs, which depreciate rapidly in any case, than they do for long-lived assets like buildings.
Rattner's chart only showed the effects within the construction industry, which absorbs only about 40% of the spending on AI data centers. Krugman's Chart 2 shows the effects on the other 60% of the spending:
I look at business investment in “tech” — information processing and software — measured as a percentage of GDP. The blue line in the graph below shows how much that measure of tech spending has changed since the beginning of 2023. I compare that with the corresponding change in all other investment. This chart clearly reveals dramatic crowding out by tech spending of all other investment:
Interestingly, given the analogies being drawn between the AI boom and the dot-com boom, Krugman shows that the effects of AI spending aren't just much bigger but also different:
Chart 3 shows the same calculation as Chart 2, but applied to the tech boom of the 1990s, and it doesn’t show any crowding out at all — in fact, non-tech investment rose:
Low interest rates imply that people will swap their claims on today’s productive potential for a song. In such cases, a debt-financed investment plan need not yield very much to pencil out. As rates rise, however, the expected return on an investment plan needed to justify the borrowing costs also goes up. Decent investments get squeezed out, then good investments, and then maybe really excellent investments that stand to deliver big economic benefits. So it goes.
We are experiencing this today, and you’ll never guess which part of the economy is responsible for the pronounced and sustained rise in desired borrowing.
But Avent argues that the "part of the economy" isn't AI but governments:
Anyway, gross government debt across advanced economies rose from 67 per cent of GDP in 2001 to 108 per cent today. So one could say that AI borrowing is crowding out other private investment, but one could also, and perhaps more reasonably, say that government borrowing is stymieing what might otherwise be a very broad-based, AI-driven, private sector investment boom.
If this turns out to be the better way to think about it, we’ll know soon enough. Firms in non-AI sectors may abandon efforts to claim more of the economy’s productive potential in the face of rising interest rates, but governments almost certainly won’t. For governments to give way, budget deficits need to fall. Accomplishing this means disappointing people who are expecting to receive claims from the government. It also means people have to give up more of their own claims to the government through higher taxes. In this fractious age, governments that do such things tend to be tossed out in favour of crazier governments which are even less able to do the needful.
Unfortunately, this likely means that we are on one of two paths. Either the AI complex ultimately smashes itself against the irresistible borrowing needs of the public sector and is forced to retrench, with potentially unpleasant consequences for growth, or rising interest rates cause one or another sovereign to slip into a market-destabilising crisis first.
The re-direction of investments toward AI would not be a problem if two conditions were satisfied. First, that the AI investments were more productive than those they crowded out. Skepticism on this question is warranted. There is a distinct lack of empirical evidence that AI is increasing productivity in the economy as a whole. Krugman expresses this skepticism:
Moreover, we shouldn’t assume that the private sector knows what it’s doing. There’s often a double standard when thinking about bad investments, in which public investments that go bad are treated as evidence that the government can’t do anything right, while private malinvestment is brushed off as no big deal.
Think about the $80 billion that Meta, formerly known as Facebook, spent on the “metaverse,” only to basically abandon the concept. A government program that spent $80 billion on a failed project would be the subject of endless Congressional hearings. Yet the waste from a failed private investment is equally real.
Second, that the enhanced productivity from the AI investments was at least as durable as that from those they crowded out. But the benefits of the 60% of the iAI investment that goes into IT hardware decay very rapidly, as I discussed in
Mind The GAAP Again and its predecessors.
So it’s clear that crowding out by the AI boom is happening on a very large scale. And unlike the risks of technological unemployment or a burst bubble, this isn’t a hypothetical risk. It’s happening right now.
Subsequently, Krugman also asked Will AI Crowd Out Jobs?, pointing out that a data center that might cost around $8B might only employ about 100 workers, mostly in low-skill jobs like security:
deploying $8 billion in capital, the cost of a data center, in a non-AI sector will, on average, support more than 10,000 jobs, while investing that money in a data center will support only around 100 jobs. Or to put it another way, money spent building data centers will, on average, create only around one percent as many jobs as investing the same sum in other industries would have created.
Van Nieuwerbergh estimates that through 2032 hyperscalers will spend more than $10 trillion on data centers — capital expenditure that, all the evidence suggests, will come at the expense of investment elsewhere in the economy. And because data centers employ so few people, a back-of-the-envelope estimate yields a number that made even my eyes bug out: the capital invested in data centers will employ roughly ten million fewer workers than that capital would have employed if invested elsewhere.
This doesn’t mean that unemployment will rise by 10 million. It means, instead, that reduced demand for labor will put downward pressure on wages and shift the balance of economic power, which has already been moving against labor, even more in capital’s favor. Even before the rise of AI, we were experiencing a rapid decline in the share of national income going to wages relative to profits, along with rising oligarchy. The anti-labor effects of AI will accelerate that trend.
For the AI companies' managements, this seems like a feature not a bug.
Rattner's and Krugman's charts show a rapid increase in the crowding out effect. But the worrying thing that Krugman doesn't mention is that the AI investments that have already had such a marked crowding out effect are small compared to those planned for the next few years.
This chart, from Big Tech will fund more than a third of its AI investments with debt in 2027, Goldman Sachs predicts by Jake Conley, reports Goldman Sachs' estimates of the scale of the investments by the hyperscalers, and how much they expect them to borrow:
hyperscalers have turned increasingly toward debt markets. In 2025, investment-grade-rated global bond issuance from the hyperscalers totaled $108 billion, per Goldman Sachs, or roughly 26% of capex. Through the first half of 2026, those companies have already issued $194 billion of IG-rated debt, with that figure expected to reach roughly $250 billion in debt issuance, or roughly 33% of capex.
In 2027, Goldman Sachs said, the share of debt-funded capex is likely to rise even further, with $400 billion in expected global IG bond issuance from the hyperscalers against an estimated $1.14 trillion in capex, or a roughly 35% share.
If these investment plans materialize, the crowding out effect will greatly accelerate. In the 18 months from January 2025 it reduced non-AI construction spending by $132B, or about six week's worth. The chart suggests that spendig will double from around $700B to around $1.4T in 2028. That might equate to crowding out 12 weeks worth of non-AI construction in 2028, or about a quarter. That would be a significant hit to the broader economy
the AI complex ultimately smashes itself against the irresistible borrowing needs of the public sector and is forced to retrench,
Zitron is skeptical that these investment plans will materialize:
In any case, AI data center debt is way more expensive now by virtue of the current state of treasuries, with costs compounded by the anxiety around them in general.
This means that any “virgin” projects — those that aren’t directly backstopped or co-signed by hyperscalers — are guaranteed to hit egregiously-high, 9% to 14% rates, which makes (as I discussed a few weeks ago in my two-part debt series) it near-impossible to make the already-questionable economics of running a data center work.
This means that any AI data center debt being raised right now is doing so under stricter credit conditions and doing so at unrealistic, unsustainable prices, if they’re going to be able to raise at all. This makes the $174 billion in debt that SB Energy needs to raise to fund its theoretical data center project with OpenAI — even when backstopped by NVIDIA — either horrendously expensive or impossible to complete, as does it mean that any future large, multi-gigawatt projects will add billions of dollars in interest payments to already-expensive debt.
OpenAI and Anthropic have made huge, non-cancelable "take-or-pay" commitments to the hyperscalers. These form the basis of a vast amount of the hyperscalers' backlog. The compute these commitments cover is for both training, a cost, and inference, which generates revenue. In Thoughts on The Curve and the Future of AI Thomas Wright reported an estimate:
that the labs spend 58% of their compute on R & D and 32% on servicing customers.
Zitron estimates that OpenAI and Anthropic represent 80% of the total demand for compute. Thus about half of the total demand for compute is OpenAI and Anthropic's training and attacking other companies. Halving their internal compute budget would allow for about a 50% increase in compute for customers with no need for additional investment. This is why they are using the claim that their products are too dangerous to release to call for a pause and regulation.
But even if the competitors could agree, and they weren't worried about the Chinese catching up, halving their training budget wouldn't work because the demand doesn't exist:
As a result of their $1.3 trillion in compute commitments, neither company can become “capital efficient” by cutting their training costs, because said training costs are the only means of further growth outside of massive price increases that are unlikely to grow their businesses.
To make matters worse, both are cutting prices to compete with each other, and per Ramp, said price cuts aren’t increasing usage:
Price cuts, flat usage, cutting R&D and revenue disappointments aren't a good look when you're planning a $2T IPO or a $30B funding round at a $1.4T valuation. Skepticism about the projected capex spending, and the increased crowding out is reasonable. But the effects on the broader market of the bubble deflating will be severe.
Panmure Liberum’s Joachim Klement said his base case is for that trade to disintegrate as soon as 2027, sending stocks sharply lower.
“My core conviction is that the AI bubble will either burst in 2027 or in 2028, so sometime in the next two years,” Klement said in an interview. Hyperscalers’ free cash flows are largely depleted, while the cost of debt is rising quickly and becoming prohibitive for these firms, he said.
Klement’s 2027 year-end target of 5,000 points for the S&P 500 implies 36% downside from current levels. ... Klement sees Europe’s Stoxx 600 falling to 430 points, more than 30% below current levels.
There are many more pessimistic forecasts. Hang on to your hats.
The open data activist and Open Knowledge Foundation board member joins us to discuss the less-told part of CKAN's 20-year history: how a small open-source project became the default for government open data, and the product, sales and community work that made it happen
I had the good fortune to attend an Archiving the Black Web
meeting last week. The main focus of the few days in St Louis was for fellows
to share stories of web archiving work that they had been involved with.
A few themes emerged for me while listening to these presentations. We
weren’t using Chatham House
Rules, but I didn’t ask anyone if I could write about their work
here, so these are broad strokes.
Fellows were often trying to contact website creators to negotiate
consent. This process seemed like it was quite time intensive, and
difficult, especially when contact needed to be established, and formal
process was being followed.
Some fellows were archiving within their existing trust networks, where
consent wasn’t such a big hurdle, and seemed to be established in a more
informal way.
Helping website creators understand what it meant for their websites to
be archived was challenging. Communicating why they would want to have a
web archive of their content was also challenging.
There was often a desire to create new websites that arranged the web
archives along with additional materials, such as oral histories, maps,
and other descriptive material.
There was also a need to limit access to a community of known users.
These communities aligned with existing trust networks. I inferred from
this that limiting access was one avenue to gaining consent.
Many of the fellows were using Browsertrix to create web archives,
and some were also using ArchiveWebPage to do more detailed
patch crawling. It seemed like there had also been some initial early
work using Archive-It.
When it came to integrating access to archived content, fellows were
often linking to the playback mechanism inside of Browsertrix itself.
Sometimes the website creators were given access to archived web
content, and offering to serve as a backup.
A few were thinking about the sustainability of the web archiving work
in terms of having their practice become more of a service. I had an
interesting side conversation about how a cooperative model could
potentially help, grow a web archiving practice as a form of mutual aid.
Given the interest in creating web sites for the archived web content,
and administering access to them, I mentioned to one of the organizers
that it would be super if the fellows could be introduced to how to
embed WACZ files in their own sites using the [ReplayWebPage] web
component. I think they were already planning on doing this, and that it
just hadn’t happened yet since the focus so far had been on creating the
web archives.
I think it’s super interesting how methods of access seems to sometimes
gate the types of material, and whose material, gets archived.
Understanding goals in terms of access, such as where the web archives
live, and who can see them, seems to shape the types of content that can
be archived? I think this has a counterpart in traditional archives
where the deed of gift often stipulates how and when records can be
accessed.
Donors introduce a dynamic element to the archival process, bringing
their own values, ideas, and interests. Examining donors through the
concept of agency, as developed in the social sciences, reveals that
they exercise significant influence on key archival functions, which in
our theoretical models are generally treated as the sole purview of
archival professionals. (Fisher, 2015)
I heard from at least one fellow that they were working with Omeka, and
that being able to embed WACZ files in Omeka could be desirable. Omeka S has a WebArchive
plugin that supports WACZ replay, but I noticed that Omeka Classic did not, and it is
still being actively used (and maintained).
Since I had some time to fill while waiting for my plane back I “used”
Claude to create a WebArchive plugin for
Omeka Classic. It was just 80 lines of PHP, and I haven’t written
PHP in over a decade, so I was grateful for the help. Once you’ve
installed the plugin you can upload WACZ files and replay them directly
in Omeka, where the data is stored, and you can use Omeka’s access
control settings to set who can see them. Everything old is new again…
I had the good fortune to attend an Archiving the Black Web
meeting last week. The main focus of the few days in St Louis was for fellows
to share stories of web archiving work that they had been involved with.
A few themes emerged for me while listening to these presentations. We
weren’t using Chatham House
Rules, but I didn’t ask anyone if I could write about their work
here, so these are broad strokes.
Fellows were often trying to contact website creators to negotiate
consent. This process seemed like it was quite time intensive, and
difficult, especially when contact needed to be established, and formal
process was being followed.
Some fellows were archiving within their existing trust networks, where
consent wasn’t such a big hurdle, and seemed to be established in a more
informal way.
Helping website creators understand what it meant for their websites to
be archived was challenging. Communicating why they would want to have a
web archive of their content was also non-intuitive.
There was often a desire to create new websites that arranged the web
archives along with additional materials, such as oral histories, maps,
and other descriptive material.
There was also a need to limit access to a community of known users.
These communities aligned with existing trust networks. I inferred from
this that limiting access was one avenue to gaining consent.
Many of the fellows were using Browsertrix to create web archives,
and some were also using ArchiveWebPage to do more detailed
patch crawling. It seemed like there had also been some initial early
work using Archive-It.
When it came to integrating access to archived content, fellows were
often linking to the playback mechanism inside of Browsertrix itself.
Sometimes the website creators were given access to archived web
content, while offering to serve as a backup.
A few were thinking about the sustainability of the web archiving work
in terms of having their practice become more of a service. I had an
interesting side conversation about how a cooperative model could
potentially help grow a web archiving practice as a form of mutual aid.
Given the interest in creating web sites for the archived web content,
and administering access to them, I mentioned to one of the organizers
that it would be super if the fellows could be introduced to how to
embed WACZ files in
their own sites using the ReplayWebPage web component. I learned
that they were already planning on doing this, and that it just hadn’t
happened yet since the focus so far had been on creating the web
archives. I still think Webrecorder’s innovation of allowing
archived web content to be replayed anywhere on the web is one of their
most transformative ideas.
I think it’s super interesting how methods of access seems to sometimes
gate the types of material, and whose material, gets archived.
Understanding goals in terms of access, such as where the web archives
live, and who can see them, seems to shape the types of content that can
be archived? I think this has a counterpart in traditional archives
where the deed of gift often stipulates how and when records can be
accessed.
Donors introduce a dynamic element to the archival process, bringing
their own values, ideas, and interests. Examining donors through the
concept of agency, as developed in the social sciences, reveals that
they exercise significant influence on key archival functions, which in
our theoretical models are generally treated as the sole purview of
archival professionals. (Fisher, 2015)
I heard from at least one fellow that they were working with Omeka, and
that being able to embed WACZ files in Omeka could be desirable. Omeka S has a WebArchive
plugin that supports WACZ replay, but I noticed that Omeka Classic did not, and it is
still being actively used (and maintained).
Since I had some time to fill while waiting for my plane back I “used”
Claude to create a WebArchive plugin for
Omeka Classic. It was just 80 lines of PHP, and I haven’t written
PHP in over a decade, so I was grateful for the help. Once you’ve
installed the plugin you can upload WACZ files and replay them directly
in Omeka, where the data is stored, and you can use Omeka’s access
control settings to set who can see them. Everything old is new again…
My recent search for a CULTIVATE postdoc (position now filled) got me thinking about what info I could share with recent and soon-to-be PhDs in information, LIS, and related areas. Here are some very generic notes related to the postdoc job search.
Some postdocs find funding by reaching out to senior researchers in the areas most related to their future work: This works best in well-funded fields (like medicine in this example ) or when a country or institution has well-developed opportunities.
Identifying fellowship funding can be an excellent way to write your own position.
I especially recommend the Marie Curie postdoc, an annual call (deadline each September) funded by the European Commission. While competitive, the funding is ample and the opportunity is prestigious.
Recent PhDs of any nationality can propose a 1-2 year postdoc with a mentor in Europe or the many “Horizon Europe Associated Countries”
Europeans (and long-time European residents) with PhDs can propose Global Fellowships: mentored for 1-2 years in a “Third Country” (such as the US) followed by a mandatory reintegration year in Europe/“Horizon Europe Associated Countries”
Fellowships tend to depend on:
the field of study
where the research will be carried out (country, institution, …)
aspects of the researcher (nationality, gender, country of residence, mobility, …)
Fellowships typically come from several places:
the hosting institution (most common when there’s a senior academic you’re visiting, or a library/archival collection that has a fund)
short-term grants from national organizations in the country where you’re based
short-term grants from national organizations in the country you’re visiting (some countries like Germany have lots of these)
groups like COST (I’ve had one from them, bottom-up funding for research groups)
professional societies
private foundations
I share a very out-of-date list I kept as a PhD student, focused on Europe (again, it is just indicative but may point to countries with robust funding for postdocs)
For librarians and library school faculty in lower and middle income countries, the Mortenson Center for International Library Programs may be a resource for training or for US-based professional development. For instance, its Associates Program is an annual program at the University of Illinois Urbana Champaign which has partial scholarships but requires substantial resources for living expenses, travel, and visa costs.
I have always been a bit obsessed with serving static web content that is portable, cheap, maintenance-free, and easy to distribute and mirror. My reasons are practical. I often work on non-profit projects in my spare time, with little or no budget. The web is also changing fast: hosting and server prices are rising, and crawlers are devouring resources, making it harder to keep dynamic applications running. There is a political side too. The governance of the web is fragile, and content can be blocked, taken down or lost for political reasons. Content that is easy to copy and mirror is much harder to make disappear.
The idea of serving digitized documents as zipped packages came to me in 2014, with deepzoom-osd-server. I later wrote about it in IIIF, tropy, canopy and SZI. Earlier this year I built mkiiif for my own needs, but one question stayed open: how do you serve this packaged content?
I knew Service Workers and HTTP range requests could be the answer, since Webrecorder uses the same pattern to replay WACZ files. But Service Workers have always been black magic to me, so I asked Claude to help me build a proof of concept. Here it is: github.com/atomotic/swiiifz, with a demo.
sw · iiif · z = Service Worker · IIIF · Zipped (pronounced "swiff-zee")
What it does
A IIIF Image API level 0 tree has one info.json and thousands of pre-cut tiles per image, plus a Presentation manifest. That normally means thousands of small files to upload, version and keep in sync.
swiiifz puts the whole tree inside one uncompressed zip on any static host. A Service Worker of about 170 lines turns it back into ordinary IIIF URLs in the browser. The viewer and IIIF clients don't know the zip exists. They request …/page-001/info.json or …/page-001/0,0,512,512/512,512/0/default.jpg like any other IIIF resource, and the worker answers by reading the file out of the zip with an HTTP Range request.
There is no server code, no build step on the host and no index file. You only need a host that supports Range requests.
How it works
Routing. A request for <id>/<path> is answered from the entry <id>/<path> inside zip/<id>.zip. Everything else goes to the network unchanged.
Opening a zip takes two requests. The worker reads the end of the file to find the zip's central directory, then fetches and parses it into an in-memory map of file names to offsets.
Reading a file takes one request. The worker fetches just that file's bytes with a Range request, so each tile costs about the same as a normal static tile.
Consistency. If the zip is replaced on the server, the worker notices (via ETag and If-Range), reopens it and retries. It never mixes bytes from two versions.
Compression. Tiles are JPEGs, so the zip is built without compression (zip -0). Compressed entries still work, because the worker can inflate them.
You can watch this in your browser's DevTools. Every tile appears twice in the Network tab: once as the virtual request answered by the Service Worker, and once as the 206 Partial Content request the worker sends to the zip.
The demo
The demo uses a 106-page Italian archival journal, ARCHIVI & COMPUTER (1995), converted with mkiiif:
Source PDF
13 MB
IIIF tree
1,910 files, 46 MB
Zip
1 file, 42 MB
Trade-offs
Pros
One file to upload, version, checksum, mirror or delete.
Replacing the zip is atomic.
No file-count limits, and git and CDNs stay happy.
Cons
The first view costs a few extra requests, plus the Service Worker's start-up time.
It only works in browsers with Service Workers. Harvesters, curl and server-side tools see a zip, not a IIIF endpoint.
It is still a proof of concept: no response caching, no ZIP64 (so a zip must stay under 4 GB), and the index is lost when the worker restarts.
The big limitation: interoperability
There is a major limitation. IIIF's interoperability is lost, because the manifest and tiles are not public, referenceable URLs. They exist only inside the browser, through the Service Worker.
Could this be solved? Perhaps the manifest should be served outside the zip. A new IIIF extension, something like "IIIF Level 0 packaged", could then tell viewers and clients how to load the content from the package.
What's next
The current demo is not truly portable. IIIF ids are absolute URLs, so the manifest is tied to the address it was built for. I have a branch where I am testing a rewrite so that the same zip works on any host.
Screenshot of the first alpha of what we initially called SmartFox and then Firefox Scholar, before we consulted 1) a lawyer, and 2) an Albanian dictionary for other options, June 2006
* * *
Today, software can be created more or less instantly with AI, for a large user base or for yourself, for any purpose or for no serious purpose at all. As Robin Sloan enticingly puts it, just “ask for what you want.” The genesis of Zotero, the “free, easy-to-use tool to help you collect, organize, annotate, cite, and share research,” used by over 20 million people in dozens of languages and countless disciplines, is the opposite of this instant gratification. It took five years, not five minutes, to produce a rough initial prototype, and years after that to refine and expand the application. If AI had existed in the early aughts, we could not have accelerated Zotero’s conception, because we did not know exactly what we wanted, and so could not have written coherent prompts for an LLM. Instead, it took a great deal of time and collaboration to develop a clear vision for what Zotero should be. But that slow formation led to software that was durable rather than ephemeral, with a strong foundation that could be built upon. As programming increasingly becomes a caffeinated bender with magical bots, this measured pace and emphasis on communal thinking likely holds an important lesson.
On this 20th anniversary of Zotero’s launch on October 5, 2006, I will do my best to recount Zotero’s pre-history, what it took to conceive, design, and build the 1.0 release. It’s a story about a bunch of historians who were knowledgeable about emerging technologies and knew how to code — as a secondary rather than primary skill — and who spent a lot of time, some of it wasted time, at a big table chatting. That conversation eventually coalesced into ideas about the future of scholarly research.
As the well-worn saying correctly asserts, failures are orphans while successes have many parents, and Zotero is no different. A proper accounting would require far more than this relatively modest post, and what follows is, unsurprisingly and necessarily, from my own perspective. My contributions to the success of Zotero date largely to the years before and immediately after the launch of the 1.0 beta, when I was an Assistant Professor of History and Director of Research at the Center for History and New Media at George Mason University. This institute is now the Roy Rosenzweig Center for History and New Media (RRCHNM), named after the visionary historian who founded it and really the entire field of digital history. Roy tragically passed away in 2007 when he was only 57, and his illness and death hang like a dark cloud over this narrative.
I succeeded Roy as the Director of RRCHNM, and from that point on, my day-to-day work on Zotero ebbed, as I handed off responsibilities to other, more capable people: Zotero’s talented lead developer Dan Stillman; Sean Takats, who has deftly overseen the massive growth of the project for much of the last two decades; many others in the orbit of RRCHNM and the non-profit Corporation for Digital Scholarship who handled outreach, support, and countless details; an international team of developers and volunteer contributors; and still more colleagues, only some of whom, regrettably, I’ve been able to highlight below. On this anniversary, you should visit Zotero’s Credits and Acknowledgments page to honor the full slate of productive collaborators. Sean Takats also has a great post on the continual improvement and influence of Zotero since its launch.
I’m proud to have been there at Zotero’s origin and early years, and to have helped bring it into existence. At the college graduation of one of my kids this past spring, an attendee who had heard that I had been involved with Zotero hugged me, unexpectedly and at length, in appreciation. No one has ever done that for my scholarly writing, or ever will.
Members of the Zotero team in late 2006. Roy Rosenzweig, months into chemotherapy, is kneeling at center; he had just gotten the Zotero license plate for his Prius to the great delight of all of us. Standing, L-R: Sean Takats, Trevor Owens, Josh Greenberg, me; Kneeling: Kari Kraus, Roy. Photo credit: Sharon Leon
* * *
Software development was not originally on the docket for the Center for History and New Media. With its now delightfully retro, turn-of-the-century name — new media rather than digital — it was largely in the business of developing sites for the web, which was just a few years old in 1994 at the center’s founding. I joined RRCHNM in January 2001, as a newly minted Ph.D., to work on a project on the history of science.
Alongside another new Ph.D. who had studied the history of technology, Jim Sparrow (now a history professor at the University of Chicago), we built a website called ECHO: Exploring and Collecting History Online, funded by the Alfred P. Sloan Foundation. “Collecting” was a new gerund for the center, and hinted at an expansion of our methods. Since science was growing exponentially but the number of historians of science was not growing at all, we thought we could use the web to help scientists self-document their work. For this, our site needed to be interactive, so we started to tinker with “tools” — small web apps — that could not only display history online, but also allow it to be uploaded, sorted, and archived.
Toward this end, I wrote several applications in PHP, then swiftly becoming the standard programming language for web applications, because it commingled well with HTML, the web’s lingua franca. One of these apps, Web Scrapbook, got a bit of traction, especially in classes, since it allowed students to capture images, links, and other resources from the web browser into a collection that could be shared with their classmates and instructor. Users clicked on a browser bookmark, which had a small bit of JavaScript embedded in it, to select the desired items and pass the data to the application. Web Scrapbook was not, shall we say, a rigorous piece of software — my initial version sent passwords, unencrypted, across the internet — and it paid only basic attention to metadata and other scholarly information that would be needed to assist in writing an article or book, and to form a proper footnote.
Be forewarned, ye who dare log into my web application c. 2002 (screenshot of the Web Scrapbook landing page)
Fortunately, at the same time, another colleague at the center, Elena Razlogova, who was the webmaster at RRCHNM while pursuing her Ph.D. (and is now a history professor at Concordia University), was working on a more scholarly application called Scribe, for taking notes and storing citations. Elena used a database application called FileMaker to build Scribe, which users downloaded and ran on their personal computers.
Elena Razlogova’s Scribe application, c. 2002
Scribe became popular among fellow historians as a good, free replacement for commercial software like EndNote. It ran locally on a Mac or PC, and had the organizational, search, metadata, and annotation capabilities that Zotero would later expand upon. Elena made several improvements to Scribe in the first few years of the century, as I worked in parallel on Web Scrapbook.
* * *
So by 2003, we knew how to write web apps and standalone apps, both helpful but both also naggingly insufficient. We were now starting most of our research on the web, and we thought that any robust research application of the future had to be connected to this environment where primary and secondary sources increasingly resided, either as metadata in library catalogs, or as full objects if they had been digitized. What we really needed was an application that had some aspects of Scribe and some aspects of Web Scrapbook: a full-fledged research tool that could exist independently of the web, but be fully cognizant of what was going on in the web browser, aware as the researcher surfed across digital collections and library catalogs.
On November 29, 2003, Roy asked Elena via email what she would like to do for a next version of Scribe. Roy, Tom Scheinfeldt, another recent history Ph.D. who had joined us to work on the September 11 Digital Archive (now a professor of digital humanities at the University of Connecticut), and I were working on a grant proposal for a new stage of ECHO, and we thought we would include some improved software tools as part of that. Elena wrote back on December 1, 2003:
Roy-- Here’s what I'd like to do:
setup to connect to online bibliographic databases more citation styles (at least APA and legal) setup to add custom citation styles setup to add more fields and types of references user/password setup to share and co-edit online bibliographies/notes setup to work locally then publish all or parts of bibliographies/notes online (like ical) foreign language support better manual online discussion list (using the software marty had installed)
best, elena
This was a good list. Tom wrote back that “it would be nice to update and ‘webify’ Scribe.” A good word: webify. Elena, Tom, Roy, and I agreed on this essential next step for Scribe: we needed to find a way to put it into the web browser, like Web Scrapbook, while retaining Scribe’s provenance as a detail-oriented research tool. As I later summarized our holy grail in a journal article:
We wanted the best of both worlds: the best parts of standalone applications and Web applications. We envisioned a tool that lived in the browser and that was very smart about what was going on in the browser, including the recognition of scholarly metadata and objects, and that could interact with elements both on the desktop, such as word processors or other programs, and via standards, services, and communication protocols to other tools and resources across the Web.
But how to merge these disparate worlds? That was not at all clear in 2003.
* * *
At RRCHNM, we had a large whiteboard with vague ideas, some of them serious and some of them silly, visible to everyone, just in case it gave someone a spark. Over group lunches, the faculty and staff at the center would often talk about new tech we had heard about, and muse about whether we could apply that tech to historical research. Our collective was growing quickly. In the summer of 2004, we were joined by, among others, Josh Greenberg, who had a Ph.D. in science and technology studies (and is now a program director at the Alfred P. Sloan Foundation); Sharon Leon, who had a Ph.D. in American history (and is now the Co-CEO at Digital Scholar); and Simon Kornblith, a young teenager (and son of a historian and friend of Roy’s) who was truly gifted at software development and many other things. (Simon would go on to get a Ph.D. in Brain and Cognitive Sciences at MIT, write important AI papers with future Nobelist Geoffrey Hinton, and work at Anthropic.)
One project that caught our eye was a spinoff of the open-source Mozilla web browser that came to be known as Firefox. That summer Firefox was in beta, and it would be officially released later that year. The new browser had much to recommend it. The original Mozilla browser, descended from Netscape and the early days of the web, had, over time, become slow and bloated. Firefox, on the other hand, was fast and lightweight. But much more exciting was how Firefox embraced and foregrounded XUL, which sounds like an alien god from a 1950s sci-fi movie, but stands for XML User Interface Language. It dawned on us that with XUL, you could — sweet mercy — customize and extend the web browser in any way you wanted.
The day Firefox was released to the world on November 9, 2004, Josh and I — always the earliest of adopters — downloaded it and started to tinker with its possibilities. That night I wrote an email to Josh, Elena, Tom, and Roy with the subject line “Firefox + XUL + AWS = OpenScribe/Online Scribe?” The AWS I was talking about was not the future cloud computing service from Amazon, but an API Amazon provided to retrieve information about books. I thought we could use a combination of a Firefox extension written in XUL and services like Amazon’s to “automatically load citation info into the fields.” Josh wrote back just before midnight:
Please pardon my language when I say "Holy crap, that's cool!"
The possibilities are astonishing - it looks like there are XPCOM wrappers for mySQL that would allow us to access a database (either remote or local), entirely replicating Scribe using a relational database (or, offering a custom "Scrapbook Browser.") Alternatively, we could chuck the database and use a Firefox interface to navigate Scribe XML data directly. Regardless, immediate cross-platform compatibility.
Either way, we could use a combination of Amazon and Proquest/ISI Citation Index to make it much easier to create bibliographic objects, and at the least create a Mac version that would use either Cocoa or more basic scripting to allow citation from within a word processor...
Neat-o!
With an open-source database instead of FileMaker to store bibliographic data and notes, and with a Firefox extension of the proper composition, all of the ingredients were within reach to create the research tool of our dreams.
That winter, with Roy as the principal investigator and Josh and me as co-directors, we applied for a grant from the Institute of Museum and Library Services. The abstract from that grant, which we titled “SmartFox: the Scholar’s Browser for Digital Collections,” displayed our new level of clarity, although we still envisioned what we wanted to make as a “set of tools” — such as the item capture tool from Web Scrapbook and Scribe's note-taking interface — rather than one app:
The web browser has become the primary means for accessing information, documents, and artifacts from libraries and museums around the country and the world, thanks in large part to the tremendous commitment these institutions have made to bringing their collections online (as either simple citations or complete text and images). Unfortunately for scholars, while tens of millions of dollars have been spent to create digital resources, far less funding and effort has been allocated for the development of tools to facilitate the use of these resources. The browser remains merely a passive window allowing one to view, but not easily collect, annotate, or manipulate these objects. Moreover, from the user’s perspective individual library and museum collections remain just that—separate websites with distinct designs and different ways of displaying their information, making traditional scholarly practices of bringing together and studying objects of interest from across these collections unnecessarily difficult.
SmartFox, a set of tools incorporated into popular, open, and free web software, will address these major problems by creating a web browser that is “smarter” in two key ways. First, one tool will enable the browser to intelligently sense when its user is viewing a digital library or museum object; this will allow the browser to capture information from the page automatically, such as the creator, title, date of creation, and copyright information. Second, another tool will store and organize this information, as well as full copies of items and web pages (not just their citation information) if so desired by the user and permitted by the institution’s site, allowing the user to sort, annotate, search, and manipulate these individualized collections created for scholarly purposes. Critically, all of this will occur within the web browser itself, not in a separate, standalone application; the web browser will be used not just to discover information, but also to collect, organize, and analyze scholarly materials.
(Zotero did indeed begin its life with a name, SmartFox, that was a riff on Firefox, and our potential trademark violation somehow got worse when we rebranded the app as Firefox Scholar in September 2005. More on that later.)
Even before IMLS funding had come through, Simon had started working with David Norton, another software developer, on a prototype of SmartFox. By July 2005, they had produced a very early “0.0.1” version, with the metadata panel on top, the notes field below, and the folders on the left; all of these would later be combined in a much better unified drawer.
SmartFox version 0.0.1 screenshot, July 27, 2005
Simon, who was paying attention not only to public releases of Firefox but also to the core code development itself, pointed out that summer that Mozilla planned to include “mozStorage” in Firefox, which could act as an interface to a database. (Firefox eventually included SQLite as its native, open-source database.)
Also in the productive summer of 2005, we hired Dan Stillman, a friend of an RRCHNM staff member, to help us with the center’s burgeoning set of servers and increasingly complex digital platforms. Dan fixed so many accumulated issues so well that by January 2006, Josh and I had asked him to join Simon and David on the dev team. (To this day, Dan is the lead developer of Zotero; what a run.) We also added Sean Takats, who had both a doctorate in French history and extensive technical experience, and became Zotero’s longstanding director in addition to his academic career in history and digital humanities; Kari Kraus, our technology evangelist, who became a professor in the College of Information Studies and the Department of English at the University of Maryland; and, later that year, Trevor Owens as our outreach coordinator. (Trevor is now the Chief Research Officer of the American Institute of Physics.)
Members of the Zotero team in late 2006. Standing, L-R: Simon Kornblith, Sean Takats, Trevor Owens, Dan Stillman, Matthew Eaton; Kneeling: me, Roy Rosenzweig, Josh Greenberg
With this team in place, development accelerated. In roughly six months, the developers produced a far more coherent design and much-improved functionality, as you can see in the June 2006 screenshot at the top of this piece. With summer always being a busy time at the center, as teaching responsibilities paused for some members and everyone focused on grant projects, the code base iterated rapidly and everything aligned for a fall 2006 release of the 1.0 version.
There was just the pesky issue of the name of the thing. I’m afraid we probably spent as much time in the summer of 2006 rethinking the name as we did fine-tuning the design. We had, by then, long abandoned “SmartFox,” and were calling the software “Firefox Scholar,” but we heard from a lawyer that any branding with “Firefox” in it would be problematic. We were also wondering whether this software was just for fancy “scholars” — it seemed to have broad applicability. We kicked around countless replacements, many of which are howlers in retrospect. Withholding who proposed each of these alternatives to protect everyone involved from embarrassment, we considered iScholar (these were the days of iPods), Scholr (these were the days of Flickr), Citopia, FreeCite, CiteHound, DynoCite, FireScribe, FireHoard, MetaFox, ClioFox, and every pun you could think of involving fires, foxes, and footnotes. The domain name also had to be available; it’s amazing how many of the bad names we came up with had somehow already been registered by other fools.
Late in the summer of 2006, as time was running out, we thought we would move beyond English, taking a cue from other new projects at the time, like Ubuntu, a Linux distribution that used a word borrowed from a family of African languages, and Wikipedia, whose name derived from wiki, the Hawaiian word for quick. Our colleague Mills Kelly, a historian of Eastern Europe, made the suggestion to consult an Albanian dictionary, which we thought was a good idea as it was unlikely that any other Americans had ever done so, for any reason, much less to find available domain names. We quickly selected the Albanian word for “to learn well or master,” zotero, which was memorable, sounded like it could be pronounced easily in many languages, and was still available with any domain ending we wanted (we registered .org/.com/.net).
On August 29, 2006, I sent Dan Stillman some Photoshop files with a Zotero logotype I created, set in Eurostile. I can neither confirm nor deny that this logotype may or may not have been inspired by Zpizza, a restaurant near where I was living in Silver Spring, Maryland, which also had a red Z at the front but used Futura Bold, a travesty that led to the restaurant closing soon thereafter. The Zotero font has changed slightly, but the logotype has survived to this day.
The first logotype for Zotero, August 29, 2006
* * *
Twenty years ago, I posted a one-sentence notification of the initial beta release of Zotero, asking people, a bit manically, to “try it out!” Try it out they did. We had 60,000 users of this new open-source research assistant within a month. The numbers escalated rapidly from there, and additional funding soon followed from the Mellon Foundation and the Alfred P. Sloan Foundation that allowed the app to sync collections across research teams and, eventually, to become independent of Firefox and work with other web browsers.
The library I now lead, like most research libraries, has workshops that teach students and faculty how to use Zotero. It’s part of the fabric of academia, and is widely used outside of the academy as well. Zotero is great, solid software. It continues to evolve in concert with the needs of its enormous user base. Its value, slow to emerge from the collective thinking of a friendly cohort of historians, has been incredibly durable. May it last another 20 years, or more.
Like the desktop version, Read Aloud on iOS reads PDFs, EPUBs, and webpage snapshots to you in high-quality, natural-sounding voices. It automatically skips in-text citations, headers and footers, and tables, and you can quickly skip ahead by sentence or paragraph or start reading from a given spot. While you’re listening, an Annotate Sentence button highlights or underlines the current sentence with a single tap, so you can make note of important points without selecting text by hand.
Read Aloud on iOS keeps reading if you lock your device or switch to another app, and you can use standard iOS methods (lock-screen controls, buttons on AirPods, Siri) to pause and resume or skip ahead by sentence. When you stop reading, Read Aloud remembers your exact position, so you can pick up later right where you left off.
The iOS version features the same voices as the desktop app. From the Zotero 9 announcement:
Read Aloud requires an internet connection and a Zotero account for high-quality voices, which we’re calling Zotero Voices. If you’d like to use Read Aloud offline, you can still use the text-to-speech voices available on your system, but the quality will be significantly degraded.
We offer two tiers of Zotero Voices: Standard and Premium. Standard voices are generated on Zotero servers, with unlimited minutes for Zotero Storage subscribers and 2 hours/month for free accounts. Premium voices are the highest-quality voices available, processed by external text-to-speech providers — they make fewer mistakes, sound more natural, support many more languages, and can handle multilingual text. Individual Zotero Storage subscribers will receive up to 2 hours of Premium usage (varying by voice) each month, and free and institutional accounts will also receive a small quota in order to try the voices out.
For now, you can request additional Premium minutes for free if you run out. We’ll provide more details on monthly allocations and options for adding additional minutes at a later date.
ts_server is a web server proposing a REST API to large language models.
They can be used for example for text completion, question answering,
classification, chat, translation, image generation, audio
transcription, speech synthesis, …
A group of French scientists sees OpenAI unjustly claiming it has solved
the mathematical Navier-Stokes problem and sounds the alarm. A handful
of tech moguls are establishing a private monopoly over cognition and
knowledge.
Software maintainers exercise considerable agency over the material in
their immediate care: code. However, in doing so they also find
themselves having to articulate dynamic, interdependent and hybrid
networks of relations which they are intimately entangled with, and
whose durability depends on the success of their ongoing, indeterminate
reconfiguration. Both the programmers and the software they maintain
must continuously navigate risks of breakage, burnout, bugs or falling
into obsolescence.
Eclectic, human-created radio programming is made possible not just
because of student devotion, but also because many campus stations house
irreplaceable collections of physical media. For example, Cleveland was
once considered the city with the second-largest Hungarian population
after Budapest, and WCSB was home to thousands of Hungarian records, a
culturally significant collection regularly played during the station’s
long-running “Hungarian Hour.”
How much LLMs memorize their training data is a central question, both
for copyright lawsuits and for AI research more broadly. But most
technical work that measures memorization doesn’t address the questions
that matter most for copyright. It tends to measure memorization over
random samples of all the training data to estimate average memorization
rates, yet copyright is about specific works. So the copyright-relevant
question is: how much of a particular book has a particular LLM
memorized?
Perhaps that’s what AI is exposing. We’ve spent years loading an
extraordinary number of responsibilities onto the humble code review:
quality gate, security check, architecture review, mentoring mechanism,
knowledge-sharing system, ownership model.
It worked, sort of, while humans could only produce code so quickly.
That constraint is disappearing. So perhaps the question isn’t how we
get the code reviewed faster. Perhaps it’s why we’re waiting until code
review to have all the important conversations in the first place.
Understand Anything is a Claude Code Plugin that analyzes your project
with a multi-agent pipeline, builds a knowledge graph of every file,
function, class, and dependency, then gives you an interactive dashboard
to explore it all visually. Stop reading code blind. Start seeing the
big picture.
This repository supports the article “Standardising Access to Digital
Platform Data: A Taxonomy for Social Research.” It provides guidelines,
platform-specific schemas, entity mappings aligned with the proposed
taxonomy, and example structures to help researchers access and use Data
Download Packages
While mass marketing is trying to instil a Fear of Missing Out hysteria,
the most rational and pragmatic approach is to strongly reject all
AI-generated contributions to your projects. For now.
Someday, we might realise that LLMs are doing good in the world, that
they are evolving toward ethical, reliable, sustainable solutions, and
that people who use them are happier (try to read that sentence again
without rolling your eyes). If that really happens, you could always
change your AI policy. It will cost you nothing.
God’s Waiting Room: a round-about selection of the cinematic, library
music, rock ‘n’ roll, psych, experimental, unclassifiable and
independent. From the mind of DJ, producer and creator of original
soundtracks, David Holmes.
The nuraghe, or nurhag, is the main type of ancient megalithic edifice
found in Sardinia, Italy, developed during the Nuragic Age between 1900
and 730 BC. Today it has come to be the symbol of Sardinia and its
distinctive culture known as the Nuragic civilization. More than 7,000
nuraghes have been found, though archeologists believe that originally
there were more than 10,000.
Automated feed with new posts from live music & unusual recording
repositories to accompany (LiveMusicArchiveUpdates?),
instigated by (bourgwick?), operated
via MastoFeed.
TL;DR: In this post, we show how an off-the-shelf LLM can make typed
decisions in a single forward pass. This approach makes it possible to
turn an LLM into a Jev-like decision model.
We evaluate the approach using GLM-5.3-Flash running on Privatemode.
Using a benchmark constructed from public data sets, we show that this
setup delivers results that are on par with TypeSafe’s Jev in terms of
decision accuracy/correctness and speed.
As a bonus, the setup with GLM-5.3-Flash enables typed decisions on
images, which is not possible with Jev
This is a long-form written guide for the fight AAC Heavyweight M1 (Savage) in the game Final Fantasy 14, where the challenger and their seven friends fight the Elezen succubus Vamp Fatale. You may call this the XeBin guide in party finder listings.
If you want to experience this fight, don't read this guide. This guide is based off of the Toxic Friends raidplan, the Hector Lectures strategies used by North America Party Finder raiding culture, and insights I have gained from running this fight in a static group. A lot of this advice is geared towards healers as I typically run as a healer (AST/SGE).
This fight assumes minimal raiding experience beyond what is required to beat the normal raids. All vocabulary and common mechanics terms will be explained inline.
If you or your static is looking for a healer that wants to prog through
ultimates, please get in touch. I've been trying to avoid using my clout to
help me through FFXIV raiding, but I've had such dogshit luck in Party Finder
(285 pulls to clear M10S) that I'll take whatever I can get at this point.
These strategies assume that every member of your group has an item level (IL) average of at least 780 (via Augmented crafted gear and an Enuo weapon). If you are under IL 780, you may need to mitigate (mit) more heavily than if you have higher item levels. Play it by ear and don't be afraid to break out Eukrasian Prognosis or Concitation if you are not sure what mits are required for each hit. This strategy guide will also tell you to drastically overmit beyond what is required in practice. This is intentional so that you know what fits where.
The theme of this fight is that every mechanic is blanketed in a coating of raidwide attacks (ones that hit everyone regardless of distance from the boss or other players). Every mechanic in this fight looks like a sandwich of the following:
Light mit raidwide
Mechanic
Another light mit raidwide or a heavy mit one for flavour
Raidwide attacks
In this fight, raidwide attacks come in three flavours:
Ignorable: Don't mit this attack, throw an AoE heal or two afterwards. Maybe fire off some shields so you don't have to care as much.
Light mit: This is enough damage to be annoying, but not enough to actually risk killing people. This is where your 10% mits, Addle, Feint, and Reprisal make a difference.
Heavy mit: If you are not careful, this is going to kill people. This is when you break out big guns like Spreadlo, Holos, Temperance, or Sun Sign. Remember: if people live with more than 1 HP, they didn't die.
Stack attacks
Stack attacks have two kinds to keep in mind:
Light mit: Mostly seen later in the fight when splitting into light parties for Undead Deathmatch or for the stack mechanic in Hell in a Cell. Use your judgement, make sure people don't die and get healed.
Akh Morn: Technically this fight doesn't have any hard casts of Akh Morn, but there are many multi-hit stack attacks that are functionally an Akh Morn. This is where you break out the big guns like Liturgy of the Bell, Panhaima, Macrocosmos, or Seraphism-boosted Spreadlo.
Tankbusters
This fight has particularly meaty tankbusters. These tankbusters really do need to be mitted for consistency. Every healer (save Scholar) has a 10% single target mit that makes a huge difference and can be the reason tanks live or die. A well-placed Haima can also make a huge difference during Hell in a Cell as every tank hit in that mechanic is a tankbuster hit.
Mit sheeting
Healer mits:
Mit class
White Mage
Astrologian
Sage
Scholar
Ignorable
Maybe Swiftcast -> Medica 3 if you feel generous
AoE heal over time
Swiftcast -> Prognosis after
Selene heals
Light mit
Plenary Indulgence
Heal over time
Kerachole
Sacred Soil / Fey Illumination
Heavy mit
Temperance -> Divine Caress
Neutral Sect -> Sun Sign -> Shields
Holos -> Physis
Seraphism -> Spreadlo
Note: healers WILL run out of mits, especially you have a Sage and they die at any point. Do not feel afraid to burn heavy mits in light mit mechanics in order to prog.
DPS / Tank mits:
Role
Light mit
Heavy mit
Akh Morn
Melee DPS
Ignore
Feint (trade with other melee)
Feint + Job gimmick
Reaper
Ignore
Feint
Arcane Crest
Tank
Ignore
Rampart or Reprisal
Reprisal + 40% mit
Caster
Ignore
Addle
Addle + Job gimmick
Pictomancer
Ignore
Addle
Addle + Tempera Coat -> Tempera Grassa
Ranged DPS
Ignore
Ignore
Job gimmick
Dancer
Ignore
Ignore
Shield Samba
Bard
Ignore
Troubadour
Troubadour if up
Fight-specific mechanics
This fight has a few mechanics that get repeated multiple times across the encounter. For brevity's sake, each of them will be explained here with backreferences so you can refer to them later.
Satisfied
Any time anyone fails a mechanic they get a 10% Damage Down debuff and Vamp Fatale gets a stack of Satisfied. If too many players get Damage Downs, you will not be able to clear the fight.
Every 4 stacks of Satisfied Vamp Fatale's raidwides hurt more, AoEs will get bigger, and her Half Moon cleaves get wider. If you get more than 16 stacks of Satisfied, you're probably going to get obliterated by the raidwides. Try to get as few stacks of Satisfied as possible.
Every so often Vamp Fatale will cast Crowd Kill which will make her gain 4 stacks of Satisfied regardless of player actions.
Half Moon
Half Moon is similar to Omega's Starboard/Larboard Wave Cannon. When she casts Half Moon, she will cleave half the arena and then immediately cleave the other half. This is a core verb in the fight and will often be woven into other mechanics. At eight (8) stacks of satisfied, Half Moon gets a lot larger and cleaves from her hitbox out instead of from halfway out.
These diagrams are based on some kinda janky simulation logic I've been
cooking up. They are simultaneously more and less strict than the actual game
logic and mostly intended to illustrate mechanics patterns rather than being
guides on how you should literally move.
For consistency: always assume that she has eight stacks of Satisfied and go way farther than you need to in order to avoid the cleaves.
You can tell which direction the cleave is going based on the position of the ball of aether Vamp Fatale is holding. If you're on the opposite side as the ball of Aether, you're good. If not, you're giving her a stack of Satisfied. If you want to practice this mechanic in particular, the normal version of this fight uses it pretty heavily.
Hardcore
Hardcore is a pair of meaty AoE tankbuster hits that target the top two entries in the aggro list. As Vamp Fatale gets more stacks of Satisfied, the tankbusters will hurt a lot more and the AoEs get bigger until it's unsolvable.
If she has more than 12 stacks of Satisfied, either this attack will kill you or one of the next ones will. If you find yourself in this situation, consider alternate strategies.
Vamp Stomp
Vamp Stomp is one of the easiest mechanics of the fight that will also cause you to wipe the most. Before I explain the mechanic, here's a high-level diagram that explains all the parts:
When you transition into this mechanic, you want to have everyone in modified clock spots with ranged on intercardinals and melee on cardinals, kinda like this:
When this mechanic starts, everyone gets a Curse of the Bombpyre debuff. Vamp Fatale will shoot out an expanding shockwave ring. When the ring hits you, it cleanses your debuff and explodes a point-blank AoE that hits you for about half your health and gives you a very short Magic Vulnerability (magic vuln) debuff.
If you get hit by any AoEs while you have a magic vuln, you will instantly die. This pierces tank invulnerability skills. Try to not get venn diagrammed.
Vamp Fatale spawns three rings of bats:
An inner ring with two bats
A middle ring with three bats
An outer ring with five bats
When bats are hit by the ring, they also explode into a point-blank AoE that hits for about half your health and gives a magic vuln.
The trick for this mechanic is to make sure everyone cleanses their debuffs without killing everyone else in the process. The best order to resolve this is tank/melee -> ranged -> healer. Keep in mind that the bats are randomly placed and will randomly go either clockwise or counter-clockwise. This will take some practice for you to know how to read your party to know when it is safe to cleanse your debuff.
Here's a few tricks I've learned to make things somewhat consistent:
If the inner ring of bats is positioned on the tanks, have melee go first.
If the inner ring of bats is positioned on the melees, have tanks go first.
If it's diagonally offset, agree who should go first with your group.
If anyone gets a magic vuln from the bats or dies from getting venn diagrammed, Vamp Fatale will gain a stack of Satisfied.
Practice this mechanic a bit until you can do it without thinking.
Basic Raidwides
There are a few other raidwide casts in this fight that are mostly the same light mit raidwide, but a few of the variants have some implications you should keep in mind:
Killer Voice: The basic light mit raidwide. Sometimes you can get away with ignoring this and dumping out heals after the hit. If you and your group are close to best in slot, you can get away with ignoring it for the most part.
Crowd Kill: Another light mit raidwide but it makes Vamp Fatale gain four stacks of Satisfied. This will be cast at key points in the fight and gradually increases the potency of all her attacks as the fight goes on. The last time she casts this, she will hard enrage and just kill the party.
Sadistic Screech: Another light mit raidwide that edits the arena. If the arena is a 4x4 grid, it turns into a 4x2 grid or vice versa.
Finale Fatale: Yet another light mit raidwide that makes the arena into a circle instead of a square. If you walk out of the circle, you get killed. Try not to do that.
Brutal Rain
Functionally, this is an Akh Morn but empowered by her stacks of Satisfied. The spread for Satisfied stacks vs Brutal Rain hits roughly translates to this:
Stacks of Satisfied
Brutal Rain hits
0-3
3
4-7
4
8-11
5
12+
6
When it gets to 5+ hits, you need to burn as many mits as possible because if you do not you WILL die. Some of my prog parties had us burn healer limit break 3 just to keep people alive during the worst of this. If you die right before this mechanic, cheese your rez immune to help soak the damage so your allies can live.
Suggestions for getting started
The normal version of this fight is very faithful to the savage version. It's actually a pretty brilliantly designed tutorial and can help you learn the savage mechanics in a low-risk environment. Most of the mechanics are basically the same as their savage counterparts with the exception of the removal of Hell in a Cell, Undead Deathmatch, the fun parts of Aetherletting, and most of the mechanics around stacks of Satisfied.
Opener
Vamp Fatale will start the fight out with a cast of Killer Voice. One healer should throw out a raidwide mit and top everyone up after the hit. In your first pull this will probably kill at least one of you. This is normal, and is a sign that you should over-mit until you get comfortable with the amount of damage you need to account for.
She will then cast Hardcore with the smallest tank AoEs in the fight. Everyone can maintain damage uptime during this mechanic. Just don't venn diagram eachother and you'll be fine.
After the Vamp Stomp, she casts Brutal Rain. Throw out an Akh Morn mit and make sure everyone is topped up for the raidwide that leads you into the next act: fighting the Coffinmaker.
Coffinmaker
Vamp Fatale changes the arena into a 4x2 grid and spawns a Coffinmaker enemy at the top of the grid. If you touch the area covered by the Coffinmaker, you instantly die. If you get hit by any Coffinmaker attacks, you get a Damage Down and Vamp Fatale gains a stack of Satisfied. Vamp Fatale doesn't throw out any auto-attacks or fire off any raidwides during this section of the fight, so healers can throw out a heal over time and focus on mechanics.
The Coffinmaker will have two saws glow blue and cast Coffinfiller. Those will become line AoEs when the cast finishes.. Vamp Fatale will also cast a Half Moon, meaning that at any point only 1/8th of the arena is safe ground.
In order to solve this:
Notice if the orb is on the north or south side of the arena. If it is on the north side, call out that south is safe and vice versa.
After the line AoEs fire, go diagonally opposite of where you were just standing because Vamp Fatale will fire off the opposite Half Moon as the Coffinmaker fires off another Coffinfiller with the saws that were just safe.
After the Coffinmaker casts Coffinfiller twice, it will cast Dead Wake, which will make it advance one square down the arena. If you don't kill the Coffinmaker before the end of the arena, it will enrage by destroying the floor, pushing everyone off, and wiping the party.
Suggested callouts:
Forward: closest to the Coffinmaker
Back: away from the Coffinmaker
One step forward: move to the next row to avoid Dead Wake crushing you
This is the only major DPS check in this fight and the first few times will murder you. You can also tolerate at least two Damage Downs before it becomes a problem.
When the Coffinmaker gets to the end of the arena, it spawns line AoEs as normal but Vamp Fatale's Half Moons go horizontal instead of vertical.
Suggested callouts:
Far left: far left column
Close left: close left column
Close right: close right column
Far right: far right column
If you kill the Coffinmaker before Vamp Fatale's last Half Moon cast, continue respecting the Half Moons. Once the Coffinmaker is dead, Vamp Fatale becomes targetable and casts Sadistic Screech to return the arena to normal and whallop everyone with damage for good measure.
Aetherletting
Vamp Fatale will cast Crowd Kill and gain four stacks of Satisfied no matter what. She will then spawn a bunch of randomly placed AoEs over the arena. Avoid them. She'll cast another Finale Fatale raidwide to turn the arena into a circle and begins casting Aetherletting.
Aetherletting has two main parts to it:
Vamp Fatale will spawn conal AoEs either clockwise or counterclockwise around the centre of the arena that inflicts massive damage, a damage down, a magic vuln, and gives her a stack of Satisfied.
Two players will be targeted with a hunt marker that leaves a cardinal/intercardinal bomb that gives you a damage down and her a stack of Satisfied if you get hit by its line AoEs.
The most common strategy for this is a technique called Static Aetherletting. With Static Aetherletting your goal is to place all the hunts so that they explode in a way that avoids the line AoEs crossing the centre of the arena. Everyone stacks dead centre under the boss. I cannot stress enough, you must be DIRECTLY UNDER THE BOSS.
If you place your markers on the safe sides of the pie slices, they will all explode and you take no damage.
Once she is done with that she will cast Hardcore. The AoEs will be bigger than last time. If you aren't sure how big they are going to be, sacrifice your uptime and run away to be safe. Use a gap closer to get back to her when you're done.
She will follow up with another Vamp Stomp into a Half Moon. Assume her Half Moon AoEs are much bigger than they are to be safe.
If anyone died during the Vamp Stomp, fire off the rez right at the end of the Half Moon because she's about to cast a Brutal Rain and the more bodies you have in the stack the less damage everyone takes.
She will cast Insatiable Thirst, then Sadistic Screech, and that brings you into the adds phase.
Adds phase
In the adds phase, the arena gets shrunk to the smaller arena size from Coffinmaker but with four additional hazards added in: buzzsaws. Buzzsaws are the most terrifying part of this mechanic. If you get a bleed from the saw, you will take about 40% damage per tick. If your healers don't realize it, you will die. Make sure to avoid the saws and let your healers know instantly if you get hit by them.
Either way in this phase she spawns three additional targets: two Fatal Flails that come out after tanks soak tank towers and a Deadly Doornail. Each flail has an enrage cast of Barbed Burst. If you don't kill the flail before it finishes its cast, it will either kill people or chunk out massive parts of people's HP and give a Damage Down debuff. If both flails pop, just throw yourself off the side because you're about to die anyways.
The Deadly Doornail will spawn a purple puddle that constantly gets bigger. This puddle gives you a small damage over time debuff and is just considerably annoying.
The main tank should take the towers on the north side of the arena and the off tank should take the towers on the south side of the arena. Tanks should pop their two minute burst on at least one of the flails.
Your kill order:
Tanks: burst the hell out of your flail as much as you can as fast as you can.
Melee: use your standard rotation on the nail until the purple circle gets bigger than max melee range, then go help your light party tank with their flail.
Ranged/Caster: focus the nail, then focus the flails.
Healers: Vamp Fatale is gonna blanket this in raidwides, so mit the raidwides.
Until you get confident with this you will need to have a caster Limit Break to kill the last flail and nail pair.
Once you kill the third set of flails/nails, she casts Sadistic Screech to restore the arena.
Hell in a Cell
This is your pot window, use your gemdraughts right now and burst the hell out of the boss with your party synergy buffs. If you didn't spend it in adds phase, pop your melee Limit Break 3.
Call out that people should align their camera to north such that the A waymark is behind the boss.
Vamp Fatale will now cast Hell in a Cell. This mechanic will require you to split up into light parties and have each light party take towers in clockwise priority relative to north: tank -> healer -> melee -> ranged. Everyone else should stack in the big gap and wait for the next cast from the boss.
Vamp Fatale will cast either Ultrasonic Amp (stack) or Ultrasonic Spread (spread). If you get Amp, stack in the big gap, mit, and heal after the hit. If you get Spread, do a light role spread with the tank in the big gap, both DPS in one smaller gap, and the healer in the other smaller gap.
After both hits, the remaining cells will explode and kill people. Try to make sure all the cells are dead before the second hit finishes.
Once that's done, the other light party gets their cells. Resolve it the same way.
Undead Deathmatch / Sanguine Scratch 1
If you are doing this fight with item level 780 and up and 12% echo, this will be one of the last mechanics you see. I've seen some parties kill during this mechanic.
Vamp Fatale spawns two party towers either north/south or east/west. Split up into light parties and soak the tower. One person not being in a tower is survivable, two is iffy, more than three is a wipe. Try to have everyone in the towers or mit like your life depends on it.
When the towers explode, they will spawn two untargetable bats that apply chains to everyone. Do not stretch the chains out or you will die. Follow the bats and avoid the proteans by moving from the safe spot to the place she just attacked. Stay close to the inside so you can move as little as possible.
When I cleared this fight, I still considered this mechanic my prog point. If
you are unsure, stop doing damage and focus on mechanics. You can have
multiple people do zero damage during this part and still kill the boss
before the enrage.
Once the bats get to 180 degrees away from where they started, they will either have a donut or circle marker above them. If they have a donut, stand under the bat. If they have a circle, stand near the boss.
If anyone dies in this mechanic, wait until the end to resurrect them. People
resurrect in the place where the caster finished the cast, and this will
make them be assigned to whatever bat the server wants, which will end up
killing them again.
After that, she casts another Brutal Rain. Mit and heal through this.
Act 7: Victory Lap
She will cast her final Vamp Stomp. You have the time to sort yourself out, so feel free to burn sprint and get to your spots.
She will also cast another Hardcore for the road. Respect this. This will be very meaty and the AoEs will be VERY LARGE. Tanks should go as far away as possible and heavily mitigate or pop their invulnerability skills.
There's one last cast of Sanguine Scratch but without the tether bats. Slide between the safe spots and keep doing damage.
Then there's one last Crowd Kill which makes her gain the buff Completely Satisfied. This will make her cast her enrage: Finale Fatale. If she finishes the cast, you die. Just kill the boss until it dies.
Win free books from the October 2026 batch of Early Reviewer titles! We’ve got 303 books this month, and a grand total of 3,321 copies to give out. Which books are you hoping to snag this month? Come tell us on Talk.
The deadline to request a copy is Sunday, October 25th at 6PM EDT.
Eligibility: Publishers do things country-by-country. This month we have publishers who can send books to the US, the UK, Canada, Australia, France, Belgium, Luxembourg, Denmark, Spain, Germany and more. Make sure to check the message on each book to see if it can be sent to your country.
Thanks to all the publishers participating this month!
The computing industry is extremely resource intensive. In terms of
materials. In terms of manufacturing. In terms of energy usage. In terms
of the knowledge and expertise required to build and design computer
hardware. In terms of needing computers to build the next
computers.
The list could go on, and it all was the case before “AI” or
LLMs. AI has just thrown tons of cash and political gasoline on the
already burning fire, that was already way out of control. Take a look
at this permacomputing
thread about how computing technology (the hardware) can continue to
exist after economic collapse, and you will start to see this picture
come into focus too.
I guess someone could say this position is a thinly disguised form of
whataboutism.
But I’m not really trying to redirect an argument against AI, so much as
I’m trying to sharpen the critique of AI as a form of centralization, by
zooming out and looking at the broader context of computing.
I say this because one of my main concerns to
date about the turn to AI in software development has been
centralization. The degree to which LLMs, and their attendant
infrastructure, centralize software development is literally ỉ̫̙̿͌̂̌n͍͔̑̇ͯ͞s̬̠̰͈̗͎̑̿ͣ̓ȃ̝̬̙͇̗̗̻͜n̲͈͉̙̽ͩ͌͐e̻̓̀͗ͭ̂ͮ̔⃝. But
this centralization is partly what these massive corporations want, or
even need, us to believe–that the model usage must be
centralized in order for it to work, and that they must
be the gate keepers entrusted to let us use it.
Whether it is by design, distillation, or really by whatever means
necessary, we are starting to see models being run in other places,
sometimes on laptops, or on mobile devices, even in cooperatively governed spaces. It
seems that maybe, just maybe, for software development, it might not be
necessary for the model to run by $EVIL_CORP. That it might be possible
to use LLM technology to make software development easier, and more fun,
more responsive to users’ needs, without running every prompt through
one (or a handful) of APIs. And without ingesting the entire Internet of
content.
The more this decentralization (d14n) happens, the more computing
returns to its always already
massively centralized condition. Computing wasn’t perfect before in
terms of d14n, and pretending like it was isn’t a sufficient critique of
the use of LLMs in software development. However, I think it is a
sufficient critique of software development more generally, and the idea
of Automating All the Things just because you (think you) can … but that
is a topic for another post.
One of the eye opening things I’ve learned in my own (albeit limited)
use of LLMs in coding has been the startling (and admittedly unwelcome)
realization of how much of the work I was doing as a software developer
was not creative at all. Transplanting patterns from one domain to
another (conceptual copy/paste), or mundane restructuring of code, or
finding bugs with unit tests. It turns out a lot of this work was
deeply repetitive, and didn’t really require much thought–or at
least not as much thought as I thought it did…
But it also helped me see what work was creative. The biggest
area for me has been being able focus on and understand what problems
are worth working on. What are the approaches I could bring to an
implementation? How can I understand my team members ideas and needs
better? How can I understand what the people using the software need
better? What areas of the codebase need attention to performance? What
areas need simplification? These were always deeply satisfying things to
work on before using LLMs in software development, but I didn’t
have as much time to devote to them because I was too busy indenting
some code, finding a bug, moving things to use some new framework, or
keeping some documentation in sync with its implementation, etc, etc.
Anyway, I guess I’ve found myself having to admit that LLMs are actually
useful for software development, not because they make drudge work go
away, but because they let you focus on the things that actually do
matter.
Let me return to where I started: that computing is always, already
centralized to a great degree. We should, nay we must, always
be looking for ways to decentralize and democratize access to computing.
This isn’t going to happen magically. It is hard work, and requires deep
levels of expertise and access to resources. But we also shouldn’t
deceive ourselves by not seeing how centralized computing as a
technology already is, and already was, before we started prompting LLMs
instead of searching Google to find the answer to a coding problem.
So how’s that for whataboutism? If you see a flaw in this argument, or
think it’s wrong headed, please drop me a line? For what it is worth, I
can’t shake the feeling that this is in fact the language of addiction.
This online event marked the conclusion of the Traceable AI Answers for Public Data project under the AI Learning Labs initiative, showcasing AI integration with open data portals. You can now watch the full event recording and explore all presentation materials.
.png)
.png)
Poster encouraging Americans to participate in the 1940 census. Source: 
Harper’s Weekly illustration of a man taking the census (1870). Source: 
Store room, Census Bureau (1909–40). Source: 
Cartoon depicting a man taking the census from an old woman (1890). Source: 
Taking the census from a group of Winnebago people in Wisconsin (c. 1911). Source: 
Sorting machine, Census Bureau (1909–40). Source: 
).
AI Benchmarking Exercise home page.
Agreement, cost, and time graphs for the models’ runs.
Descriptive statistics about the models’ runs.
Screenshot of the first alpha of what we initially called SmartFox and then Firefox Scholar, before we consulted 1) a lawyer, and 2) an Albanian dictionary for other options, June 2006
Members of the Zotero team in late 2006. Roy Rosenzweig, months into chemotherapy, is kneeling at center; he had just gotten the Zotero license plate for his Prius to the great delight of all of us. Standing, L-R: Sean Takats, Trevor Owens, Josh Greenberg, me; Kneeling: Kari Kraus, Roy. Photo credit: Sharon Leon
Be forewarned, ye who dare log into my web application c. 2002 (screenshot of the Web Scrapbook landing page)
Elena Razlogova’s Scribe application, c. 2002
SmartFox version 0.0.1 screenshot, July 27, 2005
Members of the Zotero team in late 2006. Standing, L-R: Simon Kornblith, Sean Takats, Trevor Owens, Dan Stillman, Matthew Eaton; Kneeling: me, Roy Rosenzweig, Josh Greenberg
The first logotype for Zotero, August 29, 2006
Zotero 10.0, released on August 17, 2026
