Hello person from the future! 

The Coaches’ BBQ was a series of six online unconferences for coaches that ran from August 2023 to October 2026. Most coaches weren’t familiar with the term unconference, so we called them BBQs, to signal that these events were different from an industry conference. 

The backyard BBQ was a useful metaphor to guide us to keep the event fun, casual with some structure. This helped the organizing team with decision making too:

• People asked us to create a mailing list to inform them of the next BBQ. A BBQ doesn’t have an email list, so no. 
• Would we record all the sessions and create a digital library? Also no. 

BBQ made for a fun brand for our website. Kudos to Alexander Agent and their team at Pain in The Assistant for putting our website together and social media visuals. 

The spark to start the Coaches’ BBQ was something that really pissed me off. The International Coaching Federation decided to host their first in-person conference after the pandemic in Florida. The background details are in this online petition that 760 coaches signed. 

I knew this would work better if a team of coaches designed and hosted this event. I reached out to Jeff Jacobson, who said yes right away. Autumn Wade joined the team from the first one to the fifth. Art Benoit joined from the second to the sixth. Alicia Klepper joined for the last one. 

Sessions

There were six coaches BBQs that ran from 2023-2026. Here are the programs, so you can see what type of sessions were offered and the overall vibes:

• August 2023 – this was the first BBQ! We scheduled sessions to be 60 minutes and by the end of the day us organizers had forgotten to eat and were busting to go to the bathroom. I wanted to send out a survey, but Jeff and Autumn thought that was too stuffy for a summer BBQ. Instead we asked people for 3 words to describe their experience–kinda like a group photo. The top results were fun, community and connection, and value and learning. We did what we had set out to do!
  I wrote a blog post with my reflections on the key ingredients of this event. 
• March 2024 – we shortened the sessions to 50 minutes so that there would be breaks for everyone, including us. The interest in hosting a session increased, so we offered 4 programming tracks. 
• October 2024 – we ran this one pretty much the same as March: 50 min sessions, 4 tracks. There was so much interest in presenting we added a waitlist. 
• April 2025 – one of our goals for this event was for it to be a place where coaches could do their first presentation, with the hopes that they would do more presentations or refine something they could sell as a client workshop or a paid presentation. We held one presentation track for new folks and Autumn did a bunch of outreach to find those people. 
• September 2025 – We continued to keep a presentation track for new presenters. We had 2 1/2 tracks of programming for this one. 
• October 2026 – this was the last one! There were two presentation tracks. For me, it felt like there were fewer people than previous BBQs, but the quality of connection and engagement was deeper. 

Community agreements

These are the community agreements we came up with:

These agreements worked really well. People showed up in the generous spirit of the BBQ and were great guests. 

The law of two feet was the hardest one for coaches. Several people shared that they felt like it would be rude to leave a session, especially if there were only a few participants. Each BBQ we reiterated that people had full permission to leave a session. Around BBQ #5 someone pointed out that this language is ableist and we updated it to be the Law of Mobility to highlight that participants have agency and freedom to join and leave sessions. 

We only received complaints about one person’s behaviour. Again, we leaned on the BBQ metaphor. What is our responsibility for our guests in our backyard? One of the organizers reached out to have a conversation with this person about the impact of their behaviour. 

Feedback

We all agreed that it would be too much to send a formal survey to the participants. Like, when have you been invited to someone’s backyard BBQ and then received a Surveymonkey link? (I hope the answer is “never”.) 

The first BBQ sent out a one question survey: “We want to capture the vibe of the BBQ. Think of it as our Group Photo!” Here’s the top responses:

1. Fun and laughter 
2. Community and connection
3. Learning and value

Coaches are generous people and many people wanted to offer kudos. These’s what people appreciated across all of the BBQs:

• The no-sell space. This came up more than anything else. People called it a safe container for learning and said it brought people who came to learn and share. One person said it’s powerful to learn without the sales pitch.
• Community and connection. Many people said they enjoyed meeting other coaches, made new connections and want to keep in touch with people they met.
• Learning they can use. Several people said they learned a lot in a short time and can’t wait to put the ideas into practice.
• The quality of the hosts and facilitators. People thanked the organizers and the presenters for their generosity, energy and care.

An invitation

The Coaches’ BBQ connected with a need for coaches to connect and be in community and to learn and share. Our team is no longer going to be hosting these. For me, life is life-ing and in this season my business needs more of my care and attention. I’m also hungry to connect in person and am part of the organizing team building Coaches’ Summer Camp. 

The invitation for you is to take the parts of the BBQ that resonate with you and run with it. Create your own BBQ, picnic, or invite people into your virtual or physical living room. Put your idea for connection or learning out there–go beyond the themes that show up in the major conferences. There are other coaches out there who are hungry to learn and talk about all the things. If you have a skill, or are learning something new, invite other people to join you in learning. 

As Priya Parker says, create the gathering you’ve been waiting for. 

The post A Love Letter to the Future: Documenting the Coaches’ BBQ appeared first on Tara Robertson Consulting.

The Sharing Shift, DPI as a commons, new course on School of Data, a conversation with Irina Bolychevsky, PeaceTech summit, and more.

The post A shift in how knowledge is shared – OKFN Newsletter Oct/Nov 2026 first appeared on Open Knowledge Blog.

We are using the proprietary hosted version of OpenAI whisper-1 to transcribe audio, with timestamps, into timestamped VTT.

The newer gpt-transcribe may reportedly do a better job of transcribing — but it doesn’t provide timestamps. One could probably try to use other tools to then sync the transcript, but we haven’t looked into that, not sure if it would get you anything useful, we’re trying to keep it simple.

The platform-hosted whisper-1 has a limit of 25 MB per audio file input, but no duration limit.

So when we started, when I was new to such things, I figured, great, we can downsample our oral history interview content to 16kbps mono Opus/ogg audio, and fit a lot in 25MB. This is just human voice it does not need hi-fi to transcribe.

That much was true, but it turns out we noticed error rate increasing the longer the file got. Especially for time syncing. it was making odd choices about how many seconds to include in a single VTT cue (shorter and shorter), and the timestamps were drifting from correct. Erorrs in text transcription were increasing too.

It turns out limiting to 20 minutes — even though whisper-1 has no duration limit — gave us much better results. Then we write some simple code to glue the VTT’s together into a big VTT (which is what we want), and by “we” I mean Claude handled that expeditiously.

I’m not sure how this applies to open source whisper (and it’s open source add-ons/forks), curious if anyone wants to say?

Interestingly, the newer gpt-transcribe does have a 25 minute limit in addition to a byte size limit, so perhaps this is a lesson they learned? Perhaps everyone else already knew this about whisper-1 too? But putting it on the web to try to help more people learn it quicker than I did!

If using platform-hosted OpenAI whisper-1 to transcribe audio to WebVTT, you will get better results by limiting your input to no more than 20-25 minutes, even though the service itself has no duration limit only a byte size limit.

This month we are in Paris for the Digital Public Goods Alliance Annual Members Meeting, hosted by the Government of France. The AMM is where the DPGA community comes together every year to discover, develop, use and invest in digital public goods. Our delegation will be there throughout, leading sessions concentrated on 18 November. Recent...

The post Attribution, Commons and Open Source AI: OKFN in Paris for the DPGA Annual Members Meeting first appeared on Open Knowledge Blog.

PDF version | Powerpoint Version

This presentation was delivered at eResearch 2026 in Melbourne by PT as Rosanna was unable to attend. We wrote the presentation together with the help of the Language Data Commons of Australia (LDaCA) team. Our attendance at the conference was paid by the LDaCA (University of Queensland), supported by coinvestment from the Australian Research Data Commons (ARDC). (Also published over at the LDaCA site.)

The Research Object Crate (RO-Crate) specification is now widely used for interchanging Research (data) Objects (ROs), and has been used as the basis for several projects in the Australian Research Data Commons (ARDC) Humanities and Social Sciences & Indigenous Research Data Commons (HASS&I RDC). RO-Crates are based on linked-data and offer a simple, easy-to-implement way of packaging data for interoperability in line with the FAIR principles. They offer a consistency for deposit in repositories, and re-use in analytical pipelines and as the basis for web sites. Until now the RO-Crate community has lacked a similarly simple approach to validating ROs for consistency, or configuring editing environments and other tools used to produce data packages. In this presentation we show a new approach to describing general purpose linked data schemas and specific data profiles for particular domains and systems known as RO-Crate Machine Actionable Schemas and Profiles (MASP), using examples ranging from Indigenous Knowledge systems, to describing research software implemented, all in line with World Wide Web Consortium standards for metadata profiles. We will show easy-to-use tools, and show how schemas and profiles can be maintained using easy-to-understand spreadsheets. RO-Crate MASP tooling includes generating human-readable documentation, a validator and a compatible RO-Crate editor (Crate-O).

This work came of the LDaCA project but what we’re talking about today is a general-purpose approach to linked data.

LDaCA has a range of collaborators across several universities and institutes.

Linked data helps us meet all of the challenges for language data highlighted above, from collecting and organising data to analysing it.

In the two decades since Tim Berners Lee’s publication on Linked Data principles, adopting the principles has been hard and involves specialised tooling that has not been particularly mainstream.

At LDaCA, we’re working with linked data through the use of RO-Crate or Research Object Crate, and this approach helps address the language data challenges shown here.

RO-Crate is an open Linked Data based specification for packaging, storing and interchanging data (and is thus aligned with the goals of LDaCA and other Data Commons projects).

RO-Crate was designed from the start to be accessible to developers who are not linked-data aware, using the JSON-LD (or Linked Data) variant of JSON for encoding metadata descriptions, rather than XML (which is dated and has patchy support) or more esoteric RDF (or Resource Description Framework) encodings.

We won’t go into huge detail on RO-Crate this talk, but will focus on how the semantics of RO-Crates can be formalised using Schemas and Profiles.

The frameworks we use for metadata come under schemas and profiles.

Schemas are how metadata is structured and standardised at the highest level. They can specify requirements for metadata fields, e.g. date published must have a date format value. Different disciplines can have different standards as well.

Schemas can range from very general like schema.org and Dublin Core, to more specific like the Lanaguage Data Commons Schema (LDAC) https://w3id.org/ldac/terms which focuses on language-related metadata, and private schemas like ARDS which describes clan and kinship terms for example.

Profiles are subsets of standards and are more tailored to specific use cases, for example, the profile used by LDaCA https://w3id.org/ldac/profile is designed specifically to provide guidance on how to structure and describe language collections.

The LDAC profile uses schema.org and parts of Dublin Core as its foundation for metadata, but it also pulls from the LDAC schema for terms specific to describing language data. Profiles also provide guidance on structuring collections, required metadata, access protocols and additional details about the metadata, for example, terms intended to describe a file as opposed to a person.

Whereas schemas focus on defining the metadata terms and their individual requirements, profiles provide more overall context for how the metadata should be used and guidance on structuring particular collections.

Profiles also have the ability to take metadata from various schemas and codify that for use with particular types of datasets.

The next stage in development of schemas and profiles is RO-Crate MASP, which stands for machine actionable schemas and profiles.

Previously, schemas and profiles have been created separately with scripts for generating the documentation specific to a single profile or schema, but MASP brings these together so the same scripts can be used to generate multiple documents.

This approach is currently being used for the LDAC schema for language data, as well as the schemas of ARDS Aboriginal Corporation and Batchelor Institute, both of which are closely developed with knowledge holders to describe clan and kinship terms that reflect ancient and ongoing cultural knowledge systems.

The diagram here shows how a profile is generated - it takes a JSON file of the metadata terms used in the profile which can be adapted from schemas, as well as a human-readable text file that provides more guidance and description of the profile. These two files are used by the script to generate the profile document, and that document and the JSON file become the profile crate.

This slide shows what a MASP profile component looks like “under the bonnet”. It’s JSON-LD. This is defining a property communicationMode which is defined in the Language Data Commons Schema. It's full URI is https://w3id/org/ldac/terms#communicationMode. It occurs in the domain of any schema:CreativeWork, and the value it points to is one of a set of CommunicationModeTerms.

This slide is a screenshot of the same property definition in an Excel format – it is possible to edit in either the text file or a spreadsheet, or in an RO-Crate editor.

One of the other major uses for MASP that we’re currently working on is to drive tools for managing collections.

An example of this is Collection2Crate or C2C (available soon at CrateWorks), which is a browser tool that turns a folder of files on your computer into an RO-Crate, with optional data cleaning, processing and visualisation steps.

It’s built for people organising diverse collections of files who need consistent, publishable metadata without having to become RO-Crate experts themselves.

Depending on which MASP profile you select, this will determine which fields you can use to describe your collection, which plugins are available for processing the data, what gets written out, and what counts as a valid RO-Crate.

In the screenshot, the processing options available here are those configured for the LDAC profile for language data, so selecting another profile would change the options we see in this and other sections of C2C.

MASP profiles can also be used to validate collections for deposit into repositories. The diagram shows how the JSON file in a profile crate can be used as a set of rules to validate a collection that uses this profile, meaning that any missing required metadata, errors in how the collection has been structured, and other requirements can be flagged and corrected.

Both PARADISEC, the Pacific and Regional Archive for Digital Sources in Endangered Cultures, and LDaCA share the LDAC profile for language data, and this validation method is used for collections deposited into the Language Data Commons Portal.

Another way MASP can be used is to indicate, or potentially package HTML templates that configure the design and layout of static sites for RO-Crated collections.

This method uses the ro-crate-static-site generator(available at the CrateWorks site) which takes an RO-Crate and provides a static HTML site configured according to that collection.

We can extend this further so that individual MASP profiles provide a set of templates available for collections using this profile, and these can then be further configured according to the requirements of the particular collection.

Several examples already using this method are Precarious Oral Histories, which catalogues at-risk oral history collections throughout Australia, the Arrernte Bird Stories, providing names, calls and descriptions of birds in language, and Dingo Lingo, which maps words for dingoes and dogs across Australia in Indigenous languages.

Each of these collections requires unique ways of presenting the information, including search, filtering, tabulation, geographical mapping, and audio and visual media. The static site approach, coupled with MASP profiles, allows for these unique requirements, so that while each of these sites uses the same static site generator, the configuration of each can be determined by the profile.

This diagram shows the “narrow waist” of standardisation – on the left is data preparation which is focussed on organising and describing data for two main purposes:

1. Deposit into archival repository services where it can be found,
2. Analysis; by which we mean pretty much any research process that uses the data to gain some insight. If data has been organised to a metadata profile, then systems such as C2C can turn on access to tools that work with data conforming to that profile.

Schemas:

• Provide shared terms for findability and data interoperability
• Provide guidance on structuring data
• Ensure that metadata terms are documented (by following the links to the schema)

Profiles:

• Set expectations for what can be put into an archival repository in detail (and how schema terms can be combined)
• Can have packaged resources such as display templates and pointers to data preparation tools like C2C

This MASP workflow also lends itself to reuse of RO-Crate packaged data collections with the same analysis tools for greater interoperability.

Since all deposited collections already meet the MASP validation requirements, they will function with a range of ready-to-use notebooks, libraries, visualisations, workflows and apps, meaning that the same methods and tools can be applied to multiple collections.

Following on from this, research outputs from any analysis done on RO-Crated collections can also be put through the data preparation workflow and published or archived as their own RO-Crate collections.

This workflow automatically adds provenance information in the RO-Crate, making it clear exactly what tools and parameters were used to create individual research outputs.

In terms of next steps, we aim to establish governance for MASP under the RO-Crate steering committee to ensure further support and interoperability with existing frameworks. MASP in on track to become an official RO-Crate specification.

We are also working on new tools to create and use MASPs, for example, creating new schemas and profiles by reading existing RO-Crate collections and inferring the structure, conversion of existing legacy linked data ontologies to RO-Crate, and an editor app to create and modify profiles and schemas, lots of tools are being developed and are currently jumbled together in a github repository, which will be cleaned up and separated out into multiple projects soon.

If you’d like to hear more about what LDaCA is working on, feel free to subscribe to our newsletter or get in contact with us.

Lisa Macpherson (Public Knowledge) and Anna Tumadóttir (Creative Commons) recently published Creating a New Internet Bargain to Preserve Fair Use and Protect the Commons in Tech Policy Press. They do an excellent job of describing the predicament that the web commons finds itself in, regarding AI plunder, and enclosure of said commons.

But I scratched my chin when I reached the part about proposed legislation that would require LLM providers to disclose what content they used to train their models, such as what the TRAIN ACT is trying to do. I haven’t read the text of the act, and maybe the authors have and there are some poison pills there. But they actually make a good case for disclosure, only to backpedal it:

Imagine if large technology companies had disclosed their training sources all along. That could have steered us towards quality on all fronts, limiting the mess we now find ourselves in. Instead, lawmakers around the world are considering legislation that would require AI developers to disclose the sources they use for training.

Both of our organizations are interested in the benefits of transparency. But Public Knowledge, in particular, is wary of legislation that threatens fair use by providing plaintiffs with a pre-discovery tool that would bring about a continuous wave of litigation risk.

It seems like requiring model creators to describe the data that was used to train the model is pretty critical for understanding how the models can be used? Perhaps we need better norms around how to do this, but it seems like a pretty low bar to get over.

I started wondering how much documentation of training data exists today. Hugging Face is a good place to look since it hosts a lot of open models, and the model card metadata lets a publisher say which datasets they used to train a model. These are “open” models that were designed to be downloaded and used, so you’d think that they wouldn’t have many qualms about saying where the models came from?

So, if you had to guess, what percentage of models on Hugging Face document their data that way?

Fortunately Hugging Face publishes model metadata as a Parquet dataset (librarian-bots/model_cards_with_metadata), so you can query it right from your browser using the DuckDB Web Shell. The query takes a minute or so to run.

You may notice I limited the query to base models, meaning models that don’t declare a base_model they were built from. This tips the scales in the favor of over-counting models that have datasets, since a fine-tuned model could lack dataset metadata but point a base model that does have it.

So the result?

46,171 out of 494,128 (base) models, or 9.3%, list a dataset.

To be fair, some model cards probably describe their training data in prose without filling in the metadata field. A query like this can’t see those. But the metadata field is what lets anyone trace a model back to its sources at scale, and over 90% of base models leave it empty.

Hugging Face is a place to publish open models. These are the publishers who chose to share their weights in public. You’d think they would be more interested in telling us about the data that was used. Of course the closed frontier models from the big AI companies tell us even less.

Users of all stripes would benefit from knowing what data was used to build these models. Macpherson and Tumadóttir seem to agree that disclosure “could have steered us towards quality on all fronts.” Litigation risk is a concern, but I don’t see why it should outweigh the benefit of knowing what’s in these models, especially for organizations trying to protect the web commons. If voluntary documentation sits under 10% even among open models, how do we get there without regulation?

DOI: 10.26181/34969048.v1

I was honoured with VALA's Robert D. Williamson Award at the 2026 VALA conference last week. This year, the VALA committee asked me to deliver the final keynote for the conference after they announced the award. The following is what I intended to say, and very nearly what I actually said. A recording will be available at some point in 2027.

What a privilege to be here together on Wurundjeri Country. Where elders have passed on knowledge about how to care for the land, skies and waterways for generation after generation, in an unbroken chain hundreds of generations long. I want to acknowledge that care and that knowledge-keeping by the Wurundjeri people, and thank Thane for his warm welcome to his Country at the beginning of this conference. As Thane reminded us, Australia has always been multi-cultural, made up of many cultures and many Countries. So to all Indigenous colleagues with us today, I extend that acknowledgement and respect to you. As knowledge keepers, our profession still has a lot to learn and engage with from Australia's first cultures.

VALA, you have always held a special place in my heart. In a lot of ways VALA as an organisation has made me who I am as a librarian. And don't tell the other conferences, but this has always been my favourite, and I think it’s the best library conference in Australia. To be presented today with the Williamson Award is truly an unexpected honour. But I'm not going to dwell on that because - as a middle aged white man - I am absolutely capable of talking about myself for 45 minutes. And that wouldn't be a very interesting note on which to end the conference. So instead, we're going to talk about you.

Toxic heirlooms

This is a test for anyone who was in Session 9B today - can anyone tell me why the State Library of Victoria's conservation lab has zipped these books into plastic bags with a big sign saying "PLEASE DO NOT HANDLE"?

That's right - they're full of arsenic and they'll kill you.

The inspiration for my title today came from another Hugh – Huw Sandaver, from the University of Divinity. Huw presented earlier today with Hayley on the work they have been doing to identify and record poisonous books from the Mannix Library's Goold collection. Like these books from the State Library, you really don't want to be licking your fingers to turn the page on a lot of Huw's books, and not just because he'll throw you out for damaging them.

These books are often beautiful. The greens and reds are still bright after two centuries. But the pigments they use are so carcinogenic that they can't be safely handled. That is: they look really cool, but they can't safely be used for their intended purpose. Our forebears got swept up in a trend and thought arsenic-laden pigments were the future. Or maybe they weren't sure, but felt pressure to act as if toxic books were the future because they didn't want to look out of touch. And now we're in that future. And we have to contain the mess they left with these toxic heirlooms that we can't get rid of because they're too important, but also can't use because they're too carcinogenic.

This is a real problem costing libraries real time, real money, and real emotional and physical labour. But you might have guessed already that these toxic heirlooms are also a metaphor.

Different questions

Libraries operate for the long term. That requires a very different mindset from Silicon valley boosters. The urge to embrace every new tech that appears comes from a kind of temporal narcissism – the idea that our current time is the most important and consequential time that ever has and ever will exist.

But the advantage we have over tech corporations and startups is that we only need to work out how to keep things going, rather than how to squeeze every last drop of profit out of them. That means we get to ask different questions of technology. We need to think about how the knowledge in our collections will be passed on to the next generation, not how to maximise returns in the next quarter. Instead of 90 days from now, we think – or at least we should be thinking – on scales of 90 years from now.

We talk a lot in libraries about information. But mostly people come to us for stories. But what stories are we telling ourselves?

I have been working in libraries for 25 years. If you look through professional journals, books, blog posts or conference proceedings over that entire period you'll find an almost continuous stream of doom about some new technology that was about to destroy the profession and everything we hold dear. Roxanne told us on Wednesday that this extends even further back to arguments about gas lighting in the State Library of Victoria. Every year of my career there have been suggestions for new things you could do to ensure that you, your library, or the profession at large can "remain relevant". Next to that were other stories. Stories about how we needed to "get on board". About movers and shakers embracing new technologies and "not being left behind".

The two genres

So our options have been presented as either to start wearing hoodies and embrace a new tech utopia, or be destroyed in our cardigans by incoming super-intelligent aliens.

This might sound familiar. You might even be wondering why I'm talking in the past tense. Some of these stories are being told right now.

I think these stories that we tell each other are as toxic as the books in the Goold Collection.

Because when we believe these stories we think ourselves into a box. And once we're in that box, they’re the only stories we can see. These stories tell us that we always need to be on the defensive about libraries and librarianship. They tell us that we need to spend limited resources chasing rainbows. They tell us that all the good ideas come from somewhere else. They tell us that all we can ever do is react. The tell us we have no agency.

They tell us we're not enough.

And quite frankly I'm bored with these stories. I think we need new stories.

Because stories about apocalypse, and stories about utopia – that's only two genres! And in fact, since one person's utopia is always somebody else's dystopia...

It's all the same genre!

What kind of collection is that? We're librarians: we know there are far more stories and far more genres available to us.

Friends, we need to do some collection development here!

We need better stories

If we're going to think and act as a profession inter-generationally like the Wurundjeri and other First Peoples do: then we need better stories.

Yesterday Sebastian Hammer suggested a story about libraries as “democracy’s drag anchors”. And Seb Chan closed out the day suggesting that we might build “generative instititutions”. This afternoon I want to suggest a few other stories.

So VALA, are you ready to embrace some new and better stories?

The Seamstress

This is called the Springfield Gown, and it is held at the National Museum of Australia.

It was originally made in the 1740s – probably in Spitalfields, outside of London. It came to Australia as a treasured heirloom of Mary Faithfull in 1838, a century after it was made. Then it was worn by Mary's granddaughter Clarice Anderson-Faithfull to a ball in 1929. So that's another century later. The Springfield Gown was refitted and adjusted several times over its life before it arrived at the museum. Fashions changed and different bodies needed to be accommodated over the years. Sleeves were shortened; seams adjusted; lace added; the shape was altered.

Libraries are full of this kind of refitting.

For example, everybody's favourite library technology, Machine Readable Cataloguing. We talk about this as a technology from the 1960s, but MARC has been continuously updated, morphed, split, re-merged, spun off, and transformed. We have MARC21, MARCXML, and UNIMARC. We can use binary mrc files, human-readable mrk files. Fields have been expanded, adjusted, and repurposed in response to changing needs and changing technologies.

MARC isn't new and shiny. But does that mean it's a toxic heirloom? Some people have definitely said that. But on the other hand, we're all quite happily using this standard to solve real problems around data transfer and display. Perhaps we can think of MARC as more like a hardy set of denim jeans that have been passed through the family. They're practical in lots of situations, there's a few patches, and we've sewed on some extra pockets.

Paul and Penelope told us about migrating a 20 year old library management system yesterday. And if you've ever run a major system migration yourself then you'll know that's when you really get to see all the seams: The structural braces that are normally hidden from view; the secret pockets; the newer cloth patched in; the rough stitching. It's when we see that the real reason any of it works at all is the custom scripts, the macros, and the extensions our teams have used to make the system work the way the need it to.

As Sebastian reminded us yesterday, the trick when you're migrating an old system is to focus on what you were trying to achieve with all these adjustments. A good seamstress knows when an old adjustment doesn't need to be recreated on a new dress.

The Seamstress recognises and addresses the needs of today whilst respecting the work that got us here. Refitting honours the past whilst embracing the future. Seamstressing is practical and particular to a given situation. It fits the tool to new requirements. The technology is re-made to measure. The seamstress needs deep expertise about the thing that is being refitted and what it's used for.

So the story I call The Seamstress is about the value of deep expertise and contextual awareness. It's about sustainable practice and care. It's about respect: Respect for materials and tools, and respect for quality workmanship.

Smuggler's Cove

This is a Smugglarfläta. In English, it's a "Smugglers Braid".

This one is held at the Länsmuseet Gävleborg in Sweden. You put a small tin inside the braided basket, and the toll-man is none the wiser. Smuggling things across boundaries is as old as toll gates, tariffs, and borders.

We have plenty of potential stories about smuggling in libraries.

For example: AlmEZ is a piece of middleware that sits between EZProxy and Clarivate's Alma. It's a pretty straightforward PHP script that uses the Alma API to check that a user's login credentials are correct, and fetches their user group. It then maps the Alma user group to an EZProxy user group, and hands the user and their group off to EZProxy to decide whether they get access to something.

AlmEZ is not the most sophisticated software program in the world. It's basically a glue script sitting between two larger systems. But smuggling doesn't always have to be as complicated as a narco-sub crossing the Pacific. The simple point here is that EZProxy and Alma aren't supposed to talk to each other. AlmEZ gives us a way to smuggle an Alma login into EZProxy behind the vendors' backs.

Another smuggling story could involve OpenURL as our main character.

OpenURL is a NISO standard for creating URL strings that represent a library resource. The neat trick with OpenURL is that you can create the URL without knowing whether or not the target system actually holds the item. It's essentially a standard for creating search parameter strings that are consistent between applications.

So if you're using something like Google Scholar and want to know whether you can access an item from your local library, OpenURL can sort of smuggle your metadata past the need for a fresh search or an API request, and just send a hopeful request straight to your library catalogue.

The more we think about how library technologies work in the real world, the more smuggling stories we can tell. We smuggle new controlled vocabularies into systems that weren't designed for them. We smuggle unique identifiers from reading lists into catalogue records so we have something to connect to an external query or a script.

When I worked in public libraries, we would sometimes smuggle new memberships past our own rules that insisted on government-issued ID with a street address, by simply posting new membership cards to wherever someone told us they lived.

So the Smugglers Cove story is about crafty and enterprising people. It's about finding clever ways to overcome artificial borders. And it's about making transactions frictionless.

Hanging Out

The two Sebs and Alix all talked yesterday about libraries as a place to “hang out”. Hanging out is casual. Hanging out with other people means you're comfortable together. You coordinate enough to be in the same place at the same time. You enjoy each other's company. But you're not locked in to doing it all the time. And if one or two of you start hanging out with other people, that's cool too.

Kathryn talked yesterday about KohaILS, the world's most widely-used integrated library system. Koha is built on hanging out. Different support and development companies work with libraries to maintain Koha. Library organisations from tiny schools to entire national networks use Koha. The infrastructure for managing the project is deliberately distributed and decentralised. Nobody forces anyone to work together, but they do it anyway.

We heard from Allen and Shaun on Wednesday about how the international inter-library loan system is – at least in theory – powered by communication standards allowing any library using any system to borrow from any other library using any other system. Vendors are constantly trying to steer libraries towards product-based solutions to the inter-library loan coordination problem. The standards push in the other direction by allowing you to talk to any system that uses the standard, rather than the network being restricted to libraries using the same software.

And Jaime showed us a different way of hanging out yesterday when she spoke about CAVAL's Collective Shelf project. Collective distributed collections let libraries agree to retain and share items without ceding ownership. Collective Shelf and similar approaches overseas let us expand the impact of our libraries without expanding their overall footprint. Collective Shelf is really just a series of promises we make to each other about what items we retain, how long for, and who can access them.

If we tell a story about open source software, inter-library loan or collective collections through the "hanging out" frame, then we can notice something important. Libraries don't have to coordinate any other actions or agree on anything else to make these things work. There's no sense that if you choose not to hang out with them, you're somehow "being left behind". They're not even promising to hang out together forever. They hang out when it suits them, and live separate lives the rest of the time.

Another way we help things hang out together is inside our systems themselves. When I was doing my undergraduate degree I spent a lot of time in the Morris Miller Library at the University of Tasmania's Sandy Bay campus. The Morris Miller used Library of Congress Classification. But UTas has, like many Australian universities, a history of institutional mergers. The campus libraries in the north of the state all used  - and continued to use - Dewey Decimal. UTas library coped with this perfectly fine. Two editions of the same title simply had different call numbers at different campuses. No big deal. We can also let vocabularies hang out together, running Homosaurus and AIATSIS Place Name vocabularies alongside Library of Congress Subject Headings or anything else we like.

So what can we do by telling stories framed around hanging out?

Every software company loves to talk about "scale". But all "scale" really means is forcing everything to use one single system that assumes every user has the same needs and every use case can be easily imagined by a 25 year old software engineer in San Francisco or Tel Aviv.

So when Chad Dunning Kruger starts talking to you about "scale", I want you to think about "hanging out".

Hanging out is the library world's answer to scale. It provides reach. It provides choice. It's forgiving. It's flexible. Hanging out allows us to back out when we need to, and lean in when we want to. Hanging out lets us work together when we have trust, without making promises about anything else, and without cutting off any other options.

Salvage Engineering

This is a mockup of the Lithium Hydroxide Canister that saved the lives of the three astronauts in the Apollo 13 spacecraft. The mockup was built by the very engineers who designed the original canister, and is now displayed at the National Air and Space Museum in Washington DC.

The most obvious fact about this object is that it looks like a pile of junk. That's because it was made out of plastic bags, cue cards from a three-ring reference binder, hoses from the lunar spacesuits, and a lot of duct tape. That's right – it is literally held together by duct tape. The engineers had to figure out how to get the cylindrical carbon dioxide filters used in the command module to work in the larger box-shaped spaces in the landing module. And all they had to work with was whatever was already on a very small spacecraft.

If those engineers hadn't figured out how to get a round peg to work in a square hole, the Apollo 13 astronauts wouldn't have lived long enough to get home. So we can see that salvage isn't something to be looked down on. Salvage can be heroic. That's how we can think about salvage in libraries. There's probably more of it than you imagine.

My favourite example of this is the Standard Interchange Protocol, which you probably know as SIP or perhaps SIP2. SIP was created by 3M in 1993 as a way to standardise communication between 3M checkout machines and library management systems. Eventually the protocol started being used by other checkout machine vendors and, when 3M didn't seem to object, it was used for other applications as well.

SIP is interesting to me because it fits into several of our story arcs. It's been refitted and extended so it can be used with non-3M checkout machines. It's been smuggled into unlikely scenarios - you can use SIP to book a games console or access an online resource. It hangs out happily with authentication protocols and APIs without feeling inferior. And ultimately it's been salvaged, because 3M sold their library checkout business to Biblioteca in 2015 and the protocol has officially been replaced with NCIP. SIP was never supposed to be used for many of the things we do with it, and it's not supposed to be used for anything in 2026. Notoriously the protocol has no security considerations at all. It's officially both unsupported and superseded. And yet...

Nearly every automated library in the world just sticks SIP inside an stunnel connection and carries on using it.

Because it still works.

You might have heard of OpenRefine. Librarians use it for data cleanup, transformation between formats, and in linked data workflows. It's often used in combination with MarcEdit. So you can do things like create a pipeline taking metadata from systems that don't speak MARC, reformatting it, turning it into MARC files, and then ingesting those files into a library catalogue. But did you know that OpenRefine is a piece of salvage engineering?

OpenRefine is the open source fork of a product that used to be called Google Refine.

Google Refine joined the famous Google Graveyard in 2013 because they couldn't work out how to make money from it.

But Google didn't even create Google Refine. It was originally called Freebase Gridworks, made by a company called MetaWeb. OpenRefine is still actively developed as a community project. And it's made many library data projects more efficient, or even possible at all. So where is the real talent and innovation here? Is it at Google, which simply rebadged someone else's work and then couldn't work out how to make money with it? Is it with MetaWeb, who made a tool specifically for use with their Freebase product? Or is it with the salvage engineers in countless library technical services teams who worked out how to build new data cleanup and migration pipelines? Something to ponder, perhaps.

We might also think about OpenAlex.

OpenAlex is fast becoming a serious competitor to expensive commercial scholarly indexing platforms like Web of Science and Scopus. It's run by the non-profit organisation OurResearch, formerly known as ImpactStory.

OurResearch are smart people. They are the ones responsible for Unpaywall and Unsub. But whilst they have built some really clever infrastructure, OpenAlex is partially a salvage project. It's based on the dataset that was left behind when Microsoft shuttered their Microsoft Academic project after 15 years of trying to compete with Google Scholar.

Libraries are close to this work. In June the University of California’s Digital Library funded work that improved OpenAlex's corresponding author identification by 50%.

But so what? How does talking about SIP, OpenRefine, and OpenAlex as "salvage technologies" change the way we think?

The rocket scientists got Apollo 13 to the moon. But the salvage engineers got the astronauts home. Remembering how much of what we use is already salvage helps us to resist the temptation to think that all our challenges have to be solved with brand new tech. It draws our attention to the smart colleagues who can identify spare parts and abandoned systems that are still useful. It encourages us to consider how we might keep something working, or coax an old tool back to life, and what's still useful from systems, processes and techniques that seems to be past their prime.

Most of all, it keeps us grounded. We'd all love an unlimited budget and a personal software development team. But in the real world, we have to look around at the plastic bags, ring binders and duct tape that we have to hand, and build what we need from what's available.

The Juicero

Does anyone remember the Juicero? This was a company that existed from 2013 to 2017 and – unbelievably – raised $120 million in startup funding. The basic idea of the Juicero was that you would buy an outrageously expensive juicing machine, and then go on to pay a subscription for packets of chopped fruit and vegetables to squeeze into juice. This was already a fairly ridiculous idea, but the company really started rapidly moving towards bankruptcy when Bloomberg News revealed that you could actually just squeeze the packets by hand without needing the juicing machine.

So the story I call The Juicero is a little different to the others I'm suggesting here. The Juicero is about a very complicated, expensive, and unnecessary technology.

One of my hobbies is visiting new libraries a couple of months after they officially open. I like to count how many "out of order" signs they have. There's always one or two. I genuinely would love ALIA to introduce a new library architecture award where the building isn't eligible until 10 years after it was built. So you could see whether the buildings actually work, instead of just beautiful photos of them the day after the last fittings were installed. The winner of that architecture award wouldn't have any Juiceros in it.

I was thinking about the Juicero a couple of weeks ago when our students were preparing for exams. My library has many PC monitors on desks and in group study rooms. The idea is that students share their screens and study collaboratively. But when you observe what they actually do, these screens are just in everybody's way. What is used extensively are the whiteboards and dry-erase pens we make available. And I can tell you, whiteboards are a hell of a lot cheaper, and significantly more reliable, than a digital "smart-table". I have never seen an out of order sign on a whiteboard.

Now Sae Ra asked me to "be controversial" today. So Sae Ra, here's your controversial hot take. I think the reason linked data has never really reached widespread adoption in libraries is that for most libraries, linked data is a Juicero. Linked data is very cool. It's a great idea. But is the improvement over just exchanging MARC records worth the implementation cost for most libraries?

I think, arguably not.

Whether you agree with me or not about linked data isn't really the point. The Juicero story gives us a different way to talk about linked data without it being a story about the inevitable march of progress, or about anyone "being left behind".

The Juicero story is one we can use to check ourselves when we're getting a little too excited. You can turn to your colleague and ask, "Do we actually need this, or is it a Juicero?"

Love letters to the future

Finally, it’s time to talk about love letters.

Jason Scott of the Internet Archive famously described metadata as a "love note" for the people who come after us. What Scott meant is that when we take the time to articulate what we know about something, we allow people in the future to understand what we know, and how we think.

The most powerful way humans have for passing on information is to tell stories.

So which stories we tell is important.

This afternoon I've given you some new stories to think about.

We talked about The Seamstress, who adjusts and refits things as circumstances change.

We talked about Smugglers Cove, a story where crafty entrepreneurs find ways to pass things across artificial boundaries.

We had a story about Hanging Out, where we can cooperate, but come and go freely.

In the heroic story of Salvage Engineering we saw people making new and useful things from the spare parts at hand.

And finally, we had the cautionary tale of The Juicero.

These stories can help us to write love letters to the future, and they also help us to feel less anxious about the present. Whether a technology is marketed as the next big thing doesn't tell us very much, even if it turns out to be true. It doesn't tell us about its usefulness, about its politics, or about its consequences. It doesn't tell us whether it's sustainable on any axis. It doesn't tell us anything about the technologies and systems we're already using, or how we might maintain them.

We need to ask different questions about new technologies. And we can't do that if we're stuck inside the same old metaphors and stories.

New questions

With new stories and new metaphors we could ask:

• can this be refitted to meet changing needs in the future?
• could I use this to smuggle something useful from one context to another?
• will it comfortably hang out alongside other library technologies and networks?
• will it leave usable components that can be safely salvaged and repurposed?
• is this thing a Juicero, making simple things complicated?

When we all go back to our day to day the conference high will fade. Different stories can help us to value ourselves and our work.

Whether you think of yourself as a Seamstress, a Smuggler, a Hangout Facilitator, a Salvage Engineer, or simply a Juicero Sceptic: I want you to know that it's ok to just be really good at doing something that needs to be done. In fact, it's necessary. That's how we make dresses last 300 years through multiple generations, and how we get astronauts back from lunar orbit. And it's also how we keep the miracle of our libraries running day after day after day.

These aren't the only stories we can use. You might have other suggestions. And who could be better than librarians to build a diverse and interesting collection of stories?

An Austrian story worth carrying forward. A new beginning for open data, shared knowledge, and technology that serves everyone.

The post Open Knowledge Austria starts anew first appeared on Open Knowledge Blog.

Beyond our sessions and events, the best part of our BCN tour was the people: catching up with familiar faces and meeting new ones is what keeps the open knowledge communities moving

The post Open Tech Week Recap: Meeting Our Communities in Barcelona first appeared on Open Knowledge Blog.

In my last post I was waiting on a second CD-RW drive from eBay, hoping it would let John’s TASCAM 788 export his songs to WAV. It arrived, and it half works. The 788 sometimes writes a disc and sometimes stops with the same PANIC message as before. At one point I deleted every song from the 788 except the one I was exporting, and that export worked, so I wondered if the 788 was running out of disk space. But then later exports failed anyway. After a lot of experimenting with two different drives I couldn’t find any pattern to it, which was incredibly frustrating. I could hook the TASCAM 788 to an audio interface and get each track and mix off that way, but that would take even more time.

So I’m going back to the plan I mentioned at the end of the last post: rip the raw data from each CD, extract the audio as one long WAV file, and cut the tracks apart by hand in Audacity or another DAW. It means more manual work per song, but it doesn’t depend on early-2000s hardware behaving itself.

Working on the ripping with Claude turned up a few things I’m glad we caught before imaging all the discs:

• My first images were scrambled. On macOS the regular disk device (/dev/disk4) goes through a cache that seems to mishandle these discs’ 2352-byte sectors, and about a fifth of the sectors came back as copies of their neighbors. ddrescue still reported 100% rescued. Reading the raw device (/dev/rdisk4) fixed it.
• Each sector on a data CD carries its own address and a checksum, so a script can now check every sector of every image instead of trusting the ripping tool’s percentage. A second copy made with cdrdao serves as a cross-check.
• When I first extracted audio from one disc it was all static. It turns out some of the discs hold 16-bit audio and some 24-bit, and on one the audio starts a byte later than the others. The extraction script now works out which is which, and plays me a few clips from each disc so I can hear that it’s music before moving on.

The scripts, and notes on what I tried along the way, are in the jas-discs repository on GitHub.

Claude was a big help with all of this, though maybe not in the way I expected. It wrote most of the code, but some of its earlier code was part of the problem: the first version of the rip script had the two sector sizes backwards, and the first extraction script assumed every disc was 24-bit. What helped was its habit of checking the files instead of trusting what a tool reported. It read the address in every sector of an image that ddrescue said was perfect, and found the scrambling. When I said a WAV sounded like static, it measured which reading of the bytes looked like audio, and found the 16-bit discs. Once there was a check like that, I could trust the results, whoever (or whatever) wrote the code. And when it came down to whether something was music, the test was me listening.

The images, the audio, and a checksum manifest all go into a BagIt bag, so I’ll be able to tell if anything changes when I copy them around later. I’m going to periodically upload the bags to a personal S3 bucket for safekeeping. Each disc gets four files, the raw image, ddrescue’s map of what it read, the cdrdao copy, and the extracted audio:

backups/
├── bagit.txt
├── manifest-md5.txt
└── data/
    ├── exp_number_11.bin    134M
    ├── exp_number_11.map    462B
    ├── exp_number_11.raw    154M
    ├── exp_number_11.wav    133M
    └── ...

and the manifest lists a checksum for each of them:

d29a6bb44ad27e955ba8553c9286b6b0  data/exp_number_11.raw
1380e2112a41460c4529b4d3efba4cc0  data/exp_number_11.map
505669c9b4e7c25d41526185bcb86aa8  data/exp_number_11.bin
7cefcee860cf341689eff007b78cf85e  data/exp_number_11.wav

I’ve imaged 18 discs so far, and have about 80 to go. Then comes the slow (but fun) part: listening to each long WAV and finding where one track ends and the next begins. The tracks come out back to back, but the gaps between them vary, which is why my earlier script kept cutting songs in the wrong places. I’m not that experienced in using a DAW, so this should be another phase in the experiment. I’ve been wanting to have a practical project to learn Ableton which a lot of field recording and musicians I follow in Bandcamp seem to use. But it’s kind of expensive so I might go with Reaper for now. If you have thoughts or opinions please get in touch.

Win free books from the November 2026 batch of Early Reviewer titles! We’ve got 400 books this month, and a grand total of 4,774 copies to give out. Which books are you hoping to snag this month? Come tell us on Talk.

If you haven’t already, sign up for Early Reviewers. If you’ve already signed up, please check your mailing/email address and make sure they’re correct.

» Request books here!

The deadline to request a copy is Monday, November 23rd at 6PM EST.

Eligibility: Publishers do things country-by-country. This month we have publishers who can send books to the US, Canada, the UK, Australia, Germany, Italy, Netherlands, Israel, Luxembourg, Spain and more. Make sure to check the message on each book to see if it can be sent to your country.

Thanks to all the publishers participating this month!

AD MONTEM	Alcove Press	Anchorline Press
Artemesia Publishing	Attwater Books	Autumn House Press
Bent Banana Books	Blessed Lotus Ventures LLC	BookViewCafe
CCKR Publishing, LLC	City Owl Press	Crooked Lane Books
Denwelco Studios	eSpec Books	Espresso Publishing House
Flat Sole Studio	Follow The Symbol Publishing	Gefen Publishing House
Harbor Lane Books, LLC.	Henry Holt and Company	Legacy Books Press
Lochan Dodd	Lucky Rabbit Publishing	Matri Global Publications
Miravalle Books	Mythological Creatures Publishing	N2 Creative Studios
NeoParadoxa	Neruda Archives	OC Publishing
OE Publishing	Oliver-Heber Books	Over The Tall Shelf LLC
Prolific Pulse Press LLC	PublishNation	Purple Moon Publishing
Restless Books	Ripples Media	Rook & Rowan Press
Rootstock Publishing	Running Wild Press, LLC	Shadow Dragon Press
Shilka Publishing	Simon & Schuster	Slant Books
Somewhat Grumpy Press	Templo Vivo Ediciones	University of Nevada Press
University of New Mexico Press	Valice Press	Vibrant Publishers
W4 Publishing, LLC	Wise Media Group	Éditions Croulebarbe

The Frictionless libraries for R and Python now support the Data Package v2 specification — with cross-language harmonisation, automatic version detection and continued v1 compatibility. Made possible by the NLnet Foundation, all the maintainers and broader Frictionless Data community.

The post Announcing Data Package v2 Support in Frictionless R and Python Libraries first appeared on Open Knowledge Blog.

It’s the most wonderful time of the year: the Twentieth Annual SantaThing is here at last!

This year we continue to focus on indie bookstores from around the United States. You can still order Kindle ebooks, we have Kenny’s and Blackwell’s for international orders, and also stores local to Australia, New Zealand, and Ireland.
» SIGN UP FOR SANTATHING NOW!

What is SantaThing?

SantaThing is “Secret Santa” for LibraryThing members.

How it Works

You pay $15–$50 and pick your favorite bookseller. We match you with a participant, and you play Santa by selecting books for them. Another Santa does the same for you, in secret. LibraryThing does the ordering, and you get the joy of giving AND receiving books!

Sign up once or thrice, for yourself or someone else.

Even if you don’t want to be a Santa, you can help by suggesting books for others. Click on an existing SantaThing profile to leave a suggestion.

Every year, LibraryThing members give generously to each other through SantaThing. If you’d like to donate an entry, or want to participate, but it’s just not in the budget this year, be sure to check out our Donations Thread here.

Important Dates

Sign-ups close THURSDAY, November 19th at 12pm EST. By the next day, we’ll notify you via LibraryThing message who your Santee is, and you can start picking books.

You’ll then have two weeks to pick your books, until WEDNESDAY, December 2nd at 12pm EST. As soon as the picking ends, the ordering begins, and we’ll get all the books out to you as soon as we can.

» Go sign up to become a Secret Santa now!

Supporting Indie Bookstores

We’re once again teaming up with independent bookstores from around the country to deliver your SantaThing picks, including BookPeople in Austin, TX, Longfellow Books in Portland, ME, and Powell’s Books in Portland, OR.

We’re also bringing back the following foreign retail partners: Readings for our Australian participants, Time Out Books for the Kiwi participants, and Kennys for our Irish friends.

International deliveries will be available through Kennys and Blackwell’s.

Kindle options are available to all members who are registered on Amazon.com (not .co.uk, .ca, etc.). See more information about all the stores.

Shipping

Some of our booksellers are able to offer free shipping, and some are not. Depending on your bookseller of choice, you may receive $6 less in books, to cover shipping costs. You can find details about shipping costs and holiday ordering deadlines for each of our booksellers here on the SantaThing Help page.
» Go sign up now!

Questions? Comments?

This is our TWENTIETH year of SantaThing. See the SantaThing Help page for further details and FAQ.
Feel free to ask your questions over on this Talk topic, or you can contact Kate directly at kate@librarything.com.
Happy SantaThinging!

Microsoft on Monday offered buyouts to a quarter of its work force, the latest effort by the company to rein in costs as it spends heavily on artificial intelligence.
...
The “time-limited, voluntary” program grew out of the company’s first ever buyouts earlier this year, she wrote. In April, Microsoft offered early retirement to 8,600 long-tenured U.S. employees, and about 30 percent accepted the packages and left over the summer. Microsoft said it heard from additional employees who would have taken the buyout if they had been eligible. That informed the company’s decision to create this new, expanded program.

Microsoft has already slightly shrunk its work force even as business has boomed. For several years, the company maintained a flat head count of about 228,000 employees.

But as of June 30, the end of its fiscal year, Microsoft’s work force had edged lower to 223,000 employees. Over that same fiscal year, Microsoft brought in $332 billion in revenue, up 18 percent from the previous year, and profit grew 31 percent to $134 billion.

In July, Microsoft announced it was cutting 4,800 jobs, particularly in its video game division. The company is scheduled to report quarterly results on Wednesday.

how consequential would it be for equities if the US experienced a rise in AI-linked unemployment among high earners which led to a fall in pension contributions.

In their AI-boom scenario, our economists model unemployment rising to 10% as workers are displaced by technology. Importantly, whereas past technological revolutions have displaced lower earners, this time higher-earning white-collar workers are in the firing line. And they are the ones who disproportionately contribute to the monthly pension flows into equity markets. This article puts numbers on just how material a risk this is. Spoiler alert: it’s big.

Lamont Figure 1

The (dollar-weighted) average allocation to equities across 401(k)s is 75%. Younger individuals have more, with those in their 20s having 90% on average but even those in their 60s still hold over 60%.

Assuming the 75% equity allocation also applies to new contributions, US DC plans and IRAs funnelled around $735 billion into global equity markets in 2024. Given the US weight in the global stock market and typical [target duration fund] allocations, 60-70% of that will have gone into US equities, translating into approximately $480 billion. DC and IRA pension accounts bought an estimated nearly half a trillion dollars of US equities in 2024.

Lamont Figure 3

The Congressional Budget Office’s most recent comprehensive distributional analysis of retirement-related tax expenditures — the closest available proxy for the distribution of contribution dollars — found the highest-income quintile captured 63% of the tax benefits in 2019, against 1% for the bottom quintile. The tax-benefit share will overstate the contribution share to some extent because higher earners face higher marginal tax rates. Even so, this is broadly consistent with IRS data which shows that 62% of contributions to IRAs in 2023 came from people earning over $100k a year. Only about 15% came from people earning what was then the median income of around $50k or less.

Lamont Figure 4

There is also direct evidence from within company plans. As Figure 4 shows, the best paid 20% of employees at a given firm have, on average, taken in 44% of employer contributions.

A growing body of research reaches the same broad conclusion: higher-income, white collar occupations requiring postsecondary education show the highest exposure to AI capabilities (Brynjolfsson and others (2018); Webb (2020); Muro and others (2019); Felten and others (2023); Eloundou and others (2024); Kinder and others (2024); and Hampole and others (2025)).

Dodini & Smith Chart 3

1. We find no evidence of widespread, economy-wide job displacement.
2. However, employment of young workers (ages 22–25) in AI-exposed occupations now stands 19% below where it would be had it kept pace with that of their less-exposed peers; experienced workers show no comparable gap.
3. This divergence has widened steadily since we first documented it in August 2025.
4. It operates primarily through reduced hiring of young workers rather than increased separations.

You could take the junior-first pattern that the data currently shows as reassurance for the pension channel since junior workers contribute less than managing directors. But that would be naive. Junior workers in high-paying industries are still high earners by economy-wide standards. A 25-year-old software engineer on $140,000, or a first-year banking analyst on $120,000, sits comfortably in the top quintile of all US earners. The displacement happening right now is concentrated in industries where even the juniors out-earn the median American worker by a factor of two. These are exactly the contributors identified earlier. In addition, participants in their 20s hold 90% equity on average, the highest allocation of any age cohort. So even a junior-first shock would hit a group whose pension contributions are unusually equity-heavy.

Just as importantly, junior-first describes where current AI tools are biting first. We don’t know where it will end. The substitution frontier has moved up the capability curve every year since 2022, and there is no obvious reason to expect it to stop at the entry level.

The irresistible pitch to CEOs is that they can "do more with less", or in other words they can lay off all these troublesome employees without impacting their products and sales.

A 10% hit to that exposed flow would remove roughly $20–30 billion of annual equity buying; a 25% hit would remove $50–75 billion; a 50% hit would remove $100–150 billion. In a normal year that is material. In a downturn, when buybacks, foreign demand and retail risk appetite are likely to weaken at the same time, it could matter a lot more.

• Stocks Slump as Wall Street Gets AI Wake-Up Call
• Amazon, Alphabet, SpaceX, and Meta are all to blame for the new tech stock rout
• Tech Stocks Tumble, but Broader Market Is Holding On
• Nasdaq 100 Hits Third Correction of 2026 on Unwinding Tech Stock Bets
• Meta, Alphabet, Intel Share Sales Leave Buyers Far Underwater
• US Tech Stocks Enter Correction on AI and Midterm Election Risks

Update: 4^th November 2026

Across America, workers increasingly fear that AI will take their jobs. Yet for now, the more immediate threat may be to their pay. Average hourly wages in the private sector have risen by just 3% in the past year. Prices have risen faster still, pushing down real wages. That is all the more surprising given productivity is growing healthily, which ought to support pay rather than squeeze it. During the dotcom productivity boom of the late 1990s, for instance, real hourly wages grew by more than 2.5% a year. Some Americans were even cheerful.

Source

As the technology takes over more tasks, firms may need fewer workers, weakening demand for labour and holding down wages. Such fears received fresh fuel on October 26th, when Microsoft offered buy-outs to 60,000 employees to contain costs while making gargantuan investments in ai. But evidence of widespread job losses attributable to AI remains thin. Unemployment is low; layoffs are scarce. In a recent survey by the Federal Reserve Bank of New York, just 4% of service firms using AI reported laying off workers in the past six months because of it, while roughly as many reported hiring more workers as hiring less. Researchers at Stanford University, studying payroll records for millions of Americans, have likewise found little evidence of widespread ai-driven displacement.

Even so, fear of AI could still be having an effect, by causing workers to cling to the jobs they have rather than seek new ones. Businesses’ demand for workers remains healthy, judged by job openings. But workers are reluctant to jump ship. The Economist estimates that Americans are quitting 13% less often than job openings and the unemployment rate would predict, based on the pattern of the past 25 years. In professional and business services, a sector filled with ai-exposed jobs, the quit rate is nearly a third lower than in 2019, and almost as low as it was during during the global financial crisis of 2007-09. Wage growth in the sector has tumbled, too. When workers are happy to stay, employers face less pressure to raise pay to keep them.

TL;DR indice is a new open source tool for searching web archives. Install it locally or give the demo a try. Drop me an email if you would like to login as a “curator” with write privileges.

Not too long ago I wrote about reading rooms for the archived web, or web sites that provide contextualized access to archived web content. The basic idea is that an aspirational reading room like this provides index, browse, search, and descriptive layers that archival records need in order to be found and used by researchers.

Over the past six months, with a lot of help from agentic-coding tools (mostly Claude) on the weekends, I’ve been building an experimental tool called indice to help make reading rooms like this. Since it’s World Digital Preservation Day I thought I’d share some of its design goals, and where you can take a look at it, if you are interested.

This was my first real experiment in using “AI” for software development (more on that in another post I think). I know some people will instinctively write off indice immediately as “slop” and I respect that position. But this post isn’t about that. If you’ve dabbled with agentic-coding tools at all you will know that what matters most is the design thought that you put into them. So what were these design ideas?

I guess the obvious first design principle is that users should be able to, ahem, search the web archives. Given their odd shape, and size, search has remained elusive for web archives. Internet Archive make some pockets of their collections searchable, and (I believe) have good reasons for not making more of it searchable. There are tools like SolrWayback for indexing and searching large collections of WARC files, but not everyone can set up software like this, and not everyone is working at this scale.

I wanted indice to run easily on my laptop, but also allow it to be used on a server in a more institutional setting. I wanted a single program (or executable) with embedded fulltext search capabilities, that I could run in both settings, and not an “infrastructure” where a web server, a separate indexing service cluster (e.g. Solr), job management queue, etc needed to be managed. Just a single program to run.

So you can install indice today (which should get easier in the future). I’m also currently running a demo on the smallest virtual machine (2 CPUs, 4GB of RAM) I could find at Hetzner.

If you would like to give the curator features a try shoot me an email and I can give you curator privileges.

I wanted indice to use Webrecorder’s ReplayWebPage web component which (IMHO) is the state-of-the-art when it comes to replaying the archived web. Its approach of using service workers in the web browser to intercept requests to the live web, so that it can return archived content from a WACZ file, that can be hosted anywhere on the web is ingenious, and continues to be revolutionary for web archives. It lets web archives become a media type for the web.

Emily Maemura et. al have persuasively argued that the lack of provenance in web archives makes it difficult for researchers to use them:

Both social and technical factors influence what is captured and preserved for the future, but for any given collection these are often opaque; at best, decisions and constraints are documented inconsistently. Few have addressed the specific ways in which researchers could understand the technical processes, systems, and computing environments used—as well as the human judgments and decisions that shape the archive. This remains a key barrier to the use of web archives for research purposes. (Maemura, Worby, Milligan, & Becker, 2018, p. 1224)

The descriptive metadata in a WACZ file can contain information about who created the archive, when and with what software. It includes an index of the web resources that it contains, and the “seeds” or starting points that that were used to start the crawler. And critically, it bundles the one or more WARC files that contain the archived web content, that were created as part of the same crawl. Finally the WACZ has a manifest that helps you ensure the completeness and validity of the archived content.

I wanted indice to foreground this provenance, and to use WACZ as core building block of the archive. I like to think of WACZ files as the Hollinger boxes that line shelves in brick and mortar archives. You can only load WACZ files into indice. Each WACZ is a crawl and collections of crawls are the core organizing principle of the application. Being able to see technical metadata and for archivists to describe the contents is encouraged and facilitated.

It should be said indice doesn’t create the WACZ files itself. You need to use other tools for that. However I did want to make it possible for users of Browsertrix and Archive-It to be able to import their web archives. So indice does have connectors to those services which administrators can use to pull in and index the archived content. In the case of Archive-It, which doesn’t make WACZs available, indice will assemble one for you.

You can also import WACZs that are held in cloud storage (S3, etc), or really anywhere on the web (if the server allows HTTP range requests). The content can stay, and replay from web storage, while the index lives locally with indice. This is quite nice during indexing because the CDX index can be used to only pull back indexable content, rather than needing to download the whole (possibly large) file.

I wanted curators (the people importing content to indice) to be able to extract and describe the crawls they are adding. Some of this happens automatically during indexing and some it is done manually during collection description:

• Pages: From each archived page it takes what the page says about itself: the HTML title, the meta description, keywords and author, the h1/h2 headings, the language (declared, or detected from the text when not declared), the media type, the HTTP status and the Last-Modified year. Combined with the URL, site and capture date, these let you search, filter and facet (by site, year, type and language) without reopening the archive.
• Crawls: For each crawl it records how the capture was made, taken from the WACZ datapackage.json and the WARC warcinfo records: the software, operator, user agent, robots policy, the capture date range, a tally of HTTP statuses, and the Browsertrix QA rating when there is one.
• Collections: Each collection has a finding aid built around DACS/EAD.Its fields are creator, dates, rights and subjects, plus a narrative covering scope and content, custodial history and appraisal.

One experimental, and completely optional feature is the ability to auto-generate collection level descriptions using any configured Large Language Model (LLM) that is available via OpenAI API compatible service (there are a lot of them). The indice interface lets the user know what model is being used, as well as what content and prompt is going to be sent, prior to it being sent.

I’ve configured the demo to use my Inference Cooperative account to access the Deepseek v4.1-flash open weight model, at GreenPT who provide location based carbon and energy reporting about your usage. The goal here isn’t to replace the curator and their insights, but to assist them by giving them a starting point to work from, and to describe things at a high level which otherwise might never get described at all.

And lastly, but I think the most cool feature, is the ability to annotate. When viewing content in the web archive curators can choose to add an annotation at the page or content level. These annotations are then visible to anyone else who looks at the content, and are indexed. They are stored using the Web Annotation W3C standard as JSON-L files alongside the Markdown files that make up the rest of the descriptive metadata. Yes, the DACS informed archival description isn’t stuffed away in a database, it’s a plain Markdown file on the filesystem where indice is running. You can back these up along with the WACZ data, and you could reconstitute the indice server, or build some better system to use them.

indice is meant to be a preservation relay, or a temporary software container for web archives, where the data is allowed and encouraged to outlive the software (Janée, Frew, & Moore, 2009). Because, as the saying goes, “data ages like wine, and applications like fish”.

There’s more I’d like to write about, like indice’s approach to identity management (hint, it doesn’t and instead lets you integrate with an upstream identity provider); how it needs to take care when parsing PDFs from the world wide web; how it protects against malicious replayed JavaScript from interfering with indice itself, and what’s with the name. But I guess that will need to wait. I’ve taken enough your time, and there are other World Digital Preservation stories for you to read today! If you give indice a try, either the demo, or locally, I’d love to hear from you.

References

A monthly round-up of news, upcoming working group meetings and events, and CLIR program updates from the Digital Library Federation. See all past Digests here. 

Greetings DLF Community! 

As we head into November, thank you to everyone who helped make the 2026 Virtual DLF Forum a success! We’re grateful for your participation, thoughtful feedback, and continued engagement.

I’m already looking forward to gathering in person in Long Beach for the 2027 DLF Forum. Team DLF is working on new opportunities for us to connect and collaborate. Until then, I hope you’ll stay involved with our working groups and community activities.

Thank you for everything you bring to the DLF community!

-Shaneé

This month’s news

• Forum Resources: Couldn’t join us live? Catch up on the 2026 Virtual DLF Forum by watching the keynote recording, reviewing the shared notes, and exploring presentation slides and resources in the Zenodo repository.
• Working Group Meeting: The DLF Digital Accessibility Working Group will hold its final meeting of 2026 on Monday, November 30, at 1 p.m. ET/10 a.m. PT, focusing on the Mendeley Accessibility Audit and plans for 2027. All are welcome to participate, and registration is required to receive the Zoom link.
• New Affiliate Member: Islandora is the newest CLIR affiliate! Through CLIR’s Affiliate Program, we build collaborative relationships and cross-institutional partnerships with organizations that share our mission and goals.
• Office Closure: CLIR and DLF offices will be closed November 23-27 for the Thanksgiving holiday. 

This month’s open DLF group meetings:

For the most up-to-date schedule of DLF group meetings and events (plus conferences and more), bookmark the DLF Community Calendar. Meeting dates are subject to change. Can’t find the meeting call-in information? Email us at info@diglib.org. Reminder: Team DLF working days are Monday through Thursday.

• Born-Digital Access Working Group (BDAWG): Tuesday, 11/3, 2pm ET / 11am PT.
• Digital Accessibility Working Group (DAWG): Tuesday, 11/3, 2pm ET / 11am PT. 
• AIG Cultural Assessment Working Group: Monday, 11/9, 1pm ET / 10am PT.
• AIG Metadata Assessment Group: Friday,11/13, 2pm ET / 11am PT.
• AIG User Experience Working Group: Friday, 11/20, 11am ET / 8am PT.
• Digitization Interest Group: Monday, 11/23, 2pm ET / 11am PT.
• Climate Justice Working Group: Tuesday, 11/17, 3pm ET / 12pm PT.
• DAWG Policy & Workflows: Friday, 11/27, 1pm ET / 10am PT.
• DAWG IT & Development: Monday, 11/30, 1pm ET / 10am PT.

DLF groups are open to ALL, regardless of whether or not you’re affiliated with a DLF member organization. Learn more about our working groups on our website. Interested in scheduling an upcoming working group call or reviving a past group? Check out the DLF Organizer’s Toolkit. As always, feel free to get in touch at info@diglib.org.

The post DLF Digest: November 2026 appeared first on DLF.

Source

The episode is also known for the nonsensical business plan that the gnomes of the title devise, whose three steps consist of:

1. Collect underpants
2. ?
3. Profit

which later became a common meme used to mock poorly-thought-out business and political strategies.

By 2031, annual spending on AI infrastructure could reach $1.5 trillion, including new data center infrastructure and compute capacity as well as ongoing upgrades to the installed base of GPUs, memory, and networking equipment.

If we assume that capital expenditures amount to about 25% of industry revenue (an ambitious but reasonable percentage based on trends among cloud providers), sustaining this level of investment would require an AI market approaching $6 trillion annually.
...
Together, the consumer and enterprise AI market could total between $1.2 trillion and $1.8 trillion, leaving about $4.2 trillion of new revenue to reach the $6 trillion market that we estimate will be necessary to fund the buildout (see Figure 2).

That revenue must come from new sources of economic value.

The economics required to generate ROI from AI infrastructure are demanding trillions in new revenue, not just cost savings.

The industry needs a wave of application innovation comparable with what mobile and cloud unlocked, not just productivity gains on existing workflows.

Source

In an emailed newsletter, Laura Bratton at The Information has reported on a new study

Jeremy Korst, one of the Wharton study's authors, said it was "very difficult" to find "actual, materially measurable financial impacts" from AI spending.

Anybody remember back in early 2023, when people imagined that ChatGPT was going to 10x every employee’s productivity and radically boost the GDP’s of entire countries?

So far, there’s zero evidence of anything like that happening at scale.

Source

A recent study of actual coding practices across hundreds of firms finds that human code review forms a significant “bottleneck” for the overall efficiency of AI coding tools, resulting in “little evidence that firms increase software output or reduce employment” by using them. Any efficiency increased during the actual coding phase, the study authors find, is “absorbed by downstream constraints in the production process”; as “the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments.”
...
For now, though, letting AI write your code looks like a double-edged sword, with increases in coding speed counteracted by similar increases in human code review time and effort. It’s the kind of result that makes us wonder if the considerable time and expense to get AI coding agents working is really worth it for most companies.

The company spent $56.1 million on compute costs between January and August of 2026 alone and as of August was losing “approximately $8-9 million per month,” which it said was a massive improvement over the previous year. The company spent $19.7 million on employee salaries and benefits during that same eight-month time period and that a lot of employees quit because they “began questioning Character’s long-term viability,” according to the company.

Character said that its finances had improved to only be losing $8-9 million per month by taking “significant actions to reduce operational costs and improve revenues” by spending less on compute by simplifying its AI model, laying off staff, and injecting ads into its chats.

Source

The AI cloud-computing provider’s unsecured debt due in 2032 traded below 80 cents on the dollar on Wednesday, according to Trace pricing data. That’s widely considered the threshold for when a borrower is financially strained.

At that price, the bonds yield more than 15% — a figure noted by Palihapitiya that led Musk to respond “yikes” on his social media platform.

someone wanting to buy $10mn of protection against a CoreWeave default needs to pay $1.95mn on day one, then an additional $125k each quarter.

Update: 1^stNovember, 2026

Source

The core economic flaw in the AI bull case is that the marginal cost of producing compute exceeds the marginal benefit it delivers to users, raising questions about claims of unlimited demand for data centers. How much compute and how many data centers are needed depends on the returns firms are able to generate with that compute. So far, the AI payoff for firms has been slow to materialize, implying a growing reluctance to pay for AI services. That is consistent with the ongoing slowdown in token spending among corporate customers.

Source

Ultimately, the number-one variable governing the viability of data center construction models is not physical capacity or power; it is the actual price users are willing to pay for the compute being built, and companies will only pay a high price for that compute if they are able to turn the compute into higher top-line growth, bigger profit margins or substantial cost savings, little of which is showing up at scale yet.

Source

Despite massive investments and widespread hype around artificial intelligence, the technology has delivered zero financial payoff to corporate America outside of tech. Revenue per worker is rising for the Mag 7 but is flat for the S&P 493,

Source

Profit margins are rising for the Mag 7 but are flat for the S&P 493, see the second chart below. ... With no impact on the bottom line, mainstream corporate America is pulling back and starting to view token spending not as a source of growth but as a cost center that weighs on margins.

Source

VC-backed companies are spending more and more on tokens, but mainstream corporate America does not need the latest model and is unwilling to pay for it. As long as agentic AI and token spending are confined to a small part of the economy, the macroeconomic effects of AI will be very limited,

In the hours following the release of CVE-2026-98432 for the project Linux, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a use-after-kfree where retransmitting a crafted connection request while finishing the handshake of that same connection causes funny things to happen up to and including arbitrary code execution. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Prince Julien Hoppe, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

In June 2015 I was having a few drinks with some former colleagues at a bar in Melbourne. I had recently started working at a library service on the other side of the city, and I wasn't the only one at that bar looking for a change. We were ambitious professionals who felt that the library profession generally – and the local scene in particular – was lethargic, boring, and lacking ambition. We complained that ALIA no longer had any interest in us since we had grown out of being shiny "new graduates" but were not yet seasoned veterans worthy of lifetime service pins. We bemoaned the lack of local events to connect outside of formal conferences.

It was a typical whingefest from a group of jaded mid-career professionals. This sort of thing happens every Friday night in every large city. Was the criticism of ALIA fair? Was our synopsis of the state of librarianship accurate? I think ...probably. But it also doesn't really matter. Because the important thing that came out of that tipsy discussion at a noisy bar was the determination to change the situation ourselves. To do something.

This was the birth of newCardigan.

A meetup

But what exactly was newCardigan? From the beginning it was a little vague. Initially the most tangible evidence was merely a Twitter account. We had a fairly good idea of what we wanted to experience, but only loose ideas about how to organise ourselves. And "loosely" was also the operative word on that front. The initial concept was based on a mash-up of tech industry "meetups" I had attended and "Drinking About Museums" events I had read about – one or two talks from professionals about something they were working on or thinking about, followed by drinks at a pub or bar. After a while this morphed into "tours" of venues, but the basic idea of professional discussion + drinks carried through all our events from start to finish.

Despite being started by librarians exclusively (or perhaps because it was started by librarians, at a particular time when GLAM was in vogue), newCardigan was always positioned as a "GLAM" organisation. This got a huge boost when Nik McGrath, the archivist at Melbourne Museum, joined the organising team. Nik's boundless enthusiasm and deep network of archivists and museum people opened up opportunities to connect GLAM workers from across Melbourne and beyond.

But this is all just a description of logistics and reach. It doesn't, by itself, explain why so many people were excited about newCardigan in a way rarely expressed in relation to formal professional associations. For that, we need to explore what newCardigan did inside people's heads.

A complaint

newCardigan was essentially a reaction against what we felt was a stuffy, self-important and overpriced culture of professional development and professional representation. Conference tickets were expensive and usually only available to more senior staff. The rare local events that associations ran tended to come with a steep ticket price and a few cliques that you and I weren't in. And trying to organise anything else through those associations felt like running into a brick wall of bureacracy.

We figured we knew how to run events (since many of us did this every week in our workplaces) and we may as well organise things ourselves directly. Being unfettered by any requirement to represent all views in the sectors, we could organise speakers and discussions that were much more direct in criticising practices we felt were harmful. Ideas around institutional neutrality, or the reluctance to account for our insitutions' historical and ongoing participation in cultural violence against Indigenous people and communities.

The lack of formality was deliberate. From the first year our tag-line was "JFDI". We wanted to cut through the management-by-five-subcommittees culture of our organisations, and also encourage ourselves to stop overthinking how to run things. To our surprise it turned out to be extremely easy to talk our way in to free after-hours tours of GLAM institutions. GLAM professionals love to infodump to their colleagues – who knew?

An introduction

For many newly-minted graduates, newCardigan was the first place they met people working in the profession – whether that profession was librarianship, archiving, record-keeping, curation, or some other GLAM-related field. It was also a place to learn about what people in those professions actually do. "GLAM" is usually a fairly nebulous, somewhat aspirational term. It's something peak bodies talk about when lobbying for funds, associations talk about when trying to attract more people to their conferences, and software developers talk about when aggregating the different kinds of cultural collections they can search with API calls.

newCardigan took a bottom-up approach to GLAM. Instead of discussing it conceptually, we just tried to ensure we had a mix of events and guests from across the professions, talking about what they do. Inevitably it leaned heavily towards libraries, but we still provided an opportunity to understand each other a little better across the sectors.

The social aspect was important here as well. Graduates and students trying to find their place could ask more experienced people for advice over dinner and drinks. Connections were made across institutions. It was surprising how many cardiParties were organised through a quick chat with a participant at a previous event. Some people travelled interstate just to come to one of our Friday-night events to connect with their people.

A punk scene

Mostly we just wanted to hang out and learn about cool new GLAM stuff. We didn't consciously set out to create a GLAM punk scene , and for quite a while I didn't realise that's what we were doing. It had to be explicitly pointed out to me – it had just kind of happened.

newCardigan came to hold a peculiar space within GLAM generally, and probably the library profession in particular. We were the GLAM punks who drew in anarchist GLAM-enthusiasts to our events. But we were also the important networking opportunity that LIS course coordinators encouraged their students to join. People attending our events asked for certificates to prove their professional development hours. newCardigan didn't want to be just another part of Official GLAM, or indeed to be any part of Official GLAM. But official GLAM wanted us to be, somehow, part of it.

Social activists and senior library managers alike would enquire about how newCardigan was faring when I bumped into them. One one level it was just a few dozen people hanging out regularly in Melbourne and – for several years under the leadership of Andrew Kelly – in Perth. But on another level it was a force that animated far more than that.

A signal

It was consciously not a representative or official organisation and made no claims to compete with them. Yet it was this very independence that allowed us to speak on behalf of the collective at certain points. newCardigan made public statements in support of both Marriage Equality and the Indigenous Voice to Parliament when our professional associations were either silent or making mealy-mouthed statements about civility and "both sides". I won't over-claim that newCardigan forced significant or specific changes within GLAM generally and librarianship in particular. But newCardigan's very existence was a signal. Whether that signal served as a warning or as a beacon depended on one's perspective. Either way it couldn't be ignored.

The various people who worked on the organising group, spent hours editing podcast episodes (thankyou Clare!), and chased up potential venues and speakers did it because we loved what it meant for the community we built. But after eleven years, newCardigan itself has run out of puff. Covid changed many things including norms around ad-hoc venue visits, working arrangements, and enthusiasm for large social events generally. Lots of people wanted newCardigan to continue; after a while nobody really wanted to run it. Earlier this week we formally agreed to wind it up.

An inspiration

The spirit of newCardigan will live on. Via email or in person many people have expressed to me in the last month how important newCardigan was for them and their understanding of what is possible in the GLAM professions. cardiParties were the place students came for a sense of community when online study wasn't cutting it. cardiCast was where librarians discovered a whole other world of collection management questions listening to archivists or museum curators and preparators. Our newsletter connected GLAM workers to what was happening in other parts of Australia and the world. We brought people together in Melbourne, Ballarat, Perth, Sydney and Sale. We recorded podcasts about a Golden Chiko Roll in Wagga Wagga and refugee libraries in Greece. We were welcomed by Directors through the front door. We were let in by deputy branch managers through the back door. We interviewed University Librarians and Professors. We interviewed volunteer seed library founders and part-time customer service officers. We ran events at esteemed institutions where Australia's elites love to be seen. We ran events at scrappy activist libraries that were probably in breach of fire regulations. newCardigan was large. It contained multitudes.

May a thousand flowers bloom from its memory.

The Evergreen release team are pleased to announce the availability of the Evergreen 3.18 Release Candidate.

The release candidate contains 24 new bug fix patches, 3 more complete features, and 10 additional contributors since the beta was released just 3 weeks ago.

The files and release notes are available on the downloads page: https://evergreen-ils.org/egdownloads

Merge pause is now lifted.

TL;DR Yes, user forums and the Internet Archive are extremely important for digital preservation work. LLMs also seem quite good for synthesizing information that is found in them, connecting the dots between aspects of the problem you are facing, and connecting you to other people engaged in similar work. The irony is that people may use forums less as they engage with generative AI.

Back in June I wrote about how my brother John had died, and how among the things he left behind were over 100 CD backups of music he had created in the early 2000s. Each CD was clearly labeled with a song title, and contained a multi-track backup of a single song, which (I was to learn) was made with a TASCAM 788. He had told me he had these backups years ago, but I’d never seen them until we were emptying out some of the storage space in his apartment building.

John preformed live in different small venues in Pittsburgh in the 1990s, and then in the Doylestown, PA area in the early 2000s. He left behind several guitars, and other musical equipment. But despite my encouragement on many occasions he never really released any of his music in any form, except for one song on Bandcamp, that he had recorded much later in 2022. John’s music was really important to me, even more now that he is gone, so I wanted to see if I could retrieve these songs, and perhaps put together a collection for family members, and other people who were interested in hearing his music, and remembering his creative spirit.

Unfortunately the CDs use a proprietary format, that can only be read using the TASCAM 788’s optional CD burner. As I noted in the previous post I did figure out a way to rip the raw data from the CDs and reverse engineer a program to read the tracks, as well as the final mix. It is far from perfect because the gaps between tracks are not consistent. Sometimes the program would extract part of a song as a track. It also wasn’t clear what the final mix was. Still, it is possible to pull the concatenated tracks off as a single WAV file, which was amazing to hear. But I was hopeful I could get the tracks as they existed in the backups.

John left his TASCAM 788 in the storage unit too, along with an (optional) external CD burner for creating the backups. My brother Matthew actually thankfully kept it. However the devices looked quite different in style, and I think I mistakenly thought the CD recorder was part of some old computer equipment, and donated to a local record store, along with part of his CD collection. Once I realized the CD recorder’s significance for recovering John’s music it was too late to get it back. So I bought one on Ebay for a little less that $200.

Once I got the CD burner hooked up to the TASCAM 788 I discovered that I could read the CDs and see the tracks, and listen to them, but not export them. While it’s possible to export each track by hand by connecting the audio output directly to a soundcard with digital input, there are over 100 CDs, so I was hoping I could easily extract the track and the final mix as a WAV file. My goal is to be able to view these in another DAW on my laptop instead of learning how to use the 788 itself.

In conversation with Claude I learned that I could upgrade the firmware from v1.10 to v2.02 to get a new Export to WAV function on the 788, and that would let me easily burn the tracks to a CD, which I could then read on my laptop. Claude also pointed me at a TASCAM forum where the upgrade was being discussed. Unfortunately the other TASCAM site where the upgrade was located was now gone, and while the page was in Wayback Machine, the actual ZIP file it linked to was not. While I could’ve pieced some of this information together given enough time it was actually very useful to learn about it in the context of my questions and conversation.

One of the helpful people on the forum seemed to be responding over the years to requests for the firmware update file. So I responded on the forum asking if anyone still had it available. I ended up hearing back about a month later, and was able to flash the 788, and like magic, it now has a new Export to WAV function. Since lots of copies keeps stuff safe I placed a copy of the 788 V2.02 Firmware Update up at the Internet Archive.

But, alas when I went to use the Export function it assembled the WAV file on the TASCAM but then gave a PANIC message when actually trying to write to the CD. I tried a few times hoping it was a temporary glitch, but it failed with the same result at the same time. Again according to Claude apparently CD-RW devices can retain the ability to read, but no longer be able to write due to a weakening laser diode. The CD-RW I found on Ebay did seem kind beat up honestly. So I rolled the dice and dropped another $175 on Ebay in the hopes that this one would work better. I’m still waiting for it to arrive.

I guess if this new one doesn’t work as well I will fall back to using the reverse-engineered Python script to extract WAV from the raw CD rip and work with that instead. It has been therapeutic spending time with these backups, especially because it was clear that John created them knowing they would be accessed some day. It also is heartbreaking, thinking that he’s no longer around to talk to about them.

It’s hard for me to imagine having done the work thus far without the TASCAM user forum, the help of the Wayback Machine, and helpful people on the Internet. The generative AI (in this case Claude) also made searching for this information while asking questions more cohesive. I think in the past I’ve argued that search results, and original sources are critical to understanding, and I stand by that. But being able to have a conversation about an information need, and the ability to ask for references for verification (as fraught as that is) is extremely useful. My worry is that if LLM systems don’t prioritize giving users accurate sources, and encouraging their use, then we will see forums like this degrade over time.

Our team is heading to BCN ready to promote open, sovereign and democratic technologies. Here's what you need to know to get involved and find out more.

The post OKFN at Barcelona Open Tech Week 2026: Follow Our Sessions first appeared on Open Knowledge Blog.

A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.

In the hours following the release of CVE-2026-89078 for the project Ruby, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a double free in Ruby's Onigmo regular expression parser when parsing fails after an alternation, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled patterns. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Miss Ebony Reilly, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.

In the hours following the release of CVE-2026-93577 for the project Ruby, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix an integer overflow in Onigmo's compiled-pattern size calculation, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled regular expressions. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer King Lorenz Adams, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.

In the hours following the release of CVE-2026-97229 for the project Ruby, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix an integer overflow in Onigmo's compiled-pattern size calculation, allowing memory corruption and potential remote code execution through attacker-controlled CI/CD regular expressions in GitLab. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Prof. Jayson Mayert, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

A forlorn businessman resting his head against a wall beside a window. Photo by Andrea Piacquadio, via Pexels.

In the hours following the release of CVE-2026-110264 for the project Ruby, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix an overflow of Onigmo's internal IDs in patterns containing many repeats, allowing memory corruption and potential remote code execution in applications that compile attacker-controlled regular expressions. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Miss Luella Bednar, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

The Evergreen release team is happy to announce that monthly releases 3.17.6 and 3.16.12 are available.

These releases contain fixes for both the public and staff catalog.

Files and release notes are available on the Downloads page: https://evergreen-ils.org/egdownloads/

Thanks to the October release team: Martha Driscoll (NOBLE), Michele Morgan (NOBLE), Blake Graham-Henderson (MOBIUS) Gina Monti (Bibliomation), and Andrea Buntz Neiman (Equinox); as well as everyone who contributed fixes and testing to this release.

Merge pause is now lifted.

We developed JAViD as a group project for the CS832 Human Computer Interaction course at Old Dominion University in Spring 2026, taught by Dr. Bhanuka Mahanama. The group members are Kumushini Thennakoon, Rochana Obadage, and Chameli Dommanige. Our goal was to make dual-participant gaze analysis more engaging using a visualization dashboard. This blog post describes the problem we addressed, how we applied human-computer interaction concepts, tools, and technologies, and the key takeaways from the course.

1. Problem Statement

Many eye-tracking tools focus on single-user study setups, but they offer limited support for comparing multiple synchronized gaze streams. An example of a single-user visualization dashboard is “A-DisETrac: Advanced Analytic Dashboard for Distributed Eye Tracking”. Researchers who study shared attention often write custom scripts or assemble Jupyter Notebooks that combine data cleaning, analysis, and visualization. These workflows can answer relevant research questions, but they require programming knowledge. We designed JAViD to bring the main steps of a dual-participant analysis into a browser-based dashboard. A user can load synchronized gaze data, inspect where each participant looked, examine how the distance between their gaze points changed over time, identify candidate joint-attention episodes, and export summary results. The project was guided by two questions:

Q1: How can we design effective interactive visualizations that reveal both spatial and temporal patterns of JVA from multi-participant eye-tracking data?

Q2: How does an integrated dashboard approach compare to traditional code-based analysis methods in terms of usability, learnability, and effectiveness for JVA analysis?

2. System Design

This section explains the JAViD system, including the task scenario, system architecture, design decisions, and core visualization components. The system was developed based on a screen-based visual search task, with the goal of analyzing JVA between two participants. It outlines how gaze data are prepared and processed, how usability principles inform the interface design, and how interactive visualizations support the exploration of spatial and temporal attention patterns.

2.1. Task Scenario

The task scenario involves two participants engaging in a screen-based activity, such as jointly analyzing visual content or completing a task on a shared display. Eye-tracking data from both participants are recorded as synchronized gaze points on the screen. To ensure consistent spatial alignment, the gaze data are transformed into the screen’s reference frame using a perspective transformation. ArUco markers are used to detect the screen frame in the visual field. Figure 1 shows an example of the data collection setup.

2.2. System Architecture

JAViD uses a client-server architecture built with Python, Dash, and Plotly. As shown in Figure 2, the browser provides the interface for uploading data, adjusting controls, and exploring coordinated visualizations. The Python Dash server handles data processing, joint visual attention analysis, and the construction of figures and results. Labeled request and response arrows show how these components communicate during interactive analysis.

Data collection and spatial preparation occur before the data is uploaded to JAViD. ArUco markers identify the screen frame in the scene recordings, and perspective transformation maps both participants’ gaze coordinates into a common two-dimensional reference frame. Next, users upload the prepared gaze files in CSV or Excel format with an optional stimulus image that provides context for the spatial visualizations.

The server uses pandas and NumPy to read, validate, and clean the uploaded data. The system aligns the participants’ gaze streams by timestamp and interpolates when necessary to accommodate different sampling rates. Using the aligned data, the system computes gaze distances and identifies JVA episodes according to spatial proximity and temporal criteria. They also support cross-recurrence analysis, spatial-temporal clustering using DBSCAN, and the calculation of episode measures and summary statistics.

Dash callbacks connect user actions with the required server-side operations. When users load data or change relevant analysis parameters, the browser sends the inputs to the server through HTTP requests. The callbacks invoke the necessary computations and construct updated Plotly figures, summary tables, and other results. These outputs are returned to the browser, including figure definitions encoded as JSON, where Dash components and Plotly.js display them.

The interface brings together five main coordinated views: scanpath and heatmap, temporal JVA analysis, cross-recurrence, gaze points on the stimulus, and statistical summaries. Shared time and display controls allow users to examine the same recording from complementary perspectives. Users can navigate through the recording, adjust analysis settings, and export figures or results for further analysis.

2.3. Design Decisions

The JAViD interface was designed with established usability principles in human-computer interaction to ensure accessibility for users with varying levels of technical expertise. As shown in Figure 3, the interface reflects key concepts from Nielsen’s usability heuristics and Norman’s principles of interaction design, including visibility of system status, feedback, constraints, consistency, affordances, and mappings.

2.3.1. Visibility of System Status

The interface maintains continuous awareness of the system state through status indicators. The left panel displays the currently loaded file names (e.g., “scene_5_participant_1.csv”) directly beneath each upload zone, so users always know which datasets are active. Participant ID fields (P1 ID, P2 ID) are shown persistently, making the current configuration visible at all times without requiring the user to navigate away from the main view.

2.3.2. Feedback

Immediate and informative feedback is provided at two levels. At the data level, a confirmation banner highlighted in green and reading “Loaded — P1: 81 pts | P2: 81 pts | Duration: 2000 ms” appears upon successful data loading, confirming exactly what has been ingested. At the system level, the Dash developer toolbar at the bottom of the screen surfaces runtime information including server connectivity status, callback activity, and version update notifications (e.g., “Dash update available - v4.1.0”), providing transparent feedback about the underlying application state.

2.3.3. Constraints

Input constraints are enforced to prevent user error before it occurs. File upload zones accept only .xls or .csv formats, as stated explicitly in the field labels (“Participant 1 (.xls or csv)”), preventing incompatible file types from being submitted. The screen dimension fields (Width, Height) and the Global Time Window slider are bounded inputs that restrict values to valid numerical ranges, guiding users toward correct configurations without requiring domain knowledge.

2.3.4. Consistency and Standards

The interface adheres to consistent visual and interaction conventions throughout. Both participant upload zones share identical layout, styling, and labeling patterns, reducing the cognitive load of learning one in order to use the other. Toggle controls for “Show Heatmap,” “Show Trails,” and “Show Stimulus” follow the same checkbox style, and opacity sliders for heatmap, trail, and image layers use a uniform slider component. The light mode/dark mode toggle and the 2D/3D JVA navigation buttons appear consistently in the top bar across views.

2.3.5. Affordance

Interactive elements are designed to signal their function visually. The “Drag & Drop or Browse” upload zones use a dashed-border card style that conventionally affords file-dropping behavior. The “Load Data” button is rendered with a play-arrow icon, reinforcing the action of initiating a process. Sliders for opacity and time window are rendered as draggable tracks with visible thumb handles, affording continuous adjustment. Radio button groups for “Heatmap View” and “Trails View” (P1 only, P2 only, or Both P1 & P2) afford mutually exclusive selection.

2.3.6. Mapping

Real-world scenarios are mapped into the dashboard through supportive interface elements that aid interpretation and usability. A dedicated help section is positioned at the top of the dashboard, providing users with contextual guidance on how to understand and interact with the visualizations in relation to the underlying study tasks. Additionally, the interface includes both dark and light modes to accommodate different viewing preferences and environments. Users can switch between the two modes using a button with a small symbol next to the label. A sun symbol indicates light mode, while a moon symbol indicates dark mode, allowing users to quickly identify the mode selection button.

2.4. Core Visualization Components

JAViD provides five primary visualization components, each addressing specific aspects of JVA analysis.

2.4.1. Scanpath and Heatmap View

The scanpath visualization overlays the gaze trails of both participants using distinct colors to show their sequential visual exploration (Figure 4). The heatmap represents attention intensity across the stimulus. Users can toggle between individual participant patterns and combined patterns to examine convergence and divergence in attention. The visualization also provides adjustable opacity for the heatmap and gaze trail overlays. The stimulus image is displayed as the background to provide spatial context for the gaze patterns.

2.4.2. Temporal JVA Analysis

The temporal analysis view (Figure 5) shows the dynamics of attention coordination between participants. The upper graph displays the Euclidean distance between the participants’ gaze points over time, while the lower graph shows the JVA signal computed using a sliding window. Users can configure the distance threshold used to detect JVA and adjust smoothing settings to reduce noise in the temporal signals. Sustained JVA episodes are highlighted in the visualization to help users identify periods of shared attention.

2.4.3. Cross Recurrence Plot

The cross-recurrence plot in Figure 6 represents the temporal relationship between the gaze patterns of the two participants in a two-dimensional grid. Participant 1's time is shown on the vertical axis, while Participant 2's time is shown on the horizontal axis. Color-coded cells indicate spatial proximity between the participants’ gaze points at corresponding time points. Diagonal patterns can reveal synchronized attention sequences, while vertical or horizontal patterns can indicate differences in participants' attention timing. Users can adjust the sampling frequency and interact with the plot using zoom and pan controls.

2.4.4. Gaze Points on Stimulus View

The gaze point visualization (Figure 7) shows the exact gaze locations of both participants on the stimulus at a selected point in time. Users can navigate through the recording using a time slider and view the current timestamp. Play and pause controls allow users to examine the gaze behavior through animated playback. The stimulus image is displayed in the background to provide spatial context for the gaze points.

2.4.5. Statistical Summary Dashboard

The statistical summary dashboard (Figure 8) provides quantitative measures of JVA across a session. It presents the total JVA time and percentage, the number of distinct JVA episodes, and the mean and median episode duration. The dashboard also includes spatial dispersion metrics and attention synchrony correlation. Summary statistics can be exported for further analysis and comparison across conditions or sessions.

Figure 8: Statistical summary dashboard displaying quantitative metrics of JVA, including total time, episode count, and synchrony measures.

3. Usability Evaluation of the Dashboard

To evaluate the usability of JAViD, we conducted a within-subjects comparative study. Here, participants used both JAViD and a Jupyter Notebook-based visualization workflow. The Jupyter Notebook contained the analytical code needed to create three selected visualizations available in JAViD. Participants first interacted with JAViD and completed a questionnaire that asked them to rate the dashboard functionalities and provide feedback through open-ended questions. Next, they used the Jupyter Notebook, ran the provided code, and generated the same three visualizations. After completing the notebook task, participants filled out a second questionnaire that collected their feedback and enabled a comparison between JAViD and the Jupyter Notebook workflow. We selected Jupyter Notebook as the comparison baseline because it is a common approach for generating visualizations by combining code and narrative text. This comparison allowed us to examine whether JAViD provided improvements in usability, learnability, and effectiveness over a traditional code-based workflow.

We recruited five participants for the usability study, including three males and two females. All participants were graduate students from different academic disciplines. Participants were selected to assess whether users with different levels of coding and domain knowledge could use JAViD to analyze JVA data. We summarized the feedback from the usability evaluation participants below.

3.1. Usability of JAViD

The usability study results indicate a strong positive response toward the dashboard across all evaluated dimensions. The majority of participants either agreed or strongly agreed that the system was easy to use, had a clear layout, and was easy to navigate. Participants also reported that the visualizations were interpretable and that the dashboard effectively supported their understanding of JVA. All participants indicated that the system was easy to learn, highlighting its accessibility for users with limited domain knowledge and coding experience. Figure 9 presents a summary of the findings.

Despite the overall positive feedback, the Cross Recurrence Plot was identified as the most difficult section to understand. This suggests that more complex visualizations may require additional guidance or explanation. Overall, these findings demonstrate that the dashboard provides a usable and intuitive interface for JVA analysis while also identifying opportunities to improve the interpretability of more advanced visual components.

Figure 9: Results of the usability study showing participant ratings across key usability aspects (ease of use, clarity, navigation, and interpretability).

4. User Feedback on JAViD

This section presents the participant feedback on the JAViD dashboard based on the layout, interactivity, complexity, and usability. It discusses what participants liked about the interface, the challenges they faced when interpreting the visualizations, and their suggestions for improvement. The feedback is helpful to identify the areas that worked well and ways to make the dashboard easier to understand and use.

4.1. Structure and Organization

Participants consistently praised the dashboard’s clean and well-structured layout. They noted that the organization made it easy to gain a broad overview of the data and navigate between different components. The logical grouping of visualizations supported efficient exploration and understanding of the data.

4.2. Interactivity

Interactive features, such as sliders, opacity controls, and participant toggles for P1 and P2, were highlighted as key strengths. These controls enabled users to dynamically explore the data and adjust visualizations in real time. Participants found that these features enhanced their interaction with the data and provided flexibility during the analysis.

4.3. Complexity

The Cross Recurrence Plot (CRP) was identified as the most challenging component of the dashboard. Four of the five participants reported difficulty interpreting it. This challenge was largely attributed to unfamiliarity with the underlying concept rather than issues with the interface itself. The feedback suggests that additional explanations could help users understand the purpose and interpretation of the CRP.

4.4. Domain Knowledge Gap

Some participants experienced difficulty because of limited familiarity with eye-tracking data and JVA concepts. This suggests that certain challenges were related to domain knowledge rather than the usability of the dashboard. Users who were less familiar with these concepts may require additional explanations to understand some of the visualizations and analytical measures.

4.5. Contextual Help

Participants expressed the need for additional support features to help them understand complex visualizations. Suggested features included hover tooltips, in-graph explanations, and a guided tutorial or onboarding process. These enhancements could provide users with information about the visualizations and metrics without requiring them to leave the dashboard.

“Maybe there could be a hover tutorial option where, if you hover over a specific graph or chart, it explains how the data was put into it.”

4.6. UI Sizing and Layout

Participants suggested improvements to the sizing and layout of some interface elements. Feedback indicated that font sizes and icon sizes could be increased and that some controls could be positioned closer together. In particular, participants suggested making the drag-and-drop icon more visible and placing the P1, P2, and opacity controls closer together to make them easier to access. These changes could improve readability and usability across different screen sizes.

“I felt some fonts were smaller when looking at the screen. The drag-and-drop option would be a bigger icon to find easily. Options for p1, p2, and opacity could be close by to use.”

5. Key Takeaways

The project demonstrated the practical value of JAViD while also providing several important lessons from the group development experience. The usability evaluation showed that JAViD can make JVA analysis more accessible by reducing the technical barriers associated with traditional tools such as Jupyter Notebook. Participants without programming backgrounds were able to complete analytical tasks, suggesting that the dashboard can support a wider range of users in exploring and understanding JVA data. Working as a group also helped us recognize and make effective use of each member’s strengths. Kumushini contributed domain knowledge in JVA, Rochana contributed technical skills for developing the dashboard, and Chameli contributed through background research and development of the dashboard. Through the project, we also gained a clear understanding of how human-computer interaction concepts can be translated into practical design decisions within an analytical dashboard. The project further helped us identify research gaps in JVA analysis and data visualization, particularly in making complex attention data easier to explore and interpret. The project also provided a valuable foundation for future research, as pilot studies and early-stage systems such as JAViD can help identify practical challenges, refine research questions, and guide the design of larger and more comprehensive studies.

A short video tutorial on how to use JAVID

For further details about JVA and JAViD, please refer to the links below:

GitHub repository:

• https://github.com/rochanaro/JAViD

In the hours following the release of CVE-2026-93479 for the project NGINX, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a heap buffer overflow in worker script evaluation triggered by crafted requests when regex map captures are referenced before the map output, or when a non-cacheable variable is used in certain string expressions, causing worker restarts and potentially code execution if ASLR is disabled or bypassed. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Mariano Franecki, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

"No way to prevent this" say users of only language where this regularly happens

In the hours following the release of CVE-2026-15176 for the project NGINX, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a heap buffer overflow in a worker process triggered by crafted requests when alias is combined with rewrite and $request_filename is referenced by set, return, or an access log format, causing worker restarts and potentially allowing code execution if ASLR is disabled or bypassed. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Ms. Odie Leuschke, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

"No way to prevent this" say users of only language where this regularly happens

In the hours following the release of CVE-2026-97205 for the project NGINX, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a use-after-free in a worker process when asynchronous file I/O is enabled in certain configurations and an SSI page containing include virtual commands is processed, causing memory corruption or a worker crash and potentially allowing code execution if ASLR is disabled or bypassed. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Mariano Franecki, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

Cadey

I am so tired

Take a look at the new hands-on course from School of Data, designed for educators and community practitioners. The course focuses on accuracy, source-grounded information and AI with conversational guardrails.

The post New Course: How to Build a Responsible AI Chatbot Using Open Technologies first appeared on Open Knowledge Blog.

It is good to meet with everyone at the @WebSciDL meeting.

— Md Javedul Ferdous (@jaf_ferdous) January 26, 2022

Reachable Is Not Usable

The Research Challenge

RQ1: How can proxy interfaces support efficient non-visual navigation in arbitrary websites for blind screen reader users?

RQ2: How can proxy interfaces improve non-visual interaction with structured web data, such as product lists or other data records requiring comparison?

ACM IUI, "Where #HCI meets with #AI". A premier venue for reporting outstanding research and development on intelligent user interfaces, I was honored that one of our papers was accepted.

#a11y #IUI2022

— Md Javedul Ferdous (@jaf_ferdous) May 6, 2022

Today, I will be presenting our #IUI2022 paper at the "2022 Web Science & Digital Libraries Research Group Expo" this afternoon.

#a11y

— Md Javedul Ferdous (@jaf_ferdous) April 13, 2022

A phd student, @jaf_ferdous is currently presenting his Ph.D. candidacy exam (Oral Component). The title of his presentation is "InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records."

— Mohan Krishna Sunkara (@mk344567) July 20, 2023

RQ3: How can proxy interfaces enhance blind users' ability to understand, follow, and participate in online multi-party discussions where context and reply relationships are critical?

The Proxy Interface Pattern

• sTag identifies user-defined page segments and turns them into a reusable navigation structure
• InSupport identifies data-record and auxiliary segments and turns them into a queryable comparison interface
• Standpoint identifies argumentative structure in a thread and turns it into position-scoped summaries and relation labels

Publications

• RQ1. Md Javedul Ferdous, Sami Uddin, Vikas Ashok. "Semantic Table-of-Contents for Efficient Web Screen Reading." ACM SAC 2021.
• RQ2. Md Javedul Ferdous, Hae-Na Lee, Sampath Jayarathna, Vikas Ashok. "InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records." ACM IUI 2022.
• RQ2. Md Javedul Ferdous, Hae-Na Lee, Sampath Jayarathna, Vikas Ashok. "Enabling Efficient Web Data-Record Interaction for People with Visual Impairments via Proxy Interfaces." ACM Transactions on Interactive Intelligent Systems 13(3), 2023.
• RQ3. Md Javedul Ferdous, Akshay Kolgar Nayak, Yash Prakash, Nithiya Venkatraman, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok. "Understanding Online Discussion Experiences of Blind Screen Reader Users." International Journal of Human–Computer Interaction, 2025.

• Akshay Kolgar Nayak, Yash Prakash, Md Javedul Ferdous, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok. "Examining Inclusive Computing Education for Blind Students in India." ACM SIGCSE Technical Symposium 2026.
• Satwik Ram Kodandaram, Mohan Sunkara, Md Javedul Ferdous, Faryaneh Poursardar, Vikas Ashok. "Unveiling Coyote Ads: Detecting Human Smuggling Advertisements on Social Media." ACM Hypertext 2024. (blog post)
• Utku Uckun, Rohan Tumkur Suresh, Md Javedul Ferdous, Xiaojun Bi, I.V. Ramakrishnan, Vikas Ashok. "Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users." ACM UMAP 2022. (blog post)

Human smuggling is a pressing global issue. On the US-Mexico border, 'coyotes' increasingly turn to social media to advertise their illicit services. Effective countermeasures require detecting these ads. (1/5)

— Md Javedul Ferdous (@jaf_ferdous) December 30, 2024

Teaching

Grateful to be part of the ODU data science bootcamp, surrounded by diverse minds from students to professors across various disciplines. Teaching with diversity brings challenges but immense satisfaction.

— Md Javedul Ferdous (@jaf_ferdous) May 10, 2024

Excited to serve as a judge at the Virginia State Science and Engineering Fair! A great honor to witness the innovative projects and bright minds shaping the future.

— Md Javedul Ferdous (@jaf_ferdous) April 13, 2024

Service and Community

Excited to announce that I will be serving as Associate Chair for the Research and Poster tracks at the upcoming @ACM_CSCW 2025, July cycle! Consider submitting your work to one of the premier HCI venues.

— Md Javedul Ferdous (@jaf_ferdous) June 19, 2024

Happy to be selected as a student volunteer(SV) two years in a row for #ASSETS23. It is the main venue for researchers to present research on the design, assessment, and teaching of computers for individuals with disabilities.

— Md Javedul Ferdous (@jaf_ferdous) July 13, 2023

I'm happy to announce I've been awarded the prestigious Dr. Hussein Abdel-Wahab Memorial Graduate Fellowship by the @oducs.

— Md Javedul Ferdous (@jaf_ferdous) February 6, 2022

Web archives are like time capsules of the internet, but are they accessible to everyone? ODU researchers are tackling this issue, focusing on improving usability for visually impaired users.

#a11y #GAAD

— Md Javedul Ferdous (@jaf_ferdous) May 16, 2024

The Defense

• Dr. Vikas Ashok (Director), Old Dominion University
• Dr. Michele C. Weigle, Old Dominion University
• Dr. Michael L. Nelson, Old Dominion University
• Dr. Hae-Na Lee, Michigan State University

@jaf_ferdous is now defending his PhD thesis: Proxy Interfaces for Enhancing Non-Visual Web Usability.

@accessodu @WebSciDL @oducs

— Akshay K Nayak (@AkshayKNayak7) July 23, 2026

Lessons Learned

1. Do not measure yourself against whoever is closest to finishing

2. Life does not pause for the degree

3. Keep the people at home close

4. Do the unglamorous community work, and do it early

5. Find out which part of the job you want before you have to choose

Looking Forward

Teaching at Ottawa University

Where the research goes

Staying in the community

Acknowledgment

In the hours following the release of CVE-2026-94300 for the project BIND 9 named, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix memory corruption in named triggered by replaying captured GSS-TSIG-signed UDP messages while the negotiated TSIG key remains valid, potentially allowing remote code execution when combined with crafted queries to an attacker-controlled zone. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Queen Aniya Rippin, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

In the hours following the release of CVE-2026-111813 for Apache Traffic Server, site reliability workers and systems administrators scrambled to patch systems running its experimental magick plugin. In its default unsigned configuration, the plugin lets unauthenticated clients control arguments passed to ImageMagick convert, allowing anyone to execute whatever command they want on the server. The affected plugin is written in C++, whose users were relieved that this time the attacker did not even have to corrupt memory to tell the server what to do. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Friedrich Frami, whose image processing plugin accepted instructions from anyone who could construct a URL. "It's a shame, but what can we do? There really isn't any way to know whether a stranger on the internet should be allowed to choose command arguments for a program running on our server. We made signature verification optional, so we've already done everything we can." At press time, administrators installing versions 9.2.16, 10.1.5, or 10.2.1 to fix the issue were referring to themselves and their situation as "helpless."

In the hours following the release of CVE-2026-111497 for the project Brotli, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix an out-of-bounds write in the Brotli decoder that permits remote code execution when processing malformed compressed streams. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Dr. Lester Cremin, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

In the hours following the release of CVE-2026-109618 for the project PHP, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix multiple memory safety vulnerabilities that allow attackers to execute arbitrary code on your server. This isn't something you can just mitigate and be fine. You MUST patch this. Don't try to bargain here. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Lady Miracle Kreiger, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

In the hours following the release of CVE-2026-107891 for the project musl, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix multiple integer overflows in regcomp() when compiling untrusted regular expressions, leading to undersized allocations and possible out-of-bounds heap writes, denial of service, and arbitrary code execution. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Dr. Otho Conroy, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

On October 1, 2026, the Library Innovation Lab’s Public Data Project hosted a panel of legal, technological, and policy experts at the Lewis International Law Center to consider how preservation of and access to US Census data can be appreciated and reimagined. The transcript below has been lightly edited for clarity.

Panelists:

• danah boyd, Professor of Communications, Cornell University
• Cynthia Dwork, Professor of Computer Science, Harvard University
• Ruth Greenwood, Assistant Clinical Professor of Law, Harvard Law School
• Jayshree Sarathy, Assistant Professor of Computer Science, Northeastern University
• Moderator: Molly Hardy, Project Lead, Public Data Project

Welcome

Amanda Watson: I’m Amanda Watson, the Dean for Library and Information here at Harvard Law School. Many years ago, I was sitting at a table with authors and editors, and we were playing a little game, and one of the questions was, “What’s the most niche hobby you have?” And I said, “Well, I really love census data, and I like to take it and figure out things like how close one thing is to another and what it might mean.” And a man who was written up in the New York Times for doing the Sunday New York Times crossword puzzle in pen said, “Wow, so you’re like a nerd’s nerd.”

There are a lot of things I can say to welcome this panel and start this event, but the thing I want to say is, to all of you nerds’ nerds in this sanctuary for nerds, with another sanctuary for nerds behind us: thank goodness we still care about the census. That we care to ask the question, who are we? That we care to count, and then let brilliant people do things with that data and information. We don’t always get it right, but that we still care to ask, I think, is so pivotal. So I want to just say, hey, let’s nerd out together and celebrate the census and talk about why this matters to us and how we can move forward. I’ll send it to Molly now, and thank you all again for being here.

Molly Hardy: Thanks, Amanda, for inspiring words, always. And thank you to our audience, both here in the room and online, for being here today. And most of all, thank you to our panelists, who have taken time out of their busy schedules to discuss important matters with us. So many factors converge to make this the right time for the Public Data Project to host this event. Some are joyous and some are less so.

I’ll start with the good news. First, danah boyd’s book Data Are Made Not Found: A Story of Politics, Power, and the Civil Servants Who Saved the US Census (2026) has just come out. If you’re in the area, please join her tonight at the Harvard Bookstore, where she will be in conversation with Yochai Benkler, the Berkman Professor of Entrepreneurial Legal Studies at Harvard Law School.

I’m also pleased to share that our friends at the Internet Archive’s Democracy’s Library have just digitized microfilms of the census from 1790 to 1930, as well as the Indian census rolls of 1885 to 1940. This film was originally produced by the National Archives and Records Administration. The collection holds 24,320 items and more than 47.8 million page images, and it is a delightful treasure trove for researchers of all types.

Behind us, you’re seeing a selection of photographs from the Library of Congress that will slowly move through as we discuss today. Just some of my favorite images of census workers at their job. I thought it was a particularly good tribute in light of the subject of danah’s book, which is, of course, the civil servants.

Now for the not-so-great news. The census has come under scrutiny, and not the good kind of scrutiny. The Office of Management and Budget is reviewing significant changes for the 2030 collections: one, reclassifying noncitizens as nonresidents; two, a rollback of race and ethnicity data collection standards; and three, the addition of a citizenship question. Changes to privacy protection have also been proposed, and those changes would have profound effects on how much census data can be shared.

Groups are mobilizing to call attention to the perils of these changes. If you’ve not already, I encourage you to check out a couple of them, Hands Off Our Census and The Census Counts. I’m mentioning just a few of the groups who are mobilizing as the October 13 public comment deadline draws near.

The census contains multitudes, both literal and figurative. We’re honored today to be joined by leading lights to help us place in perspective what a threat to the collection, as well as the preservation of and access to, this vital dataset must mean, and how it should be appreciated and reimagined. At this moment, I can think of no better way to appreciate public data anew as we testify and retestify to the census’s importance as what Dan Bouk simply refers to as “democracy’s data.”

Poster encouraging Americans to participate in the 1940 census. Source: Library of Congress.

Panelist Introductions

Molly Hardy: Now, without further ado, I’m going to turn to the panelists, introducing them and then asking each to share their own professional contributions and work with the census and how we understand it. All four of our panelists are incredibly accomplished, so please pardon me for keeping introductions short so that we can proceed to the heart of the matter.

First, we’ll hear from danah boyd, who is the Gary Gay Professor of Communication at Cornell University. Dr. boyd’s research focuses on the intersections of technology and society, with an eye towards how structural inequalities shape and are shaped by technologies. How fitting, then, that her latest book, Data Are Made, Not Found, is a story of the civil servants who work tirelessly, and increasingly under duress, to produce the census.

Next, we’ll hear from Cynthia Dwork, who is the Gordon McKay Professor of Computer Science here at Harvard at the John A. Paulson School of Engineering and Applied Sciences, and an affiliated faculty member here at the law school. Many of you know Cynthia for her pioneering work in differential privacy and its applications in protecting personal privacy in the census.

I encourage you to check out the editorial she published this summer in The Boston Globe, co-authored with our next panelist, Ruth Greenwood, who is an assistant clinical professor and director of the Election Law Clinic here at Harvard Law School. An expert in both election law and redistricting, Ruth previously co-directed Voting Rights and Redistricting at the Campaign Legal Center. She was lead counsel for voting rights at the Chicago Lawyers’ Committee for Civil Rights Under Law and was a redistricting fellow with the Democratic National Committee’s Voting Rights Institute.

And finally, a welcome home to Jayshree Sarathy, Assistant Professor of Computer Science at Northeastern University. Jayshree completed her PhD here at Harvard in Computer Science, where she was advised by Salil Vadhan and Jonathan Zittrain. And just to bring things full circle, in 2022, Jayshree and danah jointly published an article titled “Differential Perspectives: Epistemic Disconnects Surrounding the US Census Bureau’s Use of Differential Privacy.” Her work focuses on the sociotechnical dynamics of data science, with a focus on privacy, trust, and data access, all areas very germane to our conversation today.

So let’s please get started with danah — if you could just start by telling us about your engagements with the census.

danah boyd: Sure. Thank you all for coming out today. Indeed, this book came out last Thursday. I’m an ethnographer, which means that my work is to think about mapping cultural practice. This book is the culmination of 10 years of trying to understand what makes data legitimate, in the most high-stakes data we have. After all, if you think about it, it is a very weird thing that we have actually architected the US Constitution to anchor our representative democracy in the idea of collecting data and allocating political power accordingly.

And so for the last 230-plus years, we’ve embarked on this project of how to understand who our country is in the form of the population. This has had lots of crazy twists and turns. Margo Anderson wrote a brilliant book about the social history of the US census, so if you want to understand its history, because it’s crazy, everything you thought you understood about the US, you can actually read through the census. It’s fun.

But my goal was to understand what it takes to actually do this in real time. So I embedded inside the US Census Bureau to really appreciate what they were doing. Part of it is the recognition that we don’t often acknowledge the important work that civil servants do. They are the invisible workers. They are the people that just do the work on behalf of the American people, but they are constantly maligned.

I grew up in this country constantly being told that civil servants were stupid or malevolent, incompetent, that it could be done better by industry, that this is just the deep state. You’ve heard all of it. And civil servants unfortunately often even believe it, which is actually terrifying. But these people are so dedicated to a mission: let’s make certain that we have the best count possible. And so I wanted to tell their story. This book is character-driven. It tells their story in different ways through the various ups and downs of the 2020 census. So if you want to know about citizenship, if you want to know about residence rules, if you want to know about differential privacy, it’s here for you.

For the arguments in this book, I have three threads, and I want to highlight them to give you a sense. The first is that data are made, not found. That’s the title. But the point is that it’s not just, “Oh, the categories are socially constructed.” It’s a series of ongoing decisions, made with an eye to making the best data possible. To make that really concrete, I went back to 1880 — I was just too lazy to go further — and I looked from 1880 to 1950 at the age variable, how old are you, within census data. And it turns out that you get these spikes in years ending in zero and five, because humans round their age. They’re not 72, they’re 70.

And they don’t just round about their own age. They round about their housemate’s age. They round about their neighbor’s age. They don’t tell enumerators their actual age, and the enumerators are often too scared to ask them details. So they just look at you and are like, “You look like you’re 50. That sounds good.” And we move on. The Census Bureau receives this data, and if they published it as is, people would think they were nuts. So they go through all of these procedures, not just collecting the data but figuring out how to make it into democracy’s data. That’s the first big through line.

The next big through line is an argument about Jenga politics, which is going to be relevant today. Jenga is that wooden tower. Think about it as our administrative state, as people have been taking out pieces from it and adding pressure on top. People have been doing this for all sorts of different reasons, both Republicans and Democrats, for a long time. I tell lots of cases of where this occurs, where help is not helpful. Things that will also upend your idea of, “Oh, but participatory engagement, this is really helpful” — except when it’s not.

Civil servants are running around trying to keep that wobbly tower from falling over, but they need help. And I’d like to remind everybody, somebody will get blamed when the tower falls, and people will celebrate because it’s not their political party or whatever, but we all lose, and we all contributed to it. We need to understand how this Jenga tower is falling apart. If we care about democracy, and if we care about data, this becomes really important.

The final through line has to do with what makes data legitimate. You might have heard lots of stories about the 2020 census. You might be convinced that it was a terrible census. Here’s the thing: the data for the 2020 census were better than all data collected in the 19th and 20th centuries by the censuses. They were maybe more problematic than what happened in 2000 and 2010. But when you interrogate that, it turns out the problem is not differential privacy. It is not radical malfeasance or political dynamics. It is that a pandemic is a problem. It’s really hard to collect data during a pandemic.

The legitimacy of data isn’t rooted in some magical metric of data quality. It’s rooted in the social fabric that we build, the trust infrastructure, the ways we understand the systems.

But what does it mean that we believe these data to be crap? It’s really important to understand the consequences of our delegitimization of this. The legitimacy of data isn’t rooted in some magical metric of data quality. It’s rooted in the social fabric that we build, the trust infrastructure, the ways we understand the systems. I try to help you see which attacks on legitimacy are attacks on the social fabric. Because if we want to have nice things, we have to attend to the social fabric and not just think that it’s going to get magically fixed through law or through policy. It is going to be a lot of relational work, and that is as true for our data as it is for our broader political polarization issues.

Harper’s Weekly illustration of a man taking the census (1870). Source: Library of Congress.

Cynthia Dwork: So why don’t we start with what differential privacy is, because you may or may not know. Differential privacy is a mathematical guarantee. It says that the outcome of any statistical analysis is essentially equally likely whether any individual opts in or opts out of the dataset. So where does this likelihood come from?

The answer is we achieve differential privacy by the introduction of carefully calibrated noise. If the true answer for some enumeration is, say, 10,000, you might get an output of 10,002 or 9,996. And if the true enumeration is 10,001, you might also get those same numbers, just with slightly different probabilities. That’s where the randomness comes in, and that’s what “essentially equally likely” means. In the context of the census, this means that any collection of published statistics is essentially equally likely whether you chose to respond to the census or you decided not to. That’s all.

The idea is that if outcomes happen with essentially the same probability whether you joined or not, they’re really not leaking much information about you personally, because they wouldn’t have leaked it if you hadn’t joined. For any statistic or collection of statistics, there are zillions of potential differentially private algorithms, and the algorithms vary in their accuracy and noise trade-offs. Some might be optimized for one kind of data, some for a different kind. But the mathematical guarantee is always absolute.

So differential privacy does not mean the particular top-down algorithm that the Census Bureau used. It’s just a mathematical definition of privacy, and different computational methods will give you different instantiations of it. I wanted to mention also why it’s called “differential.” Everybody receives the same mathematical guarantee, and the term “differential” actually comes from calculus, where it refers to a small change in a quantity.

In the early days, we had a bound on these relative probabilities, and the bound had some name, say epsilon, and we didn’t know whether to call it epsilon privacy or epsilon non-privacy. Mike Schroeder suggested, “Why don’t you call it differential privacy?” And that seemed to be exactly mathematically the right thing. But everybody gets the same privacy protection. I think I’ll stop there.

Ruth Greenwood: I’m not good at math, so it’s quite intimidating to do anything after Cynthia Dwork. But I do know how to work with incredible people. My name is Ruth; I run the clinic here. I’m from Australia originally, but my career in America has been litigating voting rights and democracy cases.

When we think about democracy, we may all have different conceptions of it, but it will probably include some notion of the people and some notion of decision-makers. And in order to know who the people are in a democracy, you need information, you need data. Here I have printed out the Federal Register proposal to change the census, just because it makes me mad. There were parts where I wrote “CD,” as in “Cynthia Dwork needs to deal with this,” and other parts where I had my own thoughts.

Race is a social construct, but once you have constructed it, it has deep meaning in our society. To put your fingers in your ears and pretend it doesn’t exist gets you nowhere.

It quotes people like Justice Thomas saying that race is a social construct. So what they’re proposing to do is remove the race and ethnicity questions, among other things, as in prohibit them. As I like to say to my students, race is a social construct, but once you have constructed it, it has deep meaning in our society. To put your fingers in your ears and pretend it doesn’t exist gets you nowhere. In fact, it moves you backwards.

It’s the same thing when I talk about partisan gerrymandering. If you just look at a blank map of Massachusetts and draw some circles, without considering where there are Ds and Rs, you’re not going to know what you get. You know what’s better? Knowing what you’re going to get, caring about the values that go into that, and making an informed decision. So it’s very important to me that we have a sufficiently accurate census that also has privacy protections.

The way I came to know about Cynthia’s work was that there was this concern before the 2020 census — and this is all in danah’s book — from the nonprofit world. We really want accurate race and ethnicity data, because the Voting Rights Act — rest in peace after Callais — used to protect the right to vote for people of color. And you need to know where people of color are in order to protect them.

So all the nonprofit groups would help to do what they called “get out the count.” Let’s just make sure people get counted. But if you do something like say we’re going to have a citizenship question, that freaks everybody out. Not only do non-citizens not want to fill out a census, but citizens don’t want to fill out a census either, and then we get garbage data.

And the other side of it is, before the citizenship question came out, a nonprofit could go to communities and say, “Even if you are undocumented, you can fill out this census, and it’s important that you do, because there is this protection of privacy.” When I learned about the mathematical guarantee, I was like, “That’s pretty solid protection.” I can tell you this smart person invented this thing to protect your privacy. And so eventually all the civil rights groups helped to get out the count, to make sure that we had as accurate an understanding as we could of the race and ethnicity of people in the country.

And that led to not just, can we draw congressional districts that will elect a candidate of choice of Black voters? That’s important. But it matters for the state senate, and for the state legislature. It matters for school boards, public utility districts, port authorities. We elect so many things in America, and there are so many ways to have distortions in representation. I would prefer to be having a fight about whether this is the form of representation we want or that is the form of representation we want, rather than saying all the data is garbage and it doesn’t even matter what form of representation we want, because we couldn’t even assess whether we have it.

So that’s my interest in the census, and seeing the Federal Register notice was deeply distressing. One of the things people have said in the states after Louisiana v. Callais, which is the decision that essentially killed the Voting Rights Act, is: are there things we can do to help protect ourselves? I started saying, put into state legislation, just run your own census — this may be crazy, but people used to run their own census — if you want to have an accurate ability to protect. It won’t work for Congress, because you need to have accuracy between states, but for the sub-state level, having your own census is the only thing I can think of to be more helpful. So that’s where I’m at.

Jayshree Sarathy: I’m very excited to be in conversation with these amazing scholars who have worked on this topic longer than I have. I came first to differential privacy in my graduate work, about a decade after Cynthia and others had coined the term. I was really thinking, in 2018, 2019, about improving the statistical and algorithmic analyses that we can do with differential privacy, and at the time with an eye to how it might fit the census use cases or other use cases for federal or official statistics.

A lot of that work was in conversation and collaboration with census scientists, and I was really struck by how committed the scientists from the Census Bureau were, how knowledgeable they were, how much innovation was coming out of government. And this is beautifully documented in danah’s book. I was struck by how much work and thought was put into how we can take this definition of privacy, this commitment to privacy, and make it work with the many legal and bureaucratic and operational constraints that go into something as complex as the census.

In that process, I also had this moment of doing a deep dive into the census world and understanding the very interesting dynamics around how statistics and politics interact, and just how infrastructural the census is to our democracy as well as to science in the US. Since then, my research has shifted. In computer science, we were working on all of these exciting algorithmic techniques for differential privacy. We were excited that the Census Bureau was so invested in deploying this new technology, and it seemed like a straightforward modernization of privacy.

So I think myself and other computer scientists around me were really shocked when we saw how political and messy this became in the public. We were like, “This makes a lot of sense to do, and this is a natural progression of the technical advancements of the census. So what is happening? Why is this so contested?”

My research now really tries to understand that, and to bring it back to computer scientists. How can we rigorously think about the intersection of technology, especially privacy technology, with these very complex social and political contexts? How can we more deeply understand the context, and work on both designing tools and communication, to make sure that what we design as well-intended technologies doesn’t run into the many different problems it has run into with the census?

Part of that is really understanding, especially with the census, that it was such an eye-opening experience to work in the space, because there are so many different experts and so much knowledge, from the data users, data subjects, advocacy groups, and scientists, all working together on this project to create better data, all with that same goal. And still, there are a lot of challenges with that. So that’s how I came to the census, and my fascination is really with how we can understand and create more legitimate data.

Store room, Census Bureau (1909–40). Source: Library of Congress.

The 2030 Census, Privacy, and Trust

Molly Hardy: Great. Thank you so much to all of you for setting the table so nicely. Technology and context, that’s something we think a lot about here at the Berkman Klein Center, at the Public Data Project, at the Library Innovation Lab, and I think that’s a really good framing here.

So danah, I’m going to start with you, if I may, and ask you about the relationship between democracy and data. Do you think this moment reveals something about the limitations of public data to protect democracy, or is it better thought of as the limitations of democracy to protect public data?

danah boyd: Right. One of the things that’s so fascinating about census history is that when the idea of creating this country and anchoring a democracy to data started — you know that the moment the data made politics, politics were going to make data. That is the entire history here. Ruth gave you a little taste of the Voting Rights Act, but it’s all the way down into all of these different pieces. What has been radically different in recent years has to do with where the site of contestation is.

Let me explain, because there was a previous, really interesting fight over pieces or mechanisms of the census, known as the 1920 case. In 1920, the census came back with reasonable enough data, though it was really hard to collect data during that period because there was a pandemic. At the time, they were having all these technocratic fights. Up until then, the way that Congress did math was, “I don’t want any of my friends to lose their seats in the House. We’ll just add some more seats for the states that grew bigger faster.” It was very funny math. And so people were saying, “Hey, what if we do math better? We can’t fit all these people into the Capitol Building, so let’s stop growing the size of the House.” All of this was coming into being at that 1920 moment.

Then a Harvard economist penned an op-ed on the night before Congress was supposed to meet and decide the reapportionment of the House. Some Southern Democrats were like, “Sweet, we like this economist.” His argument was basically that Congress was doing the wrong kind of math. So the Southern Democrats said, “We can’t possibly reapportion the House until we figure out the right kind of math.” They didn’t figure out the right kind of math, and in 1929 the Senate got so annoyed with the House that they pushed through a law that we think of as the Permanent Reapportionment Act.

There are many weird things about that law, but one that is very relevant to the present is that they said, “Well, Congress is so dysfunctional. Despite the fact that the Constitution gave Congress this responsibility, we give up, and we will give the responsibility to the president to figure out apportionment, and that will go great.” That, of course, got us into some very interesting fights for 2020.

Every component piece of the 2030 census will be finalized and going to printing by January of 2029. So I want you to understand that the next two years are the most critical years in this process. And the question is, is Congress going to take responsibility?

So I just want to give you a moment of what’s happening right now. That was the story of 2020, but we are living in the story of 2030. For those who don’t know census structures, it takes 12 years to plan and produce a census. To give you a sense of the timeline, every component piece of the 2030 census will be finalized and going to printing by January of 2029. So I want you to understand that the next two years are the most critical years in this process. And the question is, is Congress going to take responsibility?

There are three things that happened this summer that I want to flag, and I think we’re probably going to drill into more detail on them. The first, which is relevant to these folks, is that every form of noise injection into the census data has been banned. How they define it is really wonky, and we should talk about Chevron and Loper Bright and a bunch of other things, but one takeaway is: no more differential privacy. The result is that the Census Bureau is now not publishing critical data pieces that we all care about, because they don’t know how to resolve these competing expectations.

The second move was the strangest report you could possibly imagine. Basically, a group of civil society actors working on behalf of pro-Trump Republican allies came into the Bureau and managed to weasel their way into this process, and used a certain set of data to produce a report that was showing all of the non-citizens that voted in the 2020 election. Some of you might know that President Trump is convinced that he won the election if it were not for these non-citizens.

Weirdness number one: the report doesn’t prove his claim. Even if these were non-citizens, it doesn’t prove it. Weirdness number two: this is an unsigned report. No civil servants were involved in the making of it, and they didn’t say who was there. No methodology was given, no explanation of error. And everybody’s like, “What the heck is this?” Both Republicans and Democrats, economists and demographers, have said, “This is insane.” And this is the huge challenge of what it means to think about legitimacy.

The final thing, which is what Ruth has been hitting at, is something called the residence rules. Every decade, the Census Bureau publishes the alterations to the previous residence rules that it wants to make in order to make the next data better. The census is responsible for counting everybody once, only once, and in the right location. The residence rules are the details of how to count people in the right location. In the last couple of decades, the Census Bureau would make tiny micro changes, put them out publicly for comment, and give you thousands of pages of research and documentation telling you why to make these micro changes, which, let’s be honest, nobody ever read. Like the worst academic papers ever. But it was — “See? We did the work. This is why it’s a good idea.”

On September 10, something else happened. New residence rules came down, and they are radical ruptures of everything, complete with a heading that is very political, zero research or explanation of why any of this matters, and all civil servants were told that they are not allowed to speak to the public. And you have 30 days to comment. Comments are due by October 13.

I’ll highlight the top three, and I’ll do it quickly because I suspect Ruth will have better thoughts on it. One is changing the rules of how we count non-citizens. This is going to be a constitutional fight, because the Constitution is clear that you count all the people, but you can make a rule that says you do it differently, and then lawyers will figure it out.

Number two is that we have a lot of laws in this country, especially from the 1960s and 1970s, that are anchored to race data. We did this because we thought we could invert the collection of race data to address discrimination. So instead of changing the laws, they’ll just undermine the existence of the data. That’s just great. That’s the second point that Ruth is starting to talk about.

The third one is even weirder, and since we’re in a geeky audience of nerdy mindsets, I’m just going to lay out one piece of it. The 1790 rule of where to count people is at their “usual place of abode.” I love that word, “abode.” The idea is that you understand where people typically live throughout the year, and that’s where you count them. That’s how we’ve been doing it ever since.

This time, it’s about where you live from January 3rd to April 1st. That’s really weird until you think about snowbirds, people who live in New York except when they live in Florida for the winter. Now they’re to be counted in Florida. Migratory farming populations live in Minnesota throughout most of the year, except for the window in which there’s no farming in Minnesota, so they go down to the South. You have all of these different populations that go to the South during that window. If it was just that, it might make a little political sense, except there’s a really weird twist: you have to match the place where you fill out your census with your tax record.

Let’s start to unpack that. Those snowbirds who go south, they like to file their taxes in Florida, so they’re going to count. We’re great on that one. Those migrant farmers, they’re just going to get erased. Oh, and college students. It turns out no parent lets their child fill out their own taxes, because they just won’t do it well, so the parent fills it out for them, usually not at the same residence as their college. It’s a way of upending procedure after procedure.

That is just some of the mess of these residence rules, with no explanation and no accounting of whether the Census Bureau could even implement this, because there are so many pieces that they can’t implement. They are all silenced. One of the reasons people are asked to write comments, and there are now thousands of them, is to show the anger and to start to have everybody appreciate what’s going on. But litigation is coming, and it’s also about getting Congress to wake up to the fact that they are responsible for this.

So that’s the fight right now. We thought we were going to fight about gerrymandering. We’re now fighting about micro-procedures, about how we actually build our data, and that’s the way in which you bore the heck out of the public and you can bulldoze through people. That’s what’s going on, which is why I thank you for coming today, and I encourage you to understand these pieces, because this is the cornerstone of our democracy.

Molly Hardy: Great. Thank you so much, danah. What you’re saying there, about the minutiae that data requires, the level of attention it requires being a way to silence people, a way to numb people, thank you for drawing some of that out. Ruth, let’s go to you, please. If you want to expand on any of the explanations that danah laid out for us, and/or talk about consequences here, about how you see this playing out.

Ruth Greenwood: Yeah. Let me start with the second part first. The implication of what danah was saying, of not counting migrant farm workers and not counting students, yet counting people who are in the South in the wintertime, is to make sure that southern states, which are traditionally more Republican, have more seats in the House, and the northern states have fewer seats in the House. That’s a direct line. But the implications are massive.

I just want to read a quote that they have in the Federal Register from an earlier case. Admittedly, judges are bad at math and stats. These guys are bad at math and stats. I’m not good at math and stats, but I appreciate people that are good at it, and you, too, can appreciate people that are good at it without being able to do an equation yourself. Anyway, they say, “Although the census is the delight of statisticians and sociologists and serves as a convenient measuring stick for the dispensing of federal funds, it was not created for these purposes.” That’s the justification. Like, look, it’s been really nice to have all this data, but all we need to do is count the number of people, and by people, we really just mean people that we care about, not people that you care about.

I think the implication is, yes, there is going to be litigation, and my concern with litigation is manifold. The main one is that I don’t trust the Supreme Court on anything. The second is that they got it right in Department of Commerce v. New York, the case that said we shouldn’t have a citizenship question, and that came at the last minute because of some files found in the attic of a deceased gerrymanderer. It made it easier for John Roberts and friends to say, “Oh, this is all getting a little bit nasty. It seems like it’s pretty racist, what’s going on. We’re not going to do it.”

Of course, they’re covering it up, and they’re covering it up more and more and more. And the Supreme Court has got to a place where they’re fine with things being relatively racist; see all their recent decisions. So even though I think we got lucky with that prior decision, I’m very worried. And even though, if I gave this as an issue spotter to my students, they’d say, “Yeah, that doesn’t work,” and there are some very obvious holes, like backing up documentation, none at all, it’s got to go through the court system. I think the only slightly useful thing is that it’s still 2026, so there is some time to go through the court system.

But I just worry that we’re going to end up with a bunch of useless data, and that, of course, will have political implications, which seems to be what they care about, but it’s going to have all these other implications as well. I’ve been in touch with lots of the legal community about submitting comments on this, and to some extent the political science community. I was just like, “Please tell me economists and big business are explaining how this is bad for them, because maybe Republicans will care about that and it will stop them from doing this.” I don’t know. I don’t know if any of you have had other outreach about comments.

Molly Hardy: Thank you, and I think we should take that back up. But I do want to return to the questions around privacy and give you, Cynthia, first a chance to respond to these changes that we’ve seen. To oversimplify it, the removal of noise. And if you would maybe speak about how you see this as a, well, I don’t want to put words in your mouth, but perhaps a distortion between the protection of privacy and the gathering of good data.

The proposed changes to the privacy protection mechanisms roll back technology to 1972. … If you refuse to add any noise, then you simply can’t release many statistics while protecting confidentiality.

Cynthia Dwork: The proposed changes to the privacy protection mechanisms roll back technology to 1972. [I have] no words. Why? The noise infusion methods that were developed over the past half century were developed in response to a demand for more data, more granular data, more useful data. There’s a fundamental mathematical conflict, a fundamental law, sometimes called the database reconstruction theorem, that says that overly accurate estimates of too many statistics is blatantly non-private. It completely destroys privacy. This was a finding from Dinur and Nissim in 2003, before differential privacy was created.

So how do you get around “overly accurate”? You add a little bit of noise. If you refuse to add any noise, then you simply can’t release many statistics while protecting confidentiality. It’s just that basic. And by law, they have to protect confidentiality.

Cartoon depicting a man taking the census from an old woman (1890). Source: Library of Congress.

danah boyd: So it’s not a choice about confidentiality or not. It’s that we can’t release the data. That’s kind of what they’re arguing.

Cynthia Dwork: That’s exactly right. And so the demand that these noise infusion methods were created to respond to will simply be unmet. And this is what danah was talking about when she says that the ACS [American Community Survey] is not being published. And it’s because they can’t.

danah boyd: To be clear, the legal reason is there, but there are actually three reasons for confidentiality. I want people to hold on to all three, because they’re important. The first is operational. It turns out people aren’t willing to provide data when it’s going to be released to other people, which is why they set this in motion back in 1840.

And it’s not just about individual respondents. The Census Bureau relies on what are called administrative data, which means other agencies provide them with data that they use for quality assurance and all of these different procedures. If the Census Bureau is unable to protect those data, those other agencies don’t give them data. So it’s operationally necessary; it is legally required; and it is morally the right thing to do, because, of course, we have a long history of moral abuses of these data.

So I want to keep that in mind, because for the Census Bureau, it’s not just a matter of whether to follow the law. Trust me, the law is a big deal. I always loved being in a bathroom at the Census Bureau because, I’m a girl, I pee sitting down, and I stare at a door. And whenever I stared at that door, [on the door] there was a reminder of Title 13 and how I had to protect privacy. So I was reminded of it many times a day. This commitment is really central to what goes on, and it’s why Cynthia’s reasonably aghast at this, because there is no way of resolving this other than to not publish data. And I’d like to think we can have nice things, starting with a democracy. So what do we do with that complexity? That’s why this is so important.

Molly Hardy: That’s great. And I want to come back to the question of what do we do. But I want us to pause for a moment here. We’re talking a lot about privacy, but I think we’re also implicitly talking about trust, about the ways in which we want and need to be able to trust this data. And so, Jayshree, I’m thinking here specifically about the work that you and danah have done about this idea of a “statistical imaginary,” which I just love. If you could talk a little bit about what that is and how it might be operational in this moment that we’re in.

Jayshree Sarathy: Absolutely. We’ve been talking about how privacy is important for data subjects, or respondents, to trust that the Census Bureau will protect their information enough for them to feel comfortable giving it. But there’s also trust in the data itself when it comes out, trust by data users and the broader public. And you come back to this question of what makes data trustworthy, what makes data legitimate. Going back to 2020, and coming at it from a perspective of differential privacy — I actually think differential privacy is a fascinating window into this question, for two reasons.

The first is that, by the structure of DP [differential privacy] — by this theorem that Cynthia referenced and these explicit trade-offs — DP requires data analysts to shift their practices around using data, and a lot of these practices are the ways in which data users feel that the data is trustworthy and legitimate.

Some of our work at OpenDP at Harvard was looking at and showing how differential privacy means that data users are no longer able to access data directly. Often they have to use an interface, or there’s noise included, and so there’s a barrier that shifts the practices by which data analysts had traditionally felt they could trust the data, work with the data, and arbitrarily explore the data before they come to analytical conclusions.

So even if differential privacy is better than previous privacy protections in terms of accuracy, and can also lead to better statistical outcomes, in terms of more robust statistical analyses or protecting against p-hacking or things like that, there is still a sense in which not being able to work with data in the ways that data analysts have been used to for decades creates a gap in trust and legitimacy. And we can’t take away those practices without replacing them with new cultures around data analysis. That’s the first thing that I think DP brings out.

The second, as you mentioned, is that because differential privacy allows us to be transparent about how we add noise, it reveals fictions about data accuracy. danah and I wrote about this, building on a long literature about imaginaries. We talked about the statistical imaginary that we saw among census data users, where they tend to treat the published data as the best enumeration of the population, as fact, rather than as the product of a long and complex process of data collection, statistical editing, and privacy protection. It’s a really sophisticated process, but one that deals with uncertainty at every step. It’s not just a count, it’s a statistical estimate.

And that’s not how the data are used. Part of that, as danah details in her book, is the many pressures on census data. It’s used for so many different things downstream, from voting rights to emergency response to just local politics. When there’s so much pressure put on this data, it’s not easy for data users to think critically about how they work with the uncertainty in the data.

So this creates an opening: now that we talk about noise infusion for differential privacy in a very transparent way, data users are asked to comment on how much error is acceptable for their use cases. And of course, their answer is none. We didn’t have to think about this before, even though error already existed in the data, for privacy protections and other reasons.

Legitimacy is not just about data accuracy. It’s not a function of any quantitative metric about the accuracy of the data … It’s really about these practices around data and the imaginaries, or the perceptions, that data users have about the data.

So, we don’t want any additional error, and we don’t want to have to think about that, and I think that’s reasonable. It really comes back to the point that legitimacy is not just about data accuracy. It’s not a function of any quantitative metric about the accuracy of the data, or even about the response rates of the census. It’s really about these practices around data and the imaginaries, or the perceptions that data users have about the data, and how we can bridge that with internal understandings of the data.

Ruth Greenwood: I just want to ask a question, whoever knows. In terms of getting rid of the injection of noise, I presume it was nefarious, but I’m wondering which type of nefarious. Was it so that we wouldn’t be able to protect privacy and therefore could only have aggregate-level, not local, data? Was it that they’re like, “I don’t understand math, so I don’t want any math in it”? Or was it something else? Do we have a sense? I don’t know who knows this.

danah boyd: So the order comes from the Commerce Department, and in particular it comes from an external organization that for a long time has been drumbeating it through their blogs. They have misunderstood it many times. They think that it’s part of the apportionment piece as well as the redistricting piece, which is amazing. So part of the order is complete misunderstanding, on top of everything else. They believed, very effectively, that by getting rid of both of these things, they could get both Republicans and Democrats who have been in favor of — or have been challenging — differential privacy on their side.

The other thing is that it gave them the logic for the Federal Register notice that is in front of you. One of the things that made this very difficult was the introduction, in 2000, of having multiple race categories as an option. It created a form of complexity in the data that was actually really hard to mask. So they can say, “Ah, we’re going to get rid of this, but because we’ve got this privacy problem, we now can justify the removal of race and ethnicity data.” It becomes part of a story. To be clear, this is not coming from civil servants. This is political all the way down, and I mean partisan in this sense.

One of the things that’s very challenging right now is that a lot of data users who are very anti-differential privacy are having this awkward moment of, “Well, we’re even more [against] not having data.” And there was this fascinating moment, because one of the historians who has been most loud about his hatred of differential privacy — a journalist managed to get him on record saying, “Well, it turns out it wasn’t actually that bad.” Which was about as much of a mea culpa as you’re going to get out of that person. Here’s the quote from the article: “After years of tests, Ruggles said he was satisfied that the amount of noise was so minuscule it wouldn’t compromise data quality.”

Ruth Greenwood: Wow. That would’ve solved a whole lot of problems 10 years ago.

Jayshree Sarathy: Right. The sad part of this is that the admission comes six years later, even though technically it was sort of obvious that differential privacy was not going to impact the quality of the data very much. But those debates have created the fodder for where we are now, and have created this opening for using differential privacy in a political way.

Ruth Greenwood: I feel like I’m partly to blame for that. I was trying to talk to the civil rights groups. I talked to you, Cynthia, and I was like, “This makes sense.” Then I talked to the civil rights groups, and they were like, “We love the thing before, which was swapping,” which, if you don’t do math, is much easier to comprehend. Instead of having one person on Liberty Island and one person in Brooklyn, you just swap them, so then you don’t know if the person on Liberty Island is or is not the race that you’ve put. I get that. Whereas when you say, “Do a cool equation that Cynthia made,” I’m like, “Mm, don’t understand that.”

And the [lack of understanding], I think, then led to civil rights groups being able to say, “Well, maybe it’s going to affect the Native American population; maybe it’s going to affect this population” — sort of general concerns. So I think you’re right, it created the environment wherein we’ve now got this. And I also think everyone’s going to band together presumably again and be like, “Oh, no, this is terrible. We want to go back to that other thing.” But it’s a shame we had to get to this point where it’s so naked.

Taking the census from a group of Winnebago people in Wisconsin (c. 1911). Source: Library of Congress.

Cynthia Dwork: We were telling them. We were the Cassandras.

danah boyd: Well, and to be clear, one of the things you’ll get out of this book, if you get a chance to pick it up, is that this ain’t new. These fights go back in fascinating ways, and Cynthia kindly let you know that the last name of the person who was the loudest about this is Steve Ruggles. Richard Ruggles, his father, who was an economist, is at the center of the 1960s fights over the National Data Center and how you should aggregate all data together because privacy is no longer a serious thing. That triggered crazy congressional hearings. So these things are not new.

The same goes for the big thing about citizenship, which we think of as this contemporary fight. It goes back to this amazing ophthalmologist named John Tanton, who read The Population Bomb in the middle part of the 20th century and concluded that what was going to happen with massive climate change is that undesirable populations, as in Black and brown people, were going to leave their geographies and invade the North, as in Europe and North America.

To prevent this from happening, he would help create the laws to make certain that you couldn’t have migration. But he wanted the census to collect citizenship data to do this. He was the one who litigated first to try to do citizenship, and the Supreme Court went back to him and was like, “This is ludicrous. The answer is to count everybody.” But the baby lawyer that he raised up in this process is a guy by the name of Kris Kobach, who would go on to be the Trump administration’s advisor on immigration policy.

Because the current fights are going to keep shape-shifting into the future. This is the weird thing about the census: you find that unbelievable amounts of American history are rooted in it in weird ways.

So one of the things that I think is important to realize is that as we’re fighting these fights, let us acknowledge the tentacles and the different pieces of the backstory, and the made-for-TV moments of the estranged daughter who got the hard drives that are part of that backstory, and understand these layers and these humans and these processes. Because the current fights are going to keep shape-shifting into the future. This is the weird thing about the census: you find that unbelievable amounts of American history are rooted in it in weird ways.

My favorite of which is that we actually started a civil war in part because of a census tabulation error. At the time, the language was “deaf, dumb, and blind.” Okay? We’ll leave that alone. But the idea was that Black populations in the South appeared to be less “dumb” than northern populations, or less “idiot” — that was the “idiot” category — than in the North. That was used by the South to justify slavery, and it was entirely an error. It was just an oopsie.

I say all of this because when you’ve built an entire democracy anchored to data and an algorithm, there are problems that emerge. So understand and appreciate those pieces. As we’re swirling around, we have to navigate all of these people. And so, what does it mean to do participatory engagement when that can be so heavily politicized? Which is what we’ve seen.

What Is to Be Done

Molly Hardy: Participatory engagement seems like a good place to be as we start to think about what is to be done, and if there are some steps that you all would like to mention. Ruth, I’m particularly interested in hearing more about how you see this playing out in the courts and why. Again, I don’t want to put words in your mouth, but you seemed so adamant that you hope it’s not all going to be played out in the courts, because that might have a particular outcome. So maybe we could start there, and then I’d love to hear from others as well.

Ruth Greenwood: On the litigation side of things, I would say people should not, and maybe they don’t, put all of their hopes in “we’ll get the Supreme Court,” and then, when it’s the Supreme Court, we’re done. I literally wrote down as I was going, “get these in depositions.” There are all things that you can do through the process of litigation, which has a public media aspect to it, and which has people having to give sworn testimony about things, where you can maybe pull out more information that is useful for the public narrative. So ultimately, if there’s a loss, I’m always about winning the loss. Even if we lose, what else have we done along the way? And I think there are other things that we can do along the way.

I want to raise two other things. When danah was talking, it’s just always a hard thing when you’re on the side of nuance. We can’t just say to you all that data are good and no data are bad. Data can be good in different ways, and you need to understand it, and it’s just a harder sell than the other side, which is, let’s have no data because it’s a problem.

When you [danah] said this has all happened before, maybe by looking at the past of what they did, we can think about how we can formulate our response better. It’s not just a playbook for the people that are trying to destroy it all, but maybe it’s a playbook for us.

Jayshree Sarathy: Talking about participatory engagement, I was talking to Cynthia about this before we started. And I’m curious about your views, all of you. It seems to me that differential privacy was announced, I think, in 2018 or thereabouts for the 2020 census. And I don’t think that two years was quite enough at all to build consensus around a new technology, like a massive participatory effort to get input on how the technology was going to be deployed and how the privacy budget would be distributed across different groups. That was just not enough time.

I think maybe, as Ruth was saying, it’s 2026, we have a little bit more time, and we also maybe are more able to be aligned. I think privacy is just inherently a hard thing to get alignment on. Every group is coming at it with privacy versus equity, or privacy versus accuracy. But now we’re talking about data availability, and I think that’s something more of us are able to get aligned on.

We built coalitions from 2018 to 2020 that I’ve been really happy with, to be able to see behind the scenes how we’re able to align on messaging across technical experts, privacy experts, and advocacy groups. But what messaging do we use to mobilize broad support against these political interferences and to support the statistical independence of the Census Bureau and the way that it produces data products? So, my hope is that maybe we have time, and maybe we are more aligned, and that’s something we can continue to build, bridging alliances across different communities around the census.

Cynthia Dwork: Jayshree mentioned this view of privacy versus accuracy, privacy versus availability. Privacy has no advocates. The only advocates for privacy were the Census Bureau. Everybody who used data for their work wanted more data, more granular data, more raw data, more whatever. Anybody who had a business that was based on repackaging census data, which is a tremendously valuable contribution, was like, “How am I going to do this? I don’t understand the technology. I don’t know if I’m going to be able to ply my trade if we go through with this, and privacy should go away.”

I remember talking very early to people in charge of heart study data at Stanford, and they were using a particular technique that suppressed zip code information. And we said, “Well, how would you feel about a technology that allowed you to give zip code information and allow you to see changes from one small area to another?” And the answer was, “Gee, I don’t know. Go away. I’m HIPAA compliant. Get lost.” So nobody’s advocating for privacy.

Also, people don’t tend to understand that the threat of harm is real. This is not a question of, “But my neighbors already know that I’m gay.” It’s somebody in a neighborhood that you did not choose to live in making a list of all of the gay households in the country to distribute to a group of people who want to go and harass gay households. It’s telling a prosecutor in Mississippi that the 14-year-old daughter that you mentioned on your census report used to be your 4-year-old son 10 years ago. It’s ICE learning the visa status of your elderly parent. The harms are real, and for some reason most of us have trouble imagining them. So I’m hoping that these conversations will at least try to make privacy something that people actually care about.

If you released perfectly accurate information and destroyed everybody’s privacy, you’ll have no chance of getting data from them in the future. It’s over. It’s going to be a very short-lived win for accuracy if that’s what happens.

I want to add one other thing, which is that there’s a time element that people often forget with privacy and statistics. They think it’s privacy versus accuracy. But if you released perfectly accurate information and destroyed everybody’s privacy, you’ll have no chance of getting data from them in the future. It’s over. It’s going to be a very short-lived win for accuracy if that’s what happens.

danah boyd: There is an election in this country in November. A new Congress will be seated in January. We don’t know who yet, but there will be a new Congress. And they’re kind of like goldfish. They don’t ever remember that they need to actually think about the census, even if they were here 10 years ago, and they need to be reminded of it. They need to understand that it’s important and that it’s their job to protect this. So this is where we think about that classic act of political activity.

The second key thing is, we mentioned participatory issues. One of the things that’s been really interesting is that in the 2000 census, Ken Prewitt, who was then the director — and we lost him this summer, and his memorial was this week, so may he rest in peace — wanted to make the census more legitimate and do more civic engagement. But one of the things he struggled with in recent years is that he never imagined how that would backfire, and how people’s recognition of the civic importance of the census, combined with their failure to understand how the systems work, would mean that they would create new forms of friction.

One of the cases that I tell in this book is what happens when a group of very well-meaning advocates try really hard to get out the count for their local community in a way that produces an undercount. So if we want to be good civic engagement folks around this more participatory work, we need to learn more of the pieces, the esoteric, the weird, the technocratic. We need to understand pieces of it, and we need to do it in a way that’s not binaristic: data are all good or all bad; the Census Bureau does it right or wrong. These kinds of binaries are causing us tremendous harm, and that’s where I go back to the social fabric piece.

One of my sort of jokes: I don’t know if you’ve ever been to Reykjavik. It’s a fascinating city. In the middle of one of its central squares, there is a Monument to the Unnamed Bureaucrat. Look it up. It’s really fun, this ridiculous statue. But it’s a country that actually recognizes that to have a functioning democracy, especially at this kind of scale, requires a functional administrative state.

One of the things I think about a lot is that when I started spending time at the Census Bureau, I’d go to the cafeteria a lot, and I’d walk in and hear 16 different languages. I started talking to a lot of these folks who were primarily junior in terms of government ranking, even though they were often much older. And I’d be like, “I’m going to guess you’re naturalized.” “Oh, yeah, I come from [another] country.” And I’d say, “Why did you come work for the US federal government as a naturalized citizen?”

And they told me over and over, “I come from a country where there’s corruption, authoritarianism,” — choose your favorite horrible story — “and I believe in democracy, and I believe in America, and I believe in the power of civil service, and so I wanted to serve.” I say this because we need to recognize the humanity of these people.

So my other invitation to you is to think about what it would take for you to adopt a civil servant. Go write a nice letter to a civil servant thanking them for their service. You can find any number of them on LinkedIn. This is a moment of just being able to acknowledge the humanity and the dedication of these people, and to figure out how you build that social fabric, because the people who are working on our collective behalf have become invisible to us. When they are invisible to us, we forget about their importance.

I’ll conclude with one thing that I loved, going back through the histories of all this. George Washington was a very important person in this country during the time in which the Constitutional Convention occurred, and he decided that his job was to be quiet, because he felt he would have too much sway if he talked at all. So he didn’t talk, except once.

We have reached the point where our systems are failing because they are at scale. The way that you respond to this is not by adding more algorithms. It is by figuring out how to build social fabrics at scale.

He stood up when they proposed that a representative in the House of Representatives should represent 40,000 people in this country. He said, “If they represent 40,000 people, we will not trust them. They need to only maximally represent 30,000 people.” I would like to remind us that our representatives currently represent a million people. We have reached the point where our systems are failing because they are at scale. The way that you respond to this is not by adding more algorithms. It is by figuring out how to build social fabrics at scale. And the way you do that is that each and every one of us takes a moment to connect and bridge, and connect and bridge, and we can have a democracy when we have a healthy and robust social fabric.

Audience and Online Q&A

Molly Hardy: Lovely. Thank you so much. We have some great questions online, but we may also have some questions in the room, and my colleague will be circulating with a microphone. Let me get started with one online, because it’s actually a perfect pickup point. Jennifer Chapman asks: “Census data is very important to the United States. Can anyone speak to the implications of the issues we are discussing today from a global perspective? US government data is used by governments, institutions, and individuals around the world. How might census data issues have impacts beyond US borders?”

danah boyd: I actually spent a bunch of the summer with different government agencies in Europe, which is always fun. First off, kudos to the team and all the people that did differential privacy, because a lot of that is rolling out globally as people realize the power of it. Let’s note that the US’s way of doing statistics is unique and weird and makes everybody else scratch their heads.

You go to Statistics Denmark, and they have a totally different set of problems, because their statistical system includes all of their healthcare, the connection into all of the places you’ve used public transportation, your credit card, your ability to get electricity, everything. Their biggest problem, and one of the things that I love, is what they call the “undead,” which is when somebody accidentally — in a dataset — is made dead before they’re physically dead, because it turns off everything. So they’ve got a different set of issues. They don’t allocate political power like we do, but they are really intrigued by these pieces.

There are also beautiful histories. There’s a wonderful scholar here named Arunabh Ghosh, who does this beautiful accounting of what happened when Mao Zedong decided that China would not allow for capitalist Western statistics, as in Bayesian statistics, and so they would count everything. This ended very badly, because it turns out counting grains of rice is a bad idea, and there was a great famine that was rooted in statistics. So there are histories of all of these other pieces that I want to acknowledge.

People are very much watching us. They do think some of our systems are completely weird. Like the French, who got rid of race in 1978. They didn’t get rid of racism, let’s be clear. They do not understand why this is an issue in the US. I explained to them that we anchored laws to it, and they’re like, “Well, why did you do that?” And I’m like, “That’s a very good question, but that’s where we’re at.”

So I think it’s notable that there are different global pieces of this, that each of those environments has its own context, and that we need to understand them in conversation. But there are also amazing groups that come together around the globe and share techniques. This is in many ways where the shared commitment around morality and privacy, which is rooted in the aftermath of World War II, has led us to a particular history, and then led many other countries to a history. That’s why the differential privacy materials are so exciting and important for places that have less highly partisan federal statistics.

Jayshree Sarathy: To danah’s point, having talked to a few other government agencies who were watching 2020 and being like, “We know that we should also implement differential privacy, but let’s see how this goes in the US,” I think it was obviously very shocking to many other governments: “Oh, okay. This is going to be the impact of lawsuits and lessons.” It doesn’t have to be. So I think governments have now learned that differential privacy doesn’t mean a terrible mess. That is a uniquely US situation, tied to how deeply political and how infrastructural the census is in the US.

Now we’re seeing more small-scale deployments in different countries, and I do think there are some lessons they’ve taken from how to communicate about differential privacy and the technical effort that it takes to build this out. Also, it’s really nice to see the focus that has remained on privacy in Europe and other countries, when what we’re hearing in the US is that people have stopped caring about privacy, especially in industry, and from the executive branch there is not that support from the top to care about privacy. So I think we can really look to other countries to take the lead.

Molly Hardy: Great. Thank you. Other questions?

Audience member: Thank you. I have a more technical question about differential privacy. Is it possible to inject it at the data collection level, such that we no longer have a problem with any government saying that we can’t release things because of privacy?

Cynthia Dwork: If you mean in some sense randomizing the data before they’re collected — yes, that’s exactly, for example, what Apple does with your iPhone and all of your devices. It’s possible to get much better accuracy if you centralize the data and have all of the raw data to work with from the beginning, and then just add a little bit of noise rather than noising up everything all along the way. But it is possible.

Sorting machine, Census Bureau (1909–40). Source: Library of Congress.

danah boyd: The other place where it’s really possible in the census case is that there are many procedures that are not just about collecting data directly from a person. There’s the administrative data piece, and there’s another procedure known as imputation. That’s when a household, they’ve knocked on [the door] six times, they’ve gone to all the proxies, and they can’t figure it out, but they see that there are cars and bikes, and they want to fill this out. They do a set of procedures known as imputation, and for various litigious reasons, they haven’t been able to modernize key parts of this. Had this been combined with differential privacy, you would actually get a much better way of going about it, and it’s another place where it can happen.

Likewise in the ways that the administrative data are used. So there are places in the system that could have been done differently than they were in the 2020 census. But many of the decisions about why they were done the way they were in the 2020 census were organizational, political, really just like, “Oh, that’s not another technical system that we have time to upgrade.” The process of making census data has so many elements to it, so there are other sites of it.

But right now the civil servants are under such duress that I don’t expect to see that for 2030. It is an interesting question, though, certainly for other organizations and other foreign governments who are looking at this from different parts of the pipeline. They’re not going to implement it in the same way the United States did, because the ways in which their systems fit together look different.

Molly Hardy: Another question?

Audience member: Hi. I’m teaching a class this term called “Decision-Driven Data.” We talk about some of these kinds of things, but I’m curious about your perspective. In the past, the census represented a deliberate effort by the US government to collect data. Now all of us are data collectors all the time, generating a tremendous amount of data, most of which goes to the private sector. I’m curious about your thoughts on what this might mean for the census, or for government data collection efforts generally, where the official data might not actually be the best data out there, because there’s just so much more available to be used.

danah boyd: I’ll tell you a story, because I had the privilege of being inside and being a professional meeting attender. I went to one of the meetings when they were finalizing the contract with the advertising firm, and the advertising firm was like, “We’ve got this. We do Coca-Cola, we do all this other stuff. We can reach everybody.” And the civil servant who was there was like, “That’s great. So how do you reach people who can’t afford a Coca-Cola?” And he’s like, “Well, we don’t.” And I’m like, “Hmm.” “And how do you reach one-year-olds who are not going to drink Coca-Cola?” “Well, we don’t.” And he just kept going.

One of the things the civil servant pointed out is, “Look, I can give us the 90 percent of the population right now without thinking. We have that trace data. We have it in all different ways. I can give us that next 9 percent with pretty good accuracy, with some reasonable margins of error in there. And then there’s all of the process of getting that last 1 percent.”

The private sector is great at the 90 percent. So is the Census Bureau. The private sector is pretty good at the next 8 or 9 percent. So is the Census Bureau. The private sector doesn’t give two swats about the last 1 percent. That’s what makes the Census Bureau unique, because its job is to count everybody once and only once, and in the right location.

We anchored democracy to the idea that we would go down to the single-digit person, and that means that getting everybody is the North Star that we anchor this process to.

If your goal is to do marketing, who cares? I’m a truck driver based on my travel schedule. Who cares? But we anchored democracy to the idea that we would go down to the single-digit person, and that means that getting everybody is the North Star that we anchor this process to. If we want to give up on that, okay, we can give up on our democracy as it’s structured. We can rewrite our Constitution. There are many different things that would change in the Constitution first. This is probably not top of my list, but there are ways of doing it. But here’s the democracy we have.

That’s the thing to also keep in mind: there are kinds of variables and details, also not in the decennial census, although it’s about to include citizenship, where the commercial sector doesn’t have these data at all, and it uses these data as the benchmark for nearly everything.

I always laugh at it because it’s more visible with the weather situation. “We don’t need NOAA” — the National Oceanic and Atmospheric Administration — “because we have the Weather Channel.” Where do you think the data comes from? You want nationally representative data, where do you think the benchmarks come from? You want a blue dot that gets you somewhere so you can arrive here, where do you think the geographic locations come from? These are the moments where census data is so much more infrastructural than many people even realize, and it’s because these data are cooked with such insane care.

It’s one of the reasons why our data as a country looks so different from many other countries that are not focusing on cooking the data and making sure everybody is counted. That’s the weirdness of the American project. So yes, there’s a lot of data, and for certain kinds of questions, commercial data is much better. But there are a lot of pieces of this where the US Census Bureau is a unique added value to things that actually affect the business community and everybody else out there.

Ruth Greenwood: I just want to add that the courts have decisions accepting the Census Bureau’s methods, but not other methods. If it becomes clear that we have garbage from the Census Bureau, I appreciate all the flow-on effects, and everybody has a benchmark. But if I’m trying to represent a community and litigate, then it takes a lot more effort to prove the veracity, or at least the usefulness, of private-sector data, so that I can get somebody in as an expert to say, “These data are telling me that this is what’s happening,” and then I can use that for my claim. Over time, we could build that up, and we could get it for non-apportionment things.

For apportionment, unless the Supreme Court changes its mind, there’s no other option except the Census Bureau. I’ve tried to use even just a little bit of survey data in addition to census data, to try to look at whether you can say that people vote differently by race, like white voters prefer some candidates and Latino voters prefer different candidates. And the judge is like, “Okay, well, not looking at that. I’ll look at the census data, that’s what we do.” And I was like, “But these surveys, they’re well-tested, and here’s all of the academic backing, and blah, blah, blah.”

So it’s not impossible. It’s just a real challenge if you’re going to try to get it into litigation. If I don’t have the race data, then I’m going to have to get it from somewhere else. I’m going to have to find a way to find out people’s race in a way that I can then prove to a court is accurate. And you know what’s really great if you’re on the defense side? “Oh my God, there’s no data, you can’t sue us. Thanks so much, bye.” It’s a really tough one. It’s also a big issue in public health, where they are looking for denominators. If you want to know disease rates by groups and wealth and things like that, you need the denominators. That comes from the census.

Molly Hardy: We’ve got another question from online that we need to wrap up soon: “The Census Bureau publishes so many data projects that aren’t the decennial census, such as the American Community Survey [ACS] and the Five-Year Economic Census. Are we seeing similar pressures on these datasets? And if so, what effects might this have?”

danah boyd: Yes. The disclosure avoidance rule from the summer applies to all products, and they can seek exceptions to it, but the process is very complicated. Some of the economics folks are seeking exceptions for a handful of things, but it’s become really egregious and difficult. Keep in mind also that the current Secretary of Commerce must approve all contracts. So even the rules are one thing, but the ability to implement relies on contracts, and if the Secretary has to approve a contract, and there’s a complete hiring freeze, the ability to implement change has become untenable.

The ruling on the Federal Register notice is explicitly about the decennial. However, one of the big open question marks has to do with the ACS, which is technically, according to the legal framework, under the decennial. So it is quite probable that many of the things that affect the decennial will directly affect it, not just because of the system. When you produce the decennial, you then produce this thing called population estimates, and that becomes the sample frame for doing the ACS. So you get this downstream effect, and it’s not quite clear, but it’s probable that the orders are going to affect them as well.

The Bureau is massively understaffed. The normal cycle for the decennial is that you start to build up mid-decade. You get a lot more staffing and a lot more funding, and you peak in the years ending in zero. Then people take those lessons and move them through all the other products as the number of workers declines. We are not seeing that rise. And there’s a lot of internal grousing about the decennial getting too much extra attention and love.

But Congress likes the things that affect its jobs, and that has always been a part of it. There are members of Congress right now who want the ACS eliminated entirely. There’s a proposal within all of this to make the ACS voluntary and optional. That would completely destroy the sample frame and make it impossible. There are other fights that are going to happen over the decennial that will affect downstream products, and the staffing is already undermined.

On top of the things the Census Bureau officially publishes, it contracts for other federal agencies to help them produce their data. We’ve already lost education statistics. We’re losing key parts of health statistics, all of these things, because of the failure to be able to do these contracts. So the statistical system is in a very precarious place. The decennial happens to be the thing you can get Congress to pay attention to. Hopefully, as we pay attention to that, we can start to see all of the other pieces of it as well.

Molly Hardy: Well, I think we’re at time, and I just want to take a moment to thank our panelists again for so much insight, so much brilliance, and for your dedication to this work. So thank you.

Read More...

Source

Figure 1: Web archiving process augmented with FidEx. Figure from https://www.usenix.org/system/files/nsdi26-zhu-jingyuan.pdf.

To improve the annotation feature for our web archiving livestreams, I have been looking at different Quality Assurance tools that detect replay problems and explain why specific resources fail to load. Most of the Quality Assurance tools that determine the quality of the archived web page do not specify the reason why each web resource failed to load. Zhu et al.’s FidEx tool and Archive-It's Wayback QA tool try to assist with this problem.

Zhu, Sun, and Madhyastha’s paper "Detecting and Diagnosing Errors in Serving Archived Web Pages" describes their approach (Figure 1) for detecting problems with replay quality that are caused by rewriting errors made by the replay system. Their approach involves comparing the layout trees (HTML DOM [Document Object Model] and CSSOM [CSS Object Model]) and the JavaScript writes for the live and archived web page. This can be an improvement over comparing screenshots of the web page, since taking a screenshot of a dynamic web page can result in significant differences in the pixel values when videos, images, and text are frequently being changed. In this case, the layout tree and the JavaScript writes should still be the same on the live and archived web page.

Video 1: Presentation video about the FidEx tool.

Zhu et al.’s presentation slides about FidEx.

Zhu et al.’s FidEx tool can detect replay problems and can output a JSON file that explains why an error occurred. Figure 2 shows an example output file which contains a list of detected errors for each archived web page. Each error specifies the exact line of JavaScript code (including replay system scripts like wombat.js) and the associated error message.

Figure 2: FidEx's pinpoint output file shows errors that occurred during replay.

Archive-It's Wayback QA is another tool that helps with identifying the reason why a resource did not load during replay. According to Yale’s Web Archiving Working Group Quality Assurance Report Archive-It's Wayback QA tool has an option for exporting a CSV file with all of the missing content. This file shows all of the data that would be shown on the Wayback QA tab (Figure 3), including the missing resource's URI-R, the URI-R for the web page that loaded the resource, the reason for the missing content, file type, size, and crawl date. The reason for the missing content is categorized as Excluded, Invalid_Access_Denied, Invalid_Unknown, Invalid_Not_Found, or Not Crawled. The Excluded category is for resources that were intentionally excluded in the missing documents list, like ads and analytics URLs.

Figure 3: Wayback QA's “Missing Documents” page. This screenshot is from a video that is embedded in Archive-It’s blog post “Archive-It Workshop Series Part 2: Reviewing & Quality Assurance”.

FidEx's pinpoint output file provides a more detailed description than Wayback QA for the errors that occurred when replaying the archived web page, but it does not specify which errors caused a specific resource from failing to load. Wayback QA’s “Possible Reason” column,  provides a high-level description of what went wrong for each web resource that failed to load. FidEx’s approach for displaying the errors that occurred during replay is useful for technical users who are familiar with inspecting the JavaScript and HTML files for a web page to see which resources failed to load because of a replay error. In contrast, Wayback QA’s high-level description of the possible error is better suited for archivists looking to quickly identify and patch missing resources.

Another difference between these tools is that FidEx helps identify problems that impact interactional and visual correspondence and Wayback QA displays errors that are associated with the completeness of the archived web page. Reyes Ayala’s grounded theory (Paper: “Correspondence as the primary measure of information quality for web archives: A human-centered grounded theory study”) defined interactional correspondence, visual correspondence, and completeness. Interactional correspondence is associated with being able to perform the same user interactions (e.g., scrolling down a web page to load more web resources or clicking the play button for a video) on the live and archived versions of the web page. Visual correspondence is the visual similarity between the live web page and the archived web page. Completeness for an archived web page is associated with the number of sub-resources that were originally on the live web page that have been archived. In a previous blog post, I surveyed Quality Assurance tools that can help with measuring visual correspondence, completeness, and interactional correspondence.

Summary

FidEx and Wayback QA can help with identifying why a sub-resource failed to load during replay of an archived web page. FidEx focuses on identifying problems caused because of the rewriting that is done by the replay system. Fixing the problems identified by FidEx can help improve the interactional and visual correspondence for an archived web page. Wayback QA assists with problems associated with the completeness of an archived web page, since it specifies possible reasons why each missing resource was not archived. 

Learning about tools like FidEx and Wayback QA helps with identifying what information would be useful to display during a web archiving livestream when an error occurs during replay. Our goal for the annotation feature is to create automated annotations that provide context information for missing sub-resources.  Currently our annotations specify when a sub-resource was not archived. In the future, we want to display potential reasons why the sub-resource failed to replay, such as specifying when a URL matching error or other replay rewriting error occurred. The approaches used by tools like FidEx and Jawa (paper: "Jawa: Web Archival in the Era of JavaScript") could assist with determining which type of rewriting error caused a sub-resource to fail to replay.

--Travis Reid (@treid803.bsky.social)

The $28.5T forecast compares to U.S. Q1 2026 nominal GDP of nearly $32T, with the estimate for the market of AI enterprise applications of $22.7T about 70% of total U.S. economic output.

1. Source

   The AI bubble isn't just an equity bubble, as Torsten Slok explains in AI Is Penetrating Every Corner of Financial Markets:
   

   AI now accounts for nearly half of all IG issuance, 87% of VC funding and a growing share of HY, underscoring how deeply the AI investment cycle has penetrated every corner of finance.

2. Bryce Elder's This is nuts upon nuts. When’s the crash? compares this bubble with its two biggest predecessors:
   

   Here’s the title page of this month’s Panmure Liberum market update from strategists Joachim Klement and Francisca Reis. Our emphasis in bold below:
   

   In 1929, the cyclically-adjusted P/E-ratio (CAPE) of the S&P 500 reached 32.6x according to Prof. Robert Shiller’s data. This was 1.8 standard deviations above trend at the time. In 2000, the CAPE reached 44.2x, or 3.3 standard deviations above trend – a clear sign of a bubble. However, as our chart below shows, earnings in both instances were within normal range, less than one standard deviation above trend.
   
   Today, the CAPE is at 41.0x, or 2.9 standard deviations above trend. Once again, we are clearly in bubble territory for stock market valuations. However, unlike in previous bubbles, we are having extremely high CAPE at a time when earnings themselves are 1.8 standard deviations above trend. In other words, we are in a valuation bubble at a time when earnings are in a bubble themselves.
   
   If we correct for the earnings bubble, the current CAPE would be 67.6x or 4.6 standard deviations above trend, a bubble that surpasses anything ever seen in US history by an extreme margin. If valuations followed a normal distribution (which they don’t, so don’t take this literally), this would happen in 0.00019% of months or once every 43,432 years.

3. But Panmure Liberum is underestimating how bad things are. Baolian Wang's The $69 Billion Mirage: How an Accounting Rule Inflated S&P 500's Q1 Earnings by 12% explains:
   

   A massive chunk of this quarter’s blockbuster “growth” didn’t come from selling more software, shipping more microchips, or delivering more packages. Instead, it came from an accounting rule that forced massive, illiquid “paper gains” onto the income statements of tech giants.
   
   In Q1 2026 alone, just three companies—Alphabet, Amazon, and Nvidia—reported a staggering $69.2 billion in non-operating windfall under their Other Income and Expenses (OI&E) lines. When you run the macro numbers, this single accounting phenomenon artificially inflated the entire S&P 500’s quarterly earnings by about 12%.

   That 12% takes the factor from 67.6 to 75.7. Wang notes that correcting for the 12%, "the Q1 2026 earnings growth rate will not be very different from the 5-year average of 16%." In other words, the bubble is feeding upon itself — increased stock prices causes increased earnings causes increased stock price ... But suppose, for example, that OpenAI were to suffer a down round. Then Alphabet, Amazon, Nvidia and others who included paper gains in the "other income" on the way up would have to include paper losses in their income on the way down, amplifying the crash. OpenAI's last round valued the company around $750B and they were planning an IPO for at least $1T, but had to postpone it.
4. Back in March Jared James Grogan published The End of the Foundation Model Era: Open-Weight Models, Sovereign AI, and Inference as Infrastructure setting out the forces changing the structure of the AI market:
   

   The foundation model era — roughly 2020 to 2025 — is over. The forces that defined it have inverted. Open source models have reached frontier performance while inference costs approach zero, exposing what was always structurally true: pre-training large language models at scale is not a durable competitive moat. The US government's formal designation of Anthropic as a supply chain risk in February 2026 accelerated a transition already underway — but did not cause it. The paper argues that the AI industry is restructuring simultaneously along four axes: economic, as the circular financing structure that inflated foundation model valuations collapses; technical, as the pre-training scaling paradigm gives way to post-training optimization, test-time compute, and agentic composition; commercial, as application-layer integrators displace the foundation model companies whose commodity they now consume; and political, as the government asserts its historic role as gatekeeper of strategic technology. These are not separate disruptions. They are one structural shift, arriving together.

   Grogan further argues that:
   

   The most consequential and least-discussed dimension is the permanent divergence between commercial AI and a classified national security AI track — built on different data, governed by different rules, and developing capabilities the public ecosystem cannot see, measure, or govern. Like every dual-use technology that has altered the calculus of state power, AI is being brought under government authority not by design but by the structural logic of what it is. The paper further argues that open-weight models are the counterintuitive instrument of sovereign control: a government that holds the weights commands the capability on its own terms, without dependence on vendor policy, financial continuity, or personnel clearance. The apparent openness of distributed model weights is, from a deploying government's sovereignty standpoint, the most governable architecture — because what cannot be withdrawn by a vendor's API policy cannot be taken away.

   Grogran implicitly assumes that AI is useful but that the current margins and thus the valuations aren't sustainable.
5. Carl Franzen's DeepSeek-V4 arrives with near state-of-the-art intelligence at 1/6th the cost of Opus 4.7, GPT-5.5 explains the impact of open-weights models on pricing:
   

   DeepSeek-V4-Pro is priced through its API at $1.74 USD per 1 million input tokens on a cache miss and $3.48 per million output tokens.
   
   That puts a simple one-million-input, one-million-output comparison at $5.22. With cached input, the input price drops to $0.145 per million tokens, bringing that same blended comparison down to $3.625.
   
   That is dramatically cheaper than the current premium pricing from OpenAI and Anthropic. GPT-5.5 is priced at $5.00 per million input tokens and $30.00 per million output tokens, for a combined $35.00 in the same simple comparison.
   
   Claude Opus 4.7 is priced at $5.00 input and $25.00 output, for a combined $30.00.

   Six times cheaper for an equivalent product is likely to cause pricing pressure on the incumbents, who need to raise not reduce prices. DeepSeek is likely also subsidizing usage, but they do have real advantages. First, they have fewer resources so are forced to to be inventive. Second, the 40% of infrastructure capex that isn't the racks is much cheaper in China. Third, the power component of opex is much cheaper and more available in China. The result is:
   

   In practical terms, DeepSeek does not need to win every leaderboard row to matter. If it can deliver near-frontier performance on many enterprise-relevant agent and reasoning tasks at roughly one-sixth to one-seventh the standard API cost of GPT-5.5 or Claude Opus 4.7, it still forces a major rethink of the economics of advanced AI deployment.
   
   DeepSeek-V4-Pro-Max is clearly the strongest open-weight model in the field right now, and it is unusually close to frontier closed systems on several practical benchmarks.
   
   While GPT-5.5 and Claude Opus 4.7 still retain the lead in most direct head-to-head comparisons across the company's benchmark charts, DeepSeek V4 Pro gets close while being dramatically cheaper and openly available.

6. Dan Davies agrees that the margins aren't sustainable but differs as to why in tokenalysis and john henry:
   

   Which then brings to mind another issue – how confident are we in the pricing power that underpins that 60% gross margin in the first place? In the last paragraph I was talking about the R&D equivalent of a price war, but the normal kind is also possible. The combination of price-sensitive B2B customers, big fixed costs and rewards going to the dominant player doesn’t suggest to me that pricing power is going to be sustainable indefinitely.
   
   But, I think there’s a danger of missing the big picture here. Which is that, when large companies are telling their employees to be sensible and use AI tokens wisely, then the game is up. The race is over and John Henry won against the steam hammer. If you need a human being in the loop to decide on the allocation of AI tokens, then all those predictions of mass redundancy are gone.

7. In A Slower AI Payoff Would Be Everyone's Problem, Torsten Slok is mainly focused on the fallout we can expect even if the bubble deflates gently. He asks
   

   But what if the payoff takes longer than consensus assumes? That question is particularly pressing given that token prices continue to decline and Chinese models are gaining ground, both in their share of the world's most-used models and in token usage, where they now lead their US counterparts among the top 20 models,

   Source

   Slok provides two charts, the first tracks market share by country of origin monthly since January 2025 among the top 50 models. It is bad for the US, showing a steady erosion of US market share until, eyeballing it, as of May 2026 it is roughly a 60/40 US/Chinese split.
   
   Clearly, the market is voting with its feet that the prices charged by US AI companies are unsustainable.
   
   

   Source

   The second compares monthly token use among the top 20 models by country of origin between May and June this year. It is much worse for the US.In May the Chinese had 80% of the US usage. In June they had 185% of the US usage. If this rate of market erosion were to continue for a few months it would be impossible for investors to continue to imagine the golden future awaiting OpenAI, Anthropic, xAI, Meta, Oracle and the neoclouds.
   
   Slok's analysis of the fallout of the bubble deflating is worth reading. This is an issue I plan to return to in a future post.

8. Source

   Kevin Walmsley's We were wrong about DeepSeek. Now Chinese AI companies export trillions of AI tokens also focuses on the erosion of US market share:
   

   Chinese AI providers are making more money; Zhipu’s revenues went up 60 times in Q1 compared to last year; Alibaba is the company behind Qwen, and their revenues are up 15 times just since the beginning of this year. The lion’s share of the profits, though, are still being realized by the hardware side; the chipmakers, at least for now. The model companies’ revenues are rising, and steeply. But so is their cost of compute.
   
   That dynamic is causing some Chinese labs to raise prices; the cost to use Tencent Cloud increased by over five times in March, and Alibaba, Zhipu, and ByteDance also hiked prices.
   
   DeepSeek, however, went the other way. Their latest version is priced at just a fourth of their introductory product, and they rolled out dynamic pricing that is aimed at corporate users, who use their models during the workday.

   Source

   This price difference is having an :
   

   But even with these price increases, Chinese models just cost far less than those on offer from Silicon Valley, and explains those big jumps in the exports, we can say, of Chinese AI tokens. The LLM’s out of China are “90% as good at 10% of the cost”, and American firms are buying more tokens from Chinese companies. In early 2025, token demand from Chinese LLM’s was about zero. Even the release of DeepSeek didn’t move the needle much, but by the end of the year the secret was out, and it’s been a choppy but steady ride up to 46%, today.

   Why would you invest in a company whose competitors were “90% as good at 10% of the cost” and which was rapidly losing market share?

9. Source

   If the margins, and thus the rational valuations, of AI companies are unsustainable, how long can the market remain irrational? In The Second Derivative: Why No One Understands the AI Boom Groundbreaker starts by examining the 2008 crash:
   

   The implicit underwriting assumption, shared by originator and borrower alike, was that the loan would never reach its reset: rising home values would manufacture equity, the borrower would refinance into a fresh teaser and the clock would start again. The structure was a treadmill, and the treadmill was powered by appreciation. It worked spectacularly while it worked. Nearly four in five subprime hybrid ARMs originated in 2003 had been refinanced away by the end of 2006.
   
   Now watch the timing. National home-price appreciation did not crash in 2006. It decelerated. The year-over-year rate of gain, which had run in the mid-to-high teens through 2004 and into early 2005, began bleeding off - still positive, still printing green, but slowing. Prices were higher than they had ever been. And yet, with prices at their peak and still rising, subprime delinquencies inflected upward. ...
   The deceleration was endogenous to the structure; the structure required ever-accelerating prices to keep refinancing its way out of its own reset schedule, and no series accelerates forever.
   
   The second derivative was always going to roll over. When it did, the first derivative followed it down through zero, negative equity spread from the margin inward, and the defaults the market insisted were caused by “falling prices” had in fact begun a year earlier, when prices were still rising but had stopped rising faster.

   Why does the 2008 analogy apply this time?:
   

   The market is pricing AI as a technology cycle when its actual anatomy is that of a credit-driven real estate cycle - which is precisely why the 2008 mechanics apply - and the two break for entirely different reasons.
   ...
   Walk down the AI build-out and every feature is a property development in disguise: a data center on entitled land, financed with debt against the structure and leased to tenants on take-or-pay terms. This is not a software business that happens to own servers. It is a real estate business that happens to compute.

   The AI build-out is being financed with debt:
   

   When a hyperscaler or a neocloud reports a record capex figure, the financial press reads it as confidence, as proof of demand. Read it instead as origination volume. Each gigawatt of committed build is a loan extended to whichever tenant has signed the take-or-pay beneath it, and the credit quality of that loan is precisely the credit quality of the tenant. The market is celebrating loan growth and calling it revenue growth.

   Loans in this market take the form of Remaining Performance Obligations (RPOs). The biggest borrower in this market is OpenAI:
   

   OpenAI has committed to pay for compute on a scale without precedent in corporate history: multi-year, take-or-pay capacity contracts whose aggregate obligations run to the hundreds of billions of dollars. Against them sits an operating business that does not yet earn a profit - revenue real, large, and growing quickly, but short of covering the company’s own cash burn and nowhere near covering the contracted payments.
   
   Those payments are therefore not serviced out of earnings. They are serviced out of financing, and financing, for a borrower in this position, is available on a single condition: that each new round price above the last.

   Does OpenAI's financing meet this requirement?:
   

   Measured as a level, OpenAI’s valuation is the most remarkable appreciation in the history of private markets - roughly $86 billion in early 2024, then about $157 billion, $300 billion, $500 billion, and approximately $852 billion by the spring of 2026. Measured as a rate of change, the same series inverts: the round-over-round step-up ran 1.83×, 1.91×, 1.67×, 1.70×, and falls to roughly 1.23× implied by the reported public-offering target. Private marks are inherently lumpy - negotiated, episodic, set by a handful of insiders - so no single step is decisive. But the trend is unmistakable: it bends down, and it bends hardest at the one mark set by the deepest, most unforgiving pool of capital - the public market. The implied IPO step-up is both the lowest in the sequence and the hardest to negotiate, and it is the one the structure must actually clear. This arithmetic is also the most probable explanation for OpenAI’s recent IPO delay.

   Source

   In the same way that the private stocks generating unrealized gains are not liquid assets, neither are the Remaining Perfoemance Obligations:
   

   An RPO is not a liquid asset; it is a forward contractual commitment - a promise of future payment in exchange for future compute. And a multi-year commitment is worth exactly the creditworthiness of the entity on the other end of it. When that entity is investment-grade and cash-generative, the backlog is what it claims to be: high-quality visibility, merely deferred. When that entity is a pre-profit company that loses tens of billions a year and can pay only by continuously refinancing its own equity valuation, the backlog is something else entirely. It is a subprime commitment, used to justify massive, un-depreciated capital expenditure, reported to shareholders as structural strength.
   
   Now price the credit quality of that book. Of roughly $2.1 trillion in aggregate contracted backlog across the four big platforms, about half - on the order of $1.05 trillion - is owed by OpenAI and Anthropic. Microsoft’s book is about 49% these two names; Oracle’s is 54%, with roughly $300 billion owed by OpenAI alone; Google’s is 43%; Amazon’s is 51%.
   
   The hyperscaler has, in economic substance, extended a concentrated, unsecured loan to cash-burning tenants. The RPO that Wall Street values as forward revenue is, in reality, a credit exposure to borrowers with no operating income.

10. Source

    Major financial institutions are similarly concerned. In their 2026 Annual Economic Report, the Bank for International Settlements writes:
    

    In the near term, the ongoing AI investment boom raises questions about the sustainability of the current economic expansion. The five largest hyperscalers are set to spend over a trillion US dollars on AI-related capital expenditure from 2025 through 2026. These commitments are outpacing earnings and the free cash flow of these firms, leading some to issue debt to raise additional financing (Graph 11.A). This investment race may be partly driven by the perception that only a small number of players with superior technology will ultimately dominate the market shares. The intense competition raises the risk of firms over-committing resources to investment projects with still uncertain returns, leaving all firms vulnerable to disappointments in AI payoffs. Model analysis based on such contest motives highlights the downside risk of current AI exuberance. As competitive pressure drives capex higher, the net economic surplus – the total payoff less investment costs – declines for the sector as a whole and could turn negative in adverse scenarios (Graph 11.B). Disappointment in returns could trigger a sudden pullback in financing and turn the capex boom into a protracted investment bust, with potential knock-on effects on financial conditions (see below).

    The BIS has noticed another looming problem:
    

    Another risk is that the AI boom runs into a supply side roadblock. The AI build-out has recently been facing growing bottlenecks in electricity, advanced semiconductors and grid equipment. Fast-growing demand for computing power is already pressuring electricity prices and input costs, with potential spillovers to inflation. Looking ahead, these temporary shortages may also amplify over-investment, as firms attempt to lock in future capacity through long-dated contracts that further expose them to any disappointments in demand.

    Unlike Groundbreaker, the BIS uses more conventional historical analogies:
    

    Historical episodes of investment booms offer instructive parallels (Graph 11.C). The canal mania of the 1830s, the British railway mania in the 1840s, the electrification exuberance of the late 1920s (roaring 20s) and the dotcom boom of the late 90s all shared one common trait: a genuine technological breakthrough that attracted capital in excess of what commercial returns could ultimately justify. These episodes ended with an eventual reversal in investment, inducing economy-wide recessions. The scale and pace of the current AI investment boom accompanied by expectations of large productivity payoffs bear resemblance to these precedents, highlighting potential downside risks in the near term.

11. And Eric Katz reports that the US Treasury Has an Internal Report Warning About the Dangers of an AI Bubble:
    

    A draft report inside the Treasury Department is set to warn of the risks posed by the artificial intelligence market, likening key aspects of it to the dotcom bubble that upended the U.S. economy when it burst in the early 2000s.
    
    The document, the existence and contents of which have not been previously reported but was obtained by NOTUS, is a significant departure from the Trump administration’s public tone, which has focused on encouraging unrelenting investment to unlock exponential growth.
    
    Career Treasury analysts found that AI firms are more deeply entrenched in the U.S. economy than their dotcom predecessors and pose significant risk to the entire system if financial conditions change, productivity goals are missed or various choke points stymie growth.

12. Vanderbilt University's Asad Ramzanali has a detailed look at the range of impacts from the burst bubble, and an set of optimistic suggestions for policy responses in After the AI Crash. He frames the problem thus:
    

    Companies are investing trillions of dollars based on tens of billions of dollars in revenues. Analysts at J.P. Morgan anticipate $5 trillion of AI infrastructure investment in the next five years. They estimate that the industry will need to generate annual revenues of $650 billion to justify this level of investment, while consultants at Bain & Co. estimate $2 trillion in needed annual revenues. Yet, OpenAI and Anthropic earned $13 billion and $4 billion, respectively, in 2025 revenues. OpenAI’s own financial expectations suggest negative cash flow until 2030, and Anthropic expects a small profit no sooner than 2029. Alphabet, Meta, Amazon, and others may experience increased marginal revenue from integrating AI into existing products, but that is far from certain.

    Note that over the next 5 years around $3T (60% of $5T) of the investment in AI infrastructure goes in to buying the hardware which should be fully depreciated over much less than 5 years. So the gap between the investments and the revenue is much bigger than it appears.
    
13. The cracks are starting to show. Reuters reports that Blackstone's QTS terminates Digital Gateway data center project in Virginia:
    

    Blackstone's QTS said on Thursday it had terminated its planned Digital Gateway data center project in Virginia and withdrawn the associated filings after years of planning and regulatory review.
    
    The data center operator has faced years of local opposition and litigation over the project, despite it being approved by the Prince William Board of County Supervisors.

14. Among the hyperscalers, Oracle is the most exposed because, as Ed Zitron noted:
    

    And Oracle ... is a company that, even before the AI bubble, was massively indebted. It just so happens that, as a result of its tryst with OpenAI, Larry Ellison saw fit to twist the debt knob to eleven.

    Now, Brody Ford reports that Oracle Warns Its Splurge on AI Data Centers May Not Pay Off:
    

    Six firms alone — including Oracle, Microsoft and Meta — have committed $850 billion for data centers leases that haven’t begun yet. Oracle holds the largest share of these commitments owing to its $300 billion Stargate contract with OpenAI.
    
    When Oracle mentions the risk of nonpayment, the unnamed elephant in the room is OpenAI. As part of the Stargate deal with the AI company, Oracle is developing massive data centers across the country to provide cloud computing power. For this plan to work, OpenAI needs to pay its Oracle Cloud Infrastructure bills.
    
    “Some of our customers may be highly leveraged and subject to their own operating and regulatory risks and, even if our credit review and analysis mechanisms work properly, we may experience risks of non-payment and non-performance in our dealings with such parties,” Oracle said in the filing.

15. A month ago Laura Benitez et al reported that SoftBank Attempt to Get $6 Billion OpenAI Margin Loan Stalls:
    

    SoftBank Group Corp.’s talks with potential creditors to raise at least $6 billion from a margin loan backed by its OpenAI stake have stalled, people familiar with the matter said, just weeks after the Japanese conglomerate cut its initial target from $10 billion.

    But now Echo Wang reports that SoftBank renews talks for $10 billion loan against OpenAI stake, adds concessions, sources say:
    

    SoftBank Group has reopened talks with a consortium of lenders for a $10 billion loan backed by its stake in OpenAI, after earlier attempts to secure a loan stalled over concerns about the difficulty of valuing private companies, two people familiar with the matter said.
    
    To make lenders more comfortable, the Japanese technology investor is offering to guarantee repayment of the loan, giving banks recourse to SoftBank if the OpenAI shares pledged as collateral lose value, the people said.

    This all seems to indicate that potential lenders, such as banks, are highly skeptical of the value of OpenAI stock.

16. Source

    Despite Grok being so bad that employees use Claude, Musk is touting SpaceX as an AI company. This resulted in the most overvalued IPO in history, which failed to raise enough money to avoid the immediate need to borrow $25B. Nir Kaissar's SpaceX Is Junk. That’s What the Bond Market Says reports on the bond market's reaction:
    

    Ratings companies and the bond market have very different views about how things are going. SpaceX’s bonds have an average rating of BBB across the three majors, Moody’s Ratings, S&P Global Ratings and Fitch Ratings, according to credit scores compiled by Bloomberg. In the alphabet soup of bond ratings, it’s the lowest grade still considered quality before falling into junk territory.
    
    The bond market has other ideas. There, quality is judged by a bond’s credit spread or the additional yield it offers above Treasuries with similar maturity. The wider the spread, the lower the quality. Corporate bonds with a BBB rating are trading at an average credit spread of 0.92 percentage point. SpaceX’s bonds, by contrast, trade at a significantly greater average spread of 1.62 percentage points across maturities, higher than BB rated junk bonds’ average spread of 1.55 percentage points.

    The bond market seems to agree with Softbank's lenders about the AI bubble.
17. OpenAI and Anthropic are competing for the next trillion-dollar IPO. Both would need to distract investors from their massive losses by focusing on growth. Recently, Anthorpic has been growing faster than OpenAI, so Keach Hagey and Berber Jin report that OpenAI Considers Drastic Price Cuts, Anticipating War for Users With Anthropic:
    

    OpenAI is considering drastically lowering the prices it charges users as it seeks to win customers from its rival Anthropic.
    
    The company is weighing significant cuts to what it charges for tokens, the unit of measurement artificial-intelligence firms use to bill for their products, according to people familiar with the matter. The move would be in anticipation of similar cuts the company expects at Anthropic, the people said.

    Now Kurt Wagner reports that Zuckerberg Pledges ‘Aggressive’ Pricing With Meta’s First Pay-to-Use AI:
    

    Meta will also introduce a new Meta Model API system, which will be used to collect fees from developers. Its API pricing is roughly 25% of the cost advertised by other top models from OpenAI and Anthropic PBC. Developers will be able to use Meta’s model for free, but only up to a point; they’ll be required to pay for access after reaching a certain token threshold, Zuckerberg said.
    
    “The pricing from some of the other labs is very extreme and has very high margins,” Zuckerberg said, underscoring that his strategy is to get Meta’s technology in front of as many people as possible. “We think that there’s a real ability to be able to offer frontier or very high-level intelligence at a much more affordable cost.”

    Aggressive pricing means Meta Model API will still be 50% more expensive than DeepSeek but not 50% better. Planning to reduce current income in the lead-up to an IPO is an unusual move, but it is a response to AI's Affordability Crisis.
18. Data center demands for power are having serious impacts elsewhere in the economy, as Jeremy Hsu reports in Data centers’ energy demand threatens Trump’s “Made in America” plan:
    

    Factory electricity bills are generally rising faster than those for other business customers or residential customers, according to a Reuters analysis. It highlighted the example of the Belden Brick Company, a 141-year-old brick manufacturer in Ohio, whose electricity bills have soared from $1,600 to $12,000 per month due to a higher monthly capacity charge in the 13-state region served by the grid operator PJM Interconnection.
    ...
    The Ohio-based steelmaker Metallus described its electricity costs as having jumped by 70 percent since 2024, leading the company to pay an extra $15 million in energy costs annually.
    
    The higher electricity costs for manufacturers coincide with the attraction of large AI data center projects with substantial electricity needs to many states in PJM territory. That data center growth has driven up PJM’s capacity prices—paid to power generators according to supply-and-demand forecasts—from $28.92 per megawatt-day in 2024 to $329.17 per megawatt-day in 2026, according to Reuters’ reporting.

19. Source

    As I see it, there are three separate markets for LLMs. First, there is an embedded market that runs on low-cost, low-power hardware and open-weights models. Small AI Models Gain Traction Around the World by David Berreby provides examples:
    

    For example, a drone-based system developed by Bala Murugan and colleagues at the Vellore Institute of Technology, in India, takes photos of cashew plants and quickly identifies those with splotches that indicate disease. All the processing takes place on the drone itself, so there’s no need for a computer on-site, nor for a connection to a central server.
    
    Using small language models trained for a specific problem, and sometimes running on cheap, low-power devices, other small-AI implementations have been developed to identify ant infestations in a Uruguayan vineyard, detect the presence of malaria-carrying mosquitoes in a number of nations, and run electrocardiograms from an Arduino device in parts of Brazil that lack access to more complex equipment.

    It isn't just that small hardware, such as the Raspberry Pi 5, is getting more powerful but also:
    

    the shrinking footprint of language models. Both Google DeepMind’s Gemma 4 (released in April) and Alibaba’s Qwen 3.5 are “fantastic” for small AI, Rovai says. Both models are “open weight,” meaning users can adjust the connections between parameters to suit their needs. This makes it easy, for example, “to take a lot of data from, say, the milk industry and retrain the model specifically on that,” Rovai says.

    The hyperscalers and AI platforms like OpenAI and Anthropic will garner no income from this market, because:
    

    “I think the future of AI is not like one giant model, at a center. I think it’s millions of small, precise models deployed at the edge, each one solving like a specific problem, a specific context,” Alonge says. This is partly because much of humanity—including people in parts of rich countries as well as the developing world—lives without access to cutting-edge frontier models. But, he says, it’s also because those models are not sustainable.
    
    “If someone is not subsidizing it, most people will not be able to afford those models. So those of us who are said to be small-AI developers are the ones who will have to build for the majority of the world,” Alonge says.

20. Second, there is a consumer market. Last month, Nvidia announced a product for the prosumer laptop market with significant LLM capabilities. @pramodchandrayan described it in NVIDIA Just Put a 120-Billion-Parameter AI Model in Your Laptop. Here’s What That Actually Changes.:
    

    At Computex 2026 in Taipei on June 1st, CEO Jensen Huang announced the RTX Spark superchip — a single piece of silicon that combines a 20-core Arm CPU, a Blackwell GPU with 6,144 CUDA cores, and 128 gigabytes of unified memory, connected by NVIDIA’s NVLink chip-to-chip interconnect. The whole package delivers up to one petaflop of AI compute in a laptop form factor.
    
    The number that matters: RTX Spark can run a 120-billion-parameter language model entirely locally, with a context window of one million tokens, without a single byte leaving your machine.
    
    To put that in perspective: GPT-3 had 175 billion parameters and required clusters of A100 GPUs to run. The model that stunned the world when it launched in 2020 is now approximately the size of what fits in a consumer laptop chip announced this week. The capability that required a data centre in 2020 is coming to a device you carry in a bag in 2026.

    Phones can already run small LLMs:
    

    In 2025, slightly more than a third of all smartphones shipped worldwide were capable of running generative AI, and that figure will reach 45 percent by the end of this year, according to the technology research firm Counterpoint. By the end of next year, slightly more than half of all smartphones will be able to run a small AI model.

    If Nvidia can already put a 120B-parameter in a laptop, it will only be a few years until phones can run a GPT-3-class model, good enough for almost all consumer needs. Apple and Google own that channel. Owning the channel is better than owning the technology. They will dominate consumer AI, and the other players will garner no revenue from this market.
    
21. Third, there is an enterprise market; all that is left to generate the revenue to service the debts fuelling the AI bubble, lets say $2T by 2030. There are a number of problems that make this unlikely.
    
    First, there are very few documented cases of LLM deployment that resulted in enough productivity improvement to cover its unsubsidized costs.
    
    Second, the Trump administration just demonstrated that deploying mission critical systems on AI platforms such as OpenAI or Anthropic means your company can be disabled at 90 minutes notice with no recourse.
    
    Third, this means that companies will have to run mission-critical LLMs on open-weight models on in-house hardware if they are not to be vulnerable to the whims of the US president.
    
    Fourth, systems such as RTX Spark show that good enough in-house hardware is likely to become relatively cheap compared to the unsubsidized cost of the AI platforms. In-house systems need much less over-provisioning for demand spikes, and because they aren't shared they don't need to be as fast.
    
    Fifth, companies need to balance the productivity benefits (if any) of mission-critical LLMs against the productivity costs they bring. These include a vastly greater attack surface, technical debt from reduced developer understanding of the software, and so on.
    
    Thus it seems likely that the hyperscalers and AI plaforms will generate far less revenue than they expect, because they will be restricted to non-mission-critical applications with lower productivity gains, and thus lower pricing power. They will thus be unable to cover the debts they are incurring to build massive data centers predicated on centralized systems dominating (an inflated estimate of) the entire enterprise market.
    
    David Wallace-Wells' Did We Make the Wrong Bet on Big A.I.? discusses a "televised rant" from Palatir's CEO Alex Karp that supports this argument:
    

    Karp had been softly floating his critique for some time, but the CNBC event looked like a proper coming out. Just one day earlier Palantir had published a kind of manifesto devoted to what it described as the all-important principle of “A.I. sovereignty.” The central argument: Companies should seek to build their own A.I. tools, not just customize those on offer from the frontier labs. This might mean relying on open-source L.L.M.s rather than the proprietary ones on which the A.I. boom has mostly been built in America, but it would amount to a liberating declaration of independence from Big A.I., which in Karp’s estimation was sucking up much more value than it was generating.

22. Source

    The Economist's How to turn compute into a financial asset reports another bad sign (my emphasis):
    

    Locking in a price with a multi-year neocloud contract insures a buyer against compute getting more expensive but not against it getting much cheaper. So as businesses around the world spend ever more on compute, they want to be able to hedge against price volatility just as they insure against changes in energy tariffs, interest rates or foreign-exchange movements—ideally in deep and liquid derivatives markets.
    
    Two startups want to help companies do this, by turning nascent indices tracking compute costs into a futures market. Silicon Data, founded in 2024 and backed by DRW, a trading firm, has paired up with CME Group, which operates large derivatives exchanges. Ornn, created by recent graduates of the Massachusetts Institute of Technology and run from a flat rather than an office just a few months ago, has paired up with Intercontinental Exchange, the parent company of the New York Stock Exchange, to do the same. Both aim to launch compute futures later this year, to be traded on their partner exchanges.

23. One characteristic of bubbles is overbuilding the infrastructure. Signs of overbuilding of data centers include that both SpaceX and Meta are now in the business of rentling their GPUs to the competition.
24. Victor Tangermann's Zuckerberg Admits That AI Is Not Working Out the Way He Imagined is on-trend:
    

    As morale is hitting rock-bottom, his company is heavily relying on its competitors' AI models to build out its own in-house tools. And despite the many billions of dollars the company has spent in its flailing efforts to keep up in the AI race, even Zuckerberg himself is now acknowledging that progress is nowhere near where he wanted it to be.
    
    As Reuters reports, Zuckerberg admitted during a town hall last week that AI agents in particular aren't progressing as fast as he anticipated, a devastating revelation following enormous layoffs that wiped out thousands of roles at the company.
    
    The "trajectory of the agentic development over at least the last four months hasn't really accelerated in the way that we expected," he said according to a recording obtained by Reuters.

25. Despite a derisory ~3% share of the enterprise LLM market, SpaceX's IPO was marketed as an enterprise AI company. The IPO has to rate as the most manipulated of all time featuring, in addition to ludicrous financial projections, a tiny float, bent rules for index inclusion, and massive conflicts of interest at the banks and the analysts. Immediately afterwards, SpaceX issued $25B in bondsa. A month later we can see how the markets respond to the first trillion-dollar "AI company".
    
    

    SPCX

    First, SPCX stock. Toby Nangle writes:
    

    We’re old enough to remember when the market cap of the lossmaking telecom SpaceX was bigger than Amazon’s. Heck, for a few precious moments it was bigger than Microsoft’s. Maybe one day it will be again, but for now the stock is down 38 per cent from its peak post-IPO valuation.
    
    Punters lucky enough to have been awarded a stock allocation at the outset are still sitting on a tasty [checks notes] 0.8 per cent paper profit at pixel time.

    And this is before the lockups start expiring and the initial tiny float greatly expands.
    
    

    SpaceX bod spread

    Second, more interesting as being much less subject to manipulation, are the bonds. They are what Toby Nangle focused on in SpaceX bond yields rocket towards junk:
    

    the full $25bn of benchmark bonds — issued across the curve — had a rocky first couple of days of trading. Checking back today, it turns out that the inauspicious beginning was just a prelude to the train wreck that has since unfolded.
    ...
    If you’d been allocated $100mn of the SpaceX 2056 bonds, you’ve turned $100mn into $90.7mn in less than a month. Sure, long-dated US Treasury bonds have fallen in value, and this general sell-off at the long end has done some of the work. But the spread on SpaceX 2056 — the additional yield you’re paid to compensate you for the risk that you don’t get repaid (among other things) has now widened from the initial +175bps to a whopping +231bps doing more than two-thirds of the work.

    Top 10 worst BBB

    The same conflicts of interest that pumped the stock caused rating agencies to give SpaceX bonds a BBB rating, one notch above junk and crucially the lowest that many major institutions are allowed to hold. But Nangle notes that:
    

    Looking only at the nine days since the bonds were included in ICE BofA indices at the end of June, this spread-widening has made SpaceX 2056 the single worst-performing US dollar triple-B benchmark bond:

    Note that the top two worst bonds are SpaceX, but the rest of the top 10 are all Oracle, another financial disater area.
    
    Despite both carrying a BBB rating, Nangle has a very dense graph showing that:
    

    when we overlay the average spread for double-B US dollar corporate bonds across different maturities (the pink line), it looks a lot like the type of risk that the market has assigned to both SpaceX and Oracle bonds is junk risk.

    SpaceX, OpenAI, Anthropic, Meta all need to raise vast amounts of debt to fund their plans for AI. With SpaceX's bonds trading as BB despite a BBB rating, this is going to be hard.
26. Tobias Mann reports that Former OpenAI CTO does what Altman won't, releases a frontier AI model that's actually open:
    

    Founded in early 2025 by former OpenAI CTO Mira Murati, Thinking Machines' first model is a big one. Weighing in at 975 billion parameters, the model requires more than two terabytes of GPU memory — a quantity present in around eight of Nvidia's B300 accelerators, or sixteen H200s — to run at its native 16-bit precision. If that's asking too much of your hardware, Thinking Machines has also released a NVFP4 quantized version of the model capable of running on half the GPUs.
    
    This makes it the largest American open weights model to date, and comparable to Chinese models like DeepSeek V4, GLM 5.2, and Kimi K2.6 in terms of size and capabilities. Take these claims with a grain of salt — gaming AI benchmarks isn't exactly difficult – but Thinking Machines says Inkling is competitive with these models in a variety of workloads, although its benchmark charts also show it trailing proprietary models like Anthropic's Claude and OpenAI's GPT.

    If you don't like Chinese open-weight models, the US ones are getting better:
    

    The model developer claims to have tuned the model to use these thinking tokens more efficiently and that Inkling therefore matches Nvidia's Nemotron 3 Ultra, up to now the largest and most capable American open weights model out there at 550 billion parameters, on Terminal Bench 2.1 using roughly a third the tokens.

27. CoreWeave's CEO Dumped Nearly 370,000 Shares for $30.8 Million. What Does That Mean for Investors? by Robert Izquierdo notes that:
    

    The disposition involved 369,489 shares with a total transaction value of ~$30.8 million, based on weighted average prices.
    
    The sale reduced the insider's total equity holdings by 11%, including the liquidation of 100% of Class A shares previously held indirectly.

    The notes for Ox Talk's video are skeptical:
    

    The AI revolution is supposed to be in its early innings. Demand for GPU computing is exploding. Microsoft has relied on CoreWeave for massive amounts of AI compute. OpenAI has signed tens of billions of dollars in long-term contracts. NVIDIA isn’t just supplying the chips, it has also invested in CoreWeave and entered into agreements that support parts of its financing and capacity.
    
    So why are CoreWeave insiders continuing to sell stock? Management says many of the sales were made under pre-arranged Rule 10b5-1 trading plans. But those plans explain how the shares were sold—not necessarily why executives continue converting stock into cash while investors are being told AI infrastructure demand has never been stronger.

    The CEO isn't the only insider selling.
28. Sharon Kits Kimathi's New York issues moratorium on data centers illustrates the political toxicicty of data centers:
    

    New York has become the first U.S. state to stop construction of large new data centers, imposing a one-year moratorium due to growing concerns over power costs, water supplies and the burden on local communities.
    
    "As data center development threatens to hike up utility bills, deplete our natural resources, and create uncertainty for New Yorkers, it's my responsibility to take action and lead," said New York Governor Kathy Hochul.
    
    She added that she would also pursue legislation to repeal sales tax exemptions for large data centers.

29. Phurichai Rungcharoenkitkul continues the Bank for International Settlements's pessimism in The AI Investment Race:
    

    The AI build-out ranks among the largest technology-driven investment booms in US history. Its scale, reliance on debt and circular equity ties raise questions about the boom’s sustainability and financial stability. We study a dynamic contest in which firms competing for a few dominant positions over-commit resources. The over-investment leaves the sector exposed to revenue disappointment that could turn boom into bust. The larger the boom, the deeper the eventual bust. The race to commit early through debt and circular financing also makes a bust more likely. Calibrated to balance sheet and deal data, the model points to over-investment of around 1.5 times the efficient level, rising to around three times where demand is less elastic. A network analysis shows that stress in one firm could cascade to others through chains of financial exposures.

30. Shirin Ghaffary and Rachel Metz report that Moonshot’s Kimi Upends Conventional Wisdom on US Lead Over China:
    

    As recently as this week, one executive at Anthropic PBC, who spoke on condition of anonymity, mused that the Claude maker’s technology was roughly six to 12 months ahead of Chinese rivals.
    
    On Friday, Moonshot AI Inc. upended those assumptions. The Chinese AI lab released Kimi K3, a more advanced open-weight model that it said outperforms all rivals except for Anthropic’s Claude Fable 5 and OpenAI’s GPT-5.6 on overall capability. The implication is that Moonshot, and by extension China, could be closing the gap faster than expected.
    ...
    Moonshot’s release could also undercut OpenAI, Anthropic and others on price at a moment when they’re confronting customers who are becoming more conscious of their surging AI spending. Some developers have turned to so-called model routing services that can seamlessly direct users to cheaper options, including from China, for specific tasks to maximize cost efficiency.

    Kimi K3 is an open-weight model.
31. If a 120B parameter model is too small for you the RTX Spark won't cut it. But Michal Malewicz explains in NVIDIA just killed big AI and… You’re the winner? that one of Nvidia's OEMs will sell you a $94K box that will run a 1T model, the Exxact Valence Nvidia DGX Station.
32. Carmen Arroyo explains why SpaceX is competing with CoreWeave not OpenAI or Anthropic in The Identity Crisis at Elon Musk’s Chaotic AI Outfit :
    

    The company’s models were using just 11% of its available computing power by April, according to an internal memo.

33. Bryce Elder's Is AI productivity growth in the room with us right now? is a fascinating deep dive into a very hard question to answer. He quotes Barclays' research:
    

    "[O]ur bottom line is that AI adoption appears gradual and steady rather than rapid and transformative, with most households and businesses still reporting limited exposure to the technology. At the same time, evidence of a structural pickup in productivity growth remains surprisingly fragile: Aggregate productivity growth has improved during the post-pandemic period, but there is a strong rationale to attribute much of this improvement to cyclical variation in utilization rather than a sustained acceleration in productive capacity. We also find little compelling evidence in the available industry-level data that industries adopting AI more rapidly are already experiencing stronger productivity growth."

    And the Fed:
    

    "productivity trends across all three levels have been relatively consistent over time, suggestive of micro-level productivity gains not adding up in aggregate"

    So Elder's post conforms to Betteridge's Law of Headlines.
34. Another instance of Betteridge's Law is Will OpenAI ever be profitable? from Leap Finance Academy. It is a very long and detailed examination of all the things that would have to happen for OpenAI to turn a profit by 2030. In summary, with my comments, the list is:
    
    • Reduce cost of inference which isn't going to happen enough because each successive frontier model uses more tokens to generate the same output.
    • Increase pricing which isn't going to happen because they already need to reduce pricing to stem the loss of customers to Anthropic, let alone to the open-weight models.
    • Increasing sources of revenue that are not inference dependent in other words the fantasy of $100B/year in advertising income, which isn't going to happen.
    • Securing and retaining human talent which requires an IPO to make their stock options worth something, which isn't going to happen it time to prevent bankruptcy.
    • Securing the funding they need to build Stargate and for working capital which isn't going to happen because even an IPO wouldn't come close to generating enough and the lenders are saying "enough".
    Even though I'm skeptical of the wishful thinking of the conclusion, this is an impressive piece of work and well worth reading.
35. Meghan Tobin reports that Even China’s A.I. Powerhouses Can’t Figure Out How to Profit Off A.I.:
    

    China’s open-source approach has spawned a crowded field of innovative and intensely competitive start-ups all offering systems at low cost.
    
    So attempts to increase revenue by charging more for access to certain models can scare away customers. Price-conscious Chinese consumers — businesses and individuals alike — are quick to hop across platforms in search of inexpensive A.I. tools.
    
    While China’s A.I. companies are searching for sustainable business models, spending is high and revenue low, said Richard Lin, a vice president at the Silicon Valley company Datastrato.
    
    “In two or three years, we will still be trying to figure out how large models can earn money,” he said.
    
    Offering low prices has helped the Chinese firms gain users, including in Silicon Valley, where many companies depend on the more affordable systems. Yet Chinese companies have struggled to translate huge numbers of global users into profits.

36. Matt O'Brien reports that Workplaces look for cheaper AI as ‘tokenmaxxing’ fades as a corporate fad:
    

    Bain & Company management consultant Jue Wang said many of the big businesses her firm advises have been taking a closer look at returns on their AI investments.
    
    “The token cost for them has been doubling, almost every other month,” she said. “Let’s say $200 per developer per month. Multiply that by 20,000 developers, which is often what we’re dealing with at these companies, and that quickly gets you to a number that is not a line item that any general manager has planned for.”
    
    Sometimes that just means not using the AI equivalent of a sledgehammer to crack a nut.

37. Both Ed Zitron in The More You Buy, The More You Lose and Torsten Slok on the Prof G Markets podcast The Market Is Running Out of Patience With AI focus on supply and demand in the credit market.
    
    A huge wave of supply of debt intended to fund the data center build-out for AI means that the price of this debt has dropped, and thus that the interest rate buyers are demanding goes up. This reduces the net pressent value of the (hypothetical) future cash flows the data centers will generate, and thus their ability to pay the interest and principal. This increases the interest rate the buyers of future debt will demand, causing a feedback loop.
    
    Both Ed and Torsten point to the increasing spreads above Treasuries that the AI debt is trading at, and the increased cost of insuring against these companies defaulting.
38. Katherine Burton et al report for Bloomberg on The 24-Hour Race to Salvage Situational Awareness’ Souring Bets:
    

    In a blink, [Leopold Aschenbrenner's] wildly successful hedge fund, Situational Awareness, was forced to sell billions of dollars of technology investments that had rapidly lost value, as nervous banks began to demand more and more collateral for his trades. Then came billionaire Ken Griffin.
    
    In less than 24 hours — which included a conversation between Griffin and Aschenbrenner — Griffin’s Citadel hedge fund reached out to Situational Awareness and snapped up the investments at a discount, according to a person familiar with the matter who asked not to be identified citing private information.
    
    It was a startling reversal for Aschenbrenner, a former researcher at OpenAI who — before starting his hedge fund roughly two years ago — had no previous investment experience. His fledging firm has watched its assets plunge from $45 billion at the start of July to about $10 billion.

    Aschenbrenner was all-in on AI.
    
39. On the Pof. G. Markets podcast legendary short-seller Jim Chanos focused on the accounting inequality inherent in the AI (and earlier dot-com) bubble. The spending of the hyperscalers and neoclouds is investment, expensed over say 5 years. But for the chip makers, construction companies and so on that spending is this year's revenue. So, overall, it looks like the ROI is better than it really i s.
40. Leap Financial Academy's The hidden financial costs of chasing AI supremacy and who is unwillingly footing the bill makes an interesting point about accounting for the Restricted Stock Units that companies use to reward their employees:
    

    US GAAP was built on the assumption that neither route would materially move profits, because which company would ever choose to grant ‘enough’ RSUs to employees that it could move the share price right?
    
    Meta has granted ~US$70bn in stocks. And Microsoft ~US$42bn. The sheer size of these numbers IS enough to move the needle materially. RSUs, granted in large quantities every year, compound into visible profit erosion and free cash flow consumed by “obligatory” share buybacks, as employees must receive their promised shares annually regardless of market conditions.
    
    As of FY25, total unvested RSU’s book value was ~US$60bn and corresponding market value was ~US$80bn. That $20bn gap is money Meta will have to find, one way or another, the moment these shares vest, either by diluting shareholders with fresh stock, or by spending real cash buying shares back to hand over instead.
    ...
    

    Source

    Using the extensive information provided by Meta’s financial notes on the RSU programme, we can back-solve the profit erosion that is being kept ‘off-the-Income-Statement’:
    
    Since December 2024, as the share price of Meta increased and the AI talent war started, ~10 points of EBIT margin per year have been given away as additional, uncaptured employee compensation.

    Source

    And this has decimated Meta's free cash flow.
    
41. Ed Zitron is back with another scathing analysis in The AI Demand Bubble. As usual, it is long and detailed. He concludes:
    

    To put things really simply, Anthropic and OpenAI are a way that hyperscalers can feed their revenue to themselves by spending money on capex, backstopping compute contracts, or doing direct equity investments.
    
    Their continued existence allows the AI bubble to continue inflating, but this can only continue as long as venture capital and hyperscalers are capable or willing to invest. There is simply not the demand — not from open source, not from other AI labs, not from self-hosting, not from anywhere — to justify the capex or the massive data center buildout.
    
    And for those arguing that there would be a dot-com bubble recovery story, I must be clear that if there isn’t demand today, it won’t magically appear tomorrow. AI GPUs will cost just as much to run in five years as they do today, as will unfinished data centers cost just as much to finish, as will electricity remain expensive, and all this will be happening after it’s easy to raise venture capital to actually buy the compute.

42. Issie Lapowsky reports that Tokenmaxxing Is Dead. Now Comes the Belt Tightening:
    

    In a recent survey of 300 business executives, 68% reported overspending their AI budget over the past year. It’s little wonder so many companies are now cracking down. Uber Technologies Inc. recently capped employees’ AI spending at $1,500 a month. Tesla Inc. reportedly set a limit of $200 a week. The swift reversal has left some workers complaining of whiplash, with Reddit forums filled with stories of engineers suddenly stymied by usage caps.

43. Source

    Luz Ding, Spe Chen and Hayley Warren have a major analysis of the threat to OPenAI and Anthropic in US Lead in the AI Race With China Is Rapidly Narrowing:
    

    Bloomberg in partnership with researchers at Vals AI, an independent AI evaluation and benchmarking platform, tested seven models from frontier Chinese and US companies to see how they performed in a real-world task. They were asked to create a fictional coffee e-commerce site called Brewberg using the same prompts. Most of the models scored 100% functional accuracy despite occasional design misses, but with very different price tags. The experiment employed the top performing models in July from Anthropic and all the Chinese firms, as well as more affordable models from OpenAI and Google.

    They all did reasonably well, but the two best were Claude Fable 5 at $48.99 and Kimi K3 at $11.99. Chinese models charging much less for almost the same performance are grabbing market share:
    

    the use of Chinese models overtook US platforms globally for the first time in June, and accounted for more than 60% of market share last month, on OpenRouter, a tech platform that offers software developers access to hundreds of AI models. It is a widely watched gauge of model usage despite tracking just a fraction of global AI consumption. The US, parts of Europe and Asia now favor Chinese labs, according to the same data.
    
    On Hugging Face, Chinese AI models account for 41.4% of generative model downloads among developers, 5 percentage points higher than US models.

44. In OpenAI’s unraveling has begun Gary Marcus notes that leaked Q2 numbers show OpenAI's quarterly revenue grew from $5.7B to $6.7B but its quarterly losses grew from $9.3B to $12.3B.
45. Michael Nuñez reports that Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs:

    Perplexity is launching Portable Computer today, a version of its agentic "Computer" platform that runs entirely on hardware users already own — starting with Nvidia's DGX Spark desktop supercomputer and Linux machines equipped with Nvidia RTX GPUs.
    
    The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user's files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.

46. Apple just announced the latest Mac hardware, as discussed by Samuel Axon in Apple’s new desktop computers are designed specifically for local AI development:
    

    Apple announced new iterations of both desktops, along with two new chips: the M6, the first 2nm chip in Apple’s M-series lineup for Macs, and the M5 Ultra, now the most powerful chip in the lineup for most things—especially AI workloads.
    
    There aren’t any major new features for either machine. This is just a specs bump. But based on how Apple is presenting these refreshes, they’re leaning hard into those use cases, which weren’t even a thought when earlier iterations were first engineered.
    
    The devices’ popularity for production inference took off after macOS 26.2 shipped last December. According to Apple’s release notes, 26.2 enabled “low-latency communication between Thunderbolt 5 hosts for use cases including distributed AI inference using MLX.” Thunderbolt 5 is a very fast wired data connection, and MLX is an open source array framework designed to help machine learning workflows take full advantage of the M-series chips’ unified memory.
    
    Since then, both hobbyists and professional developers and researchers have been essentially daisy-chaining Mac minis or Mac Studios to run inference on local large language models that are much bigger than anything that could run a single mass-market device—providing an alternative to ultra-beefy specialized hardware featuring specialized Nvidia GPUs.
    

47. IBM's latest model release continues the trend favoring local inference, as Samuel Axon describes in IBM’s new Granite 4.2 models ride the wave of interest in local LLMs:
    

    IBM has rolled out the newest models in its family of open-weight large language models designed to be downloaded and self-hosted. The newly launched Granite 4.2 comes in 3B, 8B, and 30B parameter variants.
    
    Like previous versions, IBM is taking a decoder-only approach here. These new releases offer a 128,000-token context window natively. The 8B and 30B variants (not the 3B one) also go through an agentic reinforcement-learning block; they were trained for expanded capabilities like using the terminal, searching the web, or using external tools. The 3B model supports tools too, but without the same level of specialized training.
    
    Beyond those tweaks, this release is particularly notable because, as IBM itself writes, “Granite 4.2 is the reasoning-focused release of the Granite language-model family.

48. The subhead of Bryce Elder's US data centres ‘are short six NYCs of electricity’ is:
    

    With great power comes great implausibility

    Source

    Elder reports that:
    

    Morgan Stanley analysts today revised their model for how much power would be needed to switch on the chips it projects will be sold. They now estimate, based on a total data-centre power requirement of 257 GW by 2028, a gap of between 30 and 40 per cent between US capacity and sales projections. Or to put it in layperson’s terms:
    

    New York City demands 5.5-6 GW of baseload power: we are short power by six New York Cities.

    And that’s using their mid-estimate for innovative “time-to-power” solutions. Absent such mitigation measures, the estimate rises to 10 NYCs:

    The innovative solutions include a massive expansion of Bloom Energy fuel cells, data centers alongside nuclear power plants, and other equally optimistic concepts.
    
    But Morgan Stanley are equally optinmistic about Nvidia:
    

    Fenyman, Nvidia’s next next chip architecture, is scheduled for launch in the second half of 2028. It’ll be another step change, Morgan Stanley says: the same dynamics will increase US power demand year-on-year by a further 40 per cent in 2029, Byrd and team estimate. They’re then looking at a cumulative power shortfall of 107GW (or approximately 18 NYCs).

    Morgan Stanley add one big if:
    

    “However,” the team add in bold, “all of these advancements converge on one central issue: if we cannot close the power shortfall, the expected pace of improvement and adoption may not fully materialise in line with market expectations.”
    
    They probably don’t need to spell out what happens to all those big tech off-balance-sheet guarantees in that scenario. Obviously, the faster-rising thing will eventually overwhelm the slower-rising thing,

    I think "market expectations" are out over their skis.
49. Realism is starting to slowly seep into the AI market. Omor Ibne Ehsan's Some Anthropic Investors Now See a $1.5 Trillion IPO, Not $2 Trillion is a foreshadowing:
    

    On October 15, 2026, Reuters reported, citing people familiar with the matter, that some current and prospective Anthropic investors are considering a $1.5 trillion valuation for the AI company's planned November IPO. That is below the roughly $2 trillion investors had expected.

    There are months ahead before Anthropic's valuation is set at the IPO, if it ever happens. If Anthropic's IPO fails to set the world on fire, OpenAI's delayed IPO will never happen and in will get absorbed by Microsoft.
    
    Three months ago the AI horizon looked much better and, George Hammond et al report,Anthropic secured highest junk credit rating in June by George Hammond et al:
    

    The Claude maker received an issuer rating of double B plus from KBRA in June, the highest speculative-grade score, in the run-up to securing a $35bn chip financing led by Apollo, according to documents seen by the FT.

    That was before the on-going Coreweave meltdown, the Firmus IPO failure, and so on. Anthropic wouldn't get BB+ today.