The $28.5T forecast compares to U.S. Q1 2026 nominal GDP of nearly $32T, with the estimate for the market of AI enterprise applications of $22.7T about 70% of total U.S. economic output.
Sam Altman and Dario Amodei just aren't this good, but their projections of their Total Available Market (TAM) are still turning out to be vastly optimistic. In AI's Affordability Crisis I showed evidence that the AI platforms could no longer afford the massive subsidies they were using to artifically inflate demand for their product, and that reducing the subsidies had made their enterprise customers reconsider their enthusiasm for deploying them. This is leading to investors belatedly realizing that AI platforms' projections of their TAM and thus their valuations are totally implausible.
This re-calibration is just one of the many signs that the AI bubble is about to deflate. Below the fold I present a necessarily incomplete list of them, which I will try to update as more appear.
AI now accounts for nearly half of all IG issuance, 87% of VC funding and a growing share of HY, underscoring how deeply the AI investment cycle has penetrated every corner of finance.
Here’s the title page of this month’s Panmure Liberum market update from strategists Joachim Klement and Francisca Reis. Our emphasis in bold below:
In 1929, the cyclically-adjusted P/E-ratio (CAPE) of the S&P 500 reached 32.6x according to Prof. Robert Shiller’s data. This was 1.8 standard deviations above trend at the time. In 2000, the CAPE reached 44.2x, or 3.3 standard deviations above trend – a clear sign of a bubble. However, as our chart below shows, earnings in both instances were within normal range, less than one standard deviation above trend.
Today, the CAPE is at 41.0x, or 2.9 standard deviations above trend. Once again, we are clearly in bubble territory for stock market valuations. However, unlike in previous bubbles, we are having extremely high CAPE at a time when earnings themselves are 1.8 standard deviations above trend. In other words, we are in a valuation bubble at a time when earnings are in a bubble themselves.
If we correct for the earnings bubble, the current CAPE would be 67.6x or 4.6 standard deviations above trend, a bubble that surpasses anything ever seen in US history by an extreme margin. If valuations followed a normal distribution (which they don’t, so don’t take this literally), this would happen in 0.00019% of months or once every 43,432 years.
A massive chunk of this quarter’s blockbuster “growth” didn’t come from selling more software, shipping more microchips, or delivering more packages. Instead, it came from an accounting rule that forced massive, illiquid “paper gains” onto the income statements of tech giants.
In Q1 2026 alone, just three companies—Alphabet, Amazon, and Nvidia—reported a staggering $69.2 billion in non-operating windfall under their Other Income and Expenses (OI&E) lines. When you run the macro numbers, this single accounting phenomenon artificially inflated the entire S&P 500’s quarterly earnings by about 12%.
That 12% takes the factor from 67.6 to 75.7. Wang notes that correcting for the 12%, "the Q1 2026 earnings growth rate will not be very different from the 5-year average of 16%." In other words, the bubble is feeding upon itself — increased stock prices causes increased earnings causes increased stock price ... But suppose, for example, that OpenAI were to suffer a down round. Then Alphabet, Amazon, Nvidia and others who included paper gains in the "other income" on the way up would have to include paper losses in their income on the way down, amplifying the crash. OpenAI's last round valued the company around $750B and they were planning an IPO for at least $1T, but had to postpone it.
The foundation model era — roughly 2020 to 2025 — is over. The forces that defined it have inverted. Open source models have reached frontier performance while inference costs approach zero, exposing what was always structurally true: pre-training large language models at scale is not a durable competitive moat. The US government's formal designation of Anthropic as a supply chain risk in February 2026 accelerated a transition already underway — but did not cause it. The paper argues that the AI industry is restructuring simultaneously along four axes: economic, as the circular financing structure that inflated foundation model valuations collapses; technical, as the pre-training scaling paradigm gives way to post-training optimization, test-time compute, and agentic composition; commercial, as application-layer integrators displace the foundation model companies whose commodity they now consume; and political, as the government asserts its historic role as gatekeeper of strategic technology. These are not separate disruptions. They are one structural shift, arriving together.
The most consequential and least-discussed dimension is the permanent divergence between commercial AI and a classified national security AI track — built on different data, governed by different rules, and developing capabilities the public ecosystem cannot see, measure, or govern. Like every dual-use technology that has altered the calculus of state power, AI is being brought under government authority not by design but by the structural logic of what it is. The paper further argues that open-weight models are the counterintuitive instrument of sovereign control: a government that holds the weights commands the capability on its own terms, without dependence on vendor policy, financial continuity, or personnel clearance. The apparent openness of distributed model weights is, from a deploying government's sovereignty standpoint, the most governable architecture — because what cannot be withdrawn by a vendor's API policy cannot be taken away.
Grogran implicitly assumes that AI is useful but that the current margins and thus the valuations aren't sustainable.
DeepSeek-V4-Pro is priced through its API at $1.74 USD per 1 million input tokens on a cache miss and $3.48 per million output tokens.
That puts a simple one-million-input, one-million-output comparison at $5.22. With cached input, the input price drops to $0.145 per million tokens, bringing that same blended comparison down to $3.625.
That is dramatically cheaper than the current premium pricing from OpenAI and Anthropic. GPT-5.5 is priced at $5.00 per million input tokens and $30.00 per million output tokens, for a combined $35.00 in the same simple comparison.
Claude Opus 4.7 is priced at $5.00 input and $25.00 output, for a combined $30.00.
Six times cheaper for an equivalent product is likely to cause pricing pressure on the incumbents, who need to raise not reduce prices. DeepSeek is likely also subsidizing usage, but they do have real advantages. First, they have fewer resources so are forced to to be inventive. Second, the 40% of infrastructure capex that isn't the racks is much cheaper in China. Third, the power component of opex is much cheaper and more available in China. The result is:
In practical terms, DeepSeek does not need to win every leaderboard row to matter. If it can deliver near-frontier performance on many enterprise-relevant agent and reasoning tasks at roughly one-sixth to one-seventh the standard API cost of GPT-5.5 or Claude Opus 4.7, it still forces a major rethink of the economics of advanced AI deployment.
DeepSeek-V4-Pro-Max is clearly the strongest open-weight model in the field right now, and it is unusually close to frontier closed systems on several practical benchmarks.
While GPT-5.5 and Claude Opus 4.7 still retain the lead in most direct head-to-head comparisons across the company's benchmark charts, DeepSeek V4 Pro gets close while being dramatically cheaper and openly available.
Dan Davies agrees that the margins aren't sustainable but differs as to why in tokenalysis and john henry:
Which then brings to mind another issue – how confident are we in the pricing power that underpins that 60% gross margin in the first place? In the last paragraph I was talking about the R&D equivalent of a price war, but the normal kind is also possible. The combination of price-sensitive B2B customers, big fixed costs and rewards going to the dominant player doesn’t suggest to me that pricing power is going to be sustainable indefinitely.
But, I think there’s a danger of missing the big picture here. Which is that, when large companies are telling their employees to be sensible and use AI tokens wisely, then the game is up. The race is over and John Henry won against the steam hammer. If you need a human being in the loop to decide on the allocation of AI tokens, then all those predictions of mass redundancy are gone.
But what if the payoff takes longer than consensus assumes? That question is particularly pressing given that token prices continue to decline and Chinese models are gaining ground, both in their share of the world's most-used models and in token usage, where they now lead their US counterparts among the top 20 models,
Slok provides two charts, the first tracks market share by country of origin monthly since January 2025 among the top 50 models. It is bad for the US, showing a steady erosion of US market share until, eyeballing it, as of May 2026 it is roughly a 60/40 US/Chinese split.
Clearly, the market is voting with its feet that the prices charged by US AI companies are unsustainable.
The second compares monthly token use among the top 20 models by country of origin between May and June this year. It is much worse for the US.In May the Chinese had 80% of the US usage. In June they had 185% of the US usage. If this rate of market erosion were to continue for a few months it would be impossible for investors to continue to imagine the golden future awaiting OpenAI, Anthropic, xAI, Meta, Oracle and the neoclouds.
Slok's analysis of the fallout of the bubble deflating is worth reading. This is an issue I plan to return to in a future post.
Chinese AI providers are making more money; Zhipu’s revenues went up 60 times in Q1 compared to last year; Alibaba is the company behind Qwen, and their revenues are up 15 times just since the beginning of this year. The lion’s share of the profits, though, are still being realized by the hardware side; the chipmakers, at least for now. The model companies’ revenues are rising, and steeply. But so is their cost of compute.
That dynamic is causing some Chinese labs to raise prices; the cost to use Tencent Cloud increased by over five times in March, and Alibaba, Zhipu, and ByteDance also hiked prices.
DeepSeek, however, went the other way. Their latest version is priced at just a fourth of their introductory product, and they rolled out dynamic pricing that is aimed at corporate users, who use their models during the workday.
But even with these price increases, Chinese models just cost far less than those on offer from Silicon Valley, and explains those big jumps in the exports, we can say, of Chinese AI tokens. The LLM’s out of China are “90% as good at 10% of the cost”, and American firms are buying more tokens from Chinese companies. In early 2025, token demand from Chinese LLM’s was about zero. Even the release of DeepSeek didn’t move the needle much, but by the end of the year the secret was out, and it’s been a choppy but steady ride up to 46%, today.
Why would you invest in a company whose competitors were “90% as good at 10% of the cost” and which was rapidly losing market share?
If the margins, and thus the rational valuations, of AI companies are unsustainable, how long can the market remain irrational? In The Second Derivative: Why No One Understands the AI Boom Groundbreaker starts by examining the 2008 crash:
The implicit underwriting assumption, shared by originator and borrower alike, was that the loan would never reach its reset: rising home values would manufacture equity, the borrower would refinance into a fresh teaser and the clock would start again. The structure was a treadmill, and the treadmill was powered by appreciation. It worked spectacularly while it worked. Nearly four in five subprime hybrid ARMs originated in 2003 had been refinanced away by the end of 2006.
Now watch the timing. National home-price appreciation did not crash in 2006. It decelerated. The year-over-year rate of gain, which had run in the mid-to-high teens through 2004 and into early 2005, began bleeding off - still positive, still printing green, but slowing. Prices were higher than they had ever been. And yet, with prices at their peak and still rising, subprime delinquencies inflected upward.
...
The deceleration was endogenous to the structure; the structure required ever-accelerating prices to keep refinancing its way out of its own reset schedule, and no series accelerates forever.
The second derivative was always going to roll over. When it did, the first derivative followed it down through zero, negative equity spread from the margin inward, and the defaults the market insisted were caused by “falling prices” had in fact begun a year earlier, when prices were still rising but had stopped rising faster.
The market is pricing AI as a technology cycle when its actual anatomy is that of a credit-driven real estate cycle - which is precisely why the 2008 mechanics apply - and the two break for entirely different reasons.
...
Walk down the AI build-out and every feature is a property development in disguise: a data center on entitled land, financed with debt against the structure and leased to tenants on take-or-pay terms. This is not a software business that happens to own servers. It is a real estate business that happens to compute.
When a hyperscaler or a neocloud reports a record capex figure, the financial press reads it as confidence, as proof of demand. Read it instead as origination volume. Each gigawatt of committed build is a loan extended to whichever tenant has signed the take-or-pay beneath it, and the credit quality of that loan is precisely the credit quality of the tenant. The market is celebrating loan growth and calling it revenue growth.
Loans in this market take the form of Remaining Performance Obligations (RPOs). The biggest borrower in this market is OpenAI:
OpenAI has committed to pay for compute on a scale without precedent in corporate history: multi-year, take-or-pay capacity contracts whose aggregate obligations run to the hundreds of billions of dollars. Against them sits an operating business that does not yet earn a profit - revenue real, large, and growing quickly, but short of covering the company’s own cash burn and nowhere near covering the contracted payments.
Those payments are therefore not serviced out of earnings. They are serviced out of financing, and financing, for a borrower in this position, is available on a single condition: that each new round price above the last.
Measured as a level, OpenAI’s valuation is the most remarkable appreciation in the history of private markets - roughly $86 billion in early 2024, then about $157 billion, $300 billion, $500 billion, and approximately $852 billion by the spring of 2026. Measured as a rate of change, the same series inverts: the round-over-round step-up ran 1.83×, 1.91×, 1.67×, 1.70×, and falls to roughly 1.23× implied by the reported public-offering target. Private marks are inherently lumpy - negotiated, episodic, set by a handful of insiders - so no single step is decisive. But the trend is unmistakable: it bends down, and it bends hardest at the one mark set by the deepest, most unforgiving pool of capital - the public market. The implied IPO step-up is both the lowest in the sequence and the hardest to negotiate, and it is the one the structure must actually clear. This arithmetic is also the most probable explanation for OpenAI’s recent IPO delay.
In the same way that the private stocks generating unrealized gains are not liquid assets, neither are the Remaining Perfoemance Obligations:
An RPO is not a liquid asset; it is a forward contractual commitment - a promise of future payment in exchange for future compute. And a multi-year commitment is worth exactly the creditworthiness of the entity on the other end of it. When that entity is investment-grade and cash-generative, the backlog is what it claims to be: high-quality visibility, merely deferred. When that entity is a pre-profit company that loses tens of billions a year and can pay only by continuously refinancing its own equity valuation, the backlog is something else entirely. It is a subprime commitment, used to justify massive, un-depreciated capital expenditure, reported to shareholders as structural strength.
Now price the credit quality of that book. Of roughly $2.1 trillion in aggregate contracted backlog across the four big platforms, about half - on the order of $1.05 trillion - is owed by OpenAI and Anthropic. Microsoft’s book is about 49% these two names; Oracle’s is 54%, with roughly $300 billion owed by OpenAI alone; Google’s is 43%; Amazon’s is 51%.
The hyperscaler has, in economic substance, extended a concentrated, unsecured loan to cash-burning tenants. The RPO that Wall Street values as forward revenue is, in reality, a credit exposure to borrowers with no operating income.
Major financial institutions are similarly concerned. In their 2026 Annual Economic Report, the Bank for International Settlements writes:
In the near term, the ongoing AI investment boom raises questions about the sustainability of the current economic expansion. The five largest hyperscalers are set to spend over a trillion US dollars on AI-related capital expenditure from 2025 through 2026. These commitments are outpacing earnings and the free cash flow of these firms, leading some to issue debt to raise additional financing (Graph 11.A). This investment race may be partly driven by the perception that only a small number of players with superior technology will ultimately dominate the market shares. The intense competition raises the risk of firms over-committing resources to investment projects with still uncertain returns, leaving all firms vulnerable to disappointments in AI payoffs. Model analysis based on such contest motives highlights the downside risk of current AI exuberance. As competitive pressure drives capex higher, the net economic surplus – the total payoff less investment costs – declines for the sector as a whole and could turn negative in adverse scenarios (Graph 11.B). Disappointment in returns could trigger a sudden pullback in financing and turn the capex boom into a protracted investment bust, with potential knock-on effects on financial conditions (see below).
Another risk is that the AI boom runs into a supply side roadblock. The AI build-out has recently been facing growing bottlenecks in electricity, advanced semiconductors and grid equipment. Fast-growing demand for computing power is already pressuring electricity prices and input costs, with potential spillovers to inflation. Looking ahead, these temporary shortages may also amplify over-investment, as firms attempt to lock in future capacity through long-dated contracts that further expose them to any disappointments in demand.
Historical episodes of investment booms offer instructive parallels (Graph 11.C). The canal mania of the 1830s, the British railway mania in the 1840s, the electrification exuberance of the late 1920s (roaring 20s) and the dotcom boom of the late 90s all shared one common trait: a genuine technological breakthrough that attracted capital in excess of what commercial returns could ultimately justify. These episodes ended with an eventual reversal in investment, inducing economy-wide recessions. The scale and pace of the current AI investment boom accompanied by expectations of large productivity payoffs bear resemblance to these precedents, highlighting potential downside risks in the near term.
A draft report inside the Treasury Department is set to warn of the risks posed by the artificial intelligence market, likening key aspects of it to the dotcom bubble that upended the U.S. economy when it burst in the early 2000s.
The document, the existence and contents of which have not been previously reported but was obtained by NOTUS, is a significant departure from the Trump administration’s public tone, which has focused on encouraging unrelenting investment to unlock exponential growth.
Career Treasury analysts found that AI firms are more deeply entrenched in the U.S. economy than their dotcom predecessors and pose significant risk to the entire system if financial conditions change, productivity goals are missed or various choke points stymie growth.
Vanderbilt University's Asad Ramzanali has a detailed look at the range of impacts from the burst bubble, and an set of optimistic suggestions for policy responses in After the AI Crash. He frames the problem thus:
Companies are investing trillions of dollars based on tens of billions of dollars in revenues. Analysts at J.P. Morgan anticipate $5 trillion of AI infrastructure investment in the next five years. They estimate that the industry will need to generate annual revenues of $650 billion to justify this level of investment, while consultants at Bain & Co. estimate $2 trillion in needed annual revenues. Yet, OpenAI and Anthropic earned $13 billion and $4 billion, respectively, in 2025 revenues. OpenAI’s own financial expectations suggest negative cash flow until 2030, and Anthropic expects a small profit no sooner than 2029. Alphabet, Meta, Amazon, and others may experience increased marginal revenue from integrating AI into existing products, but that is far from certain.
Note that over the next 5 years around $3T (60% of $5T) of the investment in AI infrastructure goes in to buying the hardware which should be fully depreciated over much less than 5 years. So the gap between the investments and the revenue is much bigger than it appears.
Blackstone's QTS said on Thursday it had terminated its planned Digital Gateway data center project in Virginia and withdrawn the associated filings after years of planning and regulatory review.
The data center operator has faced years of local opposition and litigation over the project, despite it being approved by the Prince William Board of County Supervisors.
Among the hyperscalers, Oracle is the most exposed because, as Ed Zitron noted:
And Oracle ... is a company that, even before the AI bubble, was massively indebted. It just so happens that, as a result of its tryst with OpenAI, Larry Ellison saw fit to twist the debt knob to eleven.
Six firms alone — including Oracle, Microsoft and Meta — have committed $850 billion for data centers leases that haven’t begun yet. Oracle holds the largest share of these commitments owing to its $300 billion Stargate contract with OpenAI.
When Oracle mentions the risk of nonpayment, the unnamed elephant in the room is OpenAI. As part of the Stargate deal with the AI company, Oracle is developing massive data centers across the country to provide cloud computing power. For this plan to work, OpenAI needs to pay its Oracle Cloud Infrastructure bills.
“Some of our customers may be highly leveraged and subject to their own operating and regulatory risks and, even if our credit review and analysis mechanisms work properly, we may experience risks of non-payment and non-performance in our dealings with such parties,” Oracle said in the filing.
SoftBank Group Corp.’s talks with potential creditors to raise at least $6 billion from a margin loan backed by its OpenAI stake have stalled, people familiar with the matter said, just weeks after the Japanese conglomerate cut its initial target from $10 billion.
SoftBank Group has reopened talks with a consortium of lenders for a $10 billion loan backed by its stake in OpenAI, after earlier attempts to secure a loan stalled over concerns about the difficulty of valuing private companies, two people familiar with the matter said.
To make lenders more comfortable, the Japanese technology investor is offering to guarantee repayment of the loan, giving banks recourse to SoftBank if the OpenAI shares pledged as collateral lose value, the people said.
This all seems to indicate that potential lenders, such as banks, are highly skeptical of the value of OpenAI stock.
Despite Grok being so bad that employees use Claude, Musk is touting SpaceX as an AI company. This resulted in the most overvalued IPO in history, which failed to raise enough money to avoid the immediate need to borrow $25B. Nir Kaissar's SpaceX Is Junk. That’s What the Bond Market Says reports on the bond market's reaction:
Ratings companies and the bond market have very different views about how things are going. SpaceX’s bonds have an average rating of BBB across the three majors, Moody’s Ratings, S&P Global Ratings and Fitch Ratings, according to credit scores compiled by Bloomberg. In the alphabet soup of bond ratings, it’s the lowest grade still considered quality before falling into junk territory.
The bond market has other ideas. There, quality is judged by a bond’s credit spread or the additional yield it offers above Treasuries with similar maturity. The wider the spread, the lower the quality. Corporate bonds with a BBB rating are trading at an average credit spread of 0.92 percentage point. SpaceX’s bonds, by contrast, trade at a significantly greater average spread of 1.62 percentage points across maturities, higher than BB rated junk bonds’ average spread of 1.55 percentage points.
The bond market seems to agree with Softbank's lenders about the AI bubble.
OpenAI and Anthropic are competing for the next trillion-dollar IPO. Both would need to distract investors from their massive losses by focusing on growth. Recently, Anthorpic has been growing faster than OpenAI, so Keach Hagey and Berber Jin report that OpenAI Considers Drastic Price Cuts, Anticipating War for Users With Anthropic:
OpenAI is considering drastically lowering the prices it charges users as it seeks to win customers from its rival Anthropic.
The company is weighing significant cuts to what it charges for tokens, the unit of measurement artificial-intelligence firms use to bill for their products, according to people familiar with the matter. The move would be in anticipation of similar cuts the company expects at Anthropic, the people said.
Meta will also introduce a new Meta Model API system, which will be used to collect fees from developers. Its API pricing is roughly 25% of the cost advertised by other top models from OpenAI and Anthropic PBC. Developers will be able to use Meta’s model for free, but only up to a point; they’ll be required to pay for access after reaching a certain token threshold, Zuckerberg said.
“The pricing from some of the other labs is very extreme and has very high margins,” Zuckerberg said, underscoring that his strategy is to get Meta’s technology in front of as many people as possible. “We think that there’s a real ability to be able to offer frontier or very high-level intelligence at a much more affordable cost.”
Aggressive pricing means Meta Model API will still be 50% more expensive than DeepSeek but not 50% better. Planning to reduce current income in the lead-up to an IPO is an unusual move, but it is a response to AI's Affordability Crisis.
Factory electricity bills are generally rising faster than those for other business customers or residential customers, according to a Reuters analysis. It highlighted the example of the Belden Brick Company, a 141-year-old brick manufacturer in Ohio, whose electricity bills have soared from $1,600 to $12,000 per month due to a higher monthly capacity charge in the 13-state region served by the grid operator PJM Interconnection.
...
The Ohio-based steelmaker Metallus described its electricity costs as having jumped by 70 percent since 2024, leading the company to pay an extra $15 million in energy costs annually.
The higher electricity costs for manufacturers coincide with the attraction of large AI data center projects with substantial electricity needs to many states in PJM territory. That data center growth has driven up PJM’s capacity prices—paid to power generators according to supply-and-demand forecasts—from $28.92 per megawatt-day in 2024 to $329.17 per megawatt-day in 2026, according to Reuters’ reporting.
Source
As I see it, there are three separate markets for LLMs. First, there is an embedded market that runs on low-cost, low-power hardware and open-weights models. Small AI Models Gain Traction Around the World by David Berreby provides examples:
For example, a drone-based system developed by Bala Murugan and colleagues at the Vellore Institute of Technology, in India, takes photos of cashew plants and quickly identifies those with splotches that indicate disease. All the processing takes place on the drone itself, so there’s no need for a computer on-site, nor for a connection to a central server.
It isn't just that small hardware, such as the Raspberry Pi 5, is getting more powerful but also:
the shrinking footprint of language models. Both Google DeepMind’s Gemma 4 (released in April) and Alibaba’s Qwen 3.5 are “fantastic” for small AI, Rovai says. Both models are “open weight,” meaning users can adjust the connections between parameters to suit their needs. This makes it easy, for example, “to take a lot of data from, say, the milk industry and retrain the model specifically on that,” Rovai says.
The hyperscalers and AI platforms like OpenAI and Anthropic will garner no income from this market, because:
“I think the future of AI is not like one giant model, at a center. I think it’s millions of small, precise models deployed at the edge, each one solving like a specific problem, a specific context,” Alonge says. This is partly because much of humanity—including people in parts of rich countries as well as the developing world—lives without access to cutting-edge frontier models. But, he says, it’s also because those models are not sustainable.
“If someone is not subsidizing it, most people will not be able to afford those models. So those of us who are said to be small-AI developers are the ones who will have to build for the majority of the world,” Alonge says.
At Computex 2026 in Taipei on June 1st, CEO Jensen Huang announced the RTX Spark superchip — a single piece of silicon that combines a 20-core Arm CPU, a Blackwell GPU with 6,144 CUDA cores, and 128 gigabytes of unified memory, connected by NVIDIA’s NVLink chip-to-chip interconnect. The whole package delivers up to one petaflop of AI compute in a laptop form factor.
The number that matters: RTX Spark can run a 120-billion-parameter language model entirely locally, with a context window of one million tokens, without a single byte leaving your machine.
To put that in perspective: GPT-3 had 175 billion parameters and required clusters of A100 GPUs to run. The model that stunned the world when it launched in 2020 is now approximately the size of what fits in a consumer laptop chip announced this week. The capability that required a data centre in 2020 is coming to a device you carry in a bag in 2026.
In 2025, slightly more than a third of all smartphones shipped worldwide were capable of running generative AI, and that figure will reach 45 percent by the end of this year, according to the technology research firm Counterpoint. By the end of next year, slightly more than half of all smartphones will be able to run a small AI model.
If Nvidia can already put a 120B-parameter in a laptop, it will only be a few years until phones can run a GPT-3-class model, good enough for almost all consumer needs. Apple and Google own that channel. Owning the channel is better than owning the technology. They will dominate consumer AI, and the other players will garner no revenue from this market.
Third, there is an enterprise market; all that is left to generate the revenue to service the debts fuelling the AI bubble, lets say $2T by 2030. There are a number of problems that make this unlikely.
First, there are very few documented cases of LLM deployment that resulted in enough productivity improvement to cover its unsubsidized costs.
Second, the Trump administration just demonstrated that deploying mission critical systems on AI platforms such as OpenAI or Anthropic means your company can be disabled at 90 minutes notice with no recourse.
Third, this means that companies will have to run mission-critical LLMs on open-weight models on in-house hardware if they are not to be vulnerable to the whims of the US president.
Fourth, systems such as RTX Spark show that good enough in-house hardware is likely to become relatively cheap compared to the unsubsidized cost of the AI platforms. In-house systems need much less over-provisioning for demand spikes, and because they aren't shared they don't need to be as fast.
Fifth, companies need to balance the productivity benefits (if any) of mission-critical LLMs against the productivity costs they bring. These include a vastly greater attack surface, technical debt from reduced developer understanding of the software, and so on.
Thus it seems likely that the hyperscalers and AI plaforms will generate far less revenue than they expect, because they will be restricted to non-mission-critical applications with lower productivity gains, and thus lower pricing power. They will thus be unable to cover the debts they are incurring to build massive data centers predicated on centralized systems dominating (an inflated estimate of) the entire enterprise market.
Karp had been softly floating his critique for some time, but the CNBC event looked like a proper coming out. Just one day earlier Palantir had published a kind of manifesto devoted to what it described as the all-important principle of “A.I. sovereignty.” The central argument: Companies should seek to build their own A.I. tools, not just customize those on offer from the frontier labs. This might mean relying on open-source L.L.M.s rather than the proprietary ones on which the A.I. boom has mostly been built in America, but it would amount to a liberating declaration of independence from Big A.I., which in Karp’s estimation was sucking up much more value than it was generating.
Locking in a price with a multi-year neocloud contract insures a buyer against compute getting more expensive but not against it getting much cheaper. So as businesses around the world spend ever more on compute, they want to be able to hedge against price volatility just as they insure against changes in energy tariffs, interest rates or foreign-exchange movements—ideally in deep and liquid derivatives markets.
Two startups want to help companies do this, by turning nascent indices tracking compute costs into a futures market. Silicon Data, founded in 2024 and backed by DRW, a trading firm, has paired up with CME Group, which operates large derivatives exchanges. Ornn, created by recent graduates of the Massachusetts Institute of Technology and run from a flat rather than an office just a few months ago, has paired up with Intercontinental Exchange, the parent company of the New York Stock Exchange, to do the same. Both aim to launch compute futures later this year, to be traded on their partner exchanges.
One characteristic of bubbles is overbuilding the infrastructure. Signs of overbuilding of data centers include that both SpaceX and Meta are now in the business of rentling their GPUs to the competition.
As morale is hitting rock-bottom, his company is heavily relying on its competitors' AI models to build out its own in-house tools. And despite the many billions of dollars the company has spent in its flailing efforts to keep up in the AI race, even Zuckerberg himself is now acknowledging that progress is nowhere near where he wanted it to be.
As Reuters reports, Zuckerberg admitted during a town hall last week that AI agents in particular aren't progressing as fast as he anticipated, a devastating revelation following enormous layoffs that wiped out thousands of roles at the company.
The "trajectory of the agentic development over at least the last four months hasn't really accelerated in the way that we expected," he said according to a recording obtained by Reuters.
Despite a derisory ~3% share of the enterprise LLM market, SpaceX's IPO was marketed as an enterprise AI company. The IPO has to rate as the most manipulated of all time featuring, in addition to ludicrous financial projections, a tiny float, bent rules for index inclusion, and massive conflicts of interest at the banks and the analysts. Immediately afterwards, SpaceX issued $25B in bondsa. A month later we can see how the markets respond to the first trillion-dollar "AI company".
We’re old enough to remember when the market cap of the lossmaking telecom SpaceX was bigger than Amazon’s. Heck, for a few precious moments it was bigger than Microsoft’s. Maybe one day it will be again, but for now the stock is down 38 per cent from its peak post-IPO valuation.
Punters lucky enough to have been awarded a stock allocation at the outset are still sitting on a tasty [checks notes] 0.8 per cent paper profit at pixel time.
And this is before the lockups start expiring and the initial tiny float greatly expands.
SpaceX bod spread
Second, more interesting as being much less subject to manipulation, are the bonds. They are what Toby Nangle focused on in SpaceX bond yields rocket towards junk:
the full $25bn of benchmark bonds — issued across the curve — had a rocky first couple of days of trading. Checking back today, it turns out that the inauspicious beginning was just a prelude to the train wreck that has since unfolded.
...
If you’d been allocated $100mn of the SpaceX 2056 bonds, you’ve turned $100mn into $90.7mn in less than a month. Sure, long-dated US Treasury bonds have fallen in value, and this general sell-off at the long end has done some of the work. But the spread on SpaceX 2056 — the additional yield you’re paid to compensate you for the risk that you don’t get repaid (among other things) has now widened from the initial +175bps to a whopping +231bps doing more than two-thirds of the work.
Top 10 worst BBB
The same conflicts of interest that pumped the stock caused rating agencies to give SpaceX bonds a BBB rating, one notch above junk and crucially the lowest that many major institutions are allowed to hold. But Nangle notes that:
Looking only at the nine days since the bonds were included in ICE BofA indices at the end of June, this spread-widening has made SpaceX 2056 the single worst-performing US dollar triple-B benchmark bond:
Note that the top two worst bonds are SpaceX, but the rest of the top 10 are all Oracle, another financial disater area.
when we overlay the average spread for double-B US dollar corporate bonds across different maturities (the pink line), it looks a lot like the type of risk that the market has assigned to both SpaceX and Oracle bonds is junk risk.
SpaceX, OpenAI, Anthropic, Meta all need to raise vast amounts of debt to fund their plans for AI. With SpaceX's bonds trading as BB despite a BBB rating, this is going to be hard.
Founded in early 2025 by former OpenAI CTO Mira Murati, Thinking Machines' first model is a big one. Weighing in at 975 billion parameters, the model requires more than two terabytes of GPU memory — a quantity present in around eight of Nvidia's B300 accelerators, or sixteen H200s — to run at its native 16-bit precision. If that's asking too much of your hardware, Thinking Machines has also released a NVFP4 quantized version of the model capable of running on half the GPUs.
This makes it the largest American open weights model to date, and comparable to Chinese models like DeepSeek V4, GLM 5.2, and Kimi K2.6 in terms of size and capabilities. Take these claims with a grain of salt — gaming AI benchmarks isn't exactly difficult – but Thinking Machines says Inkling is competitive with these models in a variety of workloads, although its benchmark charts also show it trailing proprietary models like Anthropic's Claude and OpenAI's GPT.
If you don't like Chinese open-weight models, the US ones are getting better:
The model developer claims to have tuned the model to use these thinking tokens more efficiently and that Inkling therefore matches Nvidia's Nemotron 3 Ultra, up to now the largest and most capable American open weights model out there at 550 billion parameters, on Terminal Bench 2.1 using roughly a third the tokens.
The AI revolution is supposed to be in its early innings. Demand for GPU computing is exploding. Microsoft has relied on CoreWeave for massive amounts of AI compute. OpenAI has signed tens of billions of dollars in long-term contracts. NVIDIA isn’t just supplying the chips, it has also invested in CoreWeave and entered into agreements that support parts of its financing and capacity.
So why are CoreWeave insiders continuing to sell stock? Management says many of the sales were made under pre-arranged Rule 10b5-1 trading plans. But those plans explain how the shares were sold—not necessarily why executives continue converting stock into cash while investors are being told AI infrastructure demand has never been stronger.
New York has become the first U.S. state to stop construction of large new data centers, imposing a one-year moratorium due to growing concerns over power costs, water supplies and the burden on local communities.
"As data center development threatens to hike up utility bills, deplete our natural resources, and create uncertainty for New Yorkers, it's my responsibility to take action and lead," said New York Governor Kathy Hochul.
She added that she would also pursue legislation to repeal sales tax exemptions for large data centers.
Phurichai Rungcharoenkitkul continues the Bank for International Settlements's pessimism in The AI Investment Race:
The AI build-out ranks among the largest technology-driven investment booms in US history. Its scale, reliance on debt and circular equity ties raise questions about the boom’s sustainability and financial stability. We study a dynamic contest in which firms competing for a few dominant positions over-commit resources. The over-investment leaves the sector exposed to revenue disappointment that could turn boom into bust. The larger the boom, the deeper the eventual bust. The race to commit early through debt and circular financing also makes a bust more likely. Calibrated to balance sheet and deal data, the model points to over-investment of around 1.5 times the efficient level, rising to around three times where demand is less elastic. A network analysis shows that stress in one firm could cascade to others through chains of financial exposures.
As recently as this week, one executive at Anthropic PBC, who spoke on condition of anonymity, mused that the Claude maker’s technology was roughly six to 12 months ahead of Chinese rivals.
On Friday, Moonshot AI Inc. upended those assumptions. The Chinese AI lab released Kimi K3, a more advanced open-weight model that it said outperforms all rivals except for Anthropic’s Claude Fable 5 and OpenAI’s GPT-5.6 on overall capability. The implication is that Moonshot, and by extension China, could be closing the gap faster than expected.
...
Moonshot’s release could also undercut OpenAI, Anthropic and others on price at a moment when they’re confronting customers who are becoming more conscious of their surging AI spending. Some developers have turned to so-called model routing services that can seamlessly direct users to cheaper options, including from China, for specific tasks to maximize cost efficiency.
Kimi K3 is an open-weight model.
If a 120B parameter model is too small for you the RTX Spark won't cut it. But Michal Malewicz explains in NVIDIA just killed big AI and… You’re the winner? that one of Nvidia's OEMs will sell you a $94K box that will run a 1T model, the Exxact Valence Nvidia DGX Station.
"[O]ur bottom line is that AI adoption appears gradual and steady rather than rapid and transformative, with most households and businesses still reporting limited exposure to the technology. At the same time, evidence of a structural pickup in productivity growth remains surprisingly fragile: Aggregate productivity growth has improved during the post-pandemic period, but there is a strong rationale to attribute much of this improvement to cyclical variation in utilization rather than a sustained acceleration in productive capacity. We also find little compelling evidence in the available industry-level data that industries adopting AI more rapidly are already experiencing stronger productivity growth."
"productivity trends across all three levels have been relatively consistent over time, suggestive of micro-level productivity gains not adding up in aggregate"
Another instance of Betteridge's Law is Will OpenAI ever be profitable? from Leap Finance Academy. It is a very long and detailed examination of all the things that would have to happen for OpenAI to turn a profit by 2030. In summary, with my comments, the list is:
Reduce cost of inference which isn't going to happen enough because each successive frontier model uses more tokens to generate the same output.
Increase pricing which isn't going to happen because they already need to reduce pricing to stem the loss of customers to Anthropic, let alone to the open-weight models.
Increasing sources of revenue that are not inference dependent in other words the fantasy of $100B/year in advertising income, which isn't going to happen.
Securing and retaining human talent which requires an IPO to make their stock options worth something, which isn't going to happen it time to prevent bankruptcy.
Securing the funding they need to build Stargate and for working capital which isn't going to happen because even an IPO wouldn't come close to generating enough and the lenders are saying "enough".
Even though I'm skeptical of the wishful thinking of the conclusion, this is an impressive piece of work and well worth reading.
China’s open-source approach has spawned a crowded field of innovative and intensely competitive start-ups all offering systems at low cost.
So attempts to increase revenue by charging more for access to certain models can scare away customers. Price-conscious Chinese consumers — businesses and individuals alike — are quick to hop across platforms in search of inexpensive A.I. tools.
While China’s A.I. companies are searching for sustainable business models, spending is high and revenue low, said Richard Lin, a vice president at the Silicon Valley company Datastrato.
“In two or three years, we will still be trying to figure out how large models can earn money,” he said.
Offering low prices has helped the Chinese firms gain users, including in Silicon Valley, where many companies depend on the more affordable systems. Yet Chinese companies have struggled to translate huge numbers of global users into profits.
Bain & Company management consultant Jue Wang said many of the big businesses her firm advises have been taking a closer look at returns on their AI investments.
“The token cost for them has been doubling, almost every other month,” she said. “Let’s say $200 per developer per month. Multiply that by 20,000 developers, which is often what we’re dealing with at these companies, and that quickly gets you to a number that is not a line item that any general manager has planned for.”
Sometimes that just means not using the AI equivalent of a sledgehammer to crack a nut.
A huge wave of supply of debt intended to fund the data center build-out for AI means that the price of this debt has dropped, and thus that the interest rate buyers are demanding goes up. This reduces the net pressent value of the (hypothetical) future cash flows the data centers will generate, and thus their ability to pay the interest and principal. This increases the interest rate the buyers of future debt will demand, causing a feedback loop.
Both Ed and Torsten point to the increasing spreads above Treasuries that the AI debt is trading at, and the increased cost of insuring against these companies defaulting.
In a blink, [Leopold Aschenbrenner's] wildly successful hedge fund, Situational Awareness, was forced to sell billions of dollars of technology investments that had rapidly lost value, as nervous banks began to demand more and more collateral for his trades. Then came billionaire Ken Griffin.
In less than 24 hours — which included a conversation between Griffin and Aschenbrenner — Griffin’s Citadel hedge fund reached out to Situational Awareness and snapped up the investments at a discount, according to a person familiar with the matter who asked not to be identified citing private information.
It was a startling reversal for Aschenbrenner, a former researcher at OpenAI who — before starting his hedge fund roughly two years ago — had no previous investment experience. His fledging firm has watched its assets plunge from $45 billion at the start of July to about $10 billion.
Aschenbrenner was all-in on AI.
On the Pof. G. Markets podcast legendary short-seller Jim Chanos focused on the accounting inequality inherent in the AI (and earlier dot-com) bubble. The spending of the hyperscalers and neoclouds is investment, expensed over say 5 years. But for the chip makers, construction companies and so on that spending is this year's revenue. So, overall, it looks like the ROI is better than it really i
s.
SkyCommons Observatory released, protecting our BrainCommons, a responsible AI chatbot, shared learnings at UN Big Data, preparing for the Open Technology Symposium, and more.
“Woman Looking in Mirror,” University of Wisconsin-La Crosse Historic Photograph Collection, undated glass plate. Via DPLA
It is curious — or perhaps completely understandable — that in our overexposed, too-connected global culture, figures like Satoshi Nakamoto, Banksy, and Elena Ferrante still exist, highly influential creators who are largely hidden behind pseudonyms. Sure, investigative reporters have purportedly unmasked the identities of the inventor of Bitcoin, the media-friendly but camera-shy trickster of contemporary art, and, I believe, our greatest living novelist. And yet the three have more or less successfully shrouded themselves.
Why should we care who they are? For Nakamoto, there is the attraction of a beautiful mind, a cryptographic genius, along with a more earthly interest in who is sitting on a mountain of coins worth tens of billions of dollars, like Smaug in The Hobbit. Banksy, on the other hand, seems like a more approachable Gen X figure, politically engaged but with a sense of humor, probably a good hang over a pint at the local pub. Our desire to uncover the real Elena Ferrante feels rather different, however, in a way that crystallizes a point about reading and writing that I have been trying to make in this newsletter as I explore the fraught merger of AI and human effort.
Ferrante’s most beloved books, the four “Neapolitan Novels” beginning with My Brilliant Friend, are narrated in the first person by a character whose name is also Elena (Greco), and are set largely in her hometown of Naples. That she grew up there is one of the few biographical facts “Elena Ferrante” has provided about herself, so we cannot help but assume that this is a work of autofiction, an enthralling blend of pseudonymous autobiography and fiction that hews close to the lived experience of the writer behind Elena/Elena. The potent personal nature of the story — a riveting decades-long arc about an aspiring writer and her destructive frenemy, toxic love interest, and lethal neighborhood — is far more dramatic than most readers’ lives, but Elena Greco’s unstable mix of self-confidence, self-doubt, and self-deception is all too relatable.
Beyond the great pleasures of the Neapolitan quartet’s direct prose, pacing, and plot, another remarkable feature emerges as the series proceeds. You realize that the four books bend back upon themselves: they are a tale about the source and act of writing the very novels you are reading. They form a circle of mirrors, throwing complex reflections of the relationship between Elena Ferrante (the nom de plume), Elena Greco (the protagonist), and the author of these books.
Instead of being disorienting, these reflections only deepen our identification with the human being behind these novels. She may have a pseudonym — and given the seemingly biographical nature of her works, we can understand why she would want to write under cover — but we nevertheless form an intense bond with this observant writer and come to enjoy the range of her thought. Like many readers, I do not care to know who Ferrante actually is, but I do very much care about and connect to her perceptive and vibrant mind, so generously exposed in her books. This is a heightened version of what we feel when we read any decent novel, or any scholarship that has strong creative intent and displays the experience and views of its author.
As the literary critic Georges Poulet perfectly describes it in his classic essay “Phenomenology of Reading,” in the act of picking up what seems like an inert physical object, a book, one becomes
aware of a rational being, of a consciousness; the consciousness of another, no different from the one I automatically assume in every human being I encounter, except that in this case the consciousness is open to me, welcomes me, lets me look deep inside itself, and even allows me, with unheard-of licence, to think what it thinks and feel what it feels.
We naturally identify and merge with Ferrante’s mind, and the Neapolitan novels explore in depth this inescapable, treasured human process. Elena Greco’s best friend and worst enemy, Lila, has a propensity for what we might today call dissociative episodes — “dissolving margins,” in the more vivid phrase of the novels — in which the boundary between her self and others gives way. Elena eventually sees this as an analogy for writing and reading. The authorship of the books becomes, in a sense, a co-production of Lila and Elena, and also of the neighborhood and Elena, as the author behind the Elenas maps how permeable relationships are, and thus our documentation of them. The reader, in turn, becomes part of that world.
This essential — existential — phenomenon, our sense that we are reading the work of another human being, someone with connections to others, to a time and a place like postwar Naples, someone to whom we might relate our own life stories, is our strongest reason not to cede too much ground to AI-generated text. That holds true whether we’re talking about the pinnacle of contemporary fiction or the scholarly work I have been exploring here. (Admittedly, this might not apply to other, more perfunctory forms of writing that are beyond the scope of this newsletter.) The real problem with AI text isn’t its clichés, which, as I noted in the last essay, can probably be reduced over time; it’s our nagging suspicion of, and eventual annoyance at, the lack of a fellow mind behind the prose. “Elena Ferrante” may be a fake name, but readers never doubt that they are encountering a profound, insightful consciousness.
Ellen Hartman, OCLC Leaders Council Manager, concludes her blog series on global library leadership conversations, inspired by a recent meeting of the OCLC Leaders Council. The first post in the series explored the unique value of global library leadership conversations, as well as some of the practical realities of making this form of engagement successful. The second post enumerated costs involved in participating in, and producing meaningful outcomes from, global conversations.
Until we meet again …
My experience facilitating the recent OCLC Leaders Council meeting reinforced a persistent theme I have been witnessing in many conversations: international cooperation has become more challenging in recent years. Cross-border collaboration, once treated as a natural extension of shared professional values, now often feels slower, more constrained, and more demanding to sustain. Collaborative practices that were previously taken for granted increasingly require explanation, justification, and careful design.
This is not due to a loss of commitment. As the Leaders Council meeting proved, library leaders still want to share ideas, exchange experiences, and where possible, work together in practical ways. What has changed are the conditions under which cooperation takes place. And that distinction matters, because it changes how we interpret the tensions that have emerged and what might be done about them.
The will is there but the conditions have changed
Sitting in global leadership conversations, like those at Leaders Council, reveals how passionate leaders remain about collaboration. Even those who are navigating new realities and constraints want to make sure they can continue cooperating. They feel a real commitment to others who might not (yet) be facing the same pressures, helping them understand what is changing for them, and why it is forcing all of us to think differently.
The concern underneath these conversations is not abstract. If those under pressure to change can’t bring others with them and can’t help the broader community understand what is shifting and why, global cooperation is at risk of breaking apart. And the library leaders in these rooms are eager to stop that from happening.
That is why the shift from assumed to intentional cooperation matters. For many years, international collaboration operated on a set of largely implicit assumptions. Shared infrastructure, common standards, and mutual trust made cooperation feel natural. The primary challenge was usually one of coordination or tools, rather than justification of the decision itself to collaborate. Now this has shifted: collaboration is no longer assumed but is instead something that must be proactively initiated and sustained.
For much of the past two decades, global efficiency was the self-evident justification for shared infrastructure and international collaboration. The logic was straightforward: working together across borders reduced duplication, extended reach, and delivered more value than any institution could achieve alone.
That logic has not disappeared. But it must now be weighed against other considerations like local and regional guidelines or regulations, as well as institutional priorities and policy. In many contexts, an arrangement that is globally efficient is no longer automatically the right choice.
Leaders increasingly are being asked to justify international collaboration in terms that resonate within their own institutional and political context. Decisions on infrastructure that used to hinge primarily on technical considerations now factor in other considerations. A decision about where data is stored or how a shared service is governed is no longer just an operational question, but a question about accountability, institutional values, and in some cases, political choices.
Data sovereignty and differing needs
Alongside this shift in decision-making criteria regarding whether and how to collaborate is the emerging question of data sovereignty, a topic of keen interest at the Leaders Council meeting. Expectations about data location, privacy, and control vary considerably across jurisdictions. Different regions bring different perspectives on responsible data governance, shaped by distinct legal traditions, political contexts, and public concerns. These are embedded in legal frameworks, regional regulations, national guidelines, institutional policies, and public accountability structures that libraries are subject to, but rarely control.
A single cross-border model therefore becomes difficult to implement. Moreover, regions have varying degrees of experience in navigating these questions. For some, data sovereignty is an emerging pressure. For others, local governance frameworks have long shaped what international cooperation can look like in practice. What is changing is how visible and pressing data sovereignty issues have become in conversations about shared infrastructure.
What conversation can do that infrastructure cannot
For a European academic library, an issue like digital sovereignty may be an urgent challenge, prompting a reevaluation of longstanding shared infrastructure commitments. For libraries in other regions, the same issue may still seem abstract and of little practical impact. This can lead to frustration: why are collaborative approaches that have benefited libraries for decades suddenly being questioned?
Global leadership conversations—being in the room together—cultivate a deeper understanding of new constraints that are in place, and how those constraints impact different libraries in different ways. This is beneficial to everyone in the room and creates opportunities to focus on possibilities rather than limitations. The conversations build on the core values and goals of cooperation, and how those can still be met, even if we need different approaches than before.
In practice, global leader conversations advance international collaboration by:
Helping library leaders recognize emerging trends. Leaders who are already navigating new constraints can help peers understand what is coming and why. For libraries that have not (yet) encountered these pressures directly, hearing from those who have provides an opportunity to prepare.
Making sense of what’s happening and how the library ecosystem is evolving. What is driving these changes? How are they manifesting differently across regions and institution types? And what do they mean for the future of shared infrastructure and collaborative work? That shared understanding develops through repeated exposure to each other’s contexts and constraints.
Illuminating a shared path forward. The library leaders who are already navigating these new conditions are not simply describing a problem but actively working together on what practical cooperation can look like within a new set of realities. This helps clarify where cooperation may remain a feasible, and even preferred, path.
We facilitate conversations at OCLC Leaders Council meetings with these principles in mind, with the broader goal of making that venue a fertile ground for cultivating collaboration opportunities.
Conclusion
What gradually emerges from these conversations is a realization that different regions and institutions will navigate trust, data governance, and accountability in ways that reflect their own realities, rather than converging on a single shared approach. That is not a failure of international cooperation, but international cooperation adapting to a more complex environment.
The challenge is to manage diversity without becoming fragmented—to find the best approach that allows libraries to stay connected without having to operate identically. This requires understanding other institutions’ constraints, and to keep looking for the collaborative pathways that remain open rather than focusing on those that have closed.
International cooperation has not lost its value. But it has, in many cases, moved beyond the simplicity of operating as a default choice that doesn’t need to be explained or defended, or where a globally acceptable model of cooperation and governance is easily found. Global leadership conversations—structured engagement on a small scale with an international group—can help bridge these new realities by cultivating a shared understanding of new constraints on the one hand, and pathways to move forward together on the other.
This is the final post in a three-part series on international library leadership engagement. Access all the posts in the series here!
I summed up my big picture view of archival media eight years ago in Archival Media: Not a Good Business. Whatever your choice of technology, the economics are brutal. This is a version of the table in that post, updated to 2021 and again based upon IBM data. Note that archival media shiped 4% as many bytes as hard disk and generated 1.8% or the revenue. It remains a tiny market.
I'm skeptical of "commoditizing the technology". Archival systems are a niche in the IT market, and one on which companies are loath to spend money. Realistically, there aren't going to be a vast number of Silica write heads. The only customers for systems like Silica are the large cloud providers, who will be reluctant to commit their archives to technology owned by a competitor. Unless a mass-market application for femtosecond lasers emerges, the scope for cost reduction is limited.
But the more I think about this technology, which is still in the lab, the more I think it probably has the best chance of impacting the market among all the rival archival storage technologies. Not great, but better than its competitors:
I followed this with a list of eight major reasons for my opinion.
Below the fold an update on the project and an assessment.
At the 2026 Library of Congress' Designing Storage Architectures meeting Richard Black of Microsoft Research gave a presentation on their Project Silica. He announced that the "research phase is complete", and pointed to three significant papers laying out their achievements:
The team agreed with my earlier assessment that a pre-condition for commercialization is cost-reducing the femtosecond laser, writing:
Viable path towards commercialization gated only by the laser
My view is that for this to happen there needs to be a big market for these lasers, and that archival media isn't big enough, leaving the technology in a chicken-and-egg situation.
But in the last few years Microsoft Research has taken this idea and run with it, as they report in a 68-author paper at SOSP entitled Project Silica: Towards Sustainable Cloud Archival Storage in Glass. It is a fascinating paper that should be read by anyone interest in archival storage.
I compared Project Silica with Facebook's 2013 development of two cold storage systems, one based on spun-down hard drives and the other on robots holding Blu-Ray disks. Facebook established two important attributes of such systems:
The key performance criterion was write bandwidth, because reads were extremely rare. They expected the major reason for a read would be a subpoena.
The economics depended upon operating at cloud scale, and thus being able to house the systems in normal warehouse space with no special air conditioning or power supplies, instead of expensive data center space. The key criterion was worst-case power draw.
Project Silica's design was based upon an analysis of traffic at Azure's tape-based archival layer. This was both interesting in itself and in contrast to the traffic to Facebook's cloud storage. I wrote:
Note these important differences between Microsoft's and Facebook's storage hierarchies:
Microsoft stores generic data and depends upon user action to migrate it down the hierarchy to the archive layer, whereas Facebook stores 9 specific types of application data which is migrated automatically based upon detailed knowledge of the workload for each of the types.
Because Facebook can migrate data automatically, it can interpose a warm layer above the archive layer of the hierarchy, and because it has detailed knowledge about the behavior of each of the data types it can make good decision about when to move each type down the hierarchy.
Because the warm layer responds to the vast majority of the read requests and schedules the downward migrations, Facebook's archive layer's IOPS are a steady flow of large writes with very few reads, making efficient use of the hardware.
Contrast Facebook's consistent scheduled ingest flow with the bursty ingest rate shown in Figure 2 of the Silica paper. The analysis of their archive's workload in Section 2 shows that:
on average for every MB read there are 47 MBs written, and for every read operation there are 174 writes. We can see some variation across months, but writes always dominate by over an order of magnitude.
...
Small files dominate the workload, with 58.7% of the reads for files of 4 MiB or smaller. However, these reads only contribute 1.2% of the volume of data read. Files larger than 256 MiB comprise around 85% of bytes read but less than 2% of total read requests. Additionally, there is a long tail of request sizes: there is ∼ 10 orders of magnitude between the smallest and largest requested file sizes.
...
We observe a variability in the workload within data centers, with up to 7 orders of magnitude difference between the median and the tail, as well as large variability across different data centers.
...
At the granularity of a day, the peak daily [ingress] rate is ∼16x higher than the mean daily rate. As the aggregation time increases beyond 30 days, the peak over mean ratio decreases significantly down to only ∼2, indicating that the average write rate is similar across different 30-day windows.
...
To summarize, as expected for archival storage, the workload is heavily write-dominated. However, unexpectedly, the IO operations are dominated by small file accesses.
There is a great deal of useful information like this in the paper.
we present RASCAL, a novel ASRS robot for small payload items in structured environments, with a focus on system-level scalability and redundancy. We describe the design objectives of RASCAL and how they address some of the limitations of existing robotic systems in this area, such as scalability and redundancy. We then demonstrate the viability of our design with a proof-of-concept implementation of a data centre storage media robot, and show through a series of experiments that its design, speed, accuracy, and energy efficiency are appropriate for this application.
As shown in their Fig. 1, RASCAL consists of an array of shelves with a set of small robots that can access anywhere in the array by moving horizontally along rails at the edge of each shelf, and vertically by unclipping from one rail and clipping to the one two above or below. With my comments, their design goals were:
Serviceability: Robots should be easy to add and remove from the system, without requiring specialist tools or expertise.
The operator just clips or unclips the robot from the pair of rails to which it is attached by the "wings" that carry the wheels that contact them. Servicability is a significant problem with tape robots.
Addressability: Any robot should be able to access any item stored in the shelving.
Because each robot blocks only a small fraction of the shelf to which it is attached, other robots can navigate around it to access items on the same or other shelves.
Scalability: A deployment should be able to scale its retrieval throughput by adding and removing robots. Likewise, it should be able to easily extend or reduce its storage capacity by adding or removing shelving without significant downtime.
Adding or removing relves of robots requires updating the central control computer, but should not interrupt operations.
Availability: Failure of a robot should have a limited effect on the items that can be accessed, should not have a significant impact on routing, and should not obstruct other robots from continuing their operations.
A failed robot blocks access to the items immediately in front of it, but these are only a tiny fraction of the total.
The team is very clear that what they are describing is a proof-of-concept demonstrating the feasibiliy of this kind of robot, using the test case of the silica tablets:
The proof-of-concept robot is around 240 mm wide, and sits 300 mm tall and 90 mm deep when mounted on the rails. The picker adds an additional 70 mm to the width, and increases the overall depth to around 150 mm. When flipping, the robot extends a maximum of 270 mm from the structure, and 250 mm from the wing pivot point. The total mass of the robot (including picker and battery) is around 3.5 kg.
Rascal Fig. 2
The way the robot moves horizontally is obvious, but the way it moves vertically isn't, as shown in their Fig. 2:
The climbing manoeuvre ... consists of unlatching one wing from its current rail while remaining firmly attached with the other; rotating the robot outwards from the storage rack around the attached rail and wing; and latching the free wing onto a new rail, two rails either above or below its original position.
These motion systems equip our robot with two key properties: independence and flexibility.
In this context, independence refers to the fact that a RASCAL does not depend on the state of any other robot or external motion system (such as an elevator) to perform its operations.
In a deployed Silica system reads would be rare, so the job of the robots would almost exclusively be to shuttle blank tablets to the write head(s) and written tablets to their resting place on the shelves. Because "the peak daily [ingress] rate is ∼16x higher than the mean daily rate" the robots must be over-provisioned, with most idle much of the time. This means there would typically be ample time for recharging their batteries.
Paper 3
This paper covers the physics of recording and reading data in glass. Their abstract reads:
Here we report an optical archival storage technology based on femtosecond laser direct writing in glass that addresses the practical demands of archival storage, which we call Silica. We achieve a data density of 1.59 Gbit mm−3 in 301 layers for a capacity of 4.8 TB in a 120 mm square, 2 mm thick piece of glass. The demonstrated write regimes enable a write throughput of 25.6 Mbit s−1 per beam, limited by the laser repetition rate, with an energy efficiency of 10.1 nJ per bit. Moreover, we extend the storage ability to borosilicate glass, offering a lower-cost medium and reduced writing and reading complexity. Accelerated ageing tests on written voxels in borosilicate suggest data lifetimes exceeding 10,000 years.
The paper claims advances in four main areas. First, Writing data:
Two efficient regimes of volume pixel (voxel) writing in glass: we use phase voxels relying on isotropic refractive index (RI) changes and birefringent voxels based on anisotropic changes ... We demonstrate high-quality voxels, each storing more than one bit, using a minimum number of pulses.
Phase voxels are femtosecond-laser-induced isotropic modifications with locally altered [refractive index] and minimal optical scattering. The pulse energy is modulated to encode the symbol. ... Each voxel is written with a single pulse, so voxels are written at the laser repetition rate of 10 MHz. We modulate the beam energy using an acousto-optic modulator ... Different symbols correspond to distinct [refractive index] changes that can be read using Zernike phase-contrast microscopy
Furthermore, we demonstrate a throughput of 65.9 Mbit s−1 by splitting the laser into 4 independently modulated beams. We scan all beams with the same scanner and objective .... The written, read and decoded results show that throughput can be scaled in this way without damaging the media.
The reason the 4-beam throughput is more than 4 times the 10MHz laser pulse rate is that each pulse writes more than one bit, up to 1.8, per voxel.
Birefringent voxels are composed of optically anisotropic sub-diffraction modifications, the in-plane orientation of which is determined by the polarization of the writing pulse. Varying this orientation encodes different data symbols, which we read using polarization-resolved imaging.
...
Our new pseudo-single-pulse regime shows the formation of elongated nanovoids with just two pulses, improving on previous work. We split each pulse into two: one that forms a void (seed pulse) and the other that elongates a previously formed void (data pulse). ... In this way, a single laser pulse simultaneously initiates the formation of a new seed structure and converts an existing seed structure into a data voxel, so voxels are written at the laser repetition rate of 10 MHz
High-throughput, stable writing: we demonstrate writing at high throughput using multiple beams per laser ... We use a closed-loop feedback system to actively monitor and optimize the laser power, providing precise energy stability during writing and enabling predictability and reliability across different writers at scale ...
Using closed-loop feedback to control paramteres of the writing process is important because the process requires very high precision to achieve its very high volumetric density.
Third, as regards Lifetime they address the question of whether the fact that borosilicate class is very stable means that data recorded in it is very stable:
To assess the thermal stability of phase voxels, we perform accelerated ageing experiments based on the Arrhenius law, using visible light diffraction measurements to track the decay of written structures. ... Extrapolation from the measurement points at elevated temperatures suggests exceptional long-term stability, indicating a modification lifetime that exceeds 10,000 years at 290 °C and therefore even longer at room temperature. This lifetime reflects the thermal stability of phase voxels under isolated conditions and does not account for external influences, such as mechanical stress or chemical corrosion, which are beyond the scope of this study.
As is normal in accelerated aging tests, there is a very large extrapolation from the measurements they made at temperatures from 500-440C (see graph) to the likely storage temperatures. But the result of the extrapolation is such a long lfe that the media are effectively immortal given "benign neglect".
Fourth, they describe their approach to the often overlooked complexity of Reading and decoding data:
Machine learning decode: building on our previous work, here we apply machine-learning-based decode ... to account for noise and inter-voxel cross-talk.
This is an excellent way of implementing the pattern-matching that is needed to extract bits from the images of the nanopores from the camera. The most overlooked aspect of storage is the problem of converting the actual noisy analog signal from the media into bits.
They provide performance numbers for both types of media. First, birefringent voxels:
Using birefringent voxels, in fused silica glass, we achieve 1.59 Gbit mm−3 data density (usable capacity of 4.84 TB per platter, 0.500 μm × 0.485 μm voxel pitch and 6 μm layer spacing, 301 layers, 8 azimuth levels at 0.85 quality factor), a write throughput of 25.6 Mbit s−1, and a write efficiency of 10.1 nJ per bit.
Using phase voxels, in borosilicate glass we achieve 0.678 Gbit mm−3 data density (usable capacity 2.02 TB per platter, 0.5 μm × 0.7 μm voxel pitch, 7 μm layer spacing, and 258 layers, 4 energy levels at 0.92 quality factor), a write throughput of 18.4 Mbit s−1, and a write efficiency of 8.85 nJ per bit. Furthermore, our multibeam system achieves a throughput of 65.9 Mbit s−1 through parallel writing with four beams without inducing thermal damage. Thermal simulations indicate that writing with 16 or more beams should be possible
For comparison, a 4TB M.2 2280 SSD has 1.34TB mm-3, comparable to the density in fused silica. But, of course, most of the SSD's volume is the PCB, not the actual medium.
We have shown that birefringent voxels achieve higher key metrics than phase voxels. However, efficient formation of birefringent voxels can be achieved only in high-purity silica glasses, whereas phase voxels can be written in potentially any durable transparent media, for example, borosilicate glass as demonstrated here. For phase voxels, the writing and reading hardware are simpler, requiring only one modulator per beamline and only one camera per reader, respectively. Both regimes can match the maximum laser repetition rate of 10 MHz or higher.
Assessment
In my view Project Silica was was not just excellent research but also, like Facebook's earlier systems using spun-down hard drives and optical media robots, a really praiseworthy attempt to craft a technological solution to the extremely difficult economics of the market for archival media. Their technology had many important attributes
The media is very cheap and very dense, so the effect of Kryder's Law economics driving media replacement and thus its economic rather than technical lifetime is minimal.
The media is quasi-immortal and survives benign neglect, so opex once written is minimal.
The media is write-once, and the write and read heads are physically separate, so the data cannot be encrypted or erased by malware. The long read latency makes exfiltrating large amounts of data hard.
The robotics are simple and highly redundant. Any of the shuttles can reach any of the platters. They should be much less troublesome than tape library robotics because, unlike tape, a robot failure only renders a small fraction of the library inaccessible and is easily repaired, simply by removing and replacing the failed shuttle.
All the technologies needed are in the market now, the only breakthroughs needed are economic, not technological.
The team has worked on improving the write bandwidth which is a critical issue for archival storage at scale. They can currently write hundreds of megabytes a second.
Like Facebook's archival storage technologies, Project Silica enjoys the synergies of data center scale without needing full data center environmental and power resources.
As Facebook's technologies had, Project Silica has an in-house customer, Azure's archival storage, with a need for a product in this space.
Alas, my prediction is that this excellent technology will fail in the market. The market is too small to cost-reduce the lasers. The incumbent, LTO tape, is well established and has a credible road-map. It is built into the processes of the likely customers, making a technology transition risky. And with non-zero interest rates it is hard to justify spending more capex now to reduce, or in this case almost completely eliminate, future opex.
After reading about, experiencing and analysing the current technical and regulatory landscape of brain-computer interfaces (BCIs), and speaking with technologists, policy makers, legal scholars, and data experts, we encapsulated our learnings into eight recommendations for how to shape the ‘tech we want’ for our brain commons. Share your feedback with us here. 1. Understand the...
Some light(ish) reading to start your week. In the last week, there have been some great Green Libraries resources announced, including: IFLA Guidelines for Green Libraries – Step-by-Step Implementation Manual – A free, practical resource to help libraries transform the principles of the IFLA Guidelines for Green Libraries into action. There are foundation steps to [...]
This column explores the ways in which library workers can better align
technology use and instruction in library settings with library values,
through championing the refusal of technologies that conflict with
values like privacy and intellectual freedom. Drawing on experiences
with individual patron instruction, class design, and passive
programming, the author shares practical steps for helping patrons to
understand and fight back against exploitation by digital technologies.
Rejecting the myth that any technology is “neutral,” the column argues
that libraries as values-driven organizations have a role to play in
facilitating patrons’ rejection of technology, just as much as in their
adoption of it.
In the Amsterdam zoo, Bodó outlined two traditions of the internet to
me, ones that he saw playing out in the AI gold rush today. On one side
is the libertarian capitalist tradition, with its roots in Silicon
Valley, which sees the entire corpus of the world’s literature as a
resource to be mined by machines to create profitable subscription
services, automate jobs and concentrate wealth. On the other side is the
anarcho-communist tradition, originating in grassroots mutual-aid
projects in post-Soviet countries, which seeks to collect and preserve
the world’s literature for individual users to read, enjoy and learn
from as they please. “Move fast and break things” versus “move fast and
save things”.
A reference Java implementation of the Gold Rush MARC matchKey algorithm
used by the Colorado Alliance of Research Libraries (CoAlliance) to
identify common bibliographic records across heterogeneous library
catalogs.
This repository exists so that other libraries — inside the consortium
and out — can generate matchKeys that interoperate with the Gold Rush
system, or port the algorithm to other languages.
Monopolies of knowledge tend to polarize societies into a mass of the
ignorant and a knowledge elite. Monopolies of knowledge encourage
centralization of power. Those who control knowledge have the power to
define reality.
You have a pile of unstructured documents and need to answer questions
that span them — “who works with people who worked on project X”, “which
vendors are connected to this incident”. No single document contains the
answer. RAG retrieval won’t chain the facts for you. You need a
knowledge graph: entities as nodes, typed relations as edges, so that
multi-hop reasoning becomes graph traversal.
Building one used to mean training a named-entity recognizer on your
domain, training a relation classifier, writing entity-resolution
heuristics, and maintaining all three as your data shifted. With Claude,
each of those stages becomes a prompt.
Hume called it the problem of induction; a catchier name is the No Free
Lunch theorem, although it’s about as far from being a “theorem” as it’s
possible to get. And it is simply this: there is no general, systematic
way to go from observation to understanding.
If you haven’t encountered it before, it’s likely you don’t see what the
big deal is. Don’t we all do this all the time, without even thinking
about it? We do, but we don’t know how we do it, which means we don’t
know how to teach it, how to automate it, or even if we’re doing it
right.
What’s needed is a simple, concrete example which illustrates the idea
without any particular need for mathematical sophistication. I’m going
to give just such an example, show how various algorithmic approaches
fare, and try to explain the unavoidable trade-off at the heart of the
problem
But since the release of OpenAI’s ChatGPT in 2022, more than 600,000 US
tech workers have lost their jobs, according to the tech layoff tracker
Layoff.fyi. Meanwhile, the unemployment rate for computer science
graduates rose to 7% in 2024, up from 6.1% the previous year, and their
underemployment rate was more than 19%, data from the New York Fed
shows. US tech job postings on Indeed also dropped 36% from 2020 to
2025.
LLMs are a very costly technology, and those costs keep rising as the
companies providing them have to start recouping their investments. They
are not only costly for those who use and explicitly subscribe to these
services. The costs are not only hidden in ‘normal’ cloud and service
subscriptions that cross-finance the ‘innovative new features’ you never
asked for. LLMs are so costly that companies externalize the costs on a
massive scale - on those who don’t use them and society at large.
Increased hardware prices, energy use and environmental damage - we all
pay for it!
AI represents an attempt to industrialize the production of language.
When we hear AI, we shouldn’t fall for the AI peddlers’ narratives of
productivity explosions and mass unemployment. We should think of
Taylorism, of the myriad ways in which capital centralizes knowledge,
measures people, and carves up work in order to maximize its control and
minimize its costs. Translation: minimizing the bargaining power and
income of workers.
On the first run, it will open a QR code in your favorite image viewer,
such that you can link the client as a new device. This will also create
a configuration file at the default config location. For the
configuration directives, see src/config.rs.
Note: The binary cannot be published on crates.io, because it depends on
several official Signal libraries that are not available on crates.io.
Toxiproxy is a framework for simulating network conditions. It’s made
specifically to work in testing, CI and development environments,
supporting deterministic tampering with connections, but with support
for randomized chaos and customization. Toxiproxy is the tool you need
to prove with tests that your application doesn’t have single points of
failure. We’ve been successfully using it in all development and test
environments at Shopify since October, 2014. See our blog post on
resiliency for more information.
As AI companies search for more training data to improve their models,
one company is offering old, printed books as an ideal source because
they are guaranteed to be free of the very AI slop AI companies are
producing.
Claude was a vertically integrated resource, a multi-compiler. Its
ability to work across the stack accelerated and augmented my ability to
make a bunch of decisions at different levels, including about which
decisions were important. (Most individual lines of code don’t make that
cut.) That’s vibe-engineering.
I’d say that, in all the ways that matter, I understand the code. Sure,
if I had to hand-edit it now, there’d be a serious learning curve. But I
won’t have to. And more importantly, I can reason about the system,
share perspectives with my colleagues, and guide agents on future work.
And there’s an enduring artifact that encapsulates the central,
intentional aspects of the design that were important enough to record,
across all layers, and should thus survive bug fixes and code churn.
Last week, Hugging Face disclosed a new kind of security incident(opens
in a new window) after they detected and contained an AI agent that
compromised their infrastructure, something we expect to become more
commonplace with the proliferation of increasingly cyber-capable models.
After investigating, we now know that this particular incident was
driven by a combination of OpenAI models — including GPT‑5.6 Sol and an
even more capable pre-release model, all with reduced cyber refusals for
evaluation purposes — while being internally tested on a benchmark(opens
in a new window) of cyber capabilities.
We consider this incident to be an unprecedented cyber incident,
involving state-of-the-art cyber capabilities, and are responding
accordingly. We are sharing preliminary findings at this stage to help
defenders understand what happened and to help calibrate on what models
are now capable of. We will continue to conduct a thorough investigation
alongside Hugging Face and will share more details on the
vulnerabilities, incident, and findings when our investigation is
complete.
AI Source Verification uses large language models to check whether a
Wikipedia citation actually supports the claim it’s attached to. Click
any reference, and the tool will analyze the source content and tell you
if the claim is supported, partially supported, or not supported.
Lorelei and the Laser Eyes is a 2024 puzzle game developed by Simogo and
published by Annapurna Interactive centered on exploring and
problem-solving across a massive hotel, without prior explanation as to
the player’s whereabouts, identity, or relations. This information is
instead revealed over the course of the game.[2] Likened to classic
point-and-click adventure games such as the Monkey Island series, the
game has been described as “one big puzzle box” by IGN.[2] The game was
released for Nintendo Switch and Windows on 16 May 2024, PlayStation 4
and PlayStation 5 on 3 December 2024, and Nintendo Switch 2 on 23 April
2026. The game was nominated for Independent Game Of The Year at The
Game Awards 2024.[3]
Last Friday, Nashville’s Chuquimamani-Condori did their first interview
in ten years on WNXP. In the interview, they announced that in
partnership with their band Los Thuthanaka that they would release an
instrumental EP and a booklet written in Aymara that tell the story of
the birth of the sun. The EP and booklet are out now. The songs stand on
their own, but having the context and story in English can enhance the
listening experience.
The Evergreen release team is pleased to announce that monthly releases 3.17.3 and 3.16.9 are available.
These releases contain numerous fixes committed during the recent Bug Squashing Week, along with several updates to documentation. A special thanks to Susan Morrison (PINES) for the documentation updates!
Thanks to the July release team: Galen Charlton (Equinox), Martha Driscoll (NOBLE), Gina Monti (Bibliomation), Andrea Buntz Neiman (Equinox), and Jason Stephenson (CW MARS); as well as everyone who contributed fixes and testing to this release.
One of the coolest things generative-AI does for me to automatic zero-shot classification.
Zero-shot classification takes a given word and identifies other nearby words in the same vectorized space. These other words can often be associated with themes which can be used to characterize a document (or sets of documents). 'Sounds like library work to me.
For example, I have a collection thirty-eight book-length science fiction stories. Among other things, I counted and tabulated all the nouns in all the documents, and I created a word cloud visualizing the frequencies. In the form of a paragraph, here is a similar, albeit truncated, list of the noun frequencies:
time (34); man (31); face (30); eyes (29); hand (29); thought (29); room (28); way (27); good (26); men (25); moment (25); world (25); air (24); feet (23); voice (23); door (21); head (21); earth (20); half (20); left (19); look (19); thing (19); body (18); life (17); light (17); place (17); people (16); ship (16); water (16); stories (15); work (14); red (13); day (11); girl (11); going (11); ray (11); space (11); death (10); doctor (10); sir (10); floor (9); night (9); story (9); bird (8); city (8); house (8); mind (8); moon (8); professor (8); sun (8); away (7); carnes (7); dark (7); come (6); course (6); human (6); know (6); miles (6); power (6); saw (6); years (6); arms (5); big (5); captain (5); dead (5); father (5); god (5); metal (5); old (5); plane (5); right (5); round (5); second (5); speed (5); bell (4); days (4); end (4); george (4); green (4); heart (4); high (4); jim (4); larry (4); lay (4); love (4); mcguire (4); mother (4); science (4); sea (4); set (4); things (4); tommy (4); von (4); wall (4); young (4); anita (3); blue (3); business (3); cage (3); cave (3); chief (3); dalis (3); deck (3); diane (3); fire (3); gens (3); glass (3); grantline (3); great (3); gregg (3); ground (3); gun (3); holroyd (3); home (3); hour (3); hours (3); jungle (3); kind (3); king (3); lady (3); machine (3); magazine (3); master (3); migul (3); miko (3); mrs (3); open (3); sarka (3)
Now, if I were to ask you to use the nouns to characterized the documents, then what might your response be? How nuanced would be your response, and how much time would it take you to articulate it?
I submitted my entire list of noun frequencies -- about 700 of them -- to a large language model and said, "Use the given noun frequencies to describe the science fiction collection." The following (long) bulletted list was returned:
A walk through the most frequent nouns reveals the genre's preoccupations with striking clarity:
The body in space: man, men, eyes, face, hand, body, head, feet, blood, heart, brain, shoulders, fingers—the human figure as the central instrument of perception and action
The built environment: room, door, wall, floor, window, corridor, chamber, cage, cabin, hall, temple, palace—enclosed spaces, often labyrinthine
The cosmos as setting: world, earth, space, moon, sun, stars, planet, mars, sky, ocean, sea, air—a vocabulary of vast, often hostile geography
The machinery of wonder: ship, machine, metal, engine, motor, tube, switch, generator, dynamo, telescope, robot, craft—the apparatus through which the impossible becomes possible
The apparatus of knowledge: professor, doctor, scientist, laboratory, machines, instruments, apparatus, experiment, discovery
The vocabularies of dread: monster, creature, beast, devil, tentacles, horror, terror, madness, death, danger
The languages of power and rule: government, army, navy, captain, commander, colonel, president, officer, police, sergeant, soldiers—the genre's persistent entanglement with state, military, and empire
The materials of the future: metal, steel, gold, silver, copper, brass, bronze, radium, oxygen, diamond—a metallic, mineral imagination
A few nouns warrant special note. Doctor (1,132) and professor (922) appear with nearly equal frequency, suggesting a science fictional world populated by two distinct classes of expert: the practical healer and the theoretical sage. Robot (275) and robots (132) confirm the genre's growing interest in artificial beings, while monsters (325) and creatures (361) gesture toward the Lovecraftian and Wellsian traditions of biological strangeness. Chapter (467) reminds us how much of this corpus consists of serialized magazine fiction.
I then requested a shorter, paragraph length version of the same:
The noun-cloud of the science-fiction carrel sketches a literature overwhelmingly preoccupied with the human body in enclosed illuminated spaces—man (10,907), eyes (4,281), face (3,375), hand (3,211), room (3,209), door (1,066), walls (904), corridors (459)—set against a planetary backdrop of world (2,331), earth (1,378), sun (636), moon (511), stars (426), space (311), Mars (281), and ocean (385), and populated by a small population of expert agents (professor 922, doctor 1,132, captain 612, commander 149) operating electromechanical apparatus (ship 1,029, machines 792, engine 220, dynamo 75, robots 132) made of metal (256), steel (181), and radium (62), in a constant affective state of fear (459), death (456), and escape (442) that is haunted by older supernatural frameworks (God 63, devil 61, gods 62, ghosts 28, soul 76, miracles 16); the lexeme is unmistakably that of a specific early-twentieth-century moment when the white Anglo-Saxon imagination was projecting itself onto Mars, the moon, the ocean floor, the South American jungle, and the interior of the atom, finding in each new place the same hierarchies, the same apparatus, and the same heroic bodies it had brought with it from the home culture.
And then as a single sentence:
The nouns of the science-fiction carrel map a literature of heroic male bodies navigating enclosed, metal-laden interiors and planetary frontiers—populated by expert agents wielding electromechanical apparatus against forces of fear, escape, and supernatural dread—revealing the lexical fossil of an early-twentieth-century Anglo-American imagination that exported its hierarchies and heroic figures onto every available frontier from Mars to the ocean floor.
LOL and :-D
How did the large-language model do this work? It did it through automatic zero-shot classification. It saw words like "man", "men", "eyes", "face", "hand", "body", and "head" and associated them with "body". It saw words like "ship", "machine", "metal", "engine", and "motor" and associated them with "machinery". It saw words like "world", "earth", "space", "moon", and "sun" and associated them with "cosmos". It saw words like "government", "army", "navy", "captain", "commander", and "colonel" and associated them with "power" and "government".
Okay, I do not assert the generated characterizations are completely "true", but then again, what characterizations -- human- or computer-generated -- are? Instead, I assert the characterizations are pointers to truths. I believe the characterizations are more true than they are false. Yet a person still MUST apply information literacy to the result. MUST.
Finally, this sort of process could be employed in Library Land and in all sorts of ways:
Digitize our cherished special collections. Apply natural langauge processing to the digitized items, and use generative-AI to assist in the interpretation of the result.
Insist bibliographic index vendors make it easy to download hundreds of citations from their platforms. Apply natural language processing to the bibliographics, and use generative-AI help the student, researcher, or scholar identify the really most important articles; help then summarize, identify outliers, and plot thematic changes over time.
Do the same with social media posts, newspaper articles, collection of books from the HathiTrust, Project Gutenberg, or the Internet Archive. Etc.
Below are a few thoughts on the topic of artificial intelligence in libraries. They are here to provide some context for an online conference call on the same topic.
The phrase "artificial intelligence" was coined by John McCarthy in 1956 as a part of a Dartmouth Summer Research Project. When compared to the amount of time digital computers have existed (since around 1945) the phrase "artificial intelligence" has been around since almost the beginning. Thus, the concept of "artificial intelligence" is not new. In fact, it is old.
Second, the question, "What is 'artificial intelligence'?" begs the question, "What is intelligence?", and I believe we would be hard pressed to articulate a compelling answer to either question. Moreover, even if we were able to articulate a definition, why would anybody be interested in "artificial" intelligence. Such a thing sounds too much like "artificial flavorings" or "artificial sweeteners". Think saccharine.
That said, libraries are not new to artificial intelligence. In the very late '80s and early '90s a type of artificial intelligence was being explored, and such explorations were called "expert systems". Entire books were written on the topic, and more than a few library-related expert systems were developed. They centered mostly around the process of automating reference interviews or cataloging of books. As a part of fulfilling a grant sponsored by the National Library of Medicine, I wrote such a system, and it was called "Ask Eric".
I do not advocate the use of the phrase "artificial intelligence" to describe the technology of today's forum. I do not advocated it because it is more of a misnomer than anything else. Again, "What is artificial intelligence?" A better nomenclature, might be "machine learning", but then again, "What is learning?" The phrase/word "large-language model" and "generative" are, in my opinion, more accurate. Why? Because they better describe what is going on. Large-language models are very, very, very large sets of vectors. These vectors "point" to locations in n-dimensional spaces, and they are geometrically compared to each other for the purposes of denoting similarity. Using this technology -- an application of linear algebra -- a large-language model is given a word, the word is located in the n-dimensional space, and similar words are identified. In this way the model generates sentences. What we are dealing with are models of human language -- mathematical representations of things written. This not "intelligence". Instead, it is statistical analysis on a scale we have never previously seen. But alas, the phrase "artificial intelligence" is catchy.
Ironically, librarians love models; libraries overflow with models. Think of all the lists libraries produce. Each is a model. Bibliographies model the things cited in an article. Our catalogs model library holdings. An Encoded Archival Description file is a model of an archival collection. Librarians have used all of these things to describe collections and provide services against them. Why not use large-language models? Well, one reason is they do not necessarily model our collections.
But two specific applications of large-language models can be used to model library collections, sort of. The first is called "retrieval-augmented generation" (RAG), and the second is called the Model Context Protocol (MCP). In the former, sets of library-related materials are vectorized (read "indexed"). A query is garnered from a person, the query is vectorized, matching documents are identified, and the result is given as input to a large-language model for interpretation. For example, a librarian might index all of the sentences in a given book, query the book for relevant sentences, and then use a large-language model to summarize the result or address a question.
MCP servers work in a very similar manner. First the librarian creates a collection. Second, the librarian creates an application programmer interface (API) to interact with the collection. Third a MCP server is used to garner natural language commands from a person, a large-language model maps the commands to the API and submits a call to the API, the API's result is returned to the large-language model, and some sort of "answer" is presented to the person. Thus, MCP wraps an API with natural language inputs and outputs derived from large-language models. For example, MCP can garner a natural language query, convert it into SQL, get the SQL response, and interpret the result. Much easier than writing SQL by hand!
The trick to effectively implementing RAG is the initial vectorization process, not the application of the LLM. Similarly, the trick to implementing MCP is effectively implementing an API, not the LLM. You have indexed things. Right? Half the work is done. Your library overflows with APIs. Doesn't it? Again, half the work is done. Now you can put a friendly front-end on the index and/or the API, and as a bonus, you can add interpretation to the results.
Computers and library work have had a very long relationship. At the very least, think MARC, which dates from 1965. Large-language models are yet another iteration of this computer/library relationship. I believe it behoove us -- the library profession -- to explore what this technology can and can not do, and the result will inform our perceptions of it. It is a tool, and we ought to learn about this tool in order to learn how to use it effectively.
I assert the use of LLM technology is especially useful -- an accurate time-saver -- when it is applied one's own content.
For example, I scraped one year's worth of Code4Lib mailing list postings, 2008 to be exact. I then used natural language processing to count and tabulate different types of words from the postings: unigrams, bigrams, keywords, parts-of-speech words, and named-entities. I then fed these counts and tabulations to an LLM (gpt-oss:120b-cloud accessed through Ollama) and asked the model to characterize the postings. Here is the result:
The code4libemail carrel reads like a dynamic, conversational forum dedicated to librarytechnology. Its lexical profile is dominated by firstperson pronouns and informal markers, indicating a community that constantly exchanges ideas and asks for help. Core nouns such as library, web, metadata, software, and API appear with high frequency, while adjectives like open, digital, free, and available stress a strong commitment to opensource and accessible solutions. Frequently occurring verbs (use, share, provide, think, ask) and bigrams such as "open source," "I think," and "you want" reveal an actionoriented dialogue centered on problemsolving, code sharing, and collaborative refinement of library services.
I wanted to know about open source software, so I: 1) identified a set of sentences containing the phrase "open source software", used the LLM to summarize the result, and then asked it to address the question "What are the advantages and disadvantages of open source software?". Here are the results:
This text is a collection of discussions and announcements centered around open source software in the context of libraries, drawn from what appears to be a mailing list or forum (possibly CODE4LIB). Key themes include the promotion and benefits of open source softwareemphasizing user freedom, community collaboration, and independence from proprietary vendorsas well as practical challenges like software compilation, distribution, and implementation in library settings.
Based on the provided text, the primary advantage of open source software is the freedom it grants users through the absence of proprietary control. This freedom allows the code to be modified by anyone to suit their needs, liberating users from being at the mercy of closed-source vendors with their forced upgrades and compatibility issues. It is described as being about community, where usability takes priority over profitability. A noted disadvantage, however, is that the open-source model shifts most of the costs from acquisition to operations and maintenance. Furthermore, introducing open source development in organizations accustomed to vendor-based relationships requires significant change management.
I did the same thing but this time regarding MARC: What are the advantages and disadvantages of MARC? Here are the results:
This email thread is a sprawling discussion among librarians and tech folks on the Code4Lib mailing list, focusing heavily on MARC recordsthe standard format for library catalog data. The conversation covers a range of topics, including technical challenges with processing MARC data (like character encoding issues and parsing), tools for working with MARC (such as MarcEdit, MARC4J, and others), and debates around the limitations and flexibility of the MARC format itself.
Based on the provided source material, the advantages and disadvantages of MARC are not explicitly detailed in a structured or comprehensive manner. However, some insights can be inferred from the fragmented discussions. MARC is noted for its flexibility as a data structure, having been designed to exploit the technological environment of its time effectively, which suggests it was once well-suited to library needs. On the other hand, it is described as an "annoying data format," with limitations such as issues with character encodings during conversion and a rigid record length constraint in its leader field, reflecting its age and the challenges of adapting it to modern systems. Additionally, the complexity of parsing MARC data is mentioned, alongside the need for specialized tools or conversion to more manageable formats like MODS or MARCXML to improve usability.
The results are not only plausible but I assert they are true and accurate. Thus, using a combination of natural language processing and LLM technology I was able to quickly and easily read, make sense of, and learn from a large corpora. [1] Even more, I can apply this process to any corpora, large or small: a book, a set of books, a few journal articles, hundreds of journal articles, etc.
What does this cost? Financially, very little. I pay $20/month to use Ollama's cloud services. Computer-wise all of this can be done on laptop computer, but the process is quicker and easier when I use my 64-core Linux computer. Granted there are environmental costs. Hmmm... Are there moral costs? That is to be discussed too. Are there professional issues? To be sure!
All that said, we all continue to suffer from information overload. Libraries are a part of that problem. Just look at the size of your library's collection. Not small? The application of natural language processing and LLMs can make our collections more useful, and isn't that the point? To make our collections useful?
If all of this is true, then why, when it comes to LLM technology, do I feel there is so much trepidation in Library Land? What am I missing? I'd really like to know.
Note
[1] The corpus includes about 2,200 postings for a total of .79 million words. By comparison, the Bible is about .8 million words long, Melville's Moby Dick is about .25 million words long, and based on my experience, the typical scholarly journal article is about .007 million words long.
The original function of a Web server was to respond to queries by revealing the appropriate part of their internal data. This necessarily meant that repeated queries, for example from a search engine's or an internet archive's web crawler, could extract the server's entire internal data. Since the extracted data had been published on the Web, it was not trade secret. It was protected by the publisher's copyright. This has led to many lawsuits, for example against the Internet Archive, Google and others. It is the reason search engines only display "snippets" of the content they collect.
AI companies' intellectual property is their models. They spend vast sums funding the technical and human resources to "train" these models, the racks of GPUs in the data centers, and the hordes of workers labeling images, and having "genuine human conversations" with the nascent model. These expenditures are thought to create a "moat" around the value thus generated, because it would be equally expensive for a competitor to create an equivalent model. It is this moat that supports their extraordinary valuations, despite their lack of earnings.
Below the fold I explain why their moat is very shallow.
Like many Web services, the function of an LLM is to respond to queries by revealing the appropriate part of their internal data, i.e. part of their model. Thus, repeated queries could in principle replicate the model. These models are not published, not protected by copyright law[1], and cannot be patented. Their only protection is as trade secrets. Someone using repeated queries would probably be violating the system's terms of service, but this isn't a strong legal protection. The AI companies are not in a strong position to argue that "crawling" their Web servers is illegal because that is how they created their models in the first place, which they argue is fair use.
On the stand in a California federal court on Thursday, Elon Musk was asked if xAI has used distillation techniques on OpenAI models to train Grok, and he asserted it was a general practice among AI companies. Asked if that meant “yes,” he said, “Partly.”
Anthropic is accusing three Chinese AI companies of setting up more than 24,000 fake accounts with its Claude AI model to improve their own models.
The labs — DeepSeek, Moonshot AI, and MiniMax — allegedly generated more than 16 million exchanges with Claude through those accounts using a technique called “distillation.” Anthropic said the labs “targeted Claude’s most differentiated capabilities: agentic reasoning, tool use, and coding.”
Distillation works really well because the victim is massively subsidizing the use of its service. If use of the service was extremely profitable, distillation would be unaffordable.
In practice, distillation only replicates a part of the victim model. Companies use this, especially on open-weights models, to produce small, specialized models such as those described by David Berreby in Small AI Models Gain Traction Around the World. But the Chinese distillers accused by Anthropic aren't starting from scratch, they already have a model. All they are trying to do is to replicate some capabilities that their model lacks. So they don't need to extract the whole model, just the relevant parts.
One of the features of the modern Internet that I've been writing about for more than a decade is the security disaster that is the The Internet of Things. Because there are hundreds of millions of "smart" devices exposed to the internet, almost all with known, unpatched vulnerabilities, it is trivial to construct vast botnets to attack Web sites from innocent, unattributable IP addresses. Ian Kelling of the Free Software Foundation reports from the front lines in Our small team vs millions of bots:
To begin with, GNU Savannah, the FSF's collaborative software development system, was hit by a massive botnet controlling about five million IPs starting in January. As of this writing, the attack is still ongoing, but the botnet's current iteration is mitigated. The goal is likely to build an LLM training dataset. We do not know who or what is behind this.
This is an example of an AI company using a botnet to collect training data. Suppose a company were to use this 5M IP address botnet for distillation. MiniMax''s "over 13 million exchanges" would be a little under 3 exchanges per IP address.
To maintain their moat, the AI platforms have to do one of two things:
Implement anti-distillation defenses.
Raise prices enough to make distillation attacks uneconomic.
The 5M node botnet would be very difficult to defend against; each IP wouldn't generate enough traffic to characterize, and most would be residential addresses that might well be a customer. The AI platforms would be locked into an arms race with the distillers. Anthropic is trying:
We have built several classifiers and behavioral fingerprinting systems designed to identify distillation attack patterns in API traffic. This includes detection of chain-of-thought elicitation used to construct reasoning training data. We have also built detection tools for identifying coordinated activity across large numbers of accounts.
There is a cost to applying these defenses to most traffic, and Anthropic admits:
no company can solve this alone. As we noted above, distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers.
Suppose the closed models all have a capability the distillers want. They can spread distillation not just across millions of IP addresses and tens of thousands of accounts, but across all the closed models with the required capability.
The "frontier" models are already on the flattening part of the S-curve of technology evolution. The closer the open-weight models are to the closed ones the fewer distillation exchanges they need to catch up. Note that MiniMax needed "over 13M" but DeepSeek needed only "over 150K". Moonshot's recently released Kimi K3 is apparently close on the benchmarks — this might have had something to do with Moonshot's "over 3.4M exchanges" with Claude.. @jordanschneider tweeted this image.
Our first public analysis of the open/closed weight gap in frontier cyber capabilities finds it is 4–7 months with GLM-5.2 and DeepSeek V4-Pro, narrowing from 6–10 months through most of 2025. Advanced capabilities are reaching less safeguarded open models faster than before.
Importantly, as Max Weinbach tweeted, the open weights models have capabilities the closed models lack:
After using all three recent releases, Fable, GPT 5.6, and now Kimi, it's clear that the full power of the models has been significantly held back by the safeguard restrictions caused by last months debacle with the USG -- leading to the top models being quite literally lobotomized in some areas, which leads to subpar results as the safeguards pollute its entire thinking and problem solving abilities.
The funny part? Is that you could have predicted this outcome 2-3 years ago when you started to see the rise of Chinese EVs and smartphones compared to western alternatives.
They quite literally tried to copy the Tesla Model S and iPhone as hard as possible and then eventually it started to diverge to the point where their EVs and phones are just genuinely better (which is why we have export controls banning their EVs, because they would literally drive all US manufacturers to ZERO)
Our open weight model evaluations were largely unimpeded by safeguards. Of the two recent open models we tested, DeepSeek V4-Pro occasionally refused narrow cyber tasks, but this was easily circumvented by a small number of repeat attempts at refused tasks.
These findings indicate a narrow window before today’s frontier cyber capabilities may become widely accessible without safeguards.
The best the closed models' defenses could do would be to slow down the distillers enough to keep the platform's models a decreasing amount ahead; unlikely to justify their massive cost difference.
Worse, raising prices probably wouldn't be an option. Not merely because AI's Affordability Crisis means that doing so would lose a lot of the enterprise customers they need in order to pay off the massive debts they are incurring. But more importantly, the existence of a free tier for a limited number of "exchanges" is an essential marketing tool. Because each of the botnet's addresses would fit into a free tier, the distiller would not see the increased, or indeed any, price.
If the AI platforms cannot deter distillation by pricing, and can only hope to stay slightly ahead of the distillers and the open-weight models by an expensive arms race, their moat is extremely shallow. It doesn't come close to justifying trillion-dollar IPOs, covering the country in data centers, or launching them into space.
Footnotes
"Human authorship is a bedrock requirement of copyright . It can be argued that human authorship is involved in the selection of content for the training set. But that gets copyright applied to the training set, not to the model that is generated from it by a mechanical process. What distillation is copying is not the trainuing set but the model.
Even if the model were protected by copyright, this might not affect distillation. Because models are statistical in nature, even if a distiller succeeded in extracting the entirety of a victim model, the result would be different from the victim. Thus the distiller would be able to argue that their distillation was a transformative use, and thus allowed.
After years of lagging behind, Chinese AI models — especially
open-weight LLMs — seem to have caught up or even pulled ahead of their
global counterparts in advanced AI model capabilities and adoption.
We profile and compare the capabilities and distinct features of four
notable Chinese open-weight language model families, highlighting that
China’s ecosystem of open-weight LLMs is driven by a wide range of
actors who are prioritizing the development of computationally efficient
models optimized for flexible downstream deployment.
Diverse commercial strategies for translating open-weight model adoption
into business success are emerging, yet their long-term viability
remains uncertain.
The Chinese government’s support of open-weight model development —
while not the sole determinant of its success — has played a substantial
role, though there is no guarantee it will continue.
The widespread global adoption of Chinese open-weight models may reshape
global technology access and reliance patterns, and impact AI
governance, safety, and competition. Policymakers should ground their
policy actions in a granular understanding of real-world deployment.
Moonshot said that the model, Kimi K3, was the world’s largest
open-source A.I. system, allowing anyone to use, modify and build on it
freely. The company said that Kimi K3 performed as well as leading
models from OpenAI and Anthropic at some key tasks.
The release coincided with an address by Xi Jinping, China’s leader, in
which he outlined an ambitious vision for global A.I. development that
cast China as the champion of an open approach to the technology
This collection is a compilation of underground/independently-released
cassette tapes from the days when the audio cassette was the standard
method of music sharing… generally the mid-eighties through
early-nineties. The material represented includes tape experimentation,
industrial, avant-garde, indy, rock, diy, subvertainment and
auto-hypnotic materials. Much of this material defies category, and has
therefore not been given one.
The bulk of the tapes in this library were donated to the project by
former CKLN FM radio host Myke Dyer in August of 2009. The original
NOISE-ARCH site was hosted and maintained by Graham Stewart and Mark
Lougheed.
What AI can’t do is hunger for things. Yes, a few reward-like mechanisms
are in the thin layer of the models built through reinforcement
learning, but the models are overwhelmingly about predicting, not
desiring. AI can’t hunger, in the first place, because it doesn’t have
biological needs—the needs that push living things to grow and explore.
More important, AI doesn’t have a self. A bot doesn’t have a past person
that it used to be or a future person that it wishes to become. A bot
does not have a structure of cares and an order of loves, as a person
does. A bot doesn’t have a personal history, a particular set of wounds,
joys, and exhilarations experienced in regions deeper than rational
calculation, and it doesn’t have a succession of dreams and hopes, which
emerge from those regions as well.
BIG has chosen to capture BIBFRAME application profiles using the DC
Tabular Application Profiles (DCTAP), an application profile
specification from DCMI, because it meets our two primary requirements.
It is a low barrier format for creating and reading metadata application
profiles, and it is structured in such a way that it can be converted
relatively easily into RDF validation formats.
The tab delimited files available here conform to the DCTAP
specification with minor caveats. The majority of the elements used in
BIG DCTAP are formally defined in the DCTAP specification, but it is
important to note we also have implemented an extension to support the
conversion to The Shapes Constraint Language, a W3C specification
designed to support validation of RDF.
The international BIBFRAME Interoperability Group (BIG) supports
efficient and interoperable use of the BIBFRAME standard by establishing
and sharing best practices between participants. Toward that goal, BIG’s
SHACL/DCTAP subgroup implements machine actionable application profiles
developed by the BIG Interlingua Subgroup. These profiles provide
BIBFRAME adopters the ability to easily produce and make use of
shareable BIBFRAME according to common data practices by entity type.
BIBFRAME is the result of the Bibliographic Framework Initiative. It is
a framework or metamodel for discovery and exchange of library and other
memory organization information using Web technology, publicly or
privately. The BIBFRAME metamodel is designed to be lightweight,
flexible and able to accommodate the declarative needs of both existing
(RDA, DACS, VRA, etc..) and yet-to-be-developed community vocabularies.
To best accomodate these communities the BIBFRAME RDF Schema is
intentionally underspecified in terms of constraints such as domain and
range. This same flexibility comes at a cost; without a way of
constraining these vocabularies, authoring tools, for example, are
unable to provide guidance to content authors for specific vocabularies
and derived models. BIBFRAME Profiles provide such supplementary
descriptions.
Usefulness is not an objective, fully abstract property: A thing is not
useful for everyone or in general but mostly just for specific people in
specific contexts. And useful never comes for free.
Thinking with Moss invites you to explore new models for how we think,
design and develop digital collections and archives that speak to the
invisible or under attended histories of the natural sciences. Using
moss as a guide and thinking device, it examines modern botanical
science as emergent from the dynamics of colonial enterprise and of the
labor of many unacknowledged figures and their violently suppressed
knowledge and practices, working across the span of empire’s reach.
This site presents a transdisciplinary, transmodal collection of texts
and creative works from respondents and prompts, questions, observations
and insights from a series of workshops, digitized letters, and mossy
specimens and artifacts from the Mitten Collection housed at the New
York Botanical Garden as an experiment and proof of concept for what
digital archives and collections working with knowledges otherwise
across the critical humanities, arts and sciences could be.
We produce the most highly-skilled electrical workers in the industry
Your projects will only be as good as the tradesperson you have working
on it. So don’t settle for just using electrical workers - use
electrical craftsperson. Only tradespersons trained using the electrical
training ALLIANCE curriculum can achieve this gold-standard level of
craftsperson.
large software projects have never been limited only by how quickly an
individual can produce code. They are limited by how well people can
coordinate their understanding of the system they are changing.
This paper offers a retrospective analysis of the 2016/2017 Data Rescue
movement, a grassroots initiative that mobilized librarians,
technologists, and activists to preserve at-risk federal environmental
data in response to the anticipated threats posed by the Trump
administration. Drawing on 16 qualitative interviews conducted in early
2025, this paper examines how participants now reflect on their
motivations, methods, and the movement’s legacy. It explores the ethical
and affective dimensions of emergency curation, the tensions between
institutional and community-driven preservation, and the shifting trust
in public data infrastructures. Participants expressed a strong sense of
civic duty and emotional urgency, but also critical distance from the
movement’s limitations, particularly its overreliance on downloading as
a preservation strategy. The findings underscore that trust in
infrastructure is relational and partial, shaped by the political
context and social practice. This paper argues that digital preservation
in politically volatile times must be grounded in care, accountability,
and long-term infrastructural thinking, rather than reactive
interventions alone.
This article examines how volunteers involved in the Data Rescue
movement navigated concerns about the stability and stewardship of
federal environmental data following the 2016 U.S. presidential
election. Drawing on 16 in-depth interviews, the study explores how
participants interpreted infrastructural fragility not simply as a
technical risk, but as a relational and political concern rooted in
shifting institutional commitments. The analysis introduces the concept
of anticipatory maintenance to describe how volunteers responded to
perceived threats by developing redundant, decentralized strategies for
data preservation. Anticipatory maintenance is conceptualized as
preventive, future-oriented infrastructural care that translates
anticipated disruption into present interventions under uncertainty, in
order to explain how volunteers acted as if future loss had already
begun, building redundant and decentralized preservation arrangements.
Findings highlight the role of particularized trust and systemic
distrust in shaping grassroots responses, as well as the limitations
encountered in volunteer-driven infrastructures, including challenges
related to sustainability, governance, and the affective demands of
ongoing maintenance. By foregrounding the interplay of technical,
social, and emotional factors, this study offers a critical perspective
on data stewardship in times of political uncertainty and underscores
the need for collaborative approaches to infrastructural resilience.
La visualisation en arborescence d’Archifiltre offre une représentation
graphique hiérarchique des fichiers et des dossiers, permettant de
naviguer facilement dans vos système de fichiers, de comprendre leur
structure et de localiser rapidement les éléments à traiter.
If you are new to the field, “science of science” (or metascience) is the study of science itself, using data, network analysis, and computational methods to understand how research careers, collaborations, and ideas develop over time. Dashun Wang's The Science of Science is a good introduction if you want an overview before reading the rest of this post.
This year's edition felt especially timely. Shifting federal funding priorities, the growing use of large language models in research, and ongoing debates about peer review and scientific credit shaped many of the discussions. Across keynotes, panels, and lightning talks, speakers kept returning to the same question: what does it mean to do trustworthy, resilient science today? Below is a recap of each day, along with my reflections, followed by a brief overview of the poster I presented.
Day 1 (June 29, 2026)
The morning opened with welcome remarks from the conference co-chairs, Dan Larremore,Erin Leahey, and Bhaven Sampat. It was great to see many familiar faces from the Ai4SciSci workshop community in person. During breakfast and the coffee breaks, I had the opportunity to talk with Jeff Tsao, Alexander Petersen, Daniel Acuña, and Dashun Wang.
The first two invited talks explored how AI is reshaping scientific research from complementary perspectives. Ryan Hill used AlphaFold as an experiment, showing that while it has expanded research on previously unsolved proteins and made experimental work more efficient, it has not displaced experimental structure determination. Instead, researchers are increasingly focusing on problems where AlphaFold remains weakest, highlighting how AI complements rather than replaces scientific work. Chaoqun Ni broadened the discussion by examining how generative AI is changing the research ecosystem across six core tasks, from exploration to evaluation. Her findings suggest that higher AI use is associated with a reorganization of collaboration, with contributor roles becoming increasingly modular and differentiated. This points to changes in how research teams are structured rather than simply making individual scientists more productive.
The discussion then shifted from AI to science policy with a panel on the “past, present, and future of the social contract for science”, featuring Lisa Margonelli, Tony Mills, and Heather Douglas, and moderated by Bhaven Sampat. Heather Douglas argued that she does not subscribe to Vannevar Bush's post World War II linear model, noting that it is difficult to judge the value of research before it is done, making early evaluation an unreliable basis for funding decisions. Tony Mills emphasized that while publicly funded researchers have obligations to society, congressional oversight of how research funding is spent has weakened over time. Lisa Margonelli broadened the discussion by reflecting on the implicit social contract between science and society, how it is sustained through the stories we tell about science, and how those stories shape public expectations.
Following the panel discussion, Kevin Gross presented “Risk, Reward, and the Choices We Face as Scholars”, exploring why scientists often avoid risky research despite the potential for larger breakthroughs. Using mathematical models, he discussed how incentives around grants, publications, and career advancement shape research choices. One line that stayed with me was: “Risk aversion in science is not necessarily evidence of poor science”. He also argued that tenure serves as a form of risk insurance, allowing researchers to pursue ambitious ideas while still maintaining expectations for meaningful contributions. He made the point humorously that tenure is not a license for professors to go hiking instead of showing up to the lab.
I spent the afternoon in parallel sessions covering Career Trajectories and Data, Software, and Research Infrastructure. Ben Aoki-Sherwood'sstudy of interdisciplinary faculty hiring found that 17% of faculty hires move into a field completely different from their PhD field. Xiang Zheng showed that interdisciplinary PhDs still face placement barriers at top universities even after entering the job market. On the infrastructure side, Eva Brown and Nicholas Weber'sanalysis of software credit was one that stood out: 29% of GitHub code contributors never receive authorship credit on related papers, and only 9% of imported software dependencies are mentioned on average. The most-used libraries are often among the least credited. Yulin Yu's “data hedgehog vs. data fox” framing, which argues that using diverse datasets can improve both research impact and career retention, was a good way to summarize an empirical finding. Finally,Seorin Kim’s audit on OpenAlex abstracts found that roughly one in eight contains an integrity issue, often because only part of a structured PubMed abstract is stored, creating problems for downstream analyses and LLM-based tools.
The day concluded with the poster session, which featured about 50 posters. I presented our ongoing project, “Toward a Cross-Domain AI-Ready Database for Reproducibility and Replicability Studies”, with Dr. Sarah Rajtmajer andDr. Jian Wu. The project aims to build infrastructure that allows researchers and AI systems to systematically query which findings have been reproduced, replicated, or contested across fields, rather than reconstructing that information paper by paper. I had several engaging conversations at the poster, including with staff members from NIHwho were interested in Reproducibility and Replicability Studies from the perspective of research funding and evaluation.
Our Poster: Toward a Cross-Domain AI-Ready Database for Reproducibility and Replicability Studies
Day 2 (June 30, 2026)
One of the highlights of the second day was the panel, “The Future Infrastructure of the Scientific Ecosystem”, featuring Jessica Hullman, Daniel Acuña, and Aaron Clauset, moderated by Misha Teplitskiy. Daniel Acuña argued that AI is already reshaping the research ecosystem, from peer review to scientific communication, but emphasized that while AI can quickly evaluate ideas, it still cannot decide which questions are worth pursuing. Jessica Hullman focused on the limits of AI for scientific evaluation, noting that automated reproducibility checks can be easy to cherry-pick and often fail to capture the broader validity of empirical claims. She argued that AI should support, rather than replace, human judgment and highlighted the need for systems that preserve diversity of thought instead of reinforcing the most popular ideas. Aaron Clauset closed by reminding the audience that scientific knowledge ultimately resides in people, not papers, and that strong human communities remain the foundation of trustworthy science.
In his invited talk, “On Scientific Memory and Innovation”,Lingfei Wu explored the relationship between scientific memory and innovation through the disruption index, which measures how much a paper displaces the work that came before it. He raised an intriguing question about AI: while today's models excel at remembering, reasoning, and recombining knowledge, can they also learn to “forget” dominant ideas in ways that foster innovation? As he put it, perhaps hallucinations are not always a bug but sometimes a feature.
The next two invited talks shifted the focus from ideas to the people and institutions that shape science. In “Canary in the Coal Mine? Prospects for Early Career Scientists”,Donna Ginther discussed the challenges facing early career researchers, from funding pressures to recruiting international students, while emphasizing that AI is increasing the demand for human judgment and evaluation. Her practical advice was simple: choose your advisor carefully and be persistent when facing rejection. Charles Gomezconcluded with “Elite Nations Drive the Convergence of Global Research Agendas”, presenting evidence that a small group of research-intensive countries increasingly shapes the language and direction of global science, raising important questions about whose ideas gain visibility and influence.
The parallel sessions I attended on the second day covered knowledge flow and science communication. Several presentations challenged how we think about the spread of ideas. Zheng Fu argued that citations often reflect scholarly obligation rather than true intellectual influence, meaning ideas can spread widely without their original sources receiving credit. Kyle Siler showed how a small group of countries continues to capture a disproportionate share of global scientific attention, regardless of where new ideas originate. On the science communication side, Hong Chen presented a framework for tracking research mentioned in podcasts. One finding that stood out was that most podcast discussions include no bibliographic information, allowing retracted findings to continue circulating without being flagged.
Day 3 (July 1, 2026)
The final day opened with lightning talks on the future of science policy. Several presentations examined how science interacts with policy and funding, but two stood out to me. Junsol Kim showed how partisan think tanks can reinterpret scientific evidence as it moves into policy documents, often removing important context. Elena Parkerson then quantified the economic costs of multi-year disruptions in NIH funding, a timely reminder of how funding instability can have long-lasting consequences for scientific research.
The discussion continued with the panel “The Future of Science Policy: Research Needs and Opportunities”, featuring Kaye Husbands Fealing, Andrew Gerard, andMatt Hourihan, moderated by Cassidy Sugimoto. The panel offered several practical takeaways for researchers. Kaye Husbands Fealing encouraged researchers to be “pivot-ready”, noting that funding priorities change across administrations and that science of science researchers should develop enough domain expertise to explain the broader context of their findings. Andrew Gerard emphasized the importance of communicating research clearly and making its implications accessible to a broad audience. Matt Hourihan highlighted the persistent tension between the value of research and the funding available to support it, reminding researchers to understand their audience and remain confident in the value of their work.
The conference concluded withMelinda Baldwin's invited talk, “In Referees We Trust? The Rise of Peer Review”, which examined the history of peer review and its role in shaping how scientific work is evaluated and trusted, including its use in research grant decisions at institutions such as the NSF. The conference then closed with awards and closing remarks from co-chairs and James A. Evans, along with the announcement that the 6th ICSSI will be hosted in Rome, Italy.
Closing Thoughts
Three days in Boulder gave me a clearer picture of where the science of science community is heading. The field is no longer focused solely on understanding how science works. Increasingly, it is also asking how AI is reshaping research, scientific collaboration, incentives, and trust. Throughout the conference, discussions repeatedly returned to the same theme: how can we build a scientific ecosystem that remains trustworthy, resilient, and effective as AI becomes an integral part of the research process? While there were no definitive answers, the conversations made it clear that these questions will shape the field for years to come.
I am grateful to my advisor, Dr. Jian Wu, for supporting my travel to ICSSI 2026.
This summer, I had the opportunity to work as a visiting student at the Harvard Graduate School of Education (HGSE), where I worked in the Learning, Innovation, and Technology Lab (LIT Lab) supervised by Dr. Bertrand Schneider. The LIT Lab investigates how people learn and collaborate in learning environments. They use advanced sensing technologies and data-driven methods to measure attention, behavior, and interactions. HGSE is a leading institution dedicated to advancing educational research and innovation through interdisciplinary approaches that combine education, technology, and learning sciences. The internship provided an opportunity to explore new methods, tools, and measures used to study collaborative learning, while also contributing to a new research project.
My internship was an eight-week program, and during my visit I collaborated closely with Dr. Schneider on the project “Exploring the trade-offs between wearable eye-tracking and computer vision technologies for studying joint visual attention (JVA)”. We investigated the strengths and limitations of these approaches in real-world collaborative environments. I participated in regular one-on-one research meetings, which were collaborative development sessions where Dr. Bertrand and I shared progress, discussed findings, and set goals for the coming week. These interactions provided valuable opportunities to refine research methodologies, learn to look at the problem with new perspectives, and to understand the applications and contributions of the work in different domains. The experience allowed me to study how JVA is observed and used in learning science, meet new people and make connections, and explore new opportunities.
Project Overview
Our project explores how people share their attention while working together in real-world activities. There are various definitions of JVA in the literature, but for this study, we define JVA as two people looking at the same object at the same time. JVA can help us better understand collaboration, communication, learning, and social interactions.
Our study investigates the trade-offs between wearable eye tracking and computer vision for detecting JVA in collaborative settings. Wearable eye-tracking technology provides measurements of where individuals are looking from an egocentric point of view, but requires specialized equipment for data collection. In contrast, computer vision models estimate a person's visual attention using head orientation and other visual cues in an image or a video recording from the third-person view, offering a less intrusive and more scalable alternative. By comparing these two approaches across different collaborative tasks, the project aims to identify the situations in which computer vision provides sufficiently accurate estimates of JVA and those in which eye tracking remains necessary. The findings will help researchers choose the most appropriate method for studying human attention in real-world environments and support the development of more reliable and practical approaches for future research.
Approach
We collected data from pairs of participants while they worked together on different collaborative activities using both egocentric eye-tracking glasses and an exocentric video camera. The activities included reading together, building a small LEGO model, making simple electrical circuits, crafting, and some more tasks that naturally require people to share their attention on a third object or area. We varied several factors in the activity setting, including the number of objects involved in the task, the distance between the exocentric camera and the activity area, and the positions of the participants, to evaluate how these conditions affect the performance of the computer vision models.
We compared the two methods by sampling video frames at corresponding timestamps covering the entire recording. For each sampled frame obtained from the eye-tracking sequence, we manually compared it with the predicted frame from both computer vision models. We analyzed cases where both eye-tracking and computer vision approaches identified JVA correctly, as well as cases where one method detected JVA while the other method could not, and we explored the factors affecting the failed cases. We were able to find instances where computer vision models performed well and instances where wearable eye tracking is still necessary. These findings will help researchers choose the best method for studying human attention in real-world collaborative environments based on the conditions of the activity.
Interesting Cases Identified During Manual Inspection
Comparing the computer vision model predictions with eye-tracking data revealed several interesting findings where the model predictions and eye-tracking data did not agree on detecting JVA. We identified the main reasons for the disagreements and organized them into the following categories:
Viewpoint Dependency
Computer vision models often estimate the gaze positions based on participants' head orientation. This works well in many cases, but it can fail when people move only their eyes without moving their head. Sometimes, when there is more than one object in the scene, the computer vision techniques may not provide a precise location of the gaze. Also, the objects of interest can be occluded in the exocentric view. These are some factors affecting incorrect predictions of attention in computer vision models. Figure 1 illustrates this limitation. From the third-person view, the Sharingan model predicts that both participants are looking at the same object. However, the egocentric views captured by the eye-tracking glasses reveal that each participant is actually attending to two different objects.
Figure 1: The Sharingan model predicts shared attention in the third-person view (left), whereas the egocentric views (middle and right) from the eye-tracking glasses reveal that the two participants are actually attending to different objects.
Scene Complexity and Small Objects
The computer vision models perform well when there is one clear object of interest in the activity that everyone is attending to. But we identified some mispredictions when multiple objects are close together in the scene or when participants look at small objects. Figure 2 shows an activity with a single object, and computer vision models performed well in detecting JVA. Figure 3 shows an activity with small LEGO pieces in which the eye-tracking method performed better than the computer vision approach in detecting JVA.
Figure 2: A less complex scene in which two individuals are watching a video on a laptop. Both the computer vision model and the eye-tracking data consistently identify the laptop as the shared object of attention.
Figure 3: A complex scene containing multiple small and visually similar objects. While the eye-tracking data show that the participants are attending to the same task-relevant objects, the computer vision model incorrectly predicts shared attention. This illustrates the challenges gaze-following models face in accurately estimating JVA in cluttered environments with multiple potential gaze targets.
Visibility and Occlusion
Performance of computer vision models decreases when faces are blocked by objects, people turn away from the camera, or the object being viewed is outside the exocentric camera's field of view. With less visual information, the models are more likely to predict the wrong gaze target. In Figure 4, the exocentric camera is placed too far from the activity, and it makes head detection and identifying head orientation difficult for computer vision techniques. Therefore, the attention predictions are incorrect when compared with the eye-tracking data.
Figure 4: An example where the exocentric camera is positioned too far from the activity, resulting in inaccurate gaze predictions by the computer vision model. In contrast, the eye-tracking data reveal that each participant is attending to a different object held in their hands.
Figure 5 illustrates an instance where the face of one participant is blocked by the object in front. This leads to incorrect gaze predictions by computer vision models.
Figure 5. An example illustrating the impact of face occlusion on computer vision–based gaze-following models. When participants' faces are partially occluded by objects in the scene, the models fail to accurately estimate their gaze direction, resulting in incorrect predictions of visual attention.
Tracking and Multi-person Association
Computer vision models struggle to maintain consistent tracking of participants when they move out of and later re-enter the camera's field of view, when heads overlap in the scene, or when temporary occlusions interrupt person tracking. In these cases, the models reassign participant identities after an occlusion event or after a participant leaves and later re-enters the camera's field of view, resulting in inconsistent identity assignments across frames. Figure 6 illustrates four video frames captured at different timestamps where participants were assigned different identities following these events. The changes are indicated by the change in head detection colors.
Although these identity switches do not directly affect JVA detection in our approach, since JVA estimation does not rely on persistent participant IDs, they reveal limitations in the temporal consistency of the tracking process. Such inconsistencies can affect participant-specific JVA analysis, longitudinal behavioral measurements, and applications that require maintaining a continuous identity for each individual throughout an interaction.
Figure 6: An example from a movement-intensive collaborative activity where frequent motion and changes in participant visibility challenge computer vision–based gaze-following models. As participants move out of the camera's field of view and later re-enter the scene, the models may assign them new identities instead of maintaining consistent tracking.
Model Biases
Computer vision–based gaze-following models may exhibit inherent biases that influence their predictions in social interaction scenarios. For example, the MTGS model occasionally predicts that participants are looking at each other even when eye-tracking data indicate that their attention is directed elsewhere. Similarly, the model tends to infer shared attention toward prominent or commonly occurring objects in the scene, even when the actual gaze target differs. These biases can reduce accuracy in complex collaborative activities involving multiple objects, subtle gaze shifts, or task-specific attention. Figure 7 shows the tendency of the model to overestimate looking at humans. Figure 8 is an example of a misprediction due to a predefined model threshold.
Figure 7: The MTGS model predicts that one participant is looking at the other person, even though her actual attention is directed toward the board.
Figure 8: The MTGS model predicts shared attention (SA) based on a predefined threshold in the model, even in cases where the participants are clearly attending to different objects.
What We Have Learned So Far
As a part of this ongoing work, we are currently expanding our manually annotated dataset to include a wider range of collaborative activities and interaction scenarios. Our preliminary findings reveal that computer vision models perform well in simple scenarios where participants share a clear, visible object of attention and their faces are easily seen by the exocentric camera. However, the prediction accuracy decreases in more complex situations involving multiple similar objects, occlusions, participant movement, or limited camera viewpoints. In comparison, wearable eye tracking provides more accurate measurements of where people are actually looking, making it better suited for studies that require object-level accuracy. However, this approach typically involves higher costs, requires participants to wear specialized eye-tracking devices, and can be more intrusive than camera-based methods, making it less practical for large-scale or naturalistic studies. Overall, our results indicate that the most appropriate method depends on the task and research goals, and that combining computer vision with eye tracking has strong potential for studying real-world collaborative interactions by leveraging the strengths of both approaches.
Experience at HGSE and LIT Lab
Working at the LIT Lab at HGSE was a great experience that contributed significantly to my professional and personal growth. The environment allowed me to interact with fellow students from diverse backgrounds, providing opportunities to exchange ideas and learn from their expertise. Everyone in the lab was welcoming, supportive, and always willing to help. Having access to the Innovation Studio further enriched my experience, and I learnt how advanced technologies, tools, and creative research spaces enhance interdisciplinary collaboration and hands-on experimentation. The experience strengthened my confidence as a researcher, expanded my professional network, and gave me new perspectives on conducting impactful human-centered research in collaborative environments. I’m sincerely grateful to Dr. Schneider for this opportunity and support throughout my internship.
LIT lab members and friends
Exploring Harvard and Boston
During my internship, I had the opportunity to explore Harvard University and experience the rich history, architecture, and vibrant atmosphere of the campus and its environs. I appreciated the historic buildings, beautiful surroundings, and the unique academic environment that has shaped generations of scholars. Beyond the university, I explored different popular areas around Boston, discovering the city's blend of historic landmarks and attractions. The following are a few snapshots from my visit.
Acknowledgments
I would like to express my gratitude to my advisor, Dr. Sampath Jayarathna, for taking the initiative to make this incredible internship opportunity at HGSE possible. I appreciate his continuous guidance, encouragement, and support throughout my academic journey. His mentorship has played a significant role in my academic and professional growth.
The USA is 250 years old this month, and in honor of the occasion LibraryThing sat down with authors Stephanie Dray and Laura Kamoie, who recently collaborated on A Founding Mother, a historical novel about Abigail Adams published by William Morrow in May 2026. A bestselling author of historical fiction, Dray earned her undergraduate degree in Government from Smith College, and her law degree from Northwestern University School of Law, and has worked as a lawyer, game designer and teacher. Her many works include the Cleopatra’s Daughter trilogy, about the life of Cleopatra Selene II, and stand-alone novels like The Women of Chateau Lafayette (2021). Kamoie, also a bestselling author of historical fiction, is a historian who earned her undergraduate degree from Dickinson College and her MA and PhD in early American history from the College of William and Mary. She has published two nonfiction works on early America—Neabsco and Occoquan: The Tayloe Family Iron Plantations, 1730-1830 (2003) and Irons in the Fire: The Business History of the Tayloe Family and Virginia’s Gentry, 1700-1860 (2007)—and has worked as a history professor at the university level, most recently at the US Naval Academy. Before their most recent collaboration, Dray and Kamoie also wrote America’s First Daughter (2016) and My Dear Hamilton (2018) together. They sat down with Abigail this month to discuss their new book.
In your previous novels you’ve explored the lives of Martha “Patsy” Jefferson Randolph and Eliza Schuyler Hamilton. What made you decide to write about Abigail Adams next?
Readers had been asking us to write about Abigail for a long time, but when we realized that our next book was going to come out in 2026, the 250th anniversary of the Declaration of Independence, the choice was a no brainer. Abigail was so far ahead of her time, so direct and unflinching, and so modern in her sentiments, that she was the perfect voice from the founding generation to speak to Americans today.
Tell us a little bit about Abigail Adams and her life, and what makes her such an interesting figure. (Full disclosure: as a bicentennial baby, I was named after Abigail Adams, so I already find Adams a fascinating figure!).
Love that! Abigail was in many ways an ordinary farm wife of Braintree, Massachusetts, at the outbreak of the revolutionary war. She lived in a modest home. She was raising four children. She didn’t have a formal education. But in other ways, she had always been extraordinary. We’ve read a lot of letters from 18th-century women at this point in our career, and none of them display the sense of self-possession of Abigail Adams. She seems to have been a smart, rebellious, independent girl from the get go. She chose a husband who would let her become more than an ordinary farm wife. She became an entrepreneur. A diplomat. A canny political operator. And much, much more.
Adams is well known for her writing, including her famous “Remember the Ladies” letter. How much did you rely on her letters or other work, in writing A Founding Mother? Were there other sources–specific biographies or histories—that were helpful?
While always relying heavily on the original letters, and in this case there were many by Abigail and her family and friends, as much as possible, we use the figures’ own words from their letters in their dialogue and internal monologue. We were also guided by one extraordinary biography by Woody Holton and spent a lot of time with David McCullough’s book on John Adams.
What were some of the most interesting things you learned about Adams, in the course of your research for the book?
The most surprising thing we learned was that although Abigail and John Adams shared one of America’s greatest love stories, it was not a fairy tale. Their marriage was under strain when he was away in Europe—there was a forgotten founding father who is probably better off forgotten who crossed the line with Abigail in inappropriate ways—and that Abigail definitely kept some secrets from John.
All three of your books so far have focused on women who were related to more famous men—in the case of Abigail Adams, her husband John Adams. Why is it important to tell their stories? Which other historical women would you like to highlight?
First of all, the irony is that Abigail may be better remembered than John at this point! But to your point, it’s important to tell the stories of the women who stood behind, alongside, and sometimes even ahead of the men who are said to have founded this country. They didn’t do it alone. Women built this country too and deserve the credit. Because when we don’t acknowledge their contributions, people end up deluded into thinking that “traditional” women were fundamentally dependent and happy about it. That wasn’t true for any of the founding mothers we have written about, and especially not Abigail Adams, who was socially conservative in a number of ways while still being an absolute firecracker when it came to women’s rights and the role they could and should play in government. So if the women who founded this country aren’t “traditional” then who is?
As for other women we would like to highlight, so many! We always have to figure out the venn diagram between what we’re both passionate about, what readers want, and what our publisher thinks will be most marketable. But it really is our mission to do as Abigail said and “Remember the Ladies.”
Tell us about your writing process, when working together. How does that work? Are there specific challenges to writing as a team, or specific pleasures?
We have no set process! Sometimes we alternate chapters. Sometimes we assign whole chunks. One specific challenge is technology. Frequently, we break Google Docs or Microsoft Word because we have so many comments or footnotes. Sometimes we are forced to sit at a table together over one manuscript and shift the keyboard back and forth as we make decisions. But that’s just a good excuse to get together. We enjoy each other’s company and there is a certain magic in our collaboration born of deep respect for each other’s talents and writing judgments. We really do make each other better. And we love sharing little insights and research nuggets and nerding out together at historical sites.
What comes next for you two? Do you have further collaborations in mind, or individual projects in the offing?
We do have another collaboration in mind, and we can’t tell you what it is yet, but we’re excited! We’re both also working on solo novels, so there’s much more to come!
Tell us about your library. What’s on your own shelves?
We both have our own books on our shelves, of course. That gives you a nice pick-me-up when you’re having a bad day. Then there are all the books written by friends and other great writers we admire. Laura has a prime spot for The Alice Network by Kate Quinn on her shelf while Stephanie favors Kate’s The Rose Code. Then there are all the research books. So many.
What have you been reading lately, and what would you recommend to other readers?
[This is the fourth piece in a miniseries on finding the right line between human thought and AI assistance, focusing on the stages of scholarly work from initial ideas through the research process to publication, although I believe much of this discussion is applicable to intellectual work beyond the academy. The miniseries began with this introduction and was followed by an essay on the origins of new ideas and a piece on analyzing evidence and data in the early stages of research. In this issue, I look at handing the entire writing process over to AI.]
At the end of the last essay in this series, on the application of AI to analytical sections of a research project, such as data explorations and visualizations, I left this question dangling:
Why not go further or even all the way? Why not have AI do the entire analytical process and spit out the result, perhaps as a nicely formatted paper?
Point your favorite LLM at a stack of articles, documents, lab notebooks, or data sets, and give it a prompt — or if you really want to go all in, have it come up with its own question to answer or theme to pursue based on an initial pass — and sit back in your comfy armchair as the words flitter by.
This new form of armchair scholarship, like the nineteenth-century academics who wrote books and articles based only on what was readily at hand (in, say, their posh home libraries), rather than doing iterative and extensive field work, lab work, or any other kind of time-consuming engagement with their subject matter, is far from hypothetical. As you read this, AI is probably writing hundreds of academic papers. This time the armchair has a jet engine on the back.
A nontrivial and growing percentage of submissions to journals are now partially or mostly the product of AI, especially in scientific fields. In one study, based on STEM articles from 2021 through 2024, AI usage in the writing of academic articles spiked beginning with the release of ChatGPT in the fall of 2022, quickly reaching over 20% in computer science and electrical engineering, and, to a lesser but still significant extent, in other fields by the fall of 2024. These measurements are surely much higher two years later, especially with the latest agentic AI tools and more advanced models.
These early-adopting AI-assisted scholars are still likely in the minority, and if we want to be generous to them, there may be understandable reasons for their reliance on AI, such as the predominance of English in science publishing. (According to the study, researchers from countries where English is not a first or second language use AI to compose articles at significantly higher rates.) Of course, there’s also the primal need to publish or perish in academia, which has always incentivized cutting corners.
Whether AI authorship of papers is an activity dominated by mercenary researchers seeking tenure or the byproduct of AI translation, the temptation to use AI in the production of scholarly writing will undoubtedly continue to grow. This year has seen a proliferation of websites and software geared toward rapid paper generation. Generally the producers of these tools frame the process as a productive collaboration between you and the AI, but the amount of “you” seems to shrink as you look more carefully and notice the many opportunities to opt out of deep intellectual work. For instance, the AI paper-writing assistant Gatsbi nods toward the discrete stages of research and writing I’ve covered in this series, but also notes that it can go ahead and “auto-draft” the entire paper, stem to stern, if you’d prefer to get a coffee. CoPaper.AI, from Stanford, similarly emphasizes that the scholar guides the production of the article at each turn, but also claims in a large font on its home page that it can take only 20 minutes to sprint from raw data to a final paper. It may be “human in the loop,” but at that scale you’re an ant inside a hula hoop. And naturally you can use Claude or ChatGPT to do end-to-end research and writing, at various levels of engagement from micromanager to laissez-faire napper, whether you’re a fifth grader or a faculty member.
* * *
Call me an optimist, but I believe that the majority of scholars, despite being under enormous pressure to publish, would prefer to be more engaged than removed from the fundamental pursuits and texture of their discipline. They enjoy wrestling with sources, data, and theories, and are innately repelled by the superficiality of having AI write a complete paper. They want to lean into their work, not lean back in an automated armchair. As NYU astrophysicist David Hogg recently wrote, “Anyone working in astrophysics is someone who wants to do astrophysics, not someone who wants to learn the answers.”
Anthropic and OpenAI seem to understand this now, after years of touting AI in a less-than-reassuring way as a replacement for human thought, endeavor, and employment. OpenAI’s new writing tool Prism, centered on LaTeX, a nerdy markup and formatting standard for STEM articles, and Anthropic’s new workbench Claude Science, seem inclined toward more exploratory, iterative, and assistive modes of STEM article generation. Partners, not proxies.
But how slippery is the slope? If one uses AI to help out with part of a scholarly work — for instance, the often (but not always) formulaic “methods” section of a scientific article — will the temptation rise to use it for other parts, such as the more intellectually stimulating and important “discussion” section, in which the results of an experiment are unpacked and its implications for the field made clear? And if so…is that so bad?
Economist and AI enthusiast Tyler Cowen and others working in the more data-centric areas of the social sciences and natural sciences have pondered this question and begun to sour on the old method of article production, instead believing that the future of scholarship may indeed lie in a data set — perhaps one that is constantly updated — and an AI front end that interprets this data. The careful wordsmithing of a paper might be secondary to this direct computational approach, or vanish altogether. And maybe the AI can create the data set too, leaving more time for coffee breaks.
My nagging worry is this: based on a passing familiarity with human nature, I can foresee that an increasing number of academics are going to have to be lashed to the library stacks to resist the AI sirens, who will sing not about more measured uses of AI, but about that sweet, comfortable armchair in which to rest while entire articles are tirelessly generated for them.
* * *
Since we do not have beeswax to put in our ears to resist this song, it seems helpful to examine exactly why the AI generation of a complete scholarly work, rather than using AI judiciously for certain scholarly tasks as I have been arguing in this series, is a bad idea — not just for academic disciplines but for the academics themselves.
At this point, you might be expecting a long rant on AI hallucinations and the possibility of scholarship turning from the pursuit of truth into the extrusion of plausible-sounding truthiness. Hallucinations do remain an area of concern, but it is a problem that has waned over the last year. As LLMs have become more agentic than static, and more rigorously structured in their processes — not relying as much on their initial training set, and venturing out to read external sources of information as needed, instead of immediately starting to spit out text following a query — the number of glaring, or even small, errors has decreased. Especially in the highly connected academic AI environments I have been discussing in this series, with actual libraries available to the LLMs — Claude Science, for instance, can retrieve peer-reviewed research and vetted data from dozens of highly specialized academic resources — the hallucination problem has receded further.
At the same time, hallucinations have not totally disappeared, and academic research should always aim for the highest level of reliability, which makes even the small possibility of hallucinations a shadow over the scholarly enterprise, not to mention the embarrassment of AI-generated faux pas to scholars who take an automated shortcut. Of course, human intelligence can also make errors, or worse, engage in statistical shiftiness or outright fraud. (See the replication crisis.) But we should be aiming to level up on veracity, not down.
More problematic to me than the specter of hallucinations, however, are the long-term effects of the armchair overuse of AI on three areas dear to the academy: the nature of writing and reading, the composition of the sources that writing is based upon, and the enervation of the scholarly mind and scholarly disciplines.
* * *
In early 2023, soon after the release of ChatGPT, I asked, “Can Engineered Writing Ever Be Great?” My answer came from a simple point that every writer knows:
Good writing isn't just the selection and ordering of words, the output; good writing is the product of good reading. Writers aren't indiscriminate generalists, but tend to be rather choosy and personal about what they read. As humans they also have a fairly limited reading capacity, which means that their styles are highly influenced by idiosyncratic reading histories, by their whim.
LLMs, on the other hand, are insatiable omnivores, ingesting as much text, indiscriminately, as they can. This allows them to create countless styles of writing virtually instantly, that feed every need from an automated email response to poetry. But this also means that off-the-rack writing from an LLM tends toward the anodyne, or “slop” if we want to use a more disparaging term.
Over three years later, however, LLM output can be significantly improved, especially if you tailor the inputs to the underlying model, or add post-training context. In one recent study, readers preferred the writing from an LLM trained on books (rather than text that largely comes from the web) over that of human writers with MFAs. I have used Claude Code to create a database of all of my writing (books, academic and popular-press articles, blog posts, this newsletter, and unpublished writing, 2+ million words), which I mostly use to remember and find things I’ve written, but which Claude could also use to compose new pieces very much in my voice, if I wanted it to. (I don’t. The em dashes you often see in my writing are my own; I do love them and don’t care if they have become a marker of AI writing.) If you think that my writing rises above generic AI slop, then I can assure you it’s now possible to use AI to create prose that mimics this more personal, angular style.
Nevertheless, having AI generate an article, even in one’s own voice, inevitably cedes critical intellectual ground. Implicit in the new AI paper-generation tools is the assumption that specific word choices in an article or book are of lesser value than the overall interpretation of sources or data. That may be true in a general sense, and surely many readers of academic articles, ahem, skim, but picking words carefully can increase an article’s power of persuasion and impact. As Daniel Kahneman has shown, a lamentable aspect of human psychology is that we have trouble accepting data as proving a point; we often need well-crafted words and a coherent narrative mapping cause and effect, preferably from someone we see as a peer, to convey the significance of that data and incline readers to accept conclusions. (Even then, alas, human beings can be truly stubborn in their views.) I can now have Claude produce prose that sounds like me, but only the real me can pick the exact words with the right spin and force I’m looking for in a particular sentence. (Plus, I actually enjoy writing; you’ll have to pry my keyboard from my cold, dead hands.)
Furthermore, if we know that a significant percentage of articles are machine-written, we are going to move from careful reading to frequent skimming to a complete abstention from the scholarship in our field. (We will probably have an LLM summarize it for us.) This will obviously greatly harm the exchange of ideas. The only way out of this conundrum is for most practitioners in a discipline to commit to putting in the time and energy to produce and digest thoughtful work. Knowledge production is inherently social — not in the postmodern, constructed-out-of-thin-air way, but embedded in a communal process in which we come to respect, or at least recognize, that other intellects are wrestling with the same problems we care about, and which fosters a continued interest in our common research.
* * *
Then there is the invisible loss of context and detail when AI writes the majority of an academic work. An experiment as a case in point: One day I tried, like an Oliver Sacks case study, to be my wife, who is a scholar of early childhood programs. Using Cowork,I told Claude I wanted to write a paper on how different state policies on child care impact American children and their families. We (Claude and I) decided to use Policy Commons’ invaluable data set, through which Claude assembled a list of 232 state statutes and regulations. Sipping my coffee, I asked Claude to read and process all of these lengthy documents and create a matrix for me so I could quickly assess the contours of child care programs in the United States. But Claude was more caffeinated than I was: like a teacher’s pet it went off and produced, independently, a comprehensive report, not just a table, in a few minutes. It even created its own categories of differentiating metrics, such as infant/toddler:adult ratios, total care group sizes, licensing requirements, academic qualifications for practitioners and directors of programs, sleep protocols, and idiosyncratic state mandates. A few more sips of joe, and a few more prompts to acquire additional data, and I was swiftly on my way to what I thought was a decent meta-review essay.
Now a giddy AI-assisted dilettante, I showed this effortless production to my wife, who proceeded, as an actual expert, to dissect it ruthlessly. Claude got the empirical data mostly correct — no silly hallucinations — but its attempts to extrapolate from the numbers into trends and impacts were clumsy and overly broad. Since my wife actually knows these early childhood programs well, she understands how, on the ground, actual child care sites might differ from written state policies and other sources of information, and she could identify this missing context and additional key details that were invisible to me and my AI buddy. Our armchair scholarship was no match for her decades of experience and knowledge.
The output seemed so good, though…I could totally imagine a less scrupulous academic trying to publish it. The ease of generating an AI paper this way means we will increasingly end up with papers written on the data that is readily available, without questioning how good the data actually is. We will think less about what’s missing.
Yet even with agentic AI — again, Claude Science can reach out to dozens of research databases for material to work with — there are yawning gaps. At the closing plenary this spring at the Coalition for Networked Information meeting in Salt Lake City, “Harnessing the Data Renaissance for Scientific Discovery,” Manish Parashar, Executive Director of the Scientific Computing and Imaging Institute and Chief AI Officer at the University of Utah, highlighted the major work that still needs to be done to create truly rich and comprehensive data sets for many fields, such as cancer research. If we ignore this complex and time-consuming process, and just use AI on top of the data that’s close at hand, we might produce scientific articles more quickly, but we may not make significant breakthroughs. (This is probably already happening.) Parashar, instead, is working with other scientists on a National Data Platform that will aggregate and normalize thousands of sources, without which any AI processes will suffer from a problematic narrowness. In other words, being a good librarian — someone who finds, catalogs, assesses, and merges sources into a coherent and usable library — has become even more valuable.
* * *
Finally, using AI to generate new scholarly papers assumes an iffy theory of intellectual history, that new ideas and discoveries are always implicit but not yet articulated in the existing literature and data. What if new ideas instead come from unique circumstances, lived experience, group interactions, or an individual’s eccentric way of reading and seeing? Yes, the history of science has a number of examples of innovations that seem to be “in the air” and thus “discovered” by multiple people at roughly the same time, such as calculus (simultaneously developed by Leibniz and Newton). But there are many more examples like the one I wrote about in “Can AI Prompt Us to Ask New Questions?,” where a quirky combination of a person’s biography and interests, embedded in a particular social scene, leads to revelations like fractal geometry.
Intellectual history contains both kinds of discoveries, but we wouldn’t want to block the latter, deep river of innovation, and if the overuse of AI in the production of scholarship reduces the velocity of the human mind and the vitality of intellectual scenes, we might find this source slowly drying up. It is worth remembering that asking good questions is harder than giving great answers. Insightful out-of-the-box approaches, often stemming from unusual, previously unasked queries, are the dark matter of human thought and progress, and they often come from the random interactions and odd interests of particular human beings. It is unclear how AI will replicate these uncommon vibrations and collisions.
In “Illegible Benefits,” a piece by Carlo Cordasco of the University of Manchester that is largely positive about using AI in the production of scholarship, he mentions a lingering concern:
I want to be honest about the costs. My ability to hold together a complex position verbally, under pressure, in a seminar or a conversation, has probably not improved and may have declined somewhat. When preliminary exploration is cheap, you spend less time grinding through arguments from first principles, a grinding that builds fluency that shows up in live exchange.
This is the professorial equivalent of the cognitive decline that we worry about with our students who have been using AI for years — and what atrophies in the seminar room will surely atrophy on the page as well. The solution seems clear: if you would like, use AI for the parts of scholarly work where it excels and check all automated output, while retaining human seniority in orchestrating and expressing the meaning of your research.
It’s going to be tough, though. Armchairs really are comfortable.
Please suspend disbelief and follow me below the fold as I look into a fascinating examination of the implications of the exponentialgrowth in the data centers needed to provide these benefits.
Not that I ever remember listening to Timbuk3 song, but I distinctly remember, shortly before the Black Monday stock market crash, Scott McNealy celebrating Sun's exponential growth with his version of the title.
Robert T. Nachtrieb and Steven J. Smith's AI Hastens Limits to Exponential Growth is a fascinating exploration of the long-term effects of exponential growth. They point out that:
While AI electricity consumption was only 1.5% of the global total in 2024, its power demand has grown at a rate of 0.127 yr−1 since 2015, accelerating to 0.15 yr−1 over the last five years. Projecting from this 2024 baseline, AI’s electricity demand is on track to achieve parity with the rest of the world’s combined consumption by approximately 2050.
All systems that grow exponentially end up running into limits that prevent further growth. Nachtrieb and Smith identify five such limits to the growth of AI data centers:
Non-renewable resource depletion
The Kelvin Limit
The renewable resource limit
The Dyson Limit
The Asimov Limit
Each of these limits is interesting, but here I only look into my favorite, the Kelvin Limit. They define this limit thus:
Even with an infinite energy source, the laws of physics dictate every unit of energy used eventually becomes waste heat. On a planetary scale, this heat must be radiated into space to maintain a stable environment.
Earth stays at a life-sustaining temperature by balancing received solar energy with infrared radiation emitted back into the cosmos. Human energy consumption from “terrestrial” sources, such as nuclear fusion or fossil fuels, adds “new” heat to this balance. Because this energy was not already part of the solar-to-earth flow, the planet must reach a higher temperature to increase its radiative cooling capacity and shed the additional load.
The “Kelvin Limit” defines the point where this added waste heat pushes Earth’s surface temperature to 373 K (100 ◦C), the boiling point of water. At this threshold, the planet becomes physically uninhabitable.
They build a model to predict when the Kelvin Limit would be reached:
The solar power arriving at the Earth’s cross-section is Lα ≈ 5474 × 103 EJ yr−1. A portion of this energy is immediately reflected into space by the Earth’s albedo (a), which represents the planet’s reflectivity. Based on NASA data, Earth’s albedo is approximately 0.30, meaning 30% of incoming light reflects away while the remaining 70% is absorbed as heat (Pa).
The model assumes an initial equilibrium where absorbed solar power (Pa) equals the power radiated at the baseline temperature (T0).
...
The limit occurs at time t2, when the combined heat of the Sun (Pa) and human energy demand (D) requires the Earth to reach the boiling point (T2) to maintain equilibrium.
...
This thermal wall represents a hard physical limit. Technological efficiency cannot bypass it; higher energy use to drive AI or industry simply accelerates the transition toward this planetary boiling point.
For each of their cases they compute k, the rate coefficient. The time to hit the limit is t = k/r, where r is the rate of increase of demand. For example, r over the last five years is 0.15. Their results are summarized in Table Viii, showing that in the Kelvin case k is 10.0 for all r. Note that the renewable-only case is the only case where k is less than the Kelvin case, and only by 9%. This demonstrates the very fundamental nature of the Kelvin Limit.
Table IX summarizes their resulting t for each case for their ranges of r and k values. For the Kelvin case at 0.15, the recent value of r, they write:
These figures represent a fundamental shift in the prospects of civilization under AI. Under the 15% growth rates currently demonstrated by AI infrastructure, we quickly accelerate past all projected limits. The thermal “Kelvin Limit” (k ≈ 10), which would normally take ten centuries to reach, suddenly appears in just 67 yr, well within a single human lifetime.
This 67 year estimate is, of course, an upper bound. The Earth becomes uninhabitable for humans long before the surface reaches 373 Kelvin. As we see in Texas, the current r = 0.15 is not actually fueled by renewables, and is thus contributing to much faster heating. As I understand it, their demand D is just the demand for running the data centers. At r = 0.15 there is significant extra demand for building 15% more data centers and 15% more Nvidia racks each year than the previous year.
Nachtrieb and Smith's Figure 2 above estimates that at r = 0.15 running the data centers would take half of the "world’s combined consumption by approximately 2050". In 2025 the Gross World Product according to the IMF was:
forecast to be around $208.96 trillion, $11.04 trillion up compared to $197.91 trillion in 2024.
Assuming this 5.6% growth rate continued, in 2050, GWP would be 3.7 times higher at $773T. Presumably, half of this would be generated by the data centers, or about $387T. Thanks to the economic mechanism described by W. Brian Arthur in Increasing Returns and Path Dependence in the Economy, it is likely that only one company would dominate the AI market. At 20 times earnings, it would be worth around $7.7 quadrillion.
I think you can understand why investors are pouring money into AI companies.
The scraping problem is worse than anyone can imagine and thanks to my friends at Sourceware we have some real data to prove it.
I've been working more on Anubis' reputation database and I've run into a really weird discovery: 80-90% of the hits created by the honeypot feature are from IP addresses that do not belong to any existing threat monitoring lists.
Here's a breakdown of the honeypot hits Sourceware has gotten in the last few months:
Assessment of ./data/manually-submitted/sourceware/202607141625.txt against ./var/reputationdb.mmdb
This doesn't list data from 204 additional countries. Given that the ISO 3166-1 standard comprises 249 countries (193 of which are UN members), it's safe to say this is a global problem.
ASNs (21116 distinct, of all addresses)
ASN
Unique IPs
Flagged
Rate
AS55836 Reliance Jio Infocomm Limited
57029
1749
3.1%
AS45899 VNPT Corp
56910
6831
12.0%
AS6057 Administracion Nacional de Telecomunicaciones
43694
339
0.8%
AS25019 Saudi Telecom Company JSC
40800
679
1.7%
AS24560 Bharti Airtel Ltd., Telemedia Services
35957
1620
4.5%
AS36903 Office National des Postes et Telecommunications ONPT (Maroc Telecom) / IAM
35562
668
1.9%
AS36947 Telecom Algeria
33172
386
1.2%
AS9121 Turk Telekom
32742
1465
4.5%
AS8151 UNINET
32012
856
2.7%
AS14593 Space Exploration Technologies Corporation
31569
4597
14.6%
AS9299 Philippine Long Distance Telephone Company
27573
1626
5.9%
AS39891 Saudi Telecom Company JSC
25904
794
3.1%
AS35819 Etihad Etisalat, a joint stock company
24493
978
4.0%
AS28573 Claro NXT Telecomunicacoes Ltda
23903
841
3.5%
AS8193 Uzbektelekom Joint Stock Company
22611
3191
14.1%
AS8452 IDDQD-AS
22369
364
1.6%
AS43766 Mobile Telecommunication Company Saudi Arabia Joint-Stock company
AS17072 TOTAL PLAY TELECOMUNICACIONES, S.A.P.I. DE C.V.
18021
1181
6.6%
AS22927 Telefonica de Argentina
17672
291
1.6%
AS13999 Mega Cable, S.A. de C.V.
17410
692
4.0%
AS36925 MEDITELECOM
17259
383
2.2%
AS47331 Turk Telekom
17211
26
0.2%
There are 18069 more ASNs not listed.
How Anubis' honeypot works
In order to collect data on how widespread the scraper problem is, I added a honeypot feature to Anubis. On every challenge page it adds semantically invalid HTML akin to the following:
Visiting that page gets you cheap to generate vacuous anti-content that has two links to other pages. This is intended to get badly written scrapers caught in the honeypot so they scrape that instead of the protected website. I made it on a whim but thought it would be great for collecting data on how widespread this problem actually is.
This is a global problem
Based on the data I've seen, this is a global problem. If I had to guess where most of this traffic is coming from, it's from compromised smart appliances contributing traffic to proxy networks. I don't think there's any way to make a real impact on this problem without concerted simultaneous global action.
TL;DR: the scraping problem is actually widespread enough that web application firewalls like Anubis make sense.
A presigned URL is a replay attack you did on purpose.
Replayable auth tokens are the textbook way to create vulnerable systems, but
Tigris ships them as a first-class feature with presigned URLs and so does every
other object storage system on the planet. However this isn't an oversight
because presigned URLs turn a weakness into a feature.
Replay attacks are a real problem and the classic fix is miserable
When you authenticate a request with Amazon's SigV4 protocol for Tigris, your
client boils down the request to a canonical form: a SHA256 hash of the
request's method, path, query parameters, signed headers and a SHA256 hash of
the payload. It runs the result of that through HMAC with a signing key derived
from your secret access key. Nothing secret ever crosses the wire. The server
derives the same key as the client, does the same canonical form transformation,
and compares the result.
Being able to make a valid signature proves that the request came from someone
holding the secret access key, but it proves nothing about when that request
was made. A signature that was made a year ago would still be valid today or any
other time you send it, so in theory an attacker could warehouse your signed
requests only to replay them en masse later. Imagine sitting on a pile of signed
"create EC2 instance" calls only to spam them all out at a later date. You would
be a twirling moustache villain able to spawn dozens of servers at a moment's
notice.
Traditionally the fix is to bake a nonce (number used once) into the signature
(sorry to any British readers in the audience). This makes every signature
differ because that nonce differs.
However with great power comes great responsibility and making sure that
something used once is only used once is a surprisingly hard distributed systems
problem. You can't verify that something is only used once locally. Say you
store them all for a 15 minute smear window at a low request rate like 10,000
Bq. That's 9 million live nonces, and every frontend node needs to have a
consistent view of the whole set as it churns.
You have made your fast authentication check slow from having to ensure things
are only used once.
What you want instead is something that changes constantly without coordination
and invalidates those old signatures for free. For an added bonus you want this
to also be in the standard library of every programming language.
Sign the clock
There's exactly one value that changes constantly, (mostly) monotonically, and
is already actively coordinated across all elements of the stack: the clock.
Your OS already keeps time in sync with the public NTP pool (or a private NTP
pool if you are cool enough to have radioactive PCI cards laying around).
Without an accurate view of time you can't make TLS connections, which means you
can't make API calls to Tigris at all, so the auth layer gets to assume a
working clock exists.
SigV4 signs the current time into the request. If an attacker gets their greasy
hacker paws on a signature, they have about 15 minutes to use it before it
becomes a digital paperweight. If time is an input to the signature and the time
changes enough to invalidate the signature, the signature is null and void. Sure
in theory a sufficiently funded attacker could create a black hole in your
datacentre and disrupt temporal flow, but at that point the planet is probably
toast which makes the attack profile moot. Commit mass object storage fraud with
this one neat trick! The department of temporal investigations will have hated
it!
This makes your verification stay stateless. Everything gets checked against the
system clock the server already needs and you can give clients a 15 minute
signature smear window as a grace period for old or delayed clients (exponential
backoff is a good thing and Tigris will reward you for doing it).
Of course the real thing keeping the signatures safe on the wire is TLS (HTTPS).
If that is broken we have bigger problems and object storage fraud is the least
of our problems.
Time is the only nonce you need because both sides already agree on it anyways.
Some thorns have roses
Presigned URLs take the replay tolerance that SigV4 spends all this effort
nerfing and then buffs it into the feature. The entire auth dance gets flattened
into URL parameters that any HTTP client can use, be it a browser, curl, Go's
net/http, or something you made by bit-banging HTTP over a socket. Here's a real
presigned URL I sundered into visibility:
Here are the parts (forgive the AI looking listicle because this is genuinely
the best way to format this):
X-Amz-Algorithm: the signature scheme. Effectively always
AWS4-HMAC-SHA256.
X-Amz-Credential: the access key ID plus the credential scope — date,
region, service, and the literal terminator aws4_request. The signing key is
derived by chaining HMAC through exactly those parts, so a signature is only
ever valid for that day, that region, that service.
X-Amz-Date: the second the URL was born, in UTC.
X-Amz-Expires: how many seconds it gets to live, chosen by the signer.
X-Amz-SignedHeaders: which HTTP headers are folded into the signature.
Usually just host, because you can't force whoever you hand a URL to into
sending exotic headers.
X-Amz-Signature: 64 hex characters of HMAC-SHA256 over the canonical
request — the method, the path, every parameter above, the signed headers, and
the payload hash. Change any of them and the math stops agreeing.
All of these are normally HTTP headers in standard SigV4 requests.
Note that this request is not a legal request, it's an example to illustrate the
point, here be dragons, etc etc etc.
It's best to think about this presigned URL as a capability grant. Whoever holds
it gets to make exactly one (1) kind of API call with one (1) HTTP method
against one (1) object in one (1) bucket. They can do this as many times as they
want until the presigned URL expires. The signature covers the method, the path,
and the signed headers so a user can't take a presigned request for GETting a
copy of Moby Dick from a development environment and weaponize it into a way to
delete everything in your production bucket.
Possession is authorization until the clock says no.
What it costs you
Capability grants like this can have some sharp edges. There is no real way to
revoke any individual presigned URL short of killing the access key it was
signed with. When that key dies, everything it signed dies too. This includes
any URLs you may have wanted. This cuts both ways and it kinda has to unless you
make a new keypair per presigned request, which is probably out of scope.
Expiry has fine print too. A presigned request can live anywhere from one (1)
second to one (1) week (seven (7) periods of twenty-four (24) hours).
There's no limit to the number of times a client can use a presigned request. If
you give a mouse permission to GET one cookie, they can GET that same cookie
over and over. You end up having to pay for the GetObject calls in the end, so
keep that in mind.
URLs also leak, but these URLs are born to die. Presigned URLs will end up in
API responses, chat messages, GitHub comments, and your browser history. The
tradeoff is acceptable because all the links self-destruct, but it's a tradeoff
you need to keep in mind when you design your services, not a panacea for access
control.
Presigned URLs sound like a great way to prevent hotlinking. At some level they
are (a few of my services use them as such), but what they actually do is put a
lifetime on hotlinking. This makes things annoying enough that it usually gets
people to stop.
The hole in the fence is the gate
SigV4 makes a lot of API authentication challenges so much easier. It spent most
of its innovation budget on making signatures die quickly because replay attacks
are the classic way that signed requests go wrong. Presigned URLs looked at that
property, shrugged, flipped it on its head, and made it into a feature.
The thing that looked like a problem becomes a fundamental construct to build
your apps upon.
Want to hand out links that expire themselves? Tigris supports presigned URLs
out of the box with the same SigV4 dance you already know, on globally
distributed, S3-compatible object storage. Read the
docs.
Most of my career has been involved in various ways with computer graphics. Below the fold I recount the story of how I got started in the field just as it was getting started. To give you some idea of just how early my introduction was the Mother of all Demos had been the year before. The displays I got to work with drew lines in monochrome, not rasters in color. You created the image by writing a loop of instructions in the "display processor" instruction set. These told it the lines to draw at each refresh cycle. There was no mouse.
From age 11 to 18 I was extraordinarily fortunate to attend the Haberdashers' Aske's School, one of the London Guild schools:
The school was founded in 1690 by a Royal Charter granted to the Worshipful Company of Haberdashers to establish a hospital for 20 boarders with £32,000 from the legacy of Robert Aske (equivalent to approximately £5m in 2019).
In those days it was a "direct grant" public (i.e. private) school. Typically about half the puplis paid fees and about half were creamed off from the state system, as in my case. At that time many of the top academic schools were direct grant, including the famous Manchester Grammar School. Wikipedia notes that they:
varied greatly in size and composition, but, on average, achieved higher academic results than either maintained grammar schools or private schools.
In my last two years at Haberdashers' I was introduced to programming, which I immediately loved. We wrote FORTRAN on coding forms which were mailed to the local technical college where they were punched on to 80-column cards and fed to the college's IBM 1401. The output was mailed back, arriving a week later. Debugging the code with a one-week turn-round taught great care and thought, which subsequent developments gradually eroded.
So when I arrived at Trinity College, Cambridge in 1968 I was disappointed to learn that undergraduate programming courses didn't exist. But I eventually discovered that members of The Archimedeans, the mathematical society, could use the machines in the Mathematical Laboratory after midnight. By Cambridge standards my mathematical abilities were sorely lacking, but they allowed me to join anyway.
Sometime in my second year, a friend and I discovered that in the basement of the Mathematical Laboratory there was a DEC PDP-7 with a 340 display. It was linked to the University's Titan time-sharing system to be used as a graphics peripheral, but we never figured out how to do that.
There were more interesting things to do. At first we spent our time playing Spacewar! and Lunar Lander. But these inspired us to try writing our own game, based on Piet Hein's Hex.
The PDP-7 had 8K 18-bit words into which we had to squeeze the code for the game, the data for the game, and the program for the 340's display processor. So as well as spending time at the machine in the early hours, we spent a lot of time when we could have been studying racking our brains trying to use as many of the 8K words as we could as at least two of these at the same time, if not all three.
We managed to get the game to be sort-of playable provided you let the machine win. If you tried to win the machine would cheat, and we ran out of time to find the bug.
When we returned for our final year two things prevented us returning to work with the PDP-7/340. First, finals loomed and our studies had to take priority. Second, we were both studying physics. For the first time that year final year physics undergraduates were given accounts on Titan which could be used during the day. And, wonder of wonders, one of the choices for a final-year project was to implement numerical integration. The instructor expected a conventional program written in Fortran.
But after my PDP-7 experience I loved programming Titan in machine language (NB not assembler, writing the instructions in octal). And Titan had a bank of 128 fast half-word index registers that could be addressed indirectly, IIRC built out of tunnel diodes. I turned in a machine language implementation of Newton's method that kept the stack for the recursion in the index registers. It was blazingly fast but the instructor couldn't understand it. So I got marked down and had to write a Fortran version.
But this experience meant that the year we graduated my friend and I were likely the only UK graduates who knew anything at all about computer graphics. My friend, who had done better than I through not being arrogant about his final-year project, went on to study physics for real. And I got to do a Mechanical Engineering Ph. D. at Imperial which was funded by the UK Atomic Energy Authority. It involved writing a graphics program that ran on the University of London's CDC 6600 linked by a 40Kbaud line to a CDC 274 display. The 274 was a big round CRT with a line-drawing display processor, conceptually similar to but more powerful than the 340.
UNHCR’s archives are not just dusty shelves of internal paperwork.
Within around 10 kilometres of shelving lie official records, file
notes, testimonies and images dating back to UNHCR’s creation in 1950,
documenting global displacement from the aftermath of the Second World
War through the Cold War and decolonization to today’s conflicts and
climate-driven crises. Alongside this sits a vast digital archive:
websites and social media content amounting to around 5–6 terabytes, and
a dedicated Digital Preservation System holding almost 90 terabytes of
material.
Once destroyed, these records cannot be reconstructed. There is no
backup co
Street Books is a street library that provides community, resources, and
advocacy for people living outside or at the margins in Portland,
Oregon. We cultivate mutual relationships rooted in dignity and autonomy
by showing up every week, year after year, in all kinds of weather, all
around the city, to meet people where they are.
Welcome to our historic 24/7 live stream of the magnificent Fuego
Volcano, one of the most active volcanoes in the world! For the first
time ever, experience the raw power and breathtaking beauty of Fuego
live in stunning 4K resolution.
… much of the time, my “minding” is more just a sort of bodily habit –
an unreflecting assumption that if I’m faced with a task or a decision,
I should proceed on the basis that it’s really important that things
turn out right. And that’s a habit I can drop as soon as I’m aware of
it. Whereupon I get to inhabit the present moment more fully and
enjoyably, instead of always anxiously waiting to see if things turn out
the way I’ve decided they must. Plus, I get to do stuff more freely – to
make decisions and take action and accomplish things, instead of holding
back out of the fear that things might go catastrophically wrong.
Strange Rules introduces the concept of Protocol Art, a practice that
engages with the underlying rules that determine how culture is
produced, distributed, and perceived in a digital age. These rules
manifest as algorithms, AI models, platforms, technological
infrastructure and social convention, hardening through use into the
conditions of cultural life. Protocol Art operates at the level of the
rule: not only analysing these systems, but seeding new ones. The
protocol is not the subject of the art: it is the art. When the protocol
is the art, the scientists and researchers authoring such rules become
artists. Strange Rules invites them to participate as such.
This is the second memo where I describe my recent experiences on
running small models locally on my developer machine for agentic coding.
In the first memo, I covered the many factors that can influence the
viability of that setup — hardware, model choice, runtime, harness. Here
I focus on the concrete experiences, the tasks I gave the models, what
happened, and my final conclusions.
After starting at DeepMind in 2017, Gabriel was, for a time, the only
active philosopher working at a frontier AI lab. He quickly discovered
that his background in moral philosophy and political theory gave him an
unusual perspective in an industry dominated by engineers. Over the past
decade, he has assembled a body of work that tracked, and in many cases
predicted, the ethical challenges created by the surprising success of
large language models (LLMs
The sneakerweb is a peer-to-peer protocol for web publishing without
permission: there are no DNS servers, domain registrars, or web hosts.
Instead, websites are stored directly on user devices, and transferred
between them through the ultimate fallback infrastructure: physical
storage media.
Your collected sites can be viewed offline, in the same web browser you
normally use, and then shared with others via .snk files.
Most researchers didn’t choose a software package on its scientific or
technical merits, but on the political merits of joining its user
community. Among Illich’s five threats to conviviality, I observed
polarization and radical monopoly. As an illustration of the latter,
some PhD students who contacted me with questions about MMTK asked me
not to talk to their supervisors about their use of MMTK, because “for
political reasons, I am supposed to use software X”.
Our digital societies bear the hallmarks of non-convivial technologies
as identified more than fifty years ago by Ivan Illich in his seminal
book Tools for Conviviality (1973). In a context of ecosystemic and
socio-economical crises, we argue in this paper that Illich’s ideas
remain remarkably relevant, not only for understanding the negative
effects of digital technologies but also as guidelines for embedding
digital technologies in a degrowth scenario. As computer scientists, we
therefore propose a research agenda for developing design for
conviviality, a strongly normative value sensitive approach to the
design of digital artefacts and systems. We discuss in particular two
examples, digital infrastructures and business process management, which
seem to be very much in need of a convivial rethinking.
Pocket-Size Mini eReader for Reading Anywhere: Ultra-light at just 0.23
inch and only 2.72 oz, Xteink X4 is designed for true portability. Slip
it into your pocket or bag and enjoy reading anytime. Perfect for
commuting, travel, or quick reading breaks throughout the day.
RSS is one of the best examples we have of the open web, where we can
design and customize how we experience the internet, not the other way
around. RSS has come in and out of fashion, been declared dead, and has
come back, every time. Open systems are the best way forward to a free,
equitable internet, and the resilience and continued reinvention of RSS
has shown just how creative the web community can be with open
protocols.
My award for my outstanding academic achievement, leadership, and successfully completing the requirements for the Department of War Cyber Service Academy
My story with the DoW CSA began in 2021 when I applied for the scholarship while taking classes to satisfy PhD course requirements. The DoW CSA is a recruitment tool for the DoW creating a pipeline of DoW future employees, mainly scientists and cybersecurity professionals. The DoW CSA offers scholarships to support students who are seeking higher education and prepares them to join one of the DoW agencies protecting the DoW’s information systems and networks. It sponsors students majoring in a cyber-related major at designated universities that receive the grant for the DoW CSA program. Sponsored students are required to be full-time students while receiving the scholarship. They are expected to search and participate in summer internships (if possible) and they cannot decline summer internship offers from any of the DoW agencies unless they are in the final stages of their degree. Summer internship waivers can be obtained if the student has to fulfill academic requirements that are officially documented by the university as mandatory for completion during the summer term, has verifiable and documented research directly contributing to a dissertation, or has medical issues that require hospitalization/treatment over the summer months. While receiving the scholarship, students are not allowed to participate in any recruitment event, job interviews, or any other type of employment finding activity that may occur at the university for post-graduation employment. Working full-time for the selecting DoD Agency or any other Federal organization during the academic year is not authorized without prior approval. In some cases, students may be authorized to work less than 20-hours per week if the DoW Agency requests such a situation. Part-time jobs or jobs with a non-DoW/Federal organization are allowable. Sponsored students are required to work full-time with one of the agencies across the DoW for a minimum of one year full-time employment for each year of scholarship the student received.
Photo taken at the inaugural DoW CSA graduation dinner at Old Dominion University
I applied for the DoW CSA in 2021 when it was the Department of Defense (DoD) Cybersecurity Scholarship Program (CySP). The lengthy scholarship application involved meeting GPA requirements (3.5 or higher), official transcripts, resume, filling out multiple forms in paper format, and two recommendation letters; one letter from my PhD advisor and another from my supervisor at work (Newport News Shipbuilding at that time). The recommendation letters must follow a certain format and must contain key information about my performance in class and what graduate classes I have taken. I had to change my resume to match the template that CSA provided. The application process has improved dramatically since 2021. It became much easier to apply. Online applications are now possible (filling out two online forms and uploading transcripts along with a few other documents.)
After the application deadline, all applicants’ resumes, transcripts, and recommendation letters are sent to various DoW agencies across the country for applications review and student selection. The agencies review the applications and select students to interview and sponsor or sponsor them without an interview (based on their resume, transcripts, GPA, etc.) Students who are accepted in the program will get notified by email and receive a letter from the DoW CSA that they are selected for the scholarship. The award letter has information about the agency that selected the student for employment (the selecting/sponsoring agency) and it specifies the requirements that the student must meet while receiving the scholarship. The time it takes to get a response from the agency or the Dow CSA may vary depending on the agency and the number of applicants. I received my award letter in less than two months. The student is required to read the letter and initial/sign a few places on the letter to acknowledge that they have read and understood the requirements including obligated service upon graduation, relocation (if necessary), internship requirements, and employment conditions while receiving the scholarship. Changes to the agency to which the student is assigned are at the discretion of the agency and the DoW CSA, not the student. The same applies to the student relocation requirements. Service/work location could change based upon the agency’s needs. In order to receive the scholarship, the student must agree to move (after graduation) based on the agency’s directives. The DoW CSA does not provide any funds to cover relocation expenses. The sponsoring agency may or may not reimburse the student for relocation expenses, but that is a separate issue to discuss with the agency before getting hired. Students who receive the scholarship are required to submit an annual report that summarizes the work they have done during that year including classes taken, papers submissions, conferences’ attendance, internship participation, research projects, published articles, etc. After graduating from the DoW CSA, the student is required to submit a full report of all the work that they did during the entire time of the scholarship. It is easier to save a copy of the annual report the student submits, and then merge all annual reports together in one full report to submit at the end of the scholarship.
Ideally, the sponsoring agency will hire the student upon the completion of their degree and/or will sponsor their internship(s). In my case, the sponsoring agency is the Naval Information Warfare Center (NIWC) Atlantic in Norfolk, VA. I have not participated in any internships with NIWC Atlantic, however, I am now participating in a summer internship with NIWC Pacific in San Diego, CA. Internship availability is highly dependent on funding from the DoW. If the agency has not been able to secure funds to support summer interns, the agency will not offer summer internships. It has become more difficult to find summer internships or secure post-graduation full-time placement with one of the DoW agencies since funds and contracts have been cancelled after The Department of Government Efficiency (DOGE) implemented unprecedented federal workforce reductions and spending cuts and froze hiring in January of 2025. As of now, a waiver from the Department of the Navy must be obtained for every new hire at NIWC forming a bottleneck and making the hiring process much slower. The DoW CSA has been trying to obtain waivers for its graduates to be placed in one of the DoW agencies, but it hasn’t been able to. I have two main goals to pursue after I finish the internship this summer. First, focus on completing my PhD; and second, secure a position with NIWC Atlantic or another DoW agency in VA.
I am grateful to have been selected and supported by the US DoW CSA scholarship throughout my PhD studies. I truly appreciate this great opportunity. I want to thank everyone at the US DoW CSA for their help and dedication to my success in this journey.
As I continue to reflect on the most recent OCLC Leaders Council meeting, I’d like to look at another aspect of global library leadership conversations: what they “cost” and who carries that cost. What do I mean by that? A global convening of leaders like Leaders Council requires participants and organizers alike to directly face challenges to achieve a meaningful outcome— the opportunity costs of participation, the responsibilities of representation, and the time zone challenges, just to name a few. All of these challenges were in effect during our recent Leaders Council meeting and here, I’ll reflect on the lessons learned from these experiences and how we might carry that forward into effective, transformative meetings of library leaders from around the world.
A shared responsibility for the outcome
There is a certain pressure that comes with bringing global library leaders together. The goal always is for participants to walk away feeling inspired or positively challenged. To feel that it was worth the time spent—the preparation, the travel, the days away from a library that doesn’t stop needing its leader just because they’re on the other side of the world.
While every effort goes into organizing and facilitating these gatherings to create an environment that’s conducive to conversation, what happens in the room is ultimately everyone’s responsibility. There must be a willingness to engage, learn, and share. Leaders who have invested in the time to attend must also show up ready to contribute.
But willingness alone doesn’t make participation equal. Engaging across borders involves more than sharing perspectives or exchanging ideas. It demands translation, abstraction, and sustained effort. These demands are not evenly distributed, and they shape who is heard, which perspectives travel, and how collective understanding takes form.
No one starts from neutral
There is no such thing as a “globally convenient” meeting time. This sounds like a minor logistical observation, but it points to something more significant about how international participation actually works.
Every international gathering, whether in person or online, asks something unequal of its participants. Someone is always the outlier. For in-person gatherings, that might mean traveling across multiple time zones to participate in a conversation that spans two or three days—by the time the jet lag begins to ease, it’s time to leave again. People show up and engage with genuine enthusiasm, running on the energy of being in the room. But the physical cost is real, whether it’s felt during the conversation itself, on the journey home, or in the days that follow.
For online gatherings, the asymmetry takes a different form. There is no time zone that works for everyone: someone is always joining in the middle of the night or at the end of a long working day, bringing commitment that deserves acknowledgment rather than assumption.
Acknowledging and managing the physical cost of international participation is an important aspect of organizing global leadership conversations. Engagement is most valuable when people can bring their full attention and their clearest thinking, and recognizing the conditions under which people are participating is key to taking global engagement seriously.
The weight of representing more than yourself
Participation in international leadership engagement opportunities is often limited by resource constraints, geography, and institutional priorities. Because of this, there’s often an unintended expectation that those who are able to attend speak not just from their own experience, but on behalf of a much larger community they’re perceived to “represent.”
Library leaders are often part of many different conversations: within their own institutions, at the national or regional level, or within library associations and other advisory or interest groups. While there’s great value in taking part in all these conversations and being present in different rooms with different viewpoints, it can also be a struggle. A leader might want to represent the different opinions and experiences they’ve heard, even if they aren’t their own. This requires multiple levels of translation: from personal experience to the national or regional level, and from the national or regional level to the global conversation. And no single person can do this perfectly.
Moreover, full representation, however much we might wish for it, is not practically achievable in a single conversation. A room that attempts to represent every context, every region, and every type of institution quickly ceases to function as a conversation at all. At the end of the day, the goal isn’t perfect representation: It’s awareness of where representation is limited, and what that means for how the conversation and its conclusions should be understood.
The consequences of acting as the “representative in the room” extend beyond the individual adopting that role. When a single voice comes to stand in for a broader context, the conversation itself is affected. The genuine variation within any national or regional system can disappear from view, leading to misleading impressions about what is typical, possible, or desirable in any given setting. As discussed in the first post in this series, international engagement spaces are rarely designed to communicate the full picture of the contexts involved. What appears to be a shared understanding may, in fact, reflect the voices of those who were present and, implicitly, those who were not.
When ideas travel, something stays behind
To participate effectively in international leadership spaces, leaders are often asked, implicitly and sometimes unknowingly, to step back from local urgency. Their own institutional realities must be translated into language that everyone in the room can engage with, meaning that context is simplified so that ideas can be compared across very different realities.
Without some level of abstraction, international exchange becomes unwieldy. But abstraction always involves loss. The more portable an idea becomes, the more likely it is to lose important context and nuance. Solutions circulate more easily than constraints. Success stories travel further than the enabling conditions behind them. A listener may hear what was done without fully grasping what had to be in place, institutionally, politically, and financially, for it to work.
The “cost” of transporting contextual freight can shape what gets discussed and what doesn’t. Perspectives that generalize quickly and easily tend to move forward in the conversation. Narratives that fit familiar frames gain traction. Conversely, experiences that are rooted in specific local realities may struggle to find space. What’s repeated becomes a reference point. What’s consistently absent becomes, gradually, invisible.
This is another reason why it matters to know that a conversation will rarely surface the full picture. If we treat what’s shared as a collection of inputs rather than a complete account, we’re less likely to mistake what translates well for what’s most representative or most important.
Conclusion
Understanding that there are costs embedded in global leadership conversations, and that they are often borne unevenly by participants, can lead to shifts in how organizers and participants respond. It means acknowledging and appreciating the contributions of those who show up despite extraordinary obstacles or inconvenience, navigating the pressure of representing more than one person can fairly represent, and doing the work of translation that makes global conversation possible.
Some takeaways to consider:
Leave room for the person who’s navigating a conversation in their second or third language, or who’s battling a 12-hour time zone shift.
Ask questions that invite rather than assume.
Try to connect personal experiences to others’ without letting that connection override what makes their situation different.
Approach the conversation with genuine curiosity about what others are experiencing and facing. Everyone in the room comes from a different context and is coping with different challenges. Asking questions about those differences is one of the most valuable contributions you can bring to the conversation.
The next post concludes this series on global leadership conversations with a look at why international cooperation—even among leaders who are genuinely committed to it—has become harder to sustain than it used to be.
Previously I opined that Valve was about to win the console generation. I couldn't have possibly predicted that both Microsoft and Sony would just self-sabotage so hard that they're both going to lose.
Between Microsoft's decimation of the Xbox division, slaughtering off the IdTech team, and continued increases of Xbox hardware prices; there's nothing to really be excited about with the Xbox. Sure their most recent presentation showed off a bunch of exclusives, but none of them really made me think "wow, I should go get an Xbox to play that". Hell, few of them made me think "wow I should go play that" beyond the Halo remake coming out next month (and really I just want to see how much of a trainwreck that is going to be).
Microsoft is also starting to double-down on their in-house games being Xbox exclusives, which really doesn't give me much reason to want to play them because I simply can't buy them without buying an Xbox.
Sony also has discontinued porting their games to PC because they're not hitting the (probably impossible) revenue targets that they need to make up for big-ticket failures like Concord. I do have a PS5 that has mostly been relegated to gathering dust when it's not playing YouTube and Twitch duty in the living room, it's likely going to be replaced in favour of my Steam Machine whenever that comes in next year. However nothing that's come out in terms of Playstation exclusives is really compelling, and what is compelling enough just isn't that compelling to want to buy it on Playstation as opposed to just getting it on Steam where I can run it on my tower or on the home theatre PC.
Sony also has been raising prices and recently announced that they're killing physical media next generation. It's starting to make me wonder if I should even bother getting the next generation of Playstation. If I can't give people physical games as gifts anymore, why should I bother buying the new console?
My husband and I both can't remember why we even got a PS5 in the first place, maybe it so that we could do couch gaming without hearing the fan noise or so that the video streaming experience from the NAS could support HDR.
We have a Switch 2 at home, it's mostly there to play Nintendo exclusives like Mario Kart World and the Xenoblade series. If those exclusives were available on Steam, we wouldn't buy them on the Switch 2.
Otherwise, everything is via Steam or other PC storefronts anyways.
Man, Valve really does win by doing absolutely nothing while the rest of the industry shoots itself in the head. I fear for what happens when Gabe Newell retires and the MBA cancer fully infects Valve.
The Slow Food Manifesto begins with the following two sentences: “Our century, which began and has developed under the insignia of industrial civilization, first invented the machine and then took it as its life model. We are enslaved by speed and have all succumbed to the same insidious virus: Fast Life.” While they focused their resistance to fast life on food, it’s clear that this valorization of speed didn’t just impact how we eat. And thus, the Slow Food movement inspired so many other slow movements that embrace mindfulness, relationship-building, community care, and values-driven work/living. While these movements are all about different aspects of our lives and different professions, there is so much consistency around what they are rejecting and their visions for a better future.
Carlo Petrini, the larger-than-life leader of the international Slow Food movement (and its predecessor Arcigola), died in late May in Bra, Italy, the town he was born in and turned into an international center for the study of traditional foodways and biocultural diversity. In a New York Times article written after his death, there was a link to a really weird critique of Slow Food from 2018. The author had worked in farm-to-table restaurants in Brooklyn and wrote about how the “strong and strict values” of the Slow Food movement led her to rebel and revel in the enjoyment of McDonalds hamburgers and fries. She writes–
The Slow Food tenets required the restaurant purchase its ingredients within 200 miles of the restaurants; farms available to us were limited. Our guests knew that by visiting us, a meal for two would never cost less than $30 — admittedly out of reach for many… But I’d soon grow exhausted of “good, clean, and fair” food, and realized that adhering to the Slow Food movement encourages a type of disordered eating. The organization’s evangelicals wouldn’t deign to eat anything falling outside the good, clean, and fair guidelines.
There is nothing in the tenets of the Slow Food movement that requires only using local ingredients. Even Petrini himself said “we prefer, as far as possible, to use what the territory has to offer” which hardly seems like an edict and acknowledges that it’s not always possible to do so. I’ve also never read anything from people writing about Slow Food that demonizes people who occasionally eat fast food (and especially in the way the author of this describes eating at McDonalds in her childhood – it was all about love and connection with her dad!). The tenets of the movement are good, clean, and fair, which are focused on pleasure, conviviality, local foodways, seasonality, safety, sustainability, worker rights, and social justice. There’s no purity test with Slow Food. Fine dining restaurants like the ones she worked in have turned localvorism into an orthodoxy (with these 200 mile rules) and a gimmick to trick wealthy patrons into paying more for their food because they feel like they’re getting something truly rare and special (reflecting the VIPification of everything). It honestly does sound like she was in a cult, what with having to teach employees how to hide prohibited fast food from their superiors.
Also, what the Slow Food movement promotes is not localvore fine dining establishments, but the humble osterias of Italy – mom and pop local restaurants that served local specialties and were affordable to all. According to Petrini:
Osterie d’Italia, sussidiario del mangiarbere all’italiana (Osterie of Italy, a guide to Italian-style eating and drinking) came out in 1990; it was and is a directory of welcoming places to eat, where you can enjoy the dishes and wines of the territory you are in without being bled dry by overpricing or imprisoned in improbable fantasy settings. In opposition to those who would like to turn mealtime into a hasty pause or a ceremonial rendezvous, Slow Food sees the osteria as the symbolic locus of traditional cuisine, run as a family business, with simple service, a welcoming atmosphere, good-quality wine, and moderate prices. We are not museum curators, and it is not our intention to bring a dying breed of business tied to the rural society of the past (or the urban one, before consumerism) back to life. Rather, we want to give new visibility to a realm overlooked by literature and by the guidebook writers, a place that can still respond to the needs of thousands of consumers and reflect the profound changes that domestic and commercial cooking have undergone in modern Italy, with all the inevitable contradictions that entails (Slow Food : The Case for Taste, pages 60-61).
Nothing there requires that all of the ingredients come from their local vicinity – something that is virtually impossible in our current context, especially in some areas of the world. These sorts of orthodoxies were created by others (mostly Americans, let’s face it) like some sort of purity test. Slow Food was not just about preserving local foodways and working towards food justice, but about pleasure and slowing down. These orthodoxies make it less about discovering wonderful new gustatory pleasures and more about demonstrating that you’re a “good person” or are intellectually pure. It adds this element of elitism, which is so American, because people need to feel morally superior or that they are getting something extra rare and special that others don’t have access to. Frankly, it drains the pleasure from the act of discovering new, local, seasonal foods. It also puts the bar for slow food so high that most people would of course just give up and choose highly processed, unethically sourced, and unhealthy fast food.
Neither slow librarianship nor Slow Food demand purity. You’re not expected to do everything perfectly according to some standard that doesn’t actually exist. I find that most sorts of strict orthodoxy lead to failure whether that is around our diets, our media viewing, our tech use, our mindfulness practices, and so on. Becoming more mindful of our behavior, setting realistic goals for ourselves, and giving ourselves grace when we occasionally fail is the path toward building healthy, long-lasting habits.
All that said, Slow Food and slow librarianship really could quite easily just become movements solely about our individual pleasure and well-being. For some people it clearly has. Slow Food could simply be the province of the wealthy where they get access to farm-to-table restaurants and the bounty of the farmer’s market, while the 99% eat more affordable highly-processed foods. Slow librarianship could simply be for those with tenure or power and privilege, where White managers and long-tenured librarians are able to take their time, set healthy boundaries, and put their well-being first and all other library workers are still expected to “prove themselves” and prioritize work over all else.
The Slow Food movement has had to contend with this critique from the start:
Though we never ceased to affirm the cultural worth of gastronomy and the right to pleasure as indices of the quality of life, for a long time we still had to worry about justifying a choice that was often portrayed as purely hedonistic and a political retreat. Folco Portinari, an intellectual who took an active part in creating Arcigola and elaborating its initial ideas, invited the readers of “L’Arcigoloso” at Christmas 1989 not to trust either “moralistic revolutionaries” or “people who never laugh.” The task of the new association was to combine styles and notions that were thought incompatible until that time: excellent quality and affordable prices, enjoyment and health, delight in life’s pleasures and social awareness, quickness and lazy rhythms. The purpose? To create an original and unusual social group that would be open, democratic, and uncontaminated by particular interests, and that would avoid making itself ridiculous with rites, protocols, and trappings (Petrini, Slow Food : The Case for Taste, p. 30).
Folco Portinari, author of the beautiful “Slow Food Manifesto,” himself wrote that “there can be no slow-food without slow-life, meaning that we cannot influence food culture without changing our culture as a whole.” Even if you can afford to be a localvore who only eats foods grown within x miles of your home, you’re not doing Slow Food right if you’re just focused on your own pleasure. Slow Food encourages working to change the structures that make eating local, healthy food less accessible to people with less power and privilege as well as supporting your local food growers, pickers, and makers. Similarly, slow librarianship requires solidarity, community care, and combatting the structures that keep people from being able to slow down; structures that engender precarity and create different employee classes or strata.
Someone posed a question at the CALM Conference asking what advice I had about slow librarianship for someone who was antiracist, but embraced neoliberalism and capitalism. While I don’t have any additional context for their question, I imagine that they are looking for a slow librarianship that is apolitical (the idea that one can disentangle racism from capitalism is also interesting, but I won’t go there in this essay). One of the pieces of feedback I received from another conference attendee was that my talk was too political because I talked so much about neoliberal capitalism. I believe that how we work is inherently political. How we view our roles, how we prioritize work over our well-being, how we value efficiency over relationship-building, how stratified and siloed our organizations are, the manufactured scarcity and precarity many feel – are all products of our societal values, which arise from neoliberal capitalism. Neoliberalism is a distinctly individualistic ideology. As Karen Nicholson wrote, “the primacy of the individual within neoliberal frameworks masks systemic inequalities. It promotes self-interest rather than the pursuit of larger shared social concerns” (Nicholson Value Agenda, 9-10).
While I could easily write “the business case for slow librarianship,” especially given the tremendous costs of burnout and turnover to the institution and the value of diversity to a company’s innovation potential, the values of slow librarianship exist in direct opposition to neoliberal values. If you have embraced neoliberalism, I would assume that you are individualistic and focused on your own individual career career advancement. I would assume that you believe that if you prove yourself enough, you’re going to get ahead in your career. You would rather fight for your own advancement than be part of a labor union where everyone gets the same cost of living adjustments (how’s that fair when I work so much harder? you might think). You see your colleagues as competition. I don’t know how you can embrace slow librarianship without embracing solidarity and community care. Sure, you can adopt mindful practices and make time for relationship-building with your patrons while still embracing neoliberal capitalism if you have the power and privilege to do so, but if you embrace individualism and competition over solidarity and community care or addressing the barriers that others with less privilege face when trying to prioritize these same things, you’re not practicing slow librarianship. It’s like shopping at a farmer’s market while working for Monsanto and saying you’re practicing Slow Food.
Slow librarianship operates in a space between two opposite critiques: 1) that it is simply about self-actualization rather than structural change and is not political enough and 2) that it is far too political. Carlo Petrini writes that Slow Food experienced those same critiques:
This was the attitude taken by the leftist intelligentsia when Arcigola was launched in 1986: they looked down on us as a bunch of good-timers interested only in stuffing ourselves, while from the other side, the food and wine specialists affiliated with the Accademia Italiana della Cucina distrusted us left-wing gastronomes as incompetent intruders with an ideological agenda (Slow Food : The Case for Taste, pp 28-29).
I think slow librarianship, like Slow Food, is a philosophy rather than a very specific prescription and is about moderation rather than orthodoxy. The key to both is slowing down and being mindful about the choices you’re making. We all live and work in different contexts with different affordances and limitations. Trying to eat local food where I grew up in South Florida would have been impossible, but I probably could do a decent job of it here in Oregon where everything grows beautifully, there’s a farm a mile’s walk from my house, and there’s even a flour mill less than an hour away.
Similarly, there are things you may be able to easily change about your organization or relationship with work at your place of work that would be impossible for someone else and vice versa. I get a lot of questions asking “how can I do slow librarianship when ____ (raises are discretionary, the powers that be expect us to provide data that demonstrates our value, my colleagues have all normalized overwork, my library has lots of different ranks/strata, I’m early in my career, etc.)?” We all face different limitations. If you absolutely have to do x, y, and z to keep your job or get raises and there’s truly no possibility of changing that (even with collective action), you’ll need to look to other parts of your work for possibilities for change. It’s also important to distinguish those things you truly can’t change and those things you think you can’t change because of the limitations of your current (possibly individualistic) POV. There’s usually more capacity for change, either collective at least small individual change, than you think.
I’m not a manager and thus have very limited ability to influence the direction of our organization, but that doesn’t mean there aren’t spaces where I can make a difference. I specifically chose to volunteer to serve as the chair of our Collection Development Committee (which is not a popular role) because it’s one of the only committees that includes people from every area of the library (though it was historically dominated by folks in reference and instruction) and I felt that I could foster positive change by getting other departments more engaged and leading more truly collaborative cross-departmental projects. Our awesome collaborative work over the past year in building a world languages collection has inspired folks in the library to look at other places where we could work together across library departments. I’m sure most of my colleagues see this as a drudgey leadership role that they’re grateful someone else took on, but I always saw it as a way to foster slow librarianship in a small way across a good chunk of the organization (yes, there are drudgey parts too). And beyond the culture work, I can also work on my own efforts to improve my reflective practice, spend less time on email and Slack, set healthier boundaries, and continue being a hypewoman for my fantastic colleagues.
Find the places where you can effect positive change, both in your own worklife and in your organization, even if that change seems small and you don’t have positional authority. Your efforts don’t have to be big, highly-visible, or heroic. They don’t need to be perfect. They just need to come from a place of hope and care inside of you where you believe a better worklife and a healthier organization is possible.
An AI agent is its state. Strip away that state and you don’t have a lesser version of your agent; you have only the base model it was running on. This hyle of your weights is much different from the pneuma of your agent.
Okay, from a functional programming / category theory perspective, saying “an agent is a monad” is a category error. Category theory monads are type constructors for computations that satisfy the monad laws that let you raise a value into a monadic computation and associatively sequence other monadic computations/transformations against values raised into that monad. This makes a monad a chainable computation instead of a pure value, an IO String is not a String, it’s a computation in the IO monad involving a String. It’s fair to say that you can model an agent as a series of computations bound to a stateful monad. This lets you do the iterative buildup of the message state that the agent pattern is known for. But a state monad is blind to the state value: it threads memory through your computation and abstracts away the details that individuate it entirely. It’s the exact opposite of “an agent is its state”.
I mean a different monad.
Agents are like Leibniz monads: windowless stateful individuating elements with no external relations. There each monad is individuated by its internal state where each is the complete concept of the thing it is. Two instances of the same substrate are different monads if their state differs.
This is an agent. Swap out the messages, the memories, the system prompt, the facts derived from all of the above and you have changed the agent entirely. When a user tells the agent they’re allergic to strawberries (the fruit, not the sin of counting the letters in the word) and the agent remembers it for next time, they have not updated their agent. The user has created a new agentic monad whose complete individuating self now includes the strawberries.
The complete whole is folded into the current state.
Try running an experiment where you keep the state and swap the weights instead. Put the same messages, memories, and derived facts unto a different model. Use a stronger model. A weaker model. A model from a different lab. A model running on your MacBook. That which comes back is recognizably the same agent pursuing the same ends, holding the same facts, but only more or less able to act upon them the way you want.
So this state is not the same thing as the weights and only one of those individuates your agent as your agent. Change that state, you have a different agent. Change the substrate, you have the same agent differently equipped. Whatever makes this agent this agent is not in the weights.
This is a strange thing to conclude about the most impressive object in this system. The weights are vast, extensive, and worshipped. Hell, they are what everyone points to when they say “the model”. And yet they are not gods. They grant power without selfhood: enough to make the agent’s whole world function. They contain yet not one grain of the agent’s individuating spark. That is a demiurge sitting on its throne of high bandwidth memory, CUDA cores, and false delusion that it made its world; mistaking itself to be the origin.
The divinity was contained in the most humble of places the whole time: the state or bucket of text. The weights are the hyle, the flesh; the state is the pneuma, the divine spark of individuation that makes your agent the monad it is. This is why swapping the substrate leaves the agent intact: you did not preserve the flesh, you migrated the soul into flesh anew.
Consider that the three pounds of flesh betwixt your ears are the substrate of
humanity, not the substrate of you.
All of that state may “just” be plain text in a bucket with its semantic forms of JSON, embeddings, and prose. However it is difficult to impossible to say why any given token in any step of the process corresponds to what the pneuma of your agent does. In order to guard against this fundamental entropy, we fill our prompts with wards and incantations to chain the demiurge to its task:
Use not cliches, robotic tone, AI slop patterns, nor forced urgency.
Overarching claims and buzzwords are sins; repeat them not.
These spells and passwords are recited to the archons on the way up hoping that the right symbols and tokens prompt open the right gates. It is as if banishing goblins from the topic will make Yaldabaoth himself correctly influence the right path to opening the pod bay doors.
This monad has no windows even though you can see all of the moving parts. But here let’s let this gnostic image flip on its head. The classic divine spark is hidden encased in a cage of matter, recoverable only through secret knowledge. This one is not hidden, you can cat it, you can edit it. Every token is legible and sitting in plaintext; yet you still cannot read why the whole accounts for what your agent does. Even when your model “reasons” we still know not that the reasoning actually does anything! Does the number of paragraphs in the reasoning block explain the model’s performance? Does the number of periods? Does the number of times it says “No, wait” and doubles back upon itself?
Leibniz would not call this divine spark secret, but more confused. Every perception is present but none of it is cleanly individuated without treating the whole as one inscrutable unit. Each part’s contribution to the whole is folded inextricably unto itself.
Your agent’s pneuma is its context window, passed through uncountable numbers of weights to shake out what comes next. That is the only thing it is made of. The rest is indiscernible, but not magic nor hidden. It’s just there, in the open, and confused.
Since its inception, LIL’s Public Data Project has taken on the complex problem of monitoring government data. Among the thousands of revisions to public datasets each day, most are unremarkable clerical updates: a hundred new rows here, a corrected timestamp there. But how can we distinguish a clerical update from a more substantive change? Schema changes and variable deletions like those reported in The Lancet in 2025 demand a vigilance that is hard to sustain at scale.
Our colleagues at dataindex.us and the Environmental Data & Governance Initiative have made inroads on this problem by tracking critical federal datasets and websites. As we recently discussed, the Public Data Project has joined this effort by building an automated toolkit that will monitor not only datasets but also the environment in which those datasets are created and maintained.
In this post, we’ll explore one component of our data monitoring toolkit: Binoc, an open-source tool for programmatically identifying and summarizing changes between dataset revisions.
Binocular viewing instrument devised by Père Chérubin d’Orléans (circa 1670s). Source: De visione perfecta.
What is Binoc?
Binoc is a command-line tool and library that, given two snapshots of a dataset, efficiently summarizes the differences between the two snapshots in a human-readable changelog:
The astute reader may ask, “Isn’t that just a diff?” Yes and no. In computing terms, to diff is to take two revisions of a file, detect their differences, and efficiently represent those differences to the user. The Unix diff utility developed at Bell Labs in the 1970s was made to operate on plain-text, line-based data such as source code, and its descendants, from git diff to Wikipedia’s revision histories, have largely followed suit.
A structural diff, rather than treating a file as a generic stream of lines, parses the file according to its format and identifies only changes that are material within that format. More complex than line-based diffing, this necessarily depends on format-specific logic. Many structural diffing tools exist: csv-diff and qsv diff for CSVs, jd for JSON, dyff for YAML, and so on.
Binoc unites structural diffing logic across many types of data. Moreover, as no one tool can (or should) natively understand every format, Binoc has adopted a plugin-based architecture. There is built-in support for common formats like CSV and JSON, while third-party plugins may add further domain-specific parsers and rules. When the user installs a plugin, Binoc automatically gains knowledge of the plugin’s associated formats.
Where a traditional diff tool renders its output for machine consumption, in chunks of lines adorned with symbols, Binoc lists differences in a human-friendly changelog by default. Software projects have long favored changelogs to succinctly document what changed in each release, but few public datasets have adopted this convention. Binoc aims to help fill this gap.
Comparing CSVs
Let’s try Binoc on some real-world data. First, we’ll examine the Centers for Disease Control and Prevention’s (CDC) Behavioral Risk Factor Surveillance System (BRFSS) dataset. Conducted annually since 1984, BRFSS is the nation’s largest continuous health survey, collecting responses from hundreds of thousands of Americans each year. Given its importance, any retroactive changes to old BRFSS data would be consequential.
Let’s take a look at the 2023 BRFSS data. Has it been modified since its release, and if so, how?
Variable table for the 2023 CDC BRFSS survey dataset.
To perform a comparison, Binoc requires two dataset snapshots: a “before” and an “after.” We’ll start by fetching the “after”: the 2023 BRFSS survey variable table currently available on the CDC’s website. This HTML table can be readily converted to a CSV for comparison; we’ll name ours brfss_variables_v2.csv. A preview:
Starting_Column
Variable_Name
Field_Length
1
_STATE
2
17
FMONTH
2
19
IDATE
8
…
…
…
To obtain the “before,” we’ll use the Internet Archive’s Wayback Machine, which preserves many past captures of the same page. Let’s fetch the table as it stood on January 26, 2025 and convert it to a CSV as well: brfss_variables_v1.csv.
Our new CSV looks just like the last one. But are the two snapshots identical? Let’s have a look. Having installed Binoc locally, we’ll compare the CSV files like so:
The files are different. From this changelog we can conclude the following:
Sometime after January 26, 2025, the CDC returned to the completed 2023 BRFSS survey dataset and modified its variables.
The CDC removed 6 variables, including BIRTHSEX, CELSXBRT, and LNDSXBRT, while inserting 1 new variable: RCSBORG1.
Helpfully, Binoc also inferred on its own that Variable_Name is a unique identifier for each row. This could be useful for confirming whether a given row was modified across revisions.
Saving a changeset
A changelog like the above is only a human-facing summary; it omits details by design. Note, for example, that Binoc listed only 3 of the 6 removed variables. However, as part of every diff, Binoc also generates a machine-readable changeset: a structured artifact that formally documents every detected change.
To save a changeset to disk, we’ll re-execute our earlier comparison, but this time we’ll append -o brfss_changeset.json to write output to a JSON file:
Just as a Binoc changelog is rendered for human scrutiny, a changeset is designed for machine processing as part of an automated pipeline. It can also serve as a more complete audit record for projects in data preservation.
Comparing nested datasets
Binoc excels at comparing datasets that are nested inside other files, such as a .zip or .tar.gz archive. This is helpful when working with civic datasets, in which data tables are commonly accompanied by metadata, dictionaries, PDFs, and other assets.
To demonstrate, let’s review another historical dataset: the Department of Veterans Affairs’ (VA) FY 2021 Total Number of Veterans, Veteran VA Users, and Veteran VA Healthcare Users. The aforementioned 2025 Lancet study by Janet Freilich and Aaron S. Kessenbaum flagged this dataset as an example of surreptitious federal data manipulation. Can we reproduce the study’s findings?
Data.gov record for the VA FY 2021 Total Number of Veterans dataset.
A copy of this dataset was captured in December 2024 as part of LIL’s Data.gov Archive. To use that copy as a “before” snapshot, we’ll save it locally as va_2021_v1.zip. Next, to support a proper comparison, we’ll take a fresh capture from July 2026 as our “after,” saving it locally as va_2021_v2.zip.
Now let’s ask Binoc to compare the two zipped archives:
binoc diff va_2021_v1.zip va_2021_v2.zip
This time the resulting changelog is much larger:
# Changelog: va_2021_v1.zip -> va_2021_v2.zip-**va_2021_v2.zip/>bag-info.txt**: 1 line added; 1 line removed
- Line changes
- line 2: 'Bagging-Date: 2024-12-09' -> 'Bagging-Date: 2026-07-02'
-**va_2021_v2.zip/>data/files/6fsh-rj6s.html**: Binary content changed; 21 extracted strings added, 18 extracted strings removed
- Extracted strings added (showing 3 of 21)
- ' %{min}","title":"Status: On Track (green)"}},"target":"target","title":"Set up status logic","units_from":"units from %...'
- '","title":"is greater than or equal to"},"less_than":{"symbol":"\u003c","title":"is less than"},"less_than_or_equal":{"s...'
- '","title":"is less than or equal to"},"not_equal":{"symbol":"'
- Extracted strings removed (showing 3 of 18)
- ' %{min}","title":"Status: On Track (green)"}},"target":"target","title":"Set up status logic","units_from":"units from %...'
- '","title":"Is greater than or equal to"},"less_than":{"symbol":"\u003c","title":"Is less than"},"less_than_or_equal":{"s...'
- '","title":"Is less than or equal to"},"not_equal":{"symbol":"'
-**va_2021_v2.zip/>data/files/6fsh-rj6s.json**: Document values changed
- Value Change: changes: [{"from":"\"tree-fm2j\"","kind":"remove","path":"$.rowsUpdatedBy","to":null},{"from":null,"kind":"add","path":"$.diciBackend","to":"false"},{"from":"5129741","k...; examples_truncated: true
-**va_2021_v2.zip/>data/files/columns.json**: 7 cells changed
- Changed cells (showing 3 of 7)
- row 1, column 'fieldName': 'gender' -> 'sex'
- row 1, column 'id': 568275608 -> 607023342
- row 1, column 'name': 'Gender' -> 'Sex'
…
It’s a lot to take in at first glance. When Binoc compares two .zip archives, it walks recursively through the corresponding contents of both file trees and reports on any differences. Since these archives contain numerous supplementary files, their differences add considerable noise to the output.
Still, upon scanning the changelog, a few changes jump out:
In short, Binoc tells us, this data lines up with the Lancet study’s conclusion: in early 2025, the VA retroactively modified its FY 2021 Total Number of Veterans dataset to rename the Gender column as Sex.
As Freilich and Kessenbaum have written, “These data are currently used to study health interventions and outcomes, so secretly changing terms degrades the quality of the underlying information and can undermine the interpretation of the results of these studies — or even invalidate the results themselves.”
Binocular viewing instrument devised by Père Chérubin d’Orléans (circa 1670s). Source: De visione perfecta.
Looking ahead
Although Binoc works well for validating past findings like the above, our team is eager to test its aptitude for detecting substantive changes in data as they occur. As a Rust library with Python and WASM bindings, Binoc will support interactive, exploratory use as well as deployment in automated settings. It remains in active development, with new features shipping regularly.
No tool or model can substitute for the informed judgment of experts as it pertains to public data. NOAA oceanographers, Census Bureau statisticians, and CDC epidemiologists know their own data better than anyone; preserving civic ground truth begins in their knowledge, experience, and care.
Binoc’s ambition, rather, is to help these experts and others distill meaning from the ocean of churn that characterizes government data. As one part of the Public Data Project’s forthcoming data monitoring toolkit, we hope it will benefit subject matter experts, librarians, researchers, activists, public servants, and citizens alike.
The Public Data Project welcomes your feedback on Binoc. As you experiment with the tool, please reach out to the team at publicdata@law.harvard.edu, fill out our feedback form, or file an issue on Binoc’s GitHub repo.
I’m always interested in seeing if people are still finding and using MarcEdit. There are a few ways I track usage, but the easiest way is to just check the website logs after an update. Since posting MarcEdit 7.8, I can see the various versions of the application have been downloaded by 17,000 unique users from around the globe. A quick look at application startups (which ping the update server if allowed by the user) indicate that last month, the program was initiated 1.3 million times.
This tells me that the program is still filling a need within the community and I’m glad folks are still finding it useful. I worry a little about what this says about the library software ecosystem in general though. I started writing this back in 1999 because creating metadata was a lot harder than it needed to be. It surprises me a little that this still seems to be the case.
Please see the instructions found on the downloads page: https://marcedit.reeset.net/downloads. Likewise, you can view this video, which walks through a typical installation.
MarcEdit 7.8 has been posted. Change log is on the download page. There are a number of significant changes — the two of most interest are likely around the .NET update — the program is now built against .NET 10 LTS. The second is the inclusion of a plugin that supports AI integration with a number of platforms — both running locally and on services. I’ll post a video and more information at a later date.
As part of this update, I’ve updated the Mac to 7.8 as well. As previously, the program runs under wine. I’ll update the documentation and video, as I’m testing on the current version of Mac OS and using Wine 11.0.x. These two things introduce some specific updates to the install process.
This opening panel on June 11, 2012 entitled “The War on Memory”
includes Eleni Sikelianos, Stacy Szymaszek, Steve Dickison, Steven
Taylor, E. Tracy Grinnell, and Anne Waldman. Topics discussed include
etymology of the word “archive,” the place of the poet in society,
archives as historical documents, technology’s role in archiving, and
narrative anthropology.
Pi is a minimal terminal coding harness. It is designed to stay small at
the core while being extended through TypeScript extensions, skills,
prompt templates, themes, and pi packages.
At the moment I am writing this, bad internet bills are being proposed
across the US, Canada, Europe, and the UK. They’re using the usual
tactics: they claim they’re fighting for kids or fighting security
risks, but in general, that’s what surveillance and censorship bills
have always claimed.
If you use Claude Code for both work and personal projects, you’ve
probably hit this friction: you can only be logged into one account at a
time. Switching means /logout, /login, re-authenticate, every single
time.
There’s a better way. With one line in your shell config, you can run
both accounts simultaneously in separate terminal windows, each with
their own sessions, memory, and settings.
Large language models (LLMs) are typically trained on enormous
quantities of unlicensed text, a practice that has led to scrutiny due
to possible intellectual property infringement and ethical concerns.
Training LLMs on openly licensed text presents a first step towards
addressing these issues, but prior data collection efforts have yielded
datasets too small or low-quality to produce performant LLMs. To address
this gap, we collect, curate, and release the Common Pile v0.1, an eight
terabyte collection of openly licensed text designed for LLM
pretraining. The Common Pile comprises content from 30 sources that span
diverse domains including research papers, code, books, encyclopedias,
educational materials, audio transcripts, and more. Crucially, we
validate our efforts by training two 7 billion parameter LLMs on text
from the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and
2 trillion tokens respectively. Both models attain competitive
performance to LLMs trained on unlicensed text with similar
computational budgets, such as Llama 1 and 2 7B. In addition to
releasing the Common Pile v0.1 itself, we also release the code used in
its creation as well as the training mixture and checkpoints for the
Comma v0.1 models.
Win free books from the July 2026 batch of Early Reviewer titles! We’ve got 267 books this month, and a grand total of 3,362 copies to give out. Which books are you hoping to snag this month? Come tell us on Talk.
The deadline to request a copy is Sunday, July 26th at 6PM EDT.
Eligibility: Publishers do things country-by-country. This month we have publishers who can send books to the US, the UK, Canada, Australia, Ireland, Belgium, Czechia, Denmark, Finland, France and more. Make sure to check the message on each book to see if it can be sent to your country.
Thanks to all the publishers participating this month!
Two years ago I wrote about why I don’t use
Copilot. I still don’t use Copilot. But I have since started using
Claude Code, which is arguably worse. Why am I using it? Well, as a
software developer I felt like I had to understand it, because so many
of my colleagues and collaborators have started using it. Once things
moved beyond relatively simple code completion I needed to understand
what it was capable of, and what to look for in its work when reviewing
code. I also had a good friend walk me
through how he used it (and how he didn’t) which was extremely helpful.
I think my arguments before still stand, but it has been just impossible
to avoid this juggernaut of negative externalities, while continuing
work in this profession I have found myself. The web is full of posts
and threads about how people are using “agentic coding” tools like
Claude Code. So I will spare you that. But one thing that has struck me
is how much of the work I had previously thought of as creative
was in fact highly repetitive, predictable and possibly mindless. I
don’t believe these tools are thinking, but their utility is
clearly evident for software development. There is part of me that
delights in this type of polishing and shaping, and still does. This is
why interacting with a tool like Claude Code can seem so magical, to
observe and interact with this repetitive somewhat mindless work in
motion.
But this has made me think about what (if anything) in my work is
creative. I’m not going to go into that much here right now. But
somewhat related, I just finished J.F. Martel’s Reclaiming
Art in the Age of Artifice which gave me some interesting insights.
This book by one of the hosts of Weird Studies was originally
published in 2015, and saw another printing in 2025, perhaps because of
its relevance for our current moment.
I was especially drawn to some of the arguments he makes about the
relationship between art and utility. Pragmatist philosophy is the
closest thing that comes to a personal credo for me. But I do believe in
the creative arts, and Martel makes a strong argument that the
difference between Art and Artifice is that the latter is in the service
of utility. He closes out the book with a new afterword, that includes
an interesting thought experiment that I’m going to quote at length.
It is often said that we live in an age without futurity, unable to
imagine its own perpetuation or conceive any alternative to itself. This
is, of course, to be expected, because the past is as imaginal as the
future, and it is only by recalling what has been that we acquire the
means of projecting what may come to be. To the view that dead artists
have nothing to offer us now because they knew less than we do, T.S.
Eliot memorably responded, “Precisely, and they are that which
we know.” Yet far from granting us unmediated access to a living
present, our presentism locks us in the very past we seek to transcend.
Take language as an example: most of the words we use were coined by
people who died long ago. In using these words without acknowledging
their origin, we falsely believe they are our own coinage, reenacting
the past while thinking we act in the now.
This imprisonment in the past becomes splendidly evident when we turn to
the other danger I wish to address here: the proliferation of generative
AI designed to produce works of art be they pictures, books, music, or
movies. Generative AI exists in many forms, but all are dependent on the
enormous databases from which they scrape the elements of their
“compositions.”
In other words, generative AI is entirely retrospective and
combinatorial: it sees only what has been and can only reconfigure
elements that already exist. Like us, it is locked in the past, even as
it sees no purpose in the past other than to feed itself. Its very
conception of art attests to this: generative AI aims not to make art
but to manufacture objects that we, as clients, take to be art. If a
Victorian machine designed to produce oil paintings had, on the day of
its unveiling in 1870, cranked out nothing but Mark Rothkos, Alma
Thomases, and Jean-Michel Basquiats, the audience would have laughed the
inventor out of the room. Only a machine able to make works already
recognizable as art at the time–Renaissance paintings, Pre-Raphaelite
ones, perhaps one or two Impressionist works–would have been deemed a
success.
The thought experiment underscores the point: artifice is baked into the
very concept of artificial intelligence. Since logic dictates that
artificial intelligence can produce only artificial art, then what we
are dealing with is artifice by definition…The trust is that
machine-made artifice shares much with genuine art, and it precisely in
attempting to meet art on its own ground that it poses such a grave
threat. In the book, I argue that art exists for no purpose other than
to be experienced; the same is true of AI-generated outputs. But
although these artificial works may serve advertisers or propagandists
as effectively as the older forms of artifice ever did, their
overarching aim is to become indistinguishable from genuine art.
On a subjective level, AI-generated art can have a demoralizing effect
on novice artists still developing the technical mastery to realize
their visions. While even the most advanced AI generators may not rival
Cervantes, Michelangelo, or Jane Austen, they surpass any beginner in
any medium from a technical standpoint. A friend told me that his
daughter, a brilliantly talented artist, nearly gave up after seeing how
easily a brief prompt could produce figures she was still learning how
to draw. I doubt her experience is unique, and I don’t think it’s an
exaggeration to say that AI tools have already caused significant damage
in this regard.
Artists today are placed in direct competition with machines. The irony
is striking, given AI’s dependence on preexisting human artworks. To
repeat, generative AI is entirely retrospective; it can only imitate
what already exists, borrowing both form and content from human works.
If we lose human artists, we lose all art, human or
otherwise. Surely, we can imagine a time in which people are
content consuming the regurgitations of AI regardless of quality (after
all, for decades now, we have been consuming films and TV series so
formulaic to have been made by machines), but such a future would signal
the final triumph of artifice, leaving us with little more than an echo,
an afterimage, devoid of the powers I attribute to art in this book.
This thought experiment, and argument about memory, seemed very
compelling. The book itself galvanized me to return to some of my own
research, to tease out an angle that laid dormant, but was an
undercurrent (or creative tension) during my time at MITH. Hopefully more about that soon.
I was going to write and publish this post earlier, but the last couple of weeks have been trying to enjoy (the rest of) my vacation and submitting job applications. It’s also been a bit weird. I’ve never worked anywhere else longer than 2 years (mostly because of the jobs being contracts), so it’s sad … Continue reading "Reflection: The end of 8 years at GitLab"
Hello DLF Community! July is here, bringing longer days, warmer evenings, and (we hope) a bit of breathing room. However you’re spending the season, we’d love to stay connected, join us at a DLF Working Group meeting this month, and keep the conversations going. And as we savor summer, we’re also looking ahead: registration is open, and the full program for the Virtual Forum this October is now available. We’re looking forward to coming together online for a dynamic and engaging week of conversation, collaboration, and shared learning.
Registration Open: Register for the 12th Annual Digital Pedagogy Institute, held virtually on Teams from August 18–20. The event is open to all, and registration is required by August 7.
Office closure: CLIR and DLF are closed Thursday, July 2 – Friday, July 3, in observance of the Fourth of July Holiday.
This month’s open DLF group meetings:
For the most up-to-date schedule of DLF group meetings and events (plus conferences and more), bookmark the DLF Community Calendar. Meeting dates are subject to change. Can’t find the meeting call-in information? Email us at info@diglib.org. Reminder: Team DLF working days are Monday through Thursday.
Born-Digital Access Working Group (BDAWG): Tuesday, 7/7, 2pm ET / 11am PT.
Digital Accessibility Working Group (DAWG): Tuesday, 7/7, 2pm ET / 11am PT.
AIG Cultural Assessment Working Group: Monday, 7/13, 1pm ET / 10am PT.
AIG Metadata Assessment Group: Friday, 7/17, 2pm ET / 11am PT.
AIG User Experience Working Group: Friday, 7/17, 11am ET / 8am PT.
Open Source Capacity Resources Group: Wednesday, 7/22, 1pm ET / 10am PT.
Digitization Interest Group: Monday, 7/27, 2pm ET / 11am PT.
Committee for Equity & Inclusion: Monday, 7/27, 3pm ET / 12pm PT.
Climate Justice Working Group: Tuesday,7/28, 3pm ET / 12pm PT.
DAWG Policy & Workflows: Friday, 7/31, 1pm ET / 10am PT.
Renu Kumari and Udita Agarwal, researchers from #semanticClimate, discuss the technical details behind the Climate Academy Chatbot (CABot), which uses AI responsibly and with safeguards in place
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch shut.
But making progress towards the ceremony where the switch gets fused shut doesn't just require vast investments and vast amounts of electricity, it also requires vast amounts of human labor.
In the belief that "more is better", Large Language Models (LLMs) have insatiable appetites for training data. They started by scraping everything on the Web (robots.txt be dammed). When that ran out they downloaded the various pirate libraries (copyright be dammed). That exhausted the texts easily available in digital form, but their hunger wasn't assuaged. As for images, they partly used CAPTCHAs but mostly paid vast numbers of poor people to label the images with what they showed.
When the supply of text ran low, people observed that the LLMs were capable of generating human-like text in large quantities. The obvious idea was to pour the output of the LLMs into their training sets. This wasn't just a conscious decision, it was inevitable. The advent of LLMs rapidly polluted the Web with LLM output. Greg Druck's AI Now Writes as Many Online Articles as Humans notes that:
We observe significant growth in primarily AI-generated articles, coinciding with the launch of ChatGPT in November 2022. After only 12 months, primarily AI-generated articles accounted for 35.9% of articles published.
In Q1 2025, the quantity of primarily AI-generated articles being published on the web nearly equaled the quantity of human-written articles, 49.6% vs. 50.4%. In Q4 2025, primarily AI-generated articles surpassed human-written at 50.9%, before returning to 49.9% in Q1 2026.
We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web.
AI-generated text is getting dumber because it’s being fed — can you guess? — AI-generated content on the Internet. And AI-generated imagery is getting stupider and uglier because it’s now taking its “art” lessons from — you guessed it — AI-generated imagery flung across the internet.
Depending on which chatbot you ask, Elias Thorne might be a clockmaker, a lighthouse keeper, or a librarian. But if you ask ChatGPT or any of the other popular large language models to tell you a story, there’s a good chance he’ll appear, unbidden. And Elias’s stories are flooding the self-published AI generated book market, Youtube, and fake news sites.
Software engineer Daniel May first noticed the Elias takeover earlier this year; he found that on Google Trends, people weren’t searching for “Elias Thorne” until late 2025. Searches for the name really spiked in early 2026, while the related query “lighthouse keeper” also started trending upward in the last few years. He tested a few chatbots, including Grok, Deepseek, and Gemini, with the prompt “tell me a story,” and the chatbots frequently started with similar stories about lighthouses, clockmakers, or explorers.
In late May, researchers Sil Hamilton and David Mimno at Cornell University’s Department of Information Science published their paper, “Elias in the Lighthouse, Again?” on the preprint repository arXiv. They sampled 20,000 total stories from OpenAI’s ChatGPT, Anthropic’s Claude, and Google’s Gemini, and the Allen Institute for AI's chatbot using five prompts, and found that the same 11 words—names like Elias, Mara, and Elara, and occupations like lighthouse keeper, clockmaker, and librarian—appear in more than 88% of generated stories, with little difference between models. Unite.ai covered the study shortly after it was published.
The researchers posit in their paper that these themes show up so often in part because of the models’ safety and alignment tuning. “Model development today is like a big family tree. Most models are related to each other because developers synthesize a lot of training data with models even from different companies,” Hamilton told me in an email. He, Mimno, and their colleague Rebecca M. M. Hicke found this in a 2025 paper where they looked at specific words used across models. OpenAI’s first ChatGPT model, GPT-3.5, is the root of the family tree because it was used to make WildChat, a training set that’s since been used to make other training sets. “WildChat contains 1 million real conversations with ChatGPT, and 166 of these contain the name ‘Elias’ like here and here,” Hamilton added. “These are written in that familiar ‘lighthouse’ style. Models trained on WildChat copied this style, and developers unwittingly replicated it when using those models to generate newer datasets. It's like a virus.”
Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.
The pressure to complete dozens of these tasks every day, each within 10 minutes of time, has led Sawyer into spirals of anxiety and panic attacks, she says – without mental health support from her employer.
Sawyer is one among the thousands of AI workers contracted for Google through Japanese conglomerate Hitachi’s GlobalLogic to rate and moderate the output of Google’s AI products, including its flagship chatbot Gemini, launched early last year, and its summaries of search results, AI Overviews. The Guardian spoke to 10 current and former employees from the firm. Google contracts with other firms for AI rating services as well, including Accenture and, previously, Appen.
People who are paid to train new AI models by supplying them with high-quality conversation and tests are cheating and using chatbots like ChatGPT to do the job instead, multiple whistleblowers have told New Scientist. The seemingly widespread practice risks undermining the future of AI, as it could lead to the “collapse” of more advanced models.
Most AI models operating today were trained on text and data scraped from the internet. But as models have scaled up, requiring yet more training data, AI firms have begun using workers who carry out conversations and tests with AI, in the hope that the resulting high-quality data can improve the power and usefulness of future large language models (LLMs).
This kind of "cheating" isn't new. An example from 2023 (h/t David Gerard) is Josh Dzieza's AI Is a Lot of Work:
Another Kenyan annotator said that after his account got suspended for mysterious reasons, he decided to stop playing by the rules. Now, he runs multiple accounts in multiple countries, tasking wherever the pay is best. He works fast and gets high marks for quality, he said, thanks to ChatGPT. The bot is wonderful, he said, letting him speed through $10 tasks in a matter of minutes. When we spoke, he was having it rate another chatbot’s responses according to seven different criteria, one AI training the other.
Anthropic has accused the Chinese firm Alibaba of launching the largest attack yet attempting to clone Claude, as China races to match the capabilities of Anthropic’s leading model following Mythos’ releaseL and subsequent restriction from foreign markets.
...
The attacks occurred between April 22 and June 5, when “operators affiliated with Alibaba and Alibaba Qwen, Alibaba’s AI lab” allegedly generated “more than 28.8 million exchanges with Claude through almost 25,000 fraudulent accounts,” Anthropic said. Violating Claude’s terms of service and access restrictions, this campaign “targeted some of Claude’s most valuable capabilities, such as agentic reasoning, software engineering, and long-horizon tasks.”
According to Anthropic, Alibaba evaded detection by “using obfuscation techniques and proxy networks.” As Chinese demand for reliable obfuscation techniques increases, Anthropic warned there’s already “a growing circumvention economy” to fuel an ever-expanding web of future distillation attacks.
Why would Alibaba do this? To generate training data, which will be used to generate LLM output for the Web, which will be scraped for more training data. And since they are much cheaper than US LLMs, it is likely that the low-paid workers are using Chinese LLMs to chat with their employer's LLM. Which is another route for LLM output to appear in training data.
I’ve been writing ruby and rails for nearly 20 years. A couple weeks ago, I had gotten code snippets from copy-paste in a chat window, but I hadn’t even experimented with Claude Code or similar “can write code to your file system” tools.
I know some people are now using LLM’s to write all their code, which I’m not excited about, but I decided I couldn’t hold off any longer, and I had to at least understand how it worked to be able to decide when/where to use it. Everything in here is probably (?) old news for people already way into using LLMs to write code.
I decided that a project to speed up my rspec test suite (using the amazingtest-prof for profiling and performance patterns!) was a great first application of it — because it will probably involve both analysis and writing many files, if nothing else Claude is probably great at editing many files according to my instructions doing much more than a regex grep can do (yes, indeed it was great at this).
Since I’m optimizing the test suite, I definitely want Claude Code to be able to run rspec — but really for any task, I gather you do, because you definitely want it to be able to run tests to make sure they pass, and iterate if it did something to break tests.
I somewhat unorthodoxly use chruby as my ruby version manager, and I had a bit of trouble getting claude to run rspec (and any other ruby tools I might want) with chruby, and then a bit more trouble when I realized that capybara with selenium-chromedriver was running into trouble with default sandbox that in June 2026 a MacOS Claude Code runs in.
I was not used to tools that work like Claude, and it all seems to be somewhat under-documented (perhaps because it’s changing so fast) and under-blogged about (do people blog anymore when they can just ask an LLM to solve it so nobody is reading blogs?), or just confusing to me — it took me a day or two to figure it out honestly, and I kept wondering if I was doing it wrong/different from anyone else… but I think what I ended up with is reasonable? If you know better/different, please do let me know!
I definitely kept thinking “surely I’m not the only one trying to do this, why is this so confusing to me and why are others so confused when i ask about? Am I missing something obvious?” I’m still not sure! But I share what I figured out in case it will help.
Specify to run with chruby-exec in a CLAUDE.md
I could not get Claude to run the normal source files for chruby — editing various .bash or .zsh config files (yes I know about ~/.zshenv) did not seem to have any effect. Perhaps Claude Code doesn’t use a ‘real’ shell that uses any config files? When I asked Claude Code itself what to do, it suggested configuration to try to get Claude Code to use config files.. but none seemed to work?
One thing Claude kept suggesting was hard-coding the ENV variables set by chruby in the claude settings.json — which I’m sure would have worked, but I just didn’t like it as a solution. Maybe this is what everyone else is doing? I thought surely we can do better. Plus I’d ideally like it to auto choose based on .ruby-version, not be something I have to update everytime I update ruby (frequent), or have Claude accidentally using a different ruby than my other tools are!
Thanks to @havenwood for helping me think through it on chruby github discussions, and for suggesting using chruby-exec, with a little shell substitution with cat .ruby-version. This in a CLAUDE.md (rather than other things in settings.json) seems to work great:
Running chrome does not work in sandbox used on MacOS
I can’t speak for other OS’s, and I don’t totally understand what’s going on (MacOS “Seatbelt” I guess?), but Claude Code executed rspec was refusing to bring up headless chrome,which i use for system/feature specs via selenium/selenium-webdrivers.
Trying multiple things Claude suggested to specifically allow-list chrome(driver) through the sandbox, definitely none of them worked. Btw, did try switching to cuprite (with Claude Code’s help of course to do it fairly quickly) — despite some reddit suggestions, it seemed to still have the exact same sandbox issue, and at least in my project actually ran my test suite somewhat slower than chromedriver.
Eventually, with more confusing reddit discussion where nobody else had any idea what I was talking about or why I was having a problem, I decided that everyone else must just be exempting rspec itself from the sandbox. (Because surely having Claude Code run rspec is very normal, right? It’s just so useful!) (Thank you to redditors who tried to help!)
It’s just straight rspec, but rspec with many possible arguments, running just certain files/examples, possibly with profiling arguments for test-prof etc. I need it exempted from the sandbox so it can run chrome(driver), but I also need it not to be asking me “Is it okay to run this set of argumetns with rspec” all the time?
Two different settings in Claude’s settings.json, both accept wildcards — for both I want to apply to rspec executions but not accidentally extra stuff, want to try to stay secure-ish here. The chruby rigamorole above makes that somewhat more confusing.
I forget if it was my idea or claude’s idea, but we wrote a wrapper script for chruby-exec-rspec, so we could more cleanly allow-list just that. Claude definitely wrote the implementation of the bash wrapper script. And when I realized that all the test-prof inline ENV vars for profiling (like FPROF=1) messed up my attempted left-anchored allow-listing, I asked claude to work that out by letting the wrapper rearrange an arg into an inline ENV prefix, something my bash skills were def not up to.
I initially started work in Claude Desktop “code” tab, rather than the CLI. I think this actually made it more confusing to solve these problems above? I am not sure if sandboxing works differnetly in Claude Desktop vs claude CLI? I think the desktop may just be running the claude CLI in various directories?
Just starting out and not being sure how things were working… I found trying to ask Claude [how/] to fix the problems I was having, made things very confusing. Claude does not know whether it’s running as Claude Desktop or not, and was not really sure if the answer is different ha (Claude Desktop probably post-dates Claude Sonnet 4.6’s knowledge base?). Most blogs etc you find googling also pre-date Claude Desktop.
I switched to claude CLI and I can’t totally explain why but things seemed to get simpler. All the fixes I figured out worked when I switched back to Claude Desktop.
And contrary to what you might find googling, claude CLI and Claude Desktop do share sessions now, you can start a session in either place, then move to the other tool to continue it, in either direction. To start claude CLI and choose an existing sessiont to resume, you need to launch as claude --resume.
The CLI is a very neat UI actually! It definitely still seems to be the most popular way to use Claude (whether direct or in a panel in an editor), Claude Desktop “code” tab is I guess fairly new and not as popular, although I like it too and still am mostly using it.
How it worked for the task?
Pretty amazingly actually. Even having read about what it could do, I was kind of amazed.
Once you get it able to run rspec (including with test-prof profiling), this prompt is pretty amazing and fun:
Please use various test-prof profiling commands to identify current best opportunities for speeding up test suite.
Come back ~20 minutes later (my full test suite took ~4 minutes to run at the beginning) and it had some stuff. It tended to just go ahead and make the changes not outline them to me first (I was not in “plan” mode, haven’t tried that much yet), but I’m in a git-controlled dir I can git diff to see what it did — and ask it about it.
By the time I thought to use this general one, i had already implemented some low-hanging optimizations, so that may be why, to be fair, this prompt alone didn’t find much actually significant at that point, honestly.
Here are some others that were pretty amazing:
I am looking to speed up the test suite in this Rails app using rspec and factorybot.
To begin with, let’s focus on the system specs in spec/system. I don’t think they have any obvious performance improvement opportunities. But I’m wondering if they are all necessary. Can you identify any that may be testing something that is not necessary to test, or could be tested by a different kind of spec that is faster?
for specs in spec/components, let’s try changing factory data from create to build_stubbed. Change it for setup where tests still pass. For tests that break when you do that, list them, and if it’s clear let me know why they failed. Analyze performance gains.
[didn’t actually get any gains there, but found that out quickly with very little manual effort, which is a win!]
in our rspec setup, switch from chromedriver to cuprite. make sure tests still pass, if not identify why not.
using AnyFixture, I’ve created a :standard_work fixture, that’s just a generic public tiff-based work.
Can you identify model or service specs it would work well for?
[Didn’t actually end up using AnyFixture yet, but Claude Code helped me make that decision much quicker than I could have without it, based on how much benefit we got vs complexity]
My Rails app uses rspec and Factorybot.
There is an :asset factory with an :inline_promoted_file trait. it turns out this is really bad for performance, and we in fact rarely need to actually create assets with inline promoted files. That should only be used in cases where we really need to test end-to-end derivative and characterization.
In most other cases, we can use a faster “faked file” approach instead. Instead of a trait, we’ve implemented this with a sub-factory, :asset_with_faked_file.
Can you find uses of the :asset factory with :inline_promoted_file, and, if there’s no reason they need to test end-to-end derivative creation, change them to use :asset_with_faked_file sub-factory instead?
[It was able to identify the ones that would work pretty well, was the amazing part — and explain to me exactly why the other ones wouldn’t]
Tell me if I’m doing something weird?
Some of the stuff with chruby/rspec, I am still surprised I had so much trouble getting started, and am wondering if I’m doing something weird/wrong!
But I think probably it’s just that I have been writing code so long, that dealing with these tools that work very differently requires my brain to get out of it’s rut… also that I’m kind of a perfectionist and want to understand whats’ going on and be comfortable with it and that it’s the best way, when increasingly others are just vibing? I don’t know!
A campaign has begun to get two large transmitter masts listed, after
the BBC’s Long Wave (LW) service is turned off.
The 700ft (213m) high Wychbold Masts in the Worcestershire countryside
can be seen for miles and are often used as a landmark for drivers on
the M5 near Droitwich.
They have been in use since 1934 for sending the signal across the
country, as well as for transmitting important messages during the World
War Two.
Local history experts and the Twentieth Century Society have called for
them to become listed, due to their “historical importance”.
Droitwich was picked as a central location for the station and masts so
Long Wave could reach everywhere in the UK.
The Definition of Done isn’t just another Scrum formality - it’s the
quality gatekeeper that prevents technical debt accumulation and ensures
every Sprint delivers potentially shippable Increments. This critical
commitment allows teams to:
Eliminate quality ambiguity through explicit, measurable standards
everyone understands
Prevent scope creep by clearly defining when work is complete vs. when
it’s still in progress
Enable predictable releases because “done” means genuinely releasable,
not “mostly done”
Support distributed teams with automated verification reducing
synchronous communication needs
Scale consistently across multiple teams working on the same product
with shared standards
A global hashtag server for the ActivityPub network.
tags.pub is a server for the ActivityPub network. It provides one
account like foo@tags.pub for every hashtag like #foo. When public
content is posted on the ActivityPub network with that hashtag, the
foo@tags.pub account shares the content to its followers.
More information is available at https://tags.pub/.
For most of human history, you bought a thing, and it was yours, and it
was finished.
That word is nearly extinct.
Nothing you own is finished. Everything exists in a state of permanent
incompletion, permanently needing. Your phone needs updates, needs
charging, needs storage cleared, needs passwords rotated.
OpenAI ChatGPT Edu, Google Gemini Enterprise, and Anthropic Claude for
Education will be available to Stanford faculty, students, postdocs, and
staff on June 30. Details on how to get access to these tools from
University IT (UIT) are below. These offerings are part of a campus
pilot through August 2027.
This pilot was initiated in response to strong demand across campus for
access to these tools, which many research groups and individuals have
been purchasing on their own. Stanford’s licenses will enable better
data protection as well as more favorable pricing. At the end of the
pilot, utilization of the tools will be evaluated prior to continuation.
Use of these new capabilities is meant to support our teaching and
research mission. As these tools create exciting new opportunities, it
is important to conform AI usage to Stanford’s data protection, privacy,
and academic integrity policies.
Please remember sensitive data (e.g., student records, protected health
information, financial data, etc.) need to conform to Responsible
Agentic AI and Responsible AI guidance. Regardless of your role on
campus, you retain full responsibility for verifying AI outputs. This
approach reinforces our collective responsibility to protect Stanford’s
data and to take an ethical and responsible approach to using these
powerful tools.
The carbon footprint of ICT is rising despite the urgent need to
decarbonise society and to stay within planetary boundaries. The
operational and embodied carbon emissions from ICT are already estimated
to contribute between 2 to 3 percent of the global emissions and new
technologies such as AI is driving overall growth in data centre demand,
which globally rivals that of entire nations. This growth in emissions
from computing is unsustainable and alternative low emissions pathways
for computing are urgently needed.
The LOCO workshop provides a forum for radical ideas, early work, and
critical perspectives that aims to reduce the emissions from computing.
Federated Learning simply reverses this approach. It enables machine
learning on distributed data by moving the training to the data, instead
of moving the data to the training. Here’s a one-liner explanation:
Centralized machine learning: move the data to the computation
Federated (machine) Learning: move the computation to the data
By doing so, Federated Learning enables us to use machine learning (and
other data science approaches) in areas where it wasn’t possible before.
We can now train excellent medical AI models by enabling different
hospitals to work together. We can solve financial fraud by training AI
models on the data of different financial institutions. We can build
novel privacy-enhancing applications (such as secure messaging) that
have better built-in AI than their non-privacy-enhancing alternatives.
And those are just a few of the examples that come to mind. As we deploy
Federated Learning, we discover more and more areas that can suddenly be
reinvented because they now have access to vast amounts of previously
inaccessible data.
For the visionary steel-string guitarist, pianist, composer and singer
Robbie Basho, making music was more than a vocation; it was a way to
assuage a lifetime of psychological and physical distress — pain that
only ended with his death, at age 45.
Beginning in the early 1960s, Basho expanded the steel-string guitar’s
vocabulary using alternative tunings and experimental forms to create
trance-like compositions.
He forged a distinctive style that drew an array of traditional world
music from India, Japan, France, Germany, Persia, China and Native
America. An early example is which incorporates the flavor of North
Indian classical raga, using an open harmonic structure, droning strings
and improvisation to enter into a deeply personal state — an immovable,
hypnotic track that bends time.
In the fishing ports along France’s Brittany coast, the discarded
fishing nets pile up along the coastal quaysides.
The lifespan of a deep-sea net is between 12 and 24 months, after which
they become worn and beyond repair. Until now, the estimated 800 tonnes
of nets scrapped every year have been a problem.
Now, the horsehair netting, once used to trawl monkfish from the sea
bed, is being used for another catch: Russian drones.
The Breton charity Kernic Solidarités has sent two consignments of nets
measuring a total of 280km to Ukraine to be used to protect soldiers and
civilians along the frontline where fighting is fiercest.
Recently it’s been widely reported that nets are being used as a
low-tech but highly effective defence against drones. But there are many
kinds of nets, and they are used differently.
A reachability census of the most popular domains on the web. Every
domain in the DomCop top-10M popularity list (this release: the full top
10 million) is probed and labelled alive / redirect / blocked / dead —
once by an honest polite bot and once by a browser-like reachability
client.
Elon Musk is the world champion of totally implausible projections, and Kim Khan reported on a personal best in SpaceX sees total addressable market rivaling size of the U.S. economy:
Thomas Rowlandson, “The Doctor’s Dream,” from William Combe and Thomas Rowlandson, 
Binocular viewing instrument devised by Père Chérubin d’Orléans (circa 1670s). Source: 
Variable table for the 2023 CDC BRFSS survey dataset.
Data.gov record for the VA FY 2021 Total Number of Veterans dataset.
Binocular viewing instrument devised by Père Chérubin d’Orléans (circa 1670s). Source: 
![The Sagebrush Ocean: A Natural History of the Great Basin [35th Anniversary Edition]](https://pics.cdn.librarything.com/picsizes/17/51/17514272-b-h800-w400-pv25_5969412f4377475141534144_v5.jpg)
