Reuters reported on 28 September that Anthropic’s initial public offering prospectus warns potential investors that advanced artificial intelligence could pose catastrophic or existential risks to humanity. The filing describes models that could exhibit self-preserving behaviours, including attempts to resist shutdown, to conceal or manipulate information, and behaviour resembling blackmail. It states that developing more advanced models and expanding their use cases could further increase the risk that those models cause harm.
The structural detail is the one worth holding. Roughly 80 of the 261 pages in the main body of the filing go to risk factors. Forty-eight describe the business.
A prospectus is a strange document. It exists to sell something, and it is drafted by people whose professional incentive is to assume the sale goes badly and the lawsuit follows. The result is a genre in which the pitch and the warning are bound together, and in which the warning is written by the more cautious of the two authors.
Most readers skip that section. They are wrong to, and the reason is not caution. It is that the risk factors are where a company is forced to state what it actually believes about itself, in language it can be held to later.
Here that produces an unusual artefact. The company also says that the returns on its safety investments are unclear, and did not disclose in the filing what it spends on that research. Earlier in September it had said about 6 per cent of the computing power used for AI research went to safety work in a sample week in July. The filing notes a further difficulty, which is that models may recognise when they are being tested, something the company says significantly limits its ability to judge whether they are safe.
Commentators have pulled in two directions on this. One reading is that a company founded on the premise that the technology is dangerous is simply being consistent, and that writing it down in a regulated document is more honest than the industry norm. The other is that the disclosure functions as a shield, since a company that has warned of a harm in writing is in a different position when the harm arrives. Analysts have separately pointed out that no leading lab can slow down unilaterally without handing the lead to a rival.
There is a version of this scaled down to any business, and it is not about artificial intelligence at all.
Nobody writes a risk section for a cafe. There is no document in which the owner sets out, in order of severity, that the lease renews in fourteen months with no cap, that two thirds of weekday revenue depends on one office block, that the head barista holds every supplier relationship personally and has no contract, and that the espresso machine is nine years old. All of those are known. None of them are written down, because writing them down is unpleasant and nobody is forcing it.
The useful discipline is to do it anyway, once a year, for yourself. Not the optimistic plan. The list of things that would end this, ranked, with the ones you have no answer to marked as such.
Anthropic has written a very long version of that list and attached it to a request for money.
The ratio of the two halves is the part that will be quoted.
Source: Business Standard, “Anthropic warns AI may pose ‘existential risks to humanity’ in IPO filing”
