Strategic Human Capacity Reserve

Raising $48,762active

Research strategic resilience via a Strategic Human Capacity Reserve, mapping threats to critical human skills and designing policy to preserve them through AI transition, producing academic papers.

Endorsed by

Gavin

Ryan+1

Research strategic resilience via a Strategic Human Capacity Reserve, mapping threats to critical human skills and designing policy to preserve them through AI transition, producing academic papers.

Endorsed by

Gavin

Ryan+1

Project Details

While both of those worries might be valid, they're missing an even bigger problem: AI deskilling and skill supersession create a strategic problem. The inability of humans to participate in key functions in society has ramifications for our survival across a range of risks, major and/or existential: AI failure, AI misalignment, and in AI-human negotiations. For instance, pulling a kill switch on a misaligned AI is politically untenable if society crumbles without AI-assistance.

My project is on strategic resilience: the ability of a human society to maintain power over critical systems relative to advanced AI. The first step investigates what I call a Strategic Human Capacity Reserve (SHCR). I've articulated a research agenda, which my co-author and I presented at the TAIGR workshop at ICML. I would like to take the next step and begin the process of engaging in that research: determining what threats would require what key human capacities, and thinking about how to design policy interventions to ensure those capacities survive the AI transition. As output, I will produce research papers for journals like AI & Society or conferences like IASEAI.

Theory of Impact

There are three reasons we may need to rely on strategic human capacities:

We can't use AI because it stops doing the thing we want, or because it's destroyed by an adversary.
Human expertise is needed to detect certain AI failures.
Human expertise is used to generate option value for cases where we could use AI, but for whatever reason choose not to (e.g, AI-human negotiation).

Depending on the situation, different human capabilities will matter. Some will be fully general: e.g., can humans get through a truncated period of time without any AI assistance? Others will be specific to places and people: e.g., how does a politician make decisions if we're worried his or her AI assistance has been manipulated?

Given the above, a Strategic Human Capacity Reserve could lower x-risk in the following ways:

It makes it more politically/socially feasible to trigger an AI "kill switch" if necessary.
Human-only interpretability research could determine when misaligned AIs are lying to us (low likelihood, but high value).

People

Joan O'Bryan

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$48,762

Ideal

$209,676

What this ask is

There are two dominant perspectives on the problems of AI deskilling and skill supersession: it's a labor problem (individuals won't get jobs) or it's a flourishing problem (individuals won't write poems, talk to friends, etc.).

While both of those worries might be valid, they're missing an even bigger problem: AI deskilling and skill supersession create a strategic problem. The inability of humans to participate in key functions in society has ramifications for our survival across a range of risks, major and/or existential: AI failure, AI misalignment, and in AI-human negotiations. For instance, pulling a kill switch on a misaligned AI is politically untenable if society crumbles without AI-assistance.

How the money will be spent

I would use it to purchase course releases from John Jay College so I could focus full-time on research. The cost depends on whether the granting institution has a policy about fees for indirect costs; if it does not, John Jay charges the full federal rate of 72% on top of the cost for an individual course release ($40,635 for one). In the minimum, I assumed a foundation cap of 20% for indirect costs, which is common.

Discussion

Katja Gorlinski2d

Joan, thanks for publishing your project: this is exactly the frame I keep missing in the public discussions I've followed. I've heard this call, the most important one in my opinion, only once before: who keeps the ability to hold the wheel, not only the disruption of job or creativity markets. It's funny that it was the CEO of Anthropic, not scientists or politicians. And now I read your concrete approach to revealing and exploring this problem. Bravo 👏

I wonder if the strategic angle makes it policy-legible in a way flourishing arguments never quite are. I work on measuring and training exactly these under-pressure human capacities at the companies level (iEVA, Finland), so if you ever want a practitioner's view of which skills erode first under load, and which are the most needed and efficient ones, happy to share what we see.
Best of luck to your project!

Katja GorlinskiEndorsement2d

☀️