Fairly short timelines, mildly fast takeoffs, and medium-high uncertainties. –> Looking for abstractions to help cognition-steering and value-loading. –> Grasping at / reacting to related/scary/FOMO lines of research.
An AI system could be built that’s far smarter than any human or small group of humans. This AI system could use its intelligence to defeat any non-motivation-directing safeguards, and gain control of the world and the future of humanity and other sentient life. Based on the orthogonality thesis, this amount of power would probably not, by default, be directed towards the best interests of humanity and other sentient life. Based on the idea of instrumental convergence, such an AI would destroy everything we value in its quest to fulfill its (dumb-by-default) original goal. This AI may require new insights to build, or could arise by “scaling up” existing ML architectures. This AI may “self-improve” its architecture, or it could get smarter through prosaic “hack more cloud computing power” techniques. In either case, it could start from a position of low capabilities and end up as the most powerful entity on Earth. This all could start within as few as 1.5 years from now, and will probably happen within 10 years, barring nuclear or other catastrophe.
The simplest solution to the above problem would be “don’t build superhuman AGI, at least for the near future”. However, superhuman AGI is likely to be built, on purpose or by accident, by any of a handful of groups with large amounts of computational resources and talented researchers. These groups are generally not monolithic, and contain leaders and employees who disagree (internally, with other orgs, and/or with me) about the best approach to AI alignment. (See the section “The AI Landscape” below).
Imagine if any of these groups got a box, today, that said “Input your alignment solution by USB drive, push button to get a superhuman AGI that runs on that, box expires in 1 week”. According to my threat model, humanity is unlikely to survive longer than 1 week in this scenario. This is despite the wildly varying (often quite good!) alignment-motivations and security-mindsets of these groups. On my view, this is (mainly) because none of these groups has an adequate pre-prepared response for the below “Two Subproblems”.
I forgot where this advice came from, but I followed the tip of “Take a day or so to think through AI alignment, for yourself, from scratch”. I was definitely biased by my previous readings on AI (especially by Yudkowsky and Wentworth), but I basically came away with
How do we direct the thought-patterns, goals, and development of an AI system? This is basically the rocket alignment analogy, specifically the “Newtonian mechanics”/“basic physics” part.
For many, the core difficulty and most-important-part of AI alignment is to be able to steer a mind’s cognition at all. If we get this right, we set a lower-bound on the badness of AGI X-risks (while also opening the door to S-risks from solving this subproblem and neglecting the one below, but that’s not the immediate focus).
If we could aim a superintelligent AI system at anything, what should we aim it at, and how? In the “rocket alignment” analogy, this is basically the flight plan (or the method for creating the flight plan) to get to the moon.
At first, this seems to naturally decompose into “determine values” and “encode values into the AGI”. However, I consider this to be one subproblem, because an AGI could most likely carry out the execution of either or both of those steps. But before those steps is something like “figure out what [a pointer to [the best values for an AGI]] would look like, in enough detail to point an AGI at it and expect things to go well from there.” Due to fragility-of-value, I don’t expect a real-life satisfactory solution to AI alignment to involve a human (that is, an neither-augmented-nor-simulated human) writing down the full Sheet Of Human Values and then plugging it into an AGI. However, we could end up with, say, a reliable theory of / mathematical abstraction for our values, which an AGI could then “fill in the blanks” of through observation.
If I research the above two subproblems (and/or the items in the section “What I’m Personally Learning/Researching” below), then one or both of the above subproblems will become more-solved. This could be a full end-to-end solution, a theoretical-but-proved plan, a paradigm that can be developed further, or contributions to the work of others. I am eager to help and fairly-agnostic about how.
Furthermore, I think that even if my above “Threat Model” is wrong in one or more key ways, the research I want to do would still be helpful. For example, if AI takeoff speeds were slower, I would still want research-similar-to-mine to be developed and refined quickly. If neural networks were usurped by a new AI paradigm, I would still think research-similar-to-mine could help align the new architectures.
And, of course, if you or somebody you know seems well-equipped to carry out any of this research, please steal my ideas, move the work forward, and disclose the results responsibly.
Create a manual/framework for building a friendly AGI.
Create a manual/framework for building an AGI that can be pointed at anything.
Helping with either of the above two.
Some other result that prevents AGI-caused extinction of humanity.
John Wentworth’s work on abstractions
MIRI’s work on agent foundations
Getting a broad knowledge base of mathematics, modulo my existing knowledge and short timelines.
Finding out what other nuggets of interesting research would be helpful for my goals, from cyborgism/human-researcher-intelligence-amplification to governance/large-training-moratorium to moral uncertainty to theories-of-how-sentience-and/or-consciousness-works.
Finding or developing mathematical structures that are actually useful in aligning smart AI systems (“What’s the type signature of an agent?”).
Contributing to abstraction theory to get it to a point where “pointing at things in the world” is doable.
Applying abstraction theory to understanding and changing the contents of artificial minds. This includes the ability to e.g. trace a “thought” through a neural network, from input to output, and understand its progression.
Applying abstraction theory to determining human values.
Contributing to, giving feedback on, extending, and applying the above areas I’m learning about, especially cutting-across different organizations. (For reference: I am currently in closest touch with people at Orthogonal, Conjecture, and OpenAI. I’ve also talked briefly with various alignment researchers at the EA Global SF 2022 conference.)
And again: If you or somebody you know seems well-equipped to carry out any of this research, please steal my ideas, move the work forward, and disclose the results responsibly.
These are described in more depth here. They are (in no particular order):
Mathematical concepts (getting even more “broad technical background”)
Mathematical notation/formalism knowledge
My working memory
My mental stamina
The most important constraint right now (i.e. the only real bottleneck at this time) is funding. With enough funding, I could work full-time on AI alignment, which would include solving or mitigating the other constraints.
Note that I already have a Bachelor’s degree in computer science, a minor in mathematics, and some other AI-related background (see here).
Here is how the rest of the AI alignment/safety landscape looks, to me, as of this writing:
Table 1: An Informal Assessment of Potentially-Strategically-Relevant AI and Alignment Organizations, as of late May 2023. (Note: This table may not be up-to-date.)
|Organization||One-Sentence Summary||Are they likely to cause AGI doom, including by accident?||Do they care about AGI risk? (This includes investor pressure and disagreements with my risk model!)|
|OpenAI||The most cutting-edge AGI research lab, structured as a sort of nonprofit/for-profit hybrid company, with heavy investment from Microsoft.||worrying||mostly|
|Microsoft||Tech giant, with large investments of money and cloud computing towards OpenAI.||worrying||maybe?|
|Google DeepMind||The other most cutting-edge AGI research lab, the AI arm of the search engine giant, and the original developers of the popular framework TensorFlow.||worrying||mostly|
|Google/Alphabet||Tech giant that owns DeepMind.||worrying||maybe?|
|X.AI||Elon Musk’s new AI research company.||worrying||Elon Musk|
|Meta/Facebook AI||The AI arm of the social networking giant, and the original developers of popular framework PyTorch.||worrying||no(?)|
|Conjecture||EleutherAI alumni + computational resources and funding + security mindset.||More than “kinda”, but less than “worrying”.||yes|
|MIRI (Machine Intelligence Research Institute)||The original AI alignment nonprofit, founded by Eliezer Yudkowsky, focused on formal research and fieldbuilding.||no||yes|
|Orthogonal||A new nonprofit built around an idiosyncratic approach, founded by an alumna of the Conjecture-hosted Refine alignment research incubator.||kinda||yes|
|ARC (Alignment Research Center)||The nonprofit alignment group run by Paul Christiano.||no||yes|
|Redwood Research||The nonprofit alignment group where Buck Shlegeris works.||kinda||yes|
|Ought||Product-oriented nonprofit working on factored cognition.||not much||maybe|
|Anthropic||Well-funded AI company focused on building and aligning large language models.||worrying||probably|
|The US government/military||The federal government and/or military of the United States of America.||not much, but could||probably|
|The Chinese government/military||The national government and/or military of the People’s Republic of China.||not much, but could||maybe|
|Any large technology company headquartered in China (including Baidu, Alibaba, Tencent, and others)||The most cutting-edge companies in China to have large computational resources.||not much, but could||maybe|
|Keen Technologies||AGI company with >$20M in funding, founded by John Carmack.||Never count out Carmack.||no(?)|
|Apart Research||Mostly fieldbuilding||no||probably|
|SERI MATS||Stanford/Berkeley-run program that supports independent alignment researchers like John Wentworth and Jeffrey Ladish.||no||yes|
This sub-site is hosted inside Thinking Much Better, though not necessarily licensed or authored or owned in precisely the same way. On this sub-site, the below statement overrides Thinking Much Better’s default licensing:
Unless otherwise specified on individual pages, all posts in this sub-website are licensed under a Creative Commons Attribution 4.0 International License.