强人工智能愿景：基本假设与框架

原文作者：Dario Amodei

机器翻译：Copilot

October 2024

我经常谈论强大人工智能的风险。我担任CEO的公司Anthropic，致力于研究如何减少这些风险。尽管如此，人们有时认为我是一位悲观者，认为人工智能主要是有害的。我并不这么认为。事实上，我关注风险是因为它们是实现根本性积极未来的唯一障碍。我认为大多数人低估了人工智能的潜在好处和风险。

在这篇文章中，我试图勾画出如果一切顺利的话，强大人工智能可能带来的美好前景。当然，没人能确定或精确地预测未来，而强大人工智能的影响可能比过去的技术变革更加不可预测，所以这一切难免会是猜测。但我至少希望这些猜测是有教育意义和有用的，即使大部分细节最终是错误的，我也希望能捕捉到将要发生的事情的味道。我包括了很多细节，主要是因为我认为具体的愿景比高度谨慎和抽象的愿景更能推进讨论。

首先，我想简要解释一下为什么我和Anthropic很少谈论强大人工智能的好处，以及为什么我们可能会继续主要讨论风险。特别是，我做出这种选择的原因有：

最大化杠杆作用。AI 技术的基本发展及其许多（但不是全部）好处似乎是不可避免的（除非风险让一切脱轨），并且基本上是由强大的市场力量驱动的。另一方面，风险不是预定的，我们的行动可以极大地改变其可能性。

避免宣传的印象。AI 公司谈论 AI 的所有惊人好处可能会像宣传者一样，或者好像他们试图分散对缺点的注意力。我还认为，作为原则问题，花太多时间“谈论你的书”对你的灵魂有害。

避免宏伟。我经常对许多 AI 风险公众人物（更不用说 AI 公司领导人）谈论 AGI 时代后的世界感到反感，好像他们的使命是单枪匹马地带领人民走向救赎。我认为将公司视为单方面塑造世界是危险的，将实际的技术目标视为基本上是宗教术语是危险的。

避免“科幻”包袱。尽管我认为大多数人低估了强大的 AI 的上行空间，但那些讨论激进 AI 未来的小社区通常以过于“科幻”的语调（例如，上传的思想、太空探索或普遍的赛博朋克氛围）进行讨论）。我认为这会导致人们不那么认真地对待这些说法，并使他们带有某种不现实的色彩。明确指出，问题不是所描述的技术是否可能或可能（主文章对此进行了详细讨论）——更重要的是，这种“氛围”隐含地带来了大量的文化包袱和对什么样的未来是可以接受的、各种社会问题将如何演变等未曾表述的假设。结果往往看起来像是某个小众文化的幻想，而对大多数人来说令人反感。

尽管存在上述所有问题，我仍然认为讨论强大人工智能可以带来的美好世界非常重要，同时尽量避免这些陷阱。事实上，我认为拥有一个真正鼓舞人心的未来愿景至关重要，而不仅仅是灭火计划。许多强大人工智能的影响是对抗性或危险的，但最终，我们必须有我们所争取的东西，有一种积极的结果，使每个人的生活更美好，有一个能够团结人们超越争吵并迎接挑战的目标。恐惧是一种激励因素，但还不够：我们还需要希望。强大人工智能的正面应用清单非常长（包括机器人技术、制造业、能源等），但我要集中讨论一些对改善人类生活质量最有潜力的领域。我最兴奋的五类是：

生物学和身体健康
神经科学和心理健康
经济发展和贫困
和平与治理
工作和意义

我的预测从大多数标准来看会是激进的（除了科幻“奇点”愿景），但我是真诚的。所有这些说法很可能都是错误的（重申我之前的观点），但我至少试图在不同领域的进展可能加速的半分析评估中奠定我的观点。我有幸在生物学和神经科学方面有专业经验，而我在经济发展领域是一个受过良好教育的业余爱好者，但我相信我会犯很多错误。写这篇文章让我意识到，召集一个领域专家小组（包括生物学、经济学、国际关系等）来写一个更好、更全面的版本是有价值的。最好将我的努力视为该小组的起点。

基本假设和框架

为了使这篇文章更加精确和有依据，明确我们所说的强大人工智能以及它的到来时间非常有帮助。我认为这种强大的人工智能可能在2026年出现，但也有可能需要更长时间。

在这篇文章中，我假设它会很快到来，并关注它出现后5-10年的情况。我假设这种系统的定义、能力以及它如何交互，尽管对这些方面存在争议。

我心目中的强大人工智能是一个 AI 模型，可能在形式上类似于今天的 LLM，尽管它可能基于不同的架构，可能涉及多个交互模型，并且可能以不同的方式训练，具有以下特性：

在纯粹智力方面，它在大多数相关领域（例如生物学、编程、数学、工程、写作等）比诺贝尔奖得主更聪明。这意味着它可以证明未解决的数学定理，写出极好的小说，从头开始编写复杂的代码库等。

除了作为一个“智能对话对象”，它还具有人类虚拟工作所需的所有“接口”，包括文本、音频、视频、鼠标和键盘控制，以及互联网接入。它可以通过这些接口执行任何操作、通信或远程操作，包括在互联网上采取行动、向人类发出或接受指令、订购材料、指导实验、观看视频、制作视频等。而且，它在这些任务上的技能再次超过了世界上最有能力的人类。

它不仅是被动回答问题；相反，它可以被分配需要数小时、数天或数周完成的任务，然后像一名聪明的员工一样自主完成这些任务，必要时再寻求澄清。

它没有实体（除了在计算机屏幕上存在），但可以通过计算机控制现有的物理工具、机器人或实验设备；理论上，它甚至可以为自己设计机器人或设备来使用。

用于训练模型的资源可以重新用于运行数百万个实例（预计到2027年，这与集群规模匹配），并且该模型可以以大约10倍至100倍于人类的速度吸收信息和生成操作。不过，它可能会受到物理世界或其交互软件的响应时间的限制。

每一个数百万个实例中的每一个都可以独立地执行不相关的任务，或者在需要时可以像人类合作一样一起工作，也许不同的子群体可以微调得特别擅长某些任务。

我们可以将其总结为“数据中心中的天才国度”。

显然，这样一个实体能够非常快地解决非常困难的问题，但弄清楚有多快并不简单。对我来说，两种“极端”观点都是错误的。首先，你可能会认为世界会在秒或天的时间尺度上立即被改变（“奇点”），因为优越的智能不断自我增强，几乎立即解决每一个科学、工程和操作任务。问题在于存在真实的物理和实践限制，例如构建硬件或进行生物实验。即使是一个新的天才国度也会遇到这些限制。智能可能非常强大，但它不是魔法仙尘。

其次，相反，你可能认为技术进步受现实世界数据或社会因素的饱和或速率限制，而且比人类更聪明的智能几乎不会增加什么6。对我来说，这同样不可信——我可以想到数百个科学甚至社会问题，在这些问题上，一大群聪明人会大大加快进展，尤其是如果他们不仅限于分析，而且可以在现实世界中实现事情（我们的假设天才国度可以这样做，包括通过指导或协助人类团队）。

我认为，真实情况可能是这两种极端图景的一种混合体，具体细节根据任务和领域的不同而有所变化。我相信我们需要新的框架来以一种建设性的方式思考这些细节。

经济学家经常谈论“生产要素”：例如劳动力、土地和资本。“劳动力/土地/资本的边际回报”这个短语捕捉到的思想是，在特定情况下，某个要素可能是也可能不是限制因素——例如，一个空军需要既需要飞机又需要飞行员，如果没有飞机，雇佣更多的飞行员并没有多大帮助。

我相信在AI时代，我们应该谈论“智能的边际回报”，并试图找出与智能互补的其他因素以及当智能非常高时成为限制因素的那些因素。我们不习惯以这种方式思考——问“变得更聪明对这项任务有多大帮助，以及在什么时间尺度上？”——但这似乎是概念化一个拥有非常强大人工智能的世界的正确方式。

我猜测的限制或补充智能的因素列表包括：

外界的速度。智能体需要在世界上互动地操作以完成任务并学习8。但是世界的运行速度是有限的。细胞和动物以固定速度运行，因此对它们进行实验需要一定的时间，这可能是不可压缩的。同样的道理适用于硬件、材料科学、与人沟通的任何事物，甚至是我们现有的软件基础设施。此外，在科学中，通常需要一系列实验，每个实验从上一个实验中学习或建立。所有这些意味着，完成一个重大项目——例如开发癌症治疗方法的速度——可能有一个不可压缩的最小值，即使随着智能的增加，这个最小值也不会进一步减少。

对数据的需求。有时缺乏原始数据，没有这些数据，更多的智能也无济于事。今天的粒子物理学家非常聪明，已经开发出了一系列理论，但由于粒子加速器数据非常有限，无法在它们之间做出选择。除了可能加快建造更大的加速器外，不清楚他们是否会显著提高。

内在复杂性。有些事情本质上是不可预测或混沌的，即使是最强大的AI也无法比今天的人类或计算机更好地预测或解开它们。例如，即使是非常强大的AI在一般情况下也只能在混沌系统（如三体问题）中向前预测一点9，而与今天的人类和计算机相比。

人类的限制。许多事情不能在不违反法律、不伤害人类或不扰乱社会的情况下完成。一个对齐的AI不会想要做这些事情（如果我们有一个不对齐的AI，我们又回到了讨论风险的问题）。许多社会结构效率低下甚至有害，但在尊重法律要求、临床试验的法律要求、人的行为习惯的改变或政府行为的限制下，很难改变。技术上成功但因法规或错误的恐惧而减少影响的例子包括核能、超音速飞行，甚至电梯。

物理法则。这是第一个观点的更严峻版本。有些物理定律似乎是不可打破的。不可能以超过光速旅行。布丁不会不搅动。芯片每平方厘米的晶体管数量有限，超过这个数量后会变得不可靠。计算需要一定的最小能量来擦除每个位，从而限制了世界上计算的密度。

在时间尺度上还有进一步的区分。在短期内难以克服的约束，长期来看可能会因智能而变得更具可塑性。例如，智能可以用来开发新的实验范式，使我们能够在体外学习以前需要活体动物实验才能实现的东西，或构建收集新数据所需的工具（例如更大的粒子加速器），或在道德限度内找到绕过人类限制的方法（例如，帮助改善临床试验系统，帮助创建新司法管辖区，使临床试验官僚作风减少，或通过改进科学本身，使人类临床试验变得不那么必要或更便宜）。

因此，我们应想象一种情景，其中智能最初受到其他生产要素的严重限制，但随着时间的推移，智能本身越来越多地绕过这些因素，即使它们永远不会完全消失（有些东西如物理定律是绝对的）10。关键问题在于这一切发生的速度及其顺序。

(以上为作者原文的第一部分，中译系由志愿者提供的社区服务项目，未经原作者授权，侵删。原文出处：https://darioamodei.com/machines-of-loving-grace）

Thanks to Kevin Esvelt, Parag Mallick, Stuart Ritchie, Matt Yglesias, Erik Brynjolfsson, Jim McClave, Allan Dafoe, and many people at Anthropic for reviewing drafts of this essay.

To the winners of the 2024 Nobel prize in Chemistry, for showing us all the way.

Footnotes

¹https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace ↩
²I do anticipate some minority of people’s reaction will be “this is pretty tame”. I think those people need to, in Twitter parlance, “touch grass”. But more importantly, tame is good from a societal perspective. I think there’s only so much change people can handle at once, and the pace I’m describing is probably close to the limits of what society can absorb without extreme turbulence. ↩
³I find AGI to be an imprecise term that has gathered a lot of sci-fi baggage and hype. I prefer “powerful AI” or “Expert-Level Science and Engineering” which get at what I mean without the hype. ↩
⁴In this essay, I use “intelligence” to refer to a general problem-solving capability that can be applied across diverse domains. This includes abilities like reasoning, learning, planning, and creativity. While I use “intelligence” as a shorthand throughout this essay, I acknowledge that the nature of intelligence is a complex and debated topic in cognitive science and AI research. Some researchers argue that intelligence isn’t a single, unified concept but rather a collection of separate cognitive abilities. Others contend that there’s a general factor of intelligence (g factor) underlying various cognitive skills. That’s a debate for another time. ↩
⁵This is roughly the current speed of AI systems – for example they can read a page of text in a couple seconds and write a page of text in maybe 20 seconds, which is 10-100x the speed at which humans can do these things. Over time larger models tend to make this slower but more powerful chips tend to make it faster; to date the two effects have roughly canceled out. ↩
⁶This might seem like a strawman position, but careful thinkers like Tyler Cowen and Matt Yglesias have raised it as a serious concern (though I don’t think they fully hold the view), and I don’t think it is crazy. ↩
⁷The closest economics work that I’m aware of to tackling this question is work on “general purpose technologies” and “intangible investments” that serve as complements to general purpose technologies. ↩
⁸This learning can include temporary, in-context learning, or traditional training; both will be rate-limited by the physical world. ↩
⁹In a chaotic system, small errors compound exponentially over time, so that even an enormous increase in computing power leads to only a small improvement in how far ahead it is possible to predict, and in practice measurement error may degrade this further. ↩
¹⁰Another factor is of course that powerful AI itself can potentially be used to create even more powerful AI. My assumption is that this might (in fact, probably will) occur, but that its effect will be smaller than you might imagine, precisely because of the “decreasing marginal returns to intelligence” discussed here. In other words, AI will continue to get smarter quickly, but its effect will eventually be limited by non-intelligence factors, and analyzing those is what matters most to the speed of scientific progress outside AI. ↩
¹¹These achievements have been an inspiration to me and perhaps the most powerful existing example of AI being used to transform biology. ↩
¹²“Progress in science depends on new techniques, new discoveries and new ideas, probably in that order.” – Sydney Brenner ↩
¹³Thanks to Parag Mallick for suggesting this point. ↩
¹⁴I didn’t want to clog up the text with speculation about what specific future discoveries AI-enabled science could make, but here is a brainstorm of some possibilities:
— Design of better computational tools like AlphaFold and AlphaProteo — that is, a general AI system speeding up our ability to make specialized AI computational biology tools.
— More efficient and selective CRISPR.
— More advanced cell therapies.
— Materials science and miniaturization breakthroughs leading to better implanted devices.
— Better control over stem cells, cell differentiation, and de-differentiation, and a resulting ability to regrow or reshape tissue.
— Better control over the immune system: turning it on selectively to address cancer and infectious disease, and turning it off selectively to address autoimmune diseases.↩
¹⁵AI may of course also help with being smarter about choosing what experiments to run: improving experimental design, learning more from a first round of experiments so that the second round can narrow in on key questions, and so on. ↩
¹⁶Thanks to Matthew Yglesias for suggesting this point. ↩
¹⁷Fast evolving diseases, like the multidrug resistant strains that essentially use hospitals as an evolutionary laboratory to continually improve their resistance to treatment, could be especially stubborn to deal with, and could be the kind of thing that prevents us from getting to 100%. ↩
¹⁸Note it may be hard to know that we have doubled the human lifespan within the 5-10 years. While we might have accomplished it, we may not know it yet within the study time-frame. ↩
¹⁹This is one place where I am willing, despite the obvious biological differences between curing diseases and slowing down the aging process itself, to instead look from a greater distance at the statistical trend and say “even though the details are different, I think human science would probably find a way to continue this trend; after all, smooth trends in anything complex are necessarily made by adding up very heterogeneous components. ↩
²⁰As an example, I’m told that an increase in productivity growth per year of 1% or even 0.5% would be transformative in projections related to these programs. If the ideas contemplated in this essay come to pass, productivity gains could be much larger than this. ↩
²¹The media loves to portray high status psychopaths, but the average psychopath is probably a person with poor economic prospects and poor impulse control who ends up spending significant time in prison. ↩
²²I think this is somewhat analogous to the fact that many, though likely not all, of the results we’re learning from interpretability would continue to be relevant even if some of the architectural details of our current artificial neural nets, such as the attention mechanism, were changed or replaced in some way. ↩
²³I suspect it is a bit like a classical chaotic system – beset by irreducible complexitythat has to be managed in a mostly decentralized manner. Though as I say later in this section, more modest interventions may be possible. A counterargument, made to me by economist Erik Brynjolfsson, is that large companies (such as Walmart or Uber) are starting to have enough centralized knowledge to understand consumers better than any decentralized process could, perhaps forcing us to revise Hayek’s insights about who has the best local knowledge. ↩
²⁴Thanks to Kevin Esvelt for suggesting this point. ↩
²⁵For example, cell phones were initially a technology for the rich, but quickly became very cheap with year-over-year improvements happening so fast as to obviate any advantage of buying a “luxury” cell phone, and today most people have phones of similar quality. ↩
²⁶This is the title of a forthcoming paper from RAND, that lays out roughly the strategy I describe. ↩
²⁷When the average person thinks of public institutions, they probably think of their experience with the DMV, IRS, medicare, or similar functions. Making these experiences more positive than they currently are seems like a powerful way to combat undue cynicism. ↩
²⁸Indeed, in an AI-powered world, the range of such possible challenges and projects will be much vaster than it is today. ↩
²⁹I am breaking my own rule not to make this about science fiction, but I’ve found it hard not to refer to it at least a bit. The truth is that science fiction is one of our only sources of expansive thought experiments about the future; I think it says something bad that it’s entangled so heavily with a particular narrow subculture. ↩

champaignmagazine.com