No. 98·中文精读

25 个 AI 居民的小镇,如今是财富 500 强的决策沙盘

Simulating Humans at Scale: Simile's Joon Sung Park

Simile 创始人 Joon Sung Park 认为前沿模型只是智能的 CPU,要模拟人类社会需要另一种模型。

节目音频00:00 / 38:45
中文精读由 AI 基于原始访谈改写,非官方翻译,可能存在偏差或错误;表述以原节目为准。

2023 年 2 月 13 日,斯坦福的一个虚拟小镇里,咖啡店老板 Isabella 决定办一场情人节派对。她提前一天出门张罗材料,挨个招呼熟客来玩。派对当天,一位叫 Klaus 的居民收到邀请,顺势约出了自己的暗恋对象。也有居民收到了邀请,转头就忘了。

没有人类写剧本。这 25 个居民全是 AI,做什么、说什么、跟谁交朋友,都是模型自己生成的。这个实验叫 Smallville,主持它的是当时还在斯坦福的 Joon Sung Park。

三年后,这个实验长成了一家公司,叫 Simile,定位是应用 AI 实验室,做的事情听起来更科幻:模拟人类行为和整个社会。客户名单里有 CVS,合作方里有盖洛普。他们用 1000 个模拟出来的美国人做验证,预测准确度达到了真人重测答案的 85%。

最近,Joon 在红杉资本的播客 Training Data 上,和主持人 Sonya Huang 聊了这门生意的来龙去脉。有几个判断,值得每个做产品、做研究的人听一听。

一场没人写剧本的派对

先回到 Smallville。Joon 团队的核心观察是:大语言模型的训练数据里编码了大量人类行为,来自网页、社交媒体这些地方,只要从合适的角度去问,就能挖出很多微观行为。给一个具体场景,问「这个人会怎么做」,模型能给出相当像样的答案。

他们觉得这就是造复杂智能体一直缺的那块原料,于是把这件事推到极限:给 25 个 AI 居民每人一份人设,配上记忆、规划和反思机制,让他们在一个游戏小镇里生活。早上起床,按习惯做事,上班,交朋友,跟真人一个样。

情人节派对就是涌现出来的。Isabella 的人设是咖啡店老板,她自主决定办派对、邀请顾客、布置场地。Klaus 借着派对把暗恋对象约了出来,两人真的去赴了会。整个过程没有人下达指令。

实验发成论文后,Joon 收到大量主动找上门的邮件。最早是社会科学家,想在这个平台上跑实验和随机对照试验。很快,财富 500 强的高管和董事也来了,他们看过演示,问题很直接:我们每年花大钱做调研,很多关于市场的问题今天根本没法回答,能不能在模拟里跑一遍?

从一篇综述到一条路线

Joon 的研究起点在 2020 年。那年 GPT-3 还没发布,他刚进斯坦福,跟着后来的联合创始人 Percy Liang 写了那篇著名的《基础模型的机会与风险》。写综述时,同事们惊讶于模型能做分类、能做简单生成,Joon 看到的却是另一件事:这些模型编码了人类行为。把这件事推到头,会是什么?

他出身人机交互里的社会计算方向。这个领域有个老大难问题:设计一个社交平台,界面好不好用容易测,但几十万、几亿人聚在一起会产生什么涌现现象,好坏都算,没法提前测。唯一的办法是上线实测。如果某个信息流设计更容易传播负面情绪,等你发现时,伤害已经发生了。

2022 年,也就是生成式代理论文的前一年,团队做了个叫 Social Simulacra 的项目。思路是:假设你要建一个 subreddit,怎么提前知道用户会在里面干什么?这对老手设计师都是难题。他们用 GPT-3 生成了几千个 persona 塞进去,模拟整个社区。有个模拟社区的主题是讨论匹兹堡去哪儿玩,跑着跑着,几个 persona 自己商量起来,组队规划了一次旅行。

回头看,这基本就是 2022 年版的 Moltbook,几年后这类 AI 代理社交网络真的火了起来,算是应验了。但当时团队更关心下一步:怎么从简单的 persona 升级成会随时间思考的复杂代理,以及怎么验证模拟结果靠不靠谱。

模型也在配合。GPT-3 时代没有指令微调,让模型听话得靠各种奇怪的提示词技巧。到生成式代理论文时,模型已经能听懂指令,代理才可能推理自己的记忆。Joon 的判断是,基础能力这一层,今天已经够了。

那为什么后来要离开实验室?Joon 的原话值得单拎出来:

研究是台广度优先的机器,实验室里一群聪明人各守一小块论题,有些会开出惊人的花。但研究者有个老毛病,我们出了论文就撒手了。公司是台深度优先搜索的机器,认准一座山,把资源和人聚起来,毫不犹豫往一个方向爬。

生成式代理论文发表半年后,他有了这座山。三位创始人,他本人、Percy Liang、Michael Bernstein,后两位都是他在斯坦福的导师,一起合作了五六年,正式凑在一起讨论「这能不能是家公司」。

今天的大模型,恰恰不擅长当人

接下来是整期播客里最有意思的一个判断。Joon 说,OpenAI、Anthropic 这些实验室的北极星,是造一台超级智能的理性机器,擅长解决有客观答案的问题。但人恰恰是非理性的,有主观的价值观、偏好和口味。

模型规模往上走,模拟人类行为的能力却没有同步跟上,目前的建模范式已经撞上平台期。这是 Joon 观察到的分叉:今天最强的模型,离「像人」这件事反而越来越远。要突破,需要一个专门为人类多样性设计的新前沿。

还有个更隐蔽的问题,叫言行差距。人说的和人做的,中间有条真实的缝。大模型的训练数据大部分是态度性数据,也就是大家在网上说过的话。用态度数据去预测行为,误差就出在这里。

Simile 的解法是收集行为数据,外加一类很特别的信息。他们 2024 年做验证研究时,会直接问受访者:讲讲你的人生故事。你从哪里长大、做过哪些艰难的决定,这类长尾信息,能架起一座从态度通往行为的桥。现在他们训练了一个访谈模型,用强化学习优化目标:花最少的时间,拿到关于一个人最大的信息量。至于政策立场这类离散选择题,还是用问卷,省时省数据。

对模型的定位,Joon 打了个比方。今天的前沿模型像智能单元里的 CPU,单一、理性、擅长解难题。Simile 要造的东西更像 GPU,很多小单元并行,每个单元代表一个真实子人群的观点。他们不需要超人级的模型,需要的是尽可能像人的模型。缺哪块就自研哪块,协调研究计划这类活儿则交给前沿模型。

数据上,他们手里囤着大批社会科学领域的随机对照试验,包括大量定价研究,正在训练一个字面意义上的「人类行为基础模型」:把 RCT 的行为信号编码进去,目标是能预测任何一项 RCT 的结果。客户那边也有想法,CVS 有 9000 万顾客,双方在谈怎么负责任地用上这些内部数据,造一个更贴合 CVS 人群的增强版模型。

CVS 是怎么用它的

拿 CVS 举例。买家是负责人类洞察的高级副总裁,读了 Joon 的验证论文后拍板要合作,因为他觉得自己被两件事卡住了:能实地测试的问题数量有限,以及人类社会的物理规律本身。问卷能问出态度,问不出一个决策在市场里的二阶影响。

合作怎么牵上的?这位副总裁的表亲认识 Joon。听起来偶然,但需求是真的。

流程的前半段和传统调研公司很像。客户说想理解某个人群,Simile 通过和盖洛普的战略合作去触达真人,用大约 15 分钟收集数据。诀窍在问题设计:哪些问题能在 15 分钟里最大程度还原一个人。拿到数据后生成模拟代理,加载到 SaaS 平台上,之后客户想问这群人什么都可以。

客户的使用路径通常从概念测试开始,这也是最直接的场景:新产品、新文案,扔给模拟人群听反馈。客户很快会意识到一件事,今天他们最多测 5 到 10 个想法,模拟可以把这个数字变成 1000 个想法乘以 1000 个子人群。

再往后是用法的深水区:

  • 给代理加上时间维度,让它真的去「体验」一个产品 10 分钟,回来报告看见了什么、感受如何;
  • 多代理模拟。让客户最意外的需求是模拟财报电话会,CEO 和董事会想知道听众会怎么反应,这个需求出现的频率高得超出 Joon 预期;
  • 二阶影响。一家车企的电动车卖爆了,市场对燃油车的看法会怎么变?整个产品线怎么平衡?这类问题今天没有任何工具能测,模拟可以。

Sonya 问了个尖锐的问题:为什么不直接投 1000 条 Facebook 广告,看真实的点击率?Joon 给了两个理由。一是规模和代表性,网上实验只有特定人群会响应,样本天然偏;Simile 花苦功夫保证人群的代表性,数量也不受平台人口限制。二是广告只能回答第一个问题,回答不了决策的下游连锁反应。他拿自动驾驶类比:先采真实路面数据,再让模型泛化到训练数据没覆盖的路况。一个道理,先见真人,再让模拟走得更远。

85% 是怎么算出来的

模拟人类,最要命的问题是:怎么知道它准不准?

Joon 团队做了这样一个验证:模拟 1000 个真实的美国人,覆盖美国人口结构。基准设计得很聪明,让真人隔一段时间重做同一份问卷,看他自己前后能对上多少,这是人类自洽度的天花板。模拟代理的预测准确度,达到了这个天花板的 85%。

日常运营的北极星指标是 TVD,总变差距离,衡量模拟出的答案分布和真实分布贴得多近。TVD 低于 0.15,他们认为是足以支撑决策的证据。这个阈值覆盖了客户的核心场景,包括各类 RCT。

多代理模拟更麻烦,误差会像传话游戏一样层层累积。Joon 把模拟分成两类。一类收敛,误差累积没关系,因为系统有强大的向心力。他举了网络科学的例子:只要按人类行为复制社交网络,一定会出现枢纽节点,这正是 PageRank 的核心观察,无论网页网络怎么形成,总有页面拿到指数级更多的链接。另一类发散,比如模拟一场选举,同样的输入跑不出同样的赢家,就像「一战是否不可避免」这类问题。发散类模拟的评估靠跑 100 次看分布,用 bootstrap 重采样算置信度,把各种可能结局都摆出来,让人看清机制、提前准备。

这个领域有多早期?Joon 的类比是,现在的模拟相当于推断统计的第一天。当年学界吵了很久才接受 P 小于 0.05 算科学证据,Simile 想做的,就是给整个行业定下这样的阈值和标准。

理论上限当然存在,人回答同一个问题,两次都会给出略有差异的答案,随机性是真实的。但 Joon 认为,距离上限还有大量可以榨取的性能空间。

一亿美元跑一次的模拟

聊到更大的图景,Joon 提了个人物:经济学家托马斯·谢林。上世纪七十年代,他用极简的代理模型研究种族隔离,每个代理只是一个红点或蓝点,每轮看看邻居颜色,同类占比低于阈值就搬家。就这么简单的规则,解释了宏观层面的隔离是怎么自发形成的。谢林后来拿了诺贝尔奖。

Joon 认为同样的机会现在摆在这代人面前,只是代理从红蓝点升级成了有完整人生故事的「人」。当年他就是这样说服两位导师入伙的:这件事做成了,诺贝尔奖是可以想的。

已经有经济学家来问过这些问题:银行挤兑什么时候发生?气候变化的死结是各国的集体行动困境,能不能模拟?一个民主政体崩溃前有什么信号?货币制度的起源是什么?

他甚至给这个领域画了一条成本曲线:今天的模拟又快又便宜,但可以预见一种模拟,跑一次要花一亿美元、耗时几个月,跑完能解开一个社会的基础问题。政治也一样,一项政策改了,未来五到十年会发生什么,各方都有一套说辞,模拟至少能把所有下游影响摆上桌。

结尾聊到未来,Joon 说自己受科幻小说影响很深。技术成熟的科幻社会里总有两根支柱,一根是 AGI,另一根是指导社会的模拟。五年前他不敢想,现在他觉得自己有机会去搭第二根。

联合创始人 Percy 常说,最伟大的科学突破往往始于一次了不起的测量。哈勃望远镜改变了人类理解宇宙的轨迹。模拟可以成为人类社会的哈勃望远镜。

Joon 自己的说法更直白:造一台人类社会的 CERN。自然科学已经被测量工具反复刷新,社会科学还缺它的那台机器。

明天就能用的三件事。大多数读者暂时用不上 Simile,但这期播客里有三个提醒,今天听完明天就能试:

  • 做用户调研时,少问态度题,多问行为和经历。「你会买吗」和「讲讲你上次买这东西的经历」是两种问题,后者离真实选择近得多。实在没时间,试试「讲讲你的人生故事」,Joon 团队验证过,这是信息密度最高的一类问题。
  • 测新想法时,把候选数量放大一个量级。今天测 5 个想法的预算,在模拟里够测 500 个。先广撒网,再挑几个去实地验证,顺序别反。
  • 重大决策拍板前,除了「用户喜不喜欢」,强制自己列一遍二阶影响:这个产品成了,会怎么改变用户对我其他产品的看法?市场认知会怎么动?列不出来的部分,就是你的认知盲区。

言行之间的缝,模型补得上,前提是先承认这条缝存在。这是 Smallville 那场情人节派对之外,这期播客留下的最实用的提醒。

文稿来源:sequoiacap.com;机器转写可能有识别错误,人名与数字请以音频为准。
id="podcast-transcript" data-framer-component-type="RichTextContainer" style="transform:none">IntroJoon Sung Park: I am somebody who is quite inspired by science fiction. And when you read science fiction that covers societies that have progressed far enough in its technological maturity, you always see two pillars. You have some version of AGI, and you have some version of simulations that really help guide the society. I do see an opportunity today to really take a first crack at building the simulation. I would not have said that even five years ago, but that is a conviction that we have built up over the years as we’re going deep into this research. Main conversationSonya Huang: Today we’re delighted to have Joon, founder and CEO of Simile. Simile is building an applied AI lab simulating human behavior and societies. And I’m very excited to have you here to discuss what you’re building. Joon Sung Park: Same here. Thank you for having me. Sonya Huang: Okay. Take me back to April 2023, Stanford, California, specifically Smallville, Stanford, California. What was that? Joon Sung Park: So Smallville was a project that we were running at Stanford, where the idea was that we made this observation that large language models can now encode a lot of human behavior that is embedded in its training data, from the web and social media and so forth, that if you sort of probe at the right angle, you can actually get a lot of microbehaviors out of these models. So given a very specific demonstration or description of a situation, what would person X do? And it would actually generate really interesting behaviors. We found that to be so interesting. And we found that to be the ingredient that we had been waiting for for creating really complex agentic behaviors. So Smallville actually was an experiment where we decided that if we push this as far as possible, what would a society that is created by these agents look like? So we basically created generative agents that is paired with a generative AI model with memory, planning, and reflection to basically create this lived experience of agents living in this small town. So Smallville was basically a game town of 25 agents living in it. Individual agents had a description of persona, but they would actually wake up in the morning, do their routines, go to work, actually have relationships, sort of like people would, and they would actually have emergent phenomena, like having parties and so forth. So that was the experiment that we ran. Sonya Huang: What was the most surprising thing to come out of the experiment? Joon Sung Park: So one of the surprising things was—so the experiment, the simulation itself actually took place the day before Valentine’s Day. So you actually see these agents, one of the agents actually thinking, “Well, I run a cafe.” So she’s a cafe owner. Her name’s Isabella. She goes and thinks, “It would be great if I can do a Valentine’s Day party where we invite a lot of friends, customers.” So you actually see her on the day before Valentine’s Day going around, actually gathering materials for the party, actually telling her customers, “Hey, we’re going to have this party, please come.” And on the day of Valentine’s, you actually see this immersion party that actually gets formed with all these agents coming to the … Sonya Huang: Did anyone not get invited? Joon Sung Park: Well, some of the people did get the invitation, but they forgot. That’s one thing that did happen. Some of the agents did not explicitly get invited, but we had one agent who got the invite, Klaus, who decided to ask his crush out on a date. So he would actually bring in the date, they would actually have a party at this cafe. So quite surreal. Sonya Huang: So how did you end up building Smallville in the first place? Were you studying kind of human psychology and social behavior, or was this coming from—was it coming from the kind of customer back, or was it coming from the technology out? Joon Sung Park: So my particular team has been excited about simulations, and we saw the vision of simulation failure early on. So my career as a researcher at Stanford really started back in 2020. That was the year when GPT-3 was about to come out. It wasn’t quite there yet, but it was just about to come out. We started to get its first demos. And my first year, we wrote this paper called “Opportunities and Risks of Foundation Models” alongside many of the Stanford researchers, and was led by one of my co-founders, Percy Liang, who is now the head of the Center for Foundation Models at Stanford. And when we were writing that, the part that I was really focused on was well, here’s a new class of models that we have not seen in the past, that these models that can be very generalizable in ways we didn’t quite have in the past. And I got into thinking, well, if we can imagine the kind of interaction we can create with these models, what would that be? And many of my colleagues back then were surprised that these agents or these models can do classification or simple generation. And that was really incredible to see because these models didn’t really know or didn’t really—wasn’t really taught to do that. But the part that was surprising to me wasn’t that these models can do that, because from an interaction perspective, we’ve known how to do this for a long time. The interesting part was, these models can actually encode human behavior. What does that mean if we were to push this as far as possible? So part of the tradition I come from in research included what we call social computing. And social computing within human-computer interaction really has to do with this idea of how can we build a better technological platform that would enable social interactions and collaboration? One of the most difficult challenges of building a social platform is not necessarily testing the UI/UX of the system, but it’s more about when you have tens of people, millions of people, and down the line billions of people, how do all these people come together to create the immersion phenomenon that’s both good and bad? And how can we design for scale? And so far, we didn’t really have a tool that would enable us to test for that. The only way we test it today is you basically field test it, you release your prototype, see what happens. And sometimes it actually comes at a real cost. Obviously, it’s high cost in terms of human hours and the time it takes. But at the same time, if you have a bad design, imagine you have a feed on social media that is more likely to propagate certain emotions that are negative, then obviously, that is something that we want to avoid. But this now gets tested in the field. So we wanted to see whether we can actually create a simulation that would actually let you test for this. So 2022, this was actually a year before generative agents, we worked on a paper called “Social Simulacra,” which actually really was the precursor to the agent paper that we ended up writing. The core thesis was: imagine you’re building a subreddit. You’re a designer on a subreddit. You want to see what people might do in the subreddit, which is a surprisingly hard task, even for practiced designers. And we basically decided, hey, we have this model, seems unique. Let’s use this model to create simulations of the entire subreddit. So you define the goal, you define the moderation strategies, and you populate it with thousands of—back then we didn’t call them “agents,” but we called them “personas,” but populate it with thousands of personas. This is basically ’22 version of Moltbook, which is quite interesting that it actually came back. And when we saw that, we actually got a lot of really important insights out of this. What are the good behaviors? We actually simulated a community where the entire idea was for people to discuss with each other the places to sightsee in Pittsburgh. And all of a sudden, you start to see these personas actually collaborate to actually discuss, “Hey, XYZ places are amazing. Do you want to actually go on a trip together?” And actually plan those trips live in this simulated subreddit. So that’s how we got excited. So we saw the vision and the excitement and the potential applications fairly early on. But then the work that we had to do was then demonstrating how can we go beyond simple personas to create complex agents that actually can think over time, because we want to simulate the longitudinal aspect of our society, and then actually validating that these simulations are actually accurate in practice. Sonya Huang: Was there a point of model evolution at which you felt like, okay, we’re there, the models are good enough for us to actually have a faithful representation of human society? Joon Sung Park: So GPT-3, when it came out, and “Social Simulacra” was built with GPT-3, and it was very janky. It didn’t do any instruction tuning. It did not follow your instructions. So just to have it listen to you and do what you wanted it to do, you had to do some weird tricks with prompting and so forth. But you could actually see the promise. The model actually had encoded a lot of human behavior, and you could actually see the trajectory. And when we had the generative agents paper, it wasn’t quite ChatGPT, but we now had instruction tuning. So we could actually build much more complex agents that can reason about its memory. That wasn’t really possible when we did “Social Simulacra.” And since then, of course, the models have improved. So where we are today is the models at its foundational level have reached a point where we can actually imagine building these kind of applications. Now the part that actually I do think, however, that’s quite interesting here, today, if you look at many of the large language model companies, whether it’s OpenAI, Anthropic, and many of the new labs that are getting formed, the models they are creating are models that I would consider to be—their North Star to be something that is similar to, let’s build a super intelligent machine. These machines are meant to be rational, and these machines are supposed to be really amazing at tackling problems that have an objective answer. Sonya Huang: So maybe that’s not even the best simulation of true human society then. Joon Sung Park: Turns out, people are irrational. We have a lot of subjective values, preferences and tastes. So you actually start to see divergence in model size going up, and the performance in its ability to predict and simulate human behavior. So we have sort of plateaued with current modeling paradigm, our ability to really simulate humans. So it is sort of at this starting good foundational level, but to make it really amazing, we do need the next frontier that is more geared towards actually modeling people’s diversity. Sonya Huang: Very interesting. At what point did you realize that what you did with Smallville could become a company? Joon Sung Park: Right. So again, the promise of application was something that I was very much inspired by early on by simulation with “Social Simulacra” and so forth. But the part that I realized over time is research and a company have very different functions. Research is an amazing vehicle if you want to basically do breadth research. You are in a lab surrounded by a really smart set of people. And each of the researchers own a small piece of thesis. And they go explore, some of those thesis blossom into amazing research products. But we’re not necessarily known for finishing our job. We’re not usually the ones to bring that research impact to the real world. A company is a machine for depth-first search. You have a conviction on an area, you find a hill that you want to climb. This is the vehicle that lets you put together resources and an amazing group of people to go after a singular vision without hesitation. And we got that conviction, I would say about half a year after generative agents. After the original generative agents paper, we got so much inbound interest initially from actually social scientists who wanted to run their experiments and all the RCTs on our platform. Then very soon after, many of the Fortune 500 companies who saw this demo and their board members and CEOs who sometimes visit Stanford saw that, and they started asking, “Well, we go run all of these surveys and experiments, and there’s so many research questions about the market that we cannot answer today. Can we run that in simulation?” And that started to really intrigue me, because that showed a clear line towards a real-world impact for research, which is not always the case that we have that kind of opportunity. So that is when we decided we actually want to validate the simulations are accurate, so we went out and actually created simulations of a thousand people of the US population. We demonstrated that using our architecture and the models, we can actually predict people’s behaviors 85 percent as accurately as people replicate their own. When we saw that, we thought okay, this is something that we feel comfortable providing to our users as a platform for simulating their really important decisions. So that’s when the co-founders—myself, Percy, as well as Michael Bernstein, who was a researcher and my advisor at Stanford—both of them were actually my advisors. So the three of us had been working together for five years. And now at this point, similarly, six years. But that’s when we got together to have the initial conversation of can this be a company? Sonya Huang: Got it. Amazing. Maybe walk me through a customer engagement end-to-end today. Like, who’s a canonical customer and in which department? And they come to you, what are they asking you? And what product do you—or service do you deliver to them? Joon Sung Park: Right. So maybe an example that I can give to make this concrete. So CVS has been partnering with Simile for the past, I would say, nearly half a year. And they’ve been an amazing partner. The way we initially got in touch with them is so our main buyer at CVS is the lead—it’s a senior VP who leads human insights. And the original story there he basically read my paper that validated the agent simulations and thought, “We have to bring this to CVS. Because today, we are bottlenecked by the number of questions we can field test. And we’re also bottlenecked by truly the physics of human society.” It’s one thing to ask surveys and experiments, but a totally different thing if down the line, you actually want to simulate the entire market, and actually map out all the second-order impact of the decisions you suggest to your leadership. So he’s been looking around for that solution. And his cousin happened to know me. And basically told our buyer, Sri, that the authors of the paper are actually looking to start something. So that’s how we got connected. And in this particular engagement, usually the way this goes is our customers are very much used to working with polling companies or panel companies today. And there they go and basically ask these companies, XYZ are the populations that we are interested in better understanding. Can we go run a research study of these topics? That initial stage looks very similar for Simile. So our buyers come and they tell us, we want to better understand XYZ population. Then Simile goes out and we have—through our partnership with vendors—we have a strategic partnership now with Gallup, for instance, who is a polling and panel company, where we go out, work with our vendors to actually reach out to real humans. So these simulations are grounded in real data, but reach out to those people, collect data that we believe are efficient and generalizable about that person. So imagine you have 15 minutes. What are the magical questions you can ask these people during that time? We collect that data, use that data to create agents or simulations of these people that can basically be used to answer a large number of questions that goes way beyond the original domain. We load it onto our platform, and it’s basically a SaaS product. Our customers come and they can basically ask any questions about the group of people of their interest. Sonya Huang: So interesting. It reminds me of autonomous vehicles. You know, you go and collect a bunch of data from the road and then you’re able to augment it with simulation. Is this a similar concept, or are there big differences to what you’re doing here? Joon Sung Park: It is a similar concept in the sense that, of course, with the self-driving vehicles, you want to create a model that is based on real-world physics, but you want to create a model that is generalizable beyond your training data. It needs to be generalizable into different locations with different weather conditions. Very similar concept, where what we want to create is we want to reach out to real people. And these people want to understand something fundamental about these people in a way that we cannot code into the model. Sonya Huang: I would’ve thought that the large language models would be such a good representation of the whole world that you could almost narrow it down. You could tell Claude you are a 34-year-old woman living in a bicoastal metropolitan area, and it would be able to have a faithful representation. So I’m actually surprised that you go out to Gallup. Maybe can you just explain why you have to go out and collect any real-world data at all? Joon Sung Park: Yeah. One of the big questions here is the question around say-do gap. There are things that people say, and then there are things that people actually do. And the gap there is real. And a lot of the large language models are trained on attitudinal data. Fundamentally, it is the things that people have said online that does cover a large quantity of its training data. So one of the things that Simile’s simulation platform does is actually closing that gap. So a lot of the data that we end up collecting by nature are behavioral. It also includes data that actually goes into literally questions like, “Just tell me the story of your life.” Turns out, if we understand the person’s story of your life, the kind of data you get from it is what we consider to be the long-tail information about this person. It’s not about what you’ve done in this particular moment. It’s not about very broad questions like what’s your view on politics? It’s about where you grew up, what were some of the difficult decisions you had to make in life. And what’s interesting about this data is it’s an amazing way to build a translational layer between attitudes and behavior. So we combine these kinds of datasets, but fundamentally, that’s the gap that we want to close. Sonya Huang: What sort of behavioral data do you have? Joon Sung Park: So Simile does run a lot of experiments. So kind of models that we have trained, for instance, we have a huge repo of RCTs, so randomized controlled trials that were run in social scientific contexts, that were run around pricing studies. So one of the models that we are training is basically the foundation model of human behavior in quite a literal sense. We have all the behavioral signals from RCTs. Can we actually encode that into the model so that the end outcome is a model that can basically predict the results of any RCTs? At the same time, one of the conversations that we keep on having with our customers that we’re very excited by is our customers then come in, see that potential, and their mind goes to, “Wow, we have 90 million customers, let’s say, here at CVS. How can we leverage this kind of data to create better simulations?” So there’s also a conversation around how can we, in a responsible and ethical way, leverage existing data that is also in-house for our customers, and then use that to create an augmented version of Simile’s model. So that, of course, is going to be more fine-tuned, specific to the population of these customers, but that’s the kind of data that we will be leveraging. Sonya Huang: I see. And are you doing these interviews typically by voice? Is it a survey that you fill out? What’s the modality? Joon Sung Park: So it’s a huge breadth. The quick answer here is it’s both. Interviews are fantastic if you want to get the long-tail information about people. So we actually do—in the original study that I conducted back in 2024, we’d literally ask the question, tell me the story of your life. Now the way we do it is we are training our own model, so it’s a reinforcement learning loop, but basically imagine the objective function here is how can you spend the minimum amount of time to get the maximum amount of visibility about this person? So that is one of the things that we do. So basically training an interviewer that is not really asking for factual information or an experience about a particular platform, but just what are the life stories that people have that can be used to train our own model for these agents? And then for the more factual or sort of more discrete choice questions, surveys, and so forth. These are also very efficient. These are time and data efficient, because people can fill out many of the questions in short periods of time. So for those, we actually do leverage them. For instance, if you want to just have a broad understanding of people’s viewpoints on certain topics, certain policies, and things like that. Sonya Huang: You describe yourself as an applied AI lab. How do you think about where you want to build your own models versus where you want to rely on other existing models? Joon Sung Park: So in terms of building our own model, really the core thesis here is there is an amazing model to be built that really encodes the diversity of people’s values, preferences and tastes in ways that simply a rational model cannot do. So one way I actually pose this, we’re sort of building—so imagine today’s models are akin to the CPU of the intelligence unit. It’s a single model trained on amazingly rational data that is amazing at solving very complex, objective questions. Simile’s model is much more akin to developing something that is closer to the GPU of the intelligence unit, where the idea here is we don’t actually need a model that is superhuman at Simile. In fact, we want a model that’s as human as possible, but we want to make sure that these models at the sort of individual subunits can represent the real viewpoints of different subpopulations. So where we see that gap, that’s when we go develop our own model. But at the same time, we do leverage frontier models, for instance, as a way to coordinate the research. Frontier models are amazing at coming up with a research plan. So that’s where those models actually do get leveraged. Sonya Huang: Very interesting. Are people typically coming to you with questions around new product launches? How they should be marketing their companies? Pricing? All of the above? Joon Sung Park: So it is all of the above. Sonya Huang: Okay. Joon Sung Park: Our customer journey usually does, however, start with very concrete use cases and problems they are trying to solve. Concept testing is a big one. It’s also a very straightforward one. So they have a new concept, new product idea, new market message they want to test. And they want to hear from their users what they would think about XYZ. This is one way for them to quickly test those ideas. And then the promise they quickly see is well, right now we’re very much in the practice of testing five to ten different ideas at most. But what does it look like for us to test instantly 1,000 different ideas across 1,000 different subpopulations? That’s the initial vision they see. Then we really get into the nitty-gritty details of well, where does simulation go from here? They then pretty soon start asking well, can this be used to do product testing, but not just simply submitting, like, an image, but imagine basically asking these agents, go experience this product for 10 minutes, and tell us about what you experienced, what you saw. So you’re basically adding temporal dimension. Then you go into things like multi-agent simulation. Some of our customers very routinely actually ask us to simulate their earnings call. This is actually a use case that both surprised me at first, but this is also surprisingly a common ask. Because of course, the CEOs and board members always need to think about hey, how are we going to design our earnings call? How would the audience react? So that is something that we also do. And this is very much a multi-agent simulation. Sonya Huang: It seems like there’s so many use cases that could potentially be tested once you have, like, a simulated almost customer population, right? Joon Sung Park: Yeah. Sonya Huang: I’m curious about the value of research and testing in sim versus just like, let’s say you have a new product concept that you want to test. Why not just go run 1,000 Facebook ads and you actually get the click-through rates on this stuff? Isn’t that real-world data almost more useful than the simulated data on how people might behave that you then correct for with your own models? Joon Sung Park: So it’s a great question. And I think to some extent here, the answer has to do initially with scale. And then down the line, truly the new capability that comes because you can simulate interactions. The scale question here is actually quite straightforward, where yes, you can absolutely run Facebook ads and Facebook testing. But the kind of experiments that you can run in simulation is actual behavior simulation at scale. And so you can basically pull in any number of users, doesn’t even have to be bounded by the number of population that’s available on Facebook. And it’s also much more representative, because only certain groups of people will actually respond to the online experiments. But Simile, the model that we are creating, one of the key promises is that it is representative. We do the hard work of actually getting the representative set of people, and then collecting the data that would actually represent them properly. So the scale representativeness is something that many of our users do not have easy access to. This is actually one of the common asks also that we do get, or common pain points that we have heard, where the question that many of these people have isn’t about what questions do we ask these people? But it’s about in the first place, how can we get to the population that we’re excited to talk to? That’s a huge bottleneck. Then down the line, you can actually really start to imagine—and this is something that our customers and some of the most forward-looking customers are now going into—which is, what are all the downstream implications of the decisions that you make? It’s not just about imagine you have this particular product. Do you like it or do you not like it? Would you pay for this, not pay for this? It’s not necessarily just the initial questions that we want to answer and finish. But we want to understand—imagine you’re a car company, you launched an electric vehicle in this market. Maybe the electric vehicle does really, really well. So we can help you do concept testing around marketing and the product around the electric vehicle. But what does that do to the perception of, let’s say, non-electric vehicles? Does it change the market perception? Then what does it mean for the rest of the product line? And how do you balance those kind of second-order impacts of your decision in a way that is more evidence-based? Today, there’s no way to test for this. Sonya Huang: Yeah. Joon Sung Park: You can run this in simulation. So really going beyond simply asking one question at a time, but then to think about what are the long-term implications of your decisions is something that our customers are quite excited by. Sonya Huang: I’d love to understand how you think about how predictive your model is in actually simulating real human behavior. I imagine you have a lot of evals on this. I guess, what is your North Star metric? How do you guys do on that? And what do you think is the theoretical limit? Joon Sung Park: It’s a great question. So theoretical limit—and let me just start from there—certainly does exist in the sense that humans are genuinely—there’s a lot of randomness that if you ask me the same question, I’ll actually answer the question slightly differently. So there is certainly that degree of randomness in human behavior. However, there’s a lot of gains in performance that we can have even today in the way we’re predicting people. So the measurement that we do is—so at the level of population, we measure the distribution of responses if it is more quantitative. So we actually measure total variance distance, which basically shows how close are the distributions of the ground truth versus the simulated information. And that is a metric that we run across all the use cases that our customers have. And we have certain thresholds that we believe is good enough for decision making. So TVD of let’s say less than 0.15, we believe, is actually quite strong evidence for making decisions. So that is the North Star state that we want to hit for this class of use cases that are more quantitative, that’s more question and answers. This also does cover RCTs, which is many of the core use cases our customers have. Now there’s actually a really interesting question to ask around, what about multi-agent simulation? What about all the downstream implications that we’re going to be simulating? What does the evaluation of those look like? Sonya Huang: Yeah. And then do you daisy-chain errors as you kind of—you know, if this one is 85 percent accurate and then this agent is telling another agent something and they’re—do you accumulate errors as you go towards multi-agents? Joon Sung Park: Exactly. And one of the core theses here is we basically see two categories of simulations. One simulation is what I would consider to be simulations that converge. The other categories of simulations are the simulations that diverge. And sometimes they actually coexist. And it’s really about what research questions do you have. Questions that converge, doesn’t actually matter if you have a little bit of error. Now the error cannot be obviously so dramatic that it sort of is completely detached from reality. But you actually are okay, even if the errors do compound over time, because the pull towards the convergence is strong enough that you’ll actually understand where everything would fall. A good example here actually is if you simulate a network of people, then that network will always have a hub that gets formed. This is what network scientists would call the scale-free network, for instance. This is actually what powered Google, too. One of the core observations of PageRank was that it doesn’t matter how these networks actually get formulated, you actually see some web pages that get exponentially more links that are attached to it. This is a very fundamental behavior in humans that we also see in simulated networks. And that convergence always happens as long as you’re replicating human behavior with certain threshold accuracy. Now there are questions that generally do diverge. It’s like your classical questions like was World War I inevitable or was it not? And there, it is sometimes difficult to run the same simulation over time and get the same exact outcome. Imagine you’re running a—this is not something that necessarily Simile right now is going into, but imagine you’re running a simulation of an election. Will the same person win the election every time? There are a lot of downstream implications of every single decision that does happen. So it does diverge. There, the core evaluation is around confidence. So imagine you run the simulation 100 times. How many of those times do the results come out to be X? And how can we actually use that to basically create a bootstrap resampling to calculate the confidence around the simulations? Those are some of the questions that we do ask. And a huge part of this also that the power of simulation, is then to show when it diverges, to show the diversity of possible outcomes so that people can actually look, understand the cause or mechanism of how we got to those outcomes, and prepare for those futures. So those are some of the implications of divergence in simulations. Sonya Huang: Are there any mathematical descriptions of why something would converge or diverge? I imagine if you have an average function, maybe you converge. And then if you’re splitting outcomes to a binary, then you might maybe diverge. Joon Sung Park: Yeah. So the intuition I think is close. And technically, this is also a research topic. So Simile is a company where we do go deep into this research topic, in the sense that I see simulation as a field that’s akin to developing your day one of inferential statistics. Inferential statistics scientists actually had to do a lot of discussion and research over time to decide that P less than 0.05 is actually evidence that is strong enough for science. Simile is working on setting the same kind of threshold and standards for the rest of the field. So those are the intuitions. I think that’s exactly the right intuition. In terms of actually how to make a robust mathematical equation around, like, what’s going to happen when, it is a real research frontier for simulations. Sonya Huang: Thank you for being nice about my vibe mathing. Joon Sung Park: [laughs] Sonya Huang: I’m curious, you know, it seems like a lot of Fortune 500s coming to you. I’m wondering whether there are non-existing corporate use cases that are great mysteries of our society that might become solved. And for example, I’m wondering about economics, central bank decisions. Oftentimes, I personally believe in macro, nobody knows nothing. And oftentimes a lot of the issues come about from human psychology. So to me, macroeconomics is a function of simulating human behavior at scale. You know, I’m thinking even in the venture capital use case, we often debate internally, does value accrue to this company or not? You could run the simulation of all the different layers of the AI stack and almost figure out where durability and value accrues. If you had a kind of perfect simulator of human behavior, there’s so much more you could do than serving the Fortune 500. Do you agree with that? And then if so, are you serving governments and the like? Joon Sung Park: Yeah. So it’s interesting. When we were still researching in this area, the way I actually got, back then, my advisors, Michael and Percy, excited about this was I basically told them, “Look, we do this right, there’s a Nobel Prize to be won there.” And I truly believe that. And it’s also not surprising in that your classical economics simulations, things like agent-based models that really pioneered our understanding of back in the day, the kind of topics they studied was: how does segregation happen? What are the causal mechanisms for segregation? So scholars like Thomas Schelling would actually build agent-based models that are extremely simple and rudimentary, but that showed something deep about human macro behaviors. And he, of course, went on to win a Nobel Prize. I see the same opportunity here, but in an augmented way, where back in the day, the agent-based models were very much deterministic in some sense, where you basically in this simulation of, let’s say, model of segregation from, like, 30 years ago, individual agent was simply red dot or blue dot. And every game iteration, they would look around its corner, see how many of its neighbors are of the same color. And if that threshold goes below a certain threshold, then they will decide to move to a new location. And that was it. But now we can actually create real agents that replicate the full richness of individuals and run the same kind of simulations. So the kind of questions that we can ask that goes beyond simply the commercial use cases, for instance, in the context of macroeconomics, actually, the questions that I actually did get asked from economists were things like: When does bank run happen? Or questions like climate change. One of the core blockers of climate—like, solving that issue is the collective action problem of many nations. Can we actually simulate that? Or what are the signals of a democracy that is about to collapse? Can we understand the origin story of the monetary system? These are the kinds of simulations that I do believe ought to be the North Star state of this field. And it is sort of interesting to imagine what that would actually look like in practice, right? Because these would involve very large-scale simulations with many agents interacting with each other. I do see a future where—today, this is certainly not the case. Today, a simulation is quick and fast to run. But what about simulations that take actually $100 million to run once, and could take many months to run? But when we run it, it solves one of the fundamental questions of our society. That I do think is genuinely a very exciting possibility for this field. Sonya Huang: Mm-hmm. I’m even thinking politics, for example, could be forever changed. Today everyone has an agenda of how they say some policy change will impact things. Well, why don’t we just run the simulation? Joon Sung Park: And understand all the downstream implications, and not just what’s going to happen this year, but what does it mean in the next five to ten years? Sonya Huang: Fascinating. I was going to close by asking you what makes you excited about the future? Is it what we just talked about? Or is it something else? Joon Sung Park: I am somebody who is quite inspired by science fiction. And when you read science fiction that covers societies that have progressed far enough in its technological maturity, you always see two pillars. You have some version of AGI, and you have some version of simulations that really help guide the society. I do see an opportunity today to really take a first crack at building the simulation. I would not have said that even five years ago. But that is a conviction that we have built up over the years as we’re going deep into this research. And what’s exciting is there’s a clear use case today that can serve our users. But then there’s a lot of innovation that is yet to come that I do think will build up to actually building a simulator that’s akin to the CERN of human society. And one of the things that my co-founder Percy sometimes says is you look at the greatest scientific innovations, they often start from an amazing measurement. Hubble Telescope really changed the trajectory of how we understand the universe. Simulation can be that for human society. So the thing that does excite me, there’s a lot of focus on natural sciences, but how can simulation really unlock our understanding of humanity and social sciences? And how can we actually use that to make our society be a better place? That’s exciting. Sonya Huang: Totally. I remember reading somebody was excited about, you know, there’s a small but breathtaking chance that the field of economics as we know it may actually become solved by simulation. And I’d extend that not just to be economics, but kind of everything that deals with human behavior and social sciences, which ultimately is everything around us. Joon Sung Park: Truly. Sonya Huang: Wonderful. Thank you so much for joining today and sharing the story of both Smallville and what you’re now up to at Simile. I really enjoyed the conversation. Joon Sung Park: Same here. Thank you for having me. <div class="framer-mcvrsd" id="

The race to build superintelligence is producing models that keep getting better at objective problems, but not at behaving like actual people. Joon Sung Park, founder and CEO of Simile and creator of Stanford's "Smallville" generative agents study, argues that simulating human society requires a fundamentally different kind of model. He frames today's frontier models as the "CPU of intelligence"—rational, superhuman at problems with right answers—and Simile as creating the "GPU of intelligence," built to encode the diversity of people's values, preferences, and tastes. It simulated 1,000 Americans and predicted their behavior 85% as accurately as people reproduce their own answers. CVS uses it for concept testing; some customers simulate their own earnings calls. Joon's larger bet: a "CERN of human society" that could one day model bank runs, climate cooperation, or the early signals of a collapsing democracy.

Hosted by Sonya Huang, Sequoia Capital

本集资源与出处