Foundation models– LLMs or LLM-like tools– are a compelling idea for advancing scientific discovery and democratizing computational science. But there’s a big gap between these lofty ideas and the trustworthiness of current models.
Youngsoo Choi of Lawrence Livermore National Laboratory and his colleagues are thinking about to how to close this chasm. They’re engaging with questions such as: What are the essential characteristics that define a foundation model? And how do we make sure that scientists can rely on their results?
In this conversation we discuss a position paper that Youngsoo and his colleagues wrote to outline these questions and propose starting points for consensus-based answers and the challenges in building foundation models that are robust, reliable and generalizable. That paper also describes the Data-Driven Finite Element Method, or DD-FEM, a tool that they’ve developed for combining the power of AI and large datasets with physics-based simulation.
You’ll meet:
- Youngsoo Choi is a staff scientist at Lawrence Livermore National Laboratory (LLNL) and a member of the lab’s Center for Applied Scientific Computing (CASC), which focuses on computational science research for national security problems. Youngsoo completed his Ph.D. in computational and mathematical engineering at Stanford University and carried out postdoctoral research at Stanford and Sandia National Laboratories before joining Livermore in 2017.

From the episode:
Youngsoo is the lead author of this position paper: Defining Foundation Models for Computational Science: A Call for Clarity and Rigor. LLNL posted a news article discussing this research: Shifting foundations: an AI paradigm emerges in computational science.
We discussed the finite element method and reduced-order modeling, tools computational scientists have used to organize and simplify problems so that they can apply mathematical tools to simulate them. With the finite element method, researchers define a problem and break it into small geometric cells. Traditionally those cells were defined by physics equations and could be assembled to get a large-scale understanding of the whole structure.
The data-driven finite element method (DD-FEM) takes that idea, but incorporates AI. Youngsoo says, “Instead of being handcrafted, each block is learned from data. You can then reuse these smart blocks to build larger and more complex systems.”
Related episodes:
Check out our Season 6 episodes to hear from a range of researchers about foundation models for science
For more on reproducible and reliable computational models, check out our conversation with Paulina Rodriguez.
Featured image: A contour plot of a device created via the finite element method. By Ajay B. Harish – Own work, CC BY-SA 4.0, Link
Transcript
We used otter.ai to produce this transcript. A human corrected word errors, capitalization and punctuation.
Sarah Webb 00:03
In this season of Science in Parallel, we’ve been focusing on AI for science with several episodes on foundation models. These data-driven tools are similar to the large language models that support ChatGPT and other generative AI apps. This episode talks about some of the practical challenges in applying today’s technology to achieving tomorrow’s goals.
Sarah Webb 00:30
I’m your host, Sarah Webb, and in this episode, I’m speaking with Youngsoo Choi, a computational scientist at Lawrence Livermore National Laboratory. Youngsoo works on foundation models and AI and he coauthored a position paper with a group of 11 other researchers calling for increased scientific rigor and reproducibility in foundation models and clearer parameters for defining them. Youngsoo and I talked about the differences between foundational methods and foundation models. And I asked him about the Data-Driven Finite Element Method, or DD-FEM, a strategy that he and his colleagues have developed to construct foundation models for science. You’ll find a link to their paper in this episode’s show notes at scienceinparallel.org.
Sarah Webb 01:26
Youngsoo, it is great to have you on the podcast.
Youngsoo Choi 01:29
Thank you for having me.
Sarah Webb 01:31
So, Youngsoo, we’ve been talking about foundation models a lot on the podcast, and they’re obviously a hot topic in computational science. What do you find exciting about this area of research?
Youngsoo Choi 01:42
Foundation model is a hot topic, as you said. If we can first talk about foundation models in AI context, foundation models in NLP, natural language processing, or computer vision are already transforming everyday life. You can see this in how large language models empower tools like ChatGPT or Gemini and many others, or how vision models enable self driving cars advanced medical imaging or creative tools in design and art. In many ways, they democratize access to these capabilities, skills like coding, analysis or design, that used to require years of training are now accessible to a much broader community. They also show broad generalization and often surprising emergent abilities, skills that weren’t explicitly trained, but arise from scale and diversity in training.
Youngsoo Choi 02:50
Extending this idea to computational science is very exciting. We are talking about extremely complex domains, fluid dynamics, material science, climate modeling, even healthcare, where solving problems from scratch has traditionally taken huge amount of time, data and specialized code. A true foundation model for science could carry over that broad generalization into these domains, so that instead of reinventing the wheel for every new problem, we can start from a strong, reusable base. The impact could be transformative. It could shorten discovery cycles in materials and drug development, enable more reliable climate projections, help engineers design new systems faster and democratize access to high performance simulations that today require years of expertise and access to supercomputers. In other words, it has the potential to change the pace at which science itself is done, moving us from one-off models to shared infrastructure that accelerates research across the board.
Sarah Webb 04:15
So what types of research projects have you been working on in this foundation model area?
Youngsoo Choi 04:22
Our group, the libROM team, at Lawrence Livermore National Lab, develops efficient surrogate models for computationally intensive scientific simulations. What makes us unique is that we are rooted in the traditional expertise of projection based reduced-order model so called ROMs, R-O-M-S, which are already data driven framework, but firmly grounded in numerical methods for solving PDE, partial differential equations. From there, we are extended our approach by incorporating the latest machine-learning, AI capabilities, exploiting neural network expressivity, transfer learning and automatic differentiation in PyTorch and so on. We worked on various applications, accelerating 3D printing process simulations, earthquake carbon capture simulations, various hydrodynamics, radiation particle transport and other kinetic simulations for inertial and magnetic confinement fusions, and porous media simulations and many others.
Youngsoo Choi 05:36
These projects are what led us directly into foundation model-related projects. In a sense, ROMs, reduced-order models, give us a natural bridge. They are reusable. They capture essential dynamic across many physical problems, and they can be scaled by marrying these very characteristics with modern machine learning. We’ve been working toward frameworks that aren’t just good for one narrowly defined simulations, but that can generalize across domain and conditions. A concrete example is our Data-Driven Finite Element Method, DD-FEM, project, which we introduced in the recent position paper. The goal there is to create a physics-grounded, reusable base model that can adapt across different PDEs, partial differential equations, geometries and boundary conditions, very much in the spirit of a foundation model for science. Beyond DD-FEM, our team has ongoing projects that explore scalable surrogate modeling for multiphysics systems, physics-informed operator learning and reduced-order approaches that integrate AI to achieve broad reusability. So in summary, our work started from ROMs, reduced-order models, but natural progression into scalable, generalizable and physics-consistent surrogates has led us to explore what foundation model could mean in computational science and to propose frameworks that make them real.
Sarah Webb 07:24
Well, you know, I came across your work because of this position paper and this DD-FEM, which you were just talking about. So let’s take a step back and talk a little bit about that paper. This position paper looks at some of the lack of clarity and the challenges in building rigorous models. How did the team come together to work on this paper? And why did you decide to write it?
Youngsoo Choi 07:51
I mean, as I said, the concept of foundation models is powerful, and their potential impacts on science potentially enormous. But the term foundation model itself has often been used inconsistently in computational science community, sometimes just as a buzzword, just to get attention from people. This creates a kind of semantic mess, actually. People mean very different things by it, that, in turn, cause confusion and risks diluting the scientific rigor we want to maintain. If we cannot agree on what qualifies as a foundation model, it becomes very difficult to benchmark them, to evaluate their performance fairly, or to even know what we are comparing against. Actually, when you go out and ask people in the computational science community, because you are doing the podcast on how they would define a foundation model for our field computational science, you will get a very wide range of answers. Some will say they don’t know really, because it’s a new concept. Or others might say any pretrained model, trained on a big data set, is a foundation model, but then the model might only work in a very narrow setting. Still others will point to models that are impressive in one physics regime, but that don’t really generalize across different domains. So there’s no shared understanding.
Youngsoo Choi 09:34
That’s a problem for a field that cares deeply about mathematical rigors, reproducibility, and long-term impact. That’s why we decided to write this position paper. Our team came together because we saw the term being applied loosely in both academic papers and industry talks without clear criteria. Yeah, we felt it was the right time to pose and ask, What should a foundation model mean in computational science? What traits should it absolutely have, and how should we evaluate them? By laying out a definition, a list of desirable characteristics, and even an example framework, like the Data-DrivenFinite Element Method, DD-FEM, we wanted to give the community a starting point, a reference that sparks discussion, debate and ultimately consensus.
Sarah Webb 10:35
So one set of terminology that I’d like to sort of talk about before we even get into how you’re defining foundation models, is this distinction between foundation models and foundational methods? I mean, there are lots of terms in computational science that use these words, but in really different ways. So can you talk a little bit about that before we dive into the foundation model part?
Youngsoo Choi 11:00
Yeah, the foundation models, or foundational methods in computational science has a long history, actually, just like the finite element method or finite volume methods, they are considered to be foundational methods in computational science. They are grounded in first principles such as physics laws, Navier-Stokes, Maxwell’s equations, or conservation of mass or momentum and energy, and build the methods on the top of the mathematical rigor, right. Applied math and computer science, etc, their universality comes from underlying core structure, meaning that you can apply FEM, for example, finite element method, to fluid problem or to solid mechanics, or even to stress analysis and bridges and buildings and many other problems without changing the underlying core structures. You just change the input files or problem setup, and the structure is very reusable for different problems.
Youngsoo Choi 12:09
These traits are indeed similar to foundation models in the AI context, like large language models. However, these foundational methods, finite element method or finite volume methods, don’t rely on training data. Instead, they rely on governing equations, physics, law, first principles, basically, physics dictates the structure entirely. By contrast, foundation models, in the AI sense, large-scale, pretrained models built from data: Their power comes from generalization. They capture patterns across very diverse training sets and then adapt to new tasks without being retrained from scratch. They aren’t derived from first principles, but from exposure to massive and varied data. So the key distinction is, foundational methods are principle-driven in computational science, while foundation models are data-driven in AI community. The importance of making this clear is that in computational science, we now have to reconcile these two different concepts or traditions. How do we bring the vigor and universality of foundational methods together with the flexibility and generalization of data driven foundation models? That’s exactly the space our paper is trying to define.
Sarah Webb 13:45
So we’ve been building up to actually defining foundation models for computational science. So how did you define foundation models?
Youngsoo Choi 13:54
Yes, I mean, we came up with our own definitions. We are not claiming that this is going to be the definitions, but as I said in the beginning, we just want it to start, provide the starting points. So here is our definitions. As I said before the foundation models, it’s got to be data-driven, just like in other AI applications, meaning that it must learn representations from scientific data, not just hand-coded rules, right? First Principles. Second, it has to be trained on broad distribution of scientific application types, of physical systems, not just one narrow things. Third, it must show wide generalization capabilities, meaning it works well for the things that it has not seen across different scientific problems, maybe fluid dynamics or solid mechanics or electromechanics, electromagnetism. Or across different computational domain, different geometries, or across different tasks or predictions or inverse problems. So it has to show wise generalization capabilities on those different fields, problems and tasks and fourth, without needing retrain from the scratch or a major structure modification. The whole point is that it must serve as a reusable base. That’s basically our definitions we came up with.
Sarah Webb 15:35
We’ve already been talking a bit about how that’s challenging, building rigorous, reliable, generalizable foundation models. Let’s talk through those challenges.
Youngsoo Choi 15:47
Building rigorous, reliable and generalizable foundation models for computational science would be very challenging indeed. It is because computational science has some unique characteristics that are different from natural language processing or computer visions. Here are four mountains we need to climb. There can be more, but I prepared four of them. The first: enormous data point sizes a single simulation output can be a 3D field with millions or billions of values. Each data point can require gigabytes or terabytes of memory, which makes training or even storing data sets very challenging. The second: high fidelity and physical constraints, unlike text or images, scientific data must respect conservation laws, some physical properties laws, stability, conditions and physical consistency. A model that looks good numerically may still violate the science which is not good, right? The third: diverse domains and tasks. Foundation models for science cannot just work in one physics regimes. They need to generalize across fluid dynamics, materials, climate or fusion and across tasks like fault simulations, control or inverse design, which is very challenging task to accomplish. The last finally, the mathematical rigor and reproducibility in science, it’s not enough for a model to perform well. We need error bounds, uncertainty quantifications, verifications and reproducibility to trust results. So while it’s possible to define a foundation model as one that generalizes broadly without retraining, actually building such a model in this setting is much harder than in NLP natural language processing or computer vision. That’s the gap our paper is trying to address. Instead of listing all these challenges and leaving the readers curious about is this model even exist? We actually presented a framework, potentially powerful one, called the DD-FEM, that can realize all the requirements that a true foundation model must have.
Sarah Webb 18:23
Well, let’s talk about this Data-Driven Finite Element Method, and how does it serve as an example to address these concerns?
Youngsoo Choi 18:34
Yeah, very exciting. DD-FEM, Data-Driven Finite Element Method is a framework which was inspired by classical finite element method, foundational methods in computational science, as you may know well, the FEM can be described in two ways, top down and bottom up. The top down is when you break down a global domain into small pieces, like a Lego pieces. Then you introduce polynomial basis for each piece to represent the solution in that small pieces. And then you do bottom up assembling many local pieces together to form a global domain, potentially big one, and then solve underlying physical simulations, like you actually solve underlying physics governing equations. This has very interesting analogy with how LLMs work. Large language models break down paragraphs and sentences into small pieces, that is words or tokens, which is top down right, and then apply attention mechanism or reasoning structure to form a completely new sentences or paragraphs which is bottom up. So it’s a analogous between the DD-FEM and LLMs; for example, you can think of FEM as building with Lego blocks, where the blocks are mathematical functions.
Youngsoo Choi 20:07
Now DD-FEM, Data-Driven Finite Element Method, keeps that idea, but makes each block smarter. If you’d want to think that way, instead of being handcrafted, each block is learned from data. You can then reuse these smart blocks to build larger and more complex systems. The power is then, once trained, these blocks obey physics by design, and they can be rearranged to solve new problems without retraining everything from scratch. One of the great features of DD-FEM would be its ability to decouple the data generation and training of the local data-driven basis from global assembly and solution process. Since they are decoupled and the data-driven basis is only local representations, you do not have to generate physics data from on global domain. You only need to generate data on small subdomain, which is easy to generate tons of data, meaning big data. This solves enormous data-point size challenges, which we talked about before. Once DD-FEM trains data-driven basis and form a data-driven element, it can now take the bottom up step that is assembling the data-driven elements, small pieces to form a global domain, big stuff and solve underlying physics problem, just like the traditional finite element method. Now you just heard that DD-FEM solves the underlying physics right? This means that the solution of DD-FEM will obey the physics law, by construction, which basically solved the second challenge I mentioned, that is satisfying physical constraints.
Sarah Webb 22:08
So it sounds like as opposed to having to brute force train the whole model with a unbelievably large dataset, if I’ve understood you correctly, you can train component pieces and bring all of that together.
Youngsoo Choi 22:25
Yes.
Sarah Webb 22:26
Oh, that is so cool.
Youngsoo Choi 22:28
Yeah, yeah, exactly in engineering, a lot of times we deal with a very complex geometries, right? I mean, the bridges, buildings and these days, fusions are very hot topic like tokamaks and inertial confinement, those target capsule simulations. All these geometries are complex and it’s hard to model, but traditional finite element method can handle and rigorously solve those problems on those complex geometry because they use this small pieces like element like so called tetrahedral or triangular elements. And those elements are on structures so that it can deform in a way that can when they are assembled together, they can form a complex geometry without a problem. It’s almost complete and perfect shape. They can do that with a traditional finite element method. DD-FEM, yes. I mean, you take that concept and adapt that small unit cells, like tetrahedral or the triangles, and instead of using like handcrafted polynomial basis to represent the solutions in each small pieces like tetrahedral or triangles, we use the data-driven approach to come up with a much higher quality basis representations on that small pieces so that you can enlarge the element size, implying that you can solve much larger problems than traditional final element method. And there is some stability concept. If you can take the larger elements, then that means you can take the larger time step if you are dealing with a time-dependent problem, which again, implies you can solve much longer time period simulations. So it’s not just that it solves the problem of the generating the large-scale data encountered by traditional black-box AI models, but also it brings many other advantages over traditional classical the finite element method or finite volume method, you know, the classical foundational methods in computational science.
Sarah Webb 24:53
So Young, Soo, so what are some other characteristics that DD FEM has?
Youngsoo Choi 24:58
Yeah, well, there are some. Several characteristics of the DD-FEM, and those characteristics actually address the challenges we talked about before. For example, The third challenge like which is generalizations to diverse domains and tasks, because the data-driven basis in DD-FEM is local. It is highly reusable, meaning that they can be reused to form rectangular domains or circular domains or any complex geometries. Also a single data driven basis can be used for fluid problems, structural problems, or even multiphysics problems. For example, fluid structure-interaction problems. And finally, the fourth challenge we mentioned that is mathematical rigor, because DDM framework follows closely the classical FEM framework, and it does solve underlying physics, rigorous error bounds, convergence and stability analysis can be done in a similar way that traditional FEM establish those mathematical analysis. As a matter of fact, we have published several papers that demonstrate the capabilities of DD-FEM. There we show the rigorous error-bound analysis, which gives us an upper bound of prediction error, the bounds for how bad the solution could be. So those are the characteristics of the DD-FEM.
Sarah Webb 26:30
Assuming that we can build foundation models for science, for computational science, that fulfill these important criteria, that are physics based and reliable and all of these important features, What do you see as the potential impact of that for scientists, for the scientific community?
Youngsoo Choi 26:52
Yeah, well, maybe not only for science and scientific community. Maybe this foundation models, if really realized in computational science, it may allow broad democratizations of the overall science field itself like that’s the one of the things that excites me a lot about foundation models in computational science, its potential to really democratize science. Well, I mean, as you know, traditionally, if you wanted to run high-level simulations in fluid dynamics, material science or climate modeling, you needed access to supercomputers, specialized solvers and years of training. That means only a handful of well-funded labs. Like, I mean, fortunately, I’m I belong to one of them, Lawrence Livermore National Lab, or the universities, could fully participate. But with foundation models that barrier may start to come down.
Youngsoo Choi 28:01
If we can build reusable, truly reusable physics-grounded models, then domain scientists, chemists, engineers, even students in undergrad or graduate school, can use these models without needing to reinvent a solver from scratch. Just like large language models have made coding or data analysis more accessible and foundation models could make advanced simulation tools available to people who don’t have deep numerical training right? So it’s also about access. Imagine having shared repository of pretrained models, so researchers at smaller institutions or in regions without supercomputing resources can still run meaningful simulations, maybe on their laptops, that could really accelerate overall scientific discovery, that contributions will not only come from, you know, well trained scientists, but also from ordinary, broad communities. And I think it will change the way we collaborate with shared model infrastructures, multiple groups could contribute different data or modules, and the whole community benefits. It’s very similar to what open-source software has done, but with the foundation models, much broader effect is going to bring. So democratizations here means lowering the expertise barrier, expanding access globally and accelerating the pace of discovery, which is very exciting.
Sarah Webb 29:50
Youngsoo, thank you so much for your time. It’s been such a pleasure talking with you.
Youngsoo Choi 29:55
Thank you so much for having me, Sarah.
Sarah Webb 29:58
To learn more about Youngsoo Choi and DD-FEM, and for more episodes about foundation models for science, please check out our website at scienceinparallel.org. Science in Parallel is produced by the Krell Institute and is a media project of the Department of Energy Computational Science Graduate Fellowship program. Any opinions expressed are those of the speaker and not those of their employers, the Krell Institute or the U.S. Department of Energy. Our music is by Steve O’Reilly. This episode was written and produced by Sarah Webb and edited by Susan Valot.