To celebrate achievement computational collaboration, Science in Parallel turns to the annual Gordon Bell Prizes, sponsored by the Association of Computing Machinery, and a finalist team that completed the largest-ever simulation of a fusion blanket, part of a fusion reactor. Work on this scale and at the edge of what’s possible requires technical expertise and teamwork at the highest levels.
Hear about four members of this team about the trust they’ve built over many years of developing software and using it to simulate complex physical systems on leadership-class supercomputers.
You’ll meet (clockwise from top left):
- Yu-Hsiang Lan of the University of Illinois, Urbana-Champaign (UIUC)
- Luiz Aldeia Machado of Penn State University
- Aleksander Dubas of the United Kingdom Atomic Energy Authority (UKAEA)
- Misun Min of Argonne National Laboratory

From the episode:
The Association for Computing Machinery has sponsored the annual Gordon Bell Prize since 1987.
In this project. the researchers modeled a water-cooled lead-lithium fusion blanket design for ITER, a large-scale international fusion experiment being constructed in France. It’s the first complete high-fidelity model of this system, including two billion spectral elements and two trillion grid points. The simulations required exascale systems, and were carried out on the Department of Energy’s Aurora system at Argonne Leadership Computing Facility and Frontier system at Oak Ridge Leadership Computing Facility.
These four guests are part of 16-member team who are all listed in the SC26 program abstract. During our conversation, these guests mentioned a few others by name including Elia Merzari of Penn State, Paul Fischer of UIUC and Andrew Davis of UKAEA.
Misun talked about the early development of Nek5000 and NekRS. She also mentioned the Department of Energy’s Exascale Computing Project (ECP). She, Elia and Paul were part of the design project CEED, the Center for Efficient Exascale Discretization. And Elia also worked on the ExaSML project.
Misun, Elia and Paul have all worked on several Gordon Bell Prize submissions in recent years, a 2023 finalist project.
Featured image on homepage: Rupert Eardley-Brunt, UK Atomic Energy Authority
Transcript
Transcript prepared with otter.ai followed by human copyediting.
Sarah Webb 00:04
It’s November on Science in Parallel, and time for the annual supercomputing meeting held this year in Chicago, Illinois. At the meeting, the Association for Computing Machinery will award the Gordon Bell Prizes, which honor outstanding achievements in high-performance computing. Often these projects use the world’s largest and fastest machines to solve important science and engineering problems. I’m your host Sarah Webb, and our podcast series on computational collaborations wouldn’t be complete without an episode discussing one of the finalists for the 2026 award.
Sarah Webb 00:43
In this episode, we’ll talk about exascale multiphysics for liquid metal fusion blankets, a project that used code developed at Argonne National Laboratory to do a complete, detailed simulation of a fusion blanket, part of a reactor, this conversation includes four researchers who worked on this project, bringing together their expertise in software engineering, computational mathematics, and computational physics. They are Yu-Hsiang Lan of the University of Illinois Urbana-Champaign, Luiz Aldea Machado of Penn State University, Aleksander Dubas of the United Kingdom’s Atomic Energy Authority, and Misun Min of Argonne.
Sarah Webb 01:31
I spoke with them in late October before the conference and before the winner was announced. Our conversation highlights a combination of important project features, exascale computers, cutting-edge code, and a significant research problem, fusion energy, that showcases the hardware and software. We also talk about the trust built over years of collaboration that made this project possible.
Sarah Webb 02:05
Welcome to all of you to Science in Parallel. First of all, I want for our audience to have an understanding of the physics problem that you’re working on. What are fusion blankets? Why are they important? And why do we need a big simulation to understand this computational problem?
Aleksander Dubas 02:24
My name is Aleksander Dubas, and I’m the head of computational fluids at the UK Atomic Energy Authority. So I think I can answer that one. Fusion is the ultimate source of energy, and if we can crack the problem, we can have effectively unlimitless amount of energy and solve all the world’s energy problems, but to do that there are a number of challenges that we need to overcome, and blankets are a big part of that challenge. So the favored fusion reaction that we want to take is the deuterium-tritium reaction, and deuterium is bountiful. You can find it in the water, which makes up 70% of the Earth’s surface, that’s great.
Aleksander Dubas 03:02
The tritium is slightly harder to come across, and one of the ways which we can close that fuel cycle is to breed our own tritium on site in the fusion power plant. And this is one of the things that the fusion breeder blanket does. So we take the energy from the fusion reaction use that to take lithium and split it into helium and tritium, and then we feed that back in. The other aspect of the breeder blanket is that most of the energy in the fusion reaction of deuterium and tritium comes out in neutrons, which aren’t confined by the magnetic field. So the breeder blanket has a secondary function of absorbing these neutrons, turning that energy into thermal energy that we can then use to drive steam turbines and generate the electricity.
Sarah Webb 03:49
Would anyone else like to add to what Aleks had to say?
Luiz Aldeia Machado 03:51
My name is Luiz, Luiz Aldeia Machado. I am an assistant research professor here at Penn State, and I’ve been working with multiphysics, multiscale modeling for about five six years.
Luiz Aldeia Machado 04:05
So answering the second part of it, because we are dealing with quite complex problem that will with like different physics that will be in different like spatial and temporal scales. It’s not a simple problem to solve, and this requires like this large-scale physics simulation, so the complexity of the problem kind of required us to go towards this like coupled models where we have like multiple physics interacting with each other of the time. So these end up making the problem like hard at solving on small machines.
Yu-Hsiang Lan 04:38
I guess I can add a little bit. So I am Yu-Hsiang Lan. I’m a graduate student at University of Illinois Urbana-Champaign, and I’ve been working with the team for several years as well. And I mainly focus on software development, numerical methods, and bring everything to scale.
Yu-Hsiang Lan 04:58
The liquid. Metal used inside the breeder blanket is opaque, so it is very hard to do experiments and measuring the flow profile as the traditional experimental procedure. And liquid metal is also very expensive and quite dangerous. So having a numerical tool to have a one-to-one model will save a lot of time in terms of development and also the validation process.
Sarah Webb 05:25
To work on this complex problem, the team relied on NekRS, the latest version of a computational fluid dynamics code that researchers at Argonne National Laboratory have been working on for decades.
Misun Min 05:39
My name is Misun Min. I’m senior computational mathematician at Argonne National Lab. I have been developing Nek code, which is including Nek5000, NekRS, and NekCM, NekLBM. So I’ve been working on developing this code for more than 20 years.
Sarah Webb 05:57
NekRS was developed for use on a variety of GPU-based supercomputers, as part of the Department of Energy’s Exascale Computing Project, commonly known as the ECP. Misun has worked with Elia Merzari of Penn State and Paul Fischer of the University of Illinois Urbana-Champaign on the ECP. Both Elia and Paul are among the 16 co-authors on this Gordon Bell paper, too. Leading up to this submission, Misun worked with Elia and Paul on a series of Gordon Bell submissions, including a 2023 finalist project that used NekRS to model a small modular nuclear reactor. She talked about how these projects on large DOE supercomputers led to this new research.
Misun Min 06:46
Since there are going to be different exascale architectures based on NVIDIA, AMD, and Intel GPUs, we have to consider portability and also performance. So we already had an OpenACC-based GPU version of Nek code, which ran on the full machine on Titan, we needed a more highly performing and portable version of the code because some of the machines weren’t going to support OpenACC. So it was necessary to write a new code, and that is how we started developing NekRS code. And when we had the first version of NekRS that could run on all of Summit machine, the world fastest computer at the time, we ran nuclear applications, including full core pebble-bed systems that Elia brought. We submitted the result in 2021 for Golden Bell entry. So these submissions have proven to be very good team-building exercises. During an ECP meeting in 2023, Andrew Davis from UKAEA showed interest in our ECP work and our new solver development on magnetohydrodynamics or MHD, which can be used for simulating liquid metal flows, such as in fusion tritium breather blankets. Most of the application problems require large-scale computing on supercomputing systems. So we have been always applying for INCITE and ALCC allocations.
Sarah Webb 08:14
Aleksander talked more about how he joined the project.
Aleksander Dubas 08:18
I think for me, my personal journey, it starts about eight years ago. I joined UKAEA along with Andy Davis, and we had this ambitious digital engineering program of simulating all the physics in a tokamak, excluding the plasma, just to skip it easy, you know, at the full scale, fully 3D. And we did some calculations, and we decided this this needs approximately an exascale computer, which at the time we didn’t have, and I don’t think anyone had at that point. And then fast forward a few years, and Andy gets in touch with the folks at Argonne, and we spin up this collaboration, giving us the access to the compute power we need. And then we get to this submission, where we’ve implemented a huge amount of the physics that we’re actually going to need to solve the problems at the scale we need to. We’re not quite at the full power-plant scale, but we’ve done so much of the physics. It’s really incredible to see that actually come alive.
Sarah Webb 09:14
Luiz started working with the team while he was a Ph.D. student working in Elia Merzari’s research group at Penn State. And although Yu-Hsiang is currently a PhD student at UIUC, he has been working with Misun and Paul Fischer for 10 years. He started as a predoctoral student researcher at Argonne National Laboratory, and he’s the first author of this Gordon Bell submission.
Yu-Hsiang Lan 09:41
For me. So I again, I also work with the team for quite a bit of time, and as Misun mentioned, it’s a good team-building activity and also a good training exercise. So we regularly run our code on leadership class supercomputers and developing the code. Testing it at the full-machine scale, and that has become part of my training so that we implement new algorithms, new features, and also make the code more robust with the features that application needs. And throughout this work, I have again experiments developing the new solver within NekRS, and this time it is the magnetohydrodynamic solver, and so I guess for my role of this project, I was mainly developing and validating the solver and make sure the simulation works as expected and performance meet the requirement.
Sarah Webb 10:35
Aleksander from UKAEA.
Aleksander Dubas 10:38
For us, it was a question of we’ve got this new inductionless MHD solver. We’ve got this new neutronics coupling capability, and what’s the problem that we can demonstrate the coupling of these at scale? That’s actually also a useful bit of information for a fusion engineer. And I think from that we came to the conclusion that this blanket geometry is the way forward with the physics that we put in,
Sarah Webb 11:01
And then Luiz from Penn State.
Luiz Aldeia Machado 11:03
I would say that’s also a matter of finding a problem that is scientifically challenging, and requires like a large scale simulation. Some problems we can solve with just like little sections of the problem. Sometimes you need to go the whole thing and adding like large amounts of physics and this increases the computational cost of everything, so it’s kind of natural as we start increasing the size of the problem. Then, oh, okay, I think this is kind of looking like something that would be an interesting submission for the Gordon Bell.
Sarah Webb 11:33
And finally, Yushang from UIUC.
Yu-Hsiang Lan 11:37
Right. So I think it’s also like depends on the background. So around 2021, we entered the GPU supercomputers era, and at some point we realized, okay, it is very, very good to solve a larger problem. And I think we also realized at some point we can simulate the entire reactor core. That is like no one has seen that before before the GPU supercomputers, so we just realized okay this is very good to solve a larger problem and opportunity for coupling new physics to the solver. And once we have this capability of solve larger and more complicated problem, and then we turn to the application that require these computational resources.
Sarah Webb 12:21
So I’d like to sort of set the scene of how this collaboration works day to day. I mean, obviously you are at four different institutions, and you’re just four members of a much larger team. What does the collaboration look like? How do you communicate with one another? Give me a sense of what that’s like from each of your perspectives.
Sarah Webb 12:42
Luiz.
Luiz Aldeia Machado 12:43
We use a lot of Slack. We have a Slack channel that we use to communicate. And so, for example, if I’m doing a post processing, I’m generating new figures or an animation, and I want to know the opinion of the thing, I will post it on Slack, and then we will open it and do suggestions and change things here and there. When it comes, for example, to the Penn State side of things, because we have like multiple members of the project from Penn State, it is easier for us. We can just like go to each other off and talk and discuss. But Slack was fundamental. Breaking down tasks between groups was also essential, in my opinion. Same, for example, you take care of A, you take care of B, and then we we try to converge the ideas. Also, we have frequently meetings where we discuss and you update the whole thing about know what was going on and the progress.
Sarah Webb 13:37
Aleksander.
Aleksander Dubas 13:38
So we often have weekly meetings to discuss our updates and work out what we’re doing over the next sort of weekly sprint and put things together. And then when it comes to things like a crunch time, like that that week before submission, we obviously need to communicate much more quickly. And so we’ve got the Slack for that. It works quite well working with the UK and the US, so we kind of work tag-team and shiftt. So we’ve got the UK working day, and then we hand over to the US working day. It allows us to make progress in sort of two steps every day.
Sarah Webb 14:09
And Yu-Hsiang.
Yu-Hsiang Lan 14:10
Yeah, I don’t think it’s a real special way to communicate with each other. We just find who we are trying to talk with, and then just use whatever that works the best based on the experience. Like email works fine, Slack works fine. Oh, I also like the way we put stuff on the GitHub, so everyone can just push their progress to the same locations, and you can check the history of the commit history. That’s very convenient.
Sarah Webb 14:36
And Misun.
Misun Min 14:37
Oh yeah, as well others mentioned, we have many weekly meeting set up, so we will weekly based meeting and also as needed basis would also quick phone call and quick Zooms and so on. So there are many way that we could interact very frequently.
Aleksander Dubas 14:54
And of course, annually there’s a NekRS user meeting.
Sarah Webb 14:57
What’s an example of a challenge in this project that you guys had to sort out using applied math and using your physics, using your computational science? How did you basically come together, bring your expertise together to achieve this?
Misun Min 15:14
So, main task involve several components: complex geometry, large mesh simulations with multiscale and multiphysics component coupling, performance, and also visualization, and from hardware part, the system upgrade sometimes makes things not working. So we have to also integrate the system support, and allocations to run in priority is also important. So that we always had to secure that part. So when we are going through the large problem size, for example, we also have to upgrade our code for supporting larger number of element count. We also have to make sure validation-verification for our new solver. So on the ECP, we’re tasked with running on every machine in the DOE exascale portfolio, and we always like to achieve 0.5 second per time step or less at 80% parallel efficiency, and also the visualization of the result at this scale is also major challenge. Yu-Hsiang and Luiz can speak to this.
Luiz Aldeia Machado 16:15
Yeah, we have some of the output, or let’s say for each field file that we generate that we can use, for example, for an animation. We are talking about files that can range from hundreds of gigabytes to like hundreds of terabytes of data for each file. Post processing all of this is quite challenging. We had to develop some strategies on how to break down this data, just to be able to access it and see what’s going on. When we say the scale of the problem itself was a challenge, is because it was really a challenge. Even like storing, like keeping the data that we need to run the post processing was difficult because we are talking about sometimes simulations that we’re using like four or five hundred terabytes of data to get like a few bunch of like data points.
Sarah Webb 17:09
Yu-Hsiang, would you like to go next?
Yu-Hsiang Lan 17:11
Okay, so it’s a very complicated device from geometry, physics, numerics, and scales. And to be honest, I don’t have the specialty to cover all of it, so we just have to break down each component. Like for physics, we talk to people who understand MHD physics, and we regularly have a meeting discussing about validation and whether the solution looks physical. And for performance, we talk to Paul, Misun because we have collected the performance on NekRS on several machines, so we know how to understand the performance of the code and just have to figure out if it’s reasonable for this particular problem at this particular scale. And yeah, as Luiz mentioned, size becomes a huge challenging part. So we also have to think ahead. So we develop some features to make our life easier. So one of them is the on-the-fly mesh refinement. So meshing is a very challenging part, and handling big files is also very error-prone. You can easily make mistake by just copy and paste the data, and therefore developing workflow that can simplify it and make it more robust is also very important. And the other thing is, I would say, in situ visualizations, because we really don’t have time to analyze every single data. So to have an ability to quickly sample the simulations at runtime that help us save the time debugging and give us a quick overview of what’s going on for the simulations.
Sarah Webb 18:48
Aleksander.
Aleksander Dubas 18:49
Yeah, the in situ visualization, I think that solved a number of challenges. I remember in a previous year we were working on some visualizations. I think about four days before the deadline, and our cluster went down, or at least the visualization nodes went down, and so we had a absolute scramble, tried to find a friend with a cluster that was working. And we did actually manage to find another cluster, move the data across, and get the visualizations done, but it was a it was a challenge, and with the in situ visualization, that’s nicely solved.
Sarah Webb 19:23
Anything else that you would highlight, and in addition to what the others have said?
Aleksander Dubas 19:27
Yeah, coupling can be a challenge. So, for example, we’ve got fluid activation calculation, where we’ve got neutronics, which has a Monte Carlo method in a Monte Carlo space, and we need to translate this into an input that we can then solve in a finite element space, and to do so in a way that is accurate is a challenge. And then managing that coupling, and then going the other way. So from our fluid activation calculation, we actually want to know. In this particular case, we were want. To know what our gamma decay source terms were, and that was a case of taking that scalar field in fluid transport back, and getting that field back out again, and that was a challenge. So yeah, the the biggest challenge is coupling the different domains and doing so both computationally and getting that to scale, and also to get it to be physically accurate, and so I think this is going to be the sort of research aspect for years to come in in the field of multi physics.
Sarah Webb 20:30
Yu-Hsiang added.
Yu-Hsiang Lan 20:31
The physics is challenging, but for my part, a more technical part, I would say debugging or even running the case is very challenging at full-machine scales. So how it works is you typically only have maybe five chances, and you have about eight-hours window to submit the full machine jobs. And of course, most of the jobs fails because the system can have some issues, and also sometimes we made mistakes. So we really don’t have a lot of times to run these big machine simulations, and it’s also very stressful, I would say, because you have to kind of figure out what’s the next job you are going to submit within the very short timeframe. You just have to be very creative, and several things is become very useful. So one thing is we print a lot of diagnostic information into a log file. That turns out to save my day multiple times because we don’t have to re round it and we can still collecting useful information. Every one of us has used the code to some degree, so they bring different perspective and kind of understand why these issues can happen, and we will have to just figure out how to narrow it down, and then debugging at the either smaller scale or having the alternative. Having alternatives is actually very important as well. So, as a software developer standpoint, extra options require extra manpower to maintain solver options, and typically we only need one method that works eventually. But when you hit some unknown error, having the backup options really save the day to make everything go through.
Sarah Webb 22:13
From Luiz
Luiz Aldeia Machado 22:14
I would agree that debugging is the most challenge part, at least for me, and balancing the expectation of queuing your job, waiting the queue, then it starts to run, and then you figure out that you know messed up something, and the code the simulation crashes because a single line that you did it wrong. This makes me very anxious sometimes, you know, especially when you put this big simulation to run. It’s like, okay, is it going to run? Is it like failing? And if it fails, it was like because of me, is something going on the computer and the machine? So debugging and and getting the cases running is quite a challenge.
Sarah Webb 22:54
And Misun.
Misun Min 22:55
I think we went through also these issues of coming from the system upgrade. We have no idea what’s going on there. Once they do system upgrade, sometimes we cannot run. We have the same issue for several months for the big-scale run, the full machine run. But then suddenly it stops. All problem resolved after they do re-upgrade one more time. So yeah, at that scale, we don’t see that smaller scale. 2000 node, you don’t see 4000 node. You don’t see, but it could happen more than 6000 node, and so on. So that is quite frustration when the problem is coming from not the code, when the problem is coming from the system itself.
Sarah Webb 23:38
Right, and and so it sounds like you’ve got outside issues that could be affecting your your work that you have no control over.
Aleksander Dubas 23:46
Yeah,
Sarah Webb 23:47
Aleksander.
Aleksander Dubas 23:48
I mean that that’s what working at the cutting edge of hardware is it’s a real challenge because you sometimes you run a simulation and it fails. It’s like, well, is that an issue with the firmware? Was that a compiler issue? Or is that something I did, and it could be any of those three, and you just got to work your way through methodically to try and work out which one it is.
Sarah Webb 24:07
So I want to follow that question up with asking you, what is the most rewarding part of this work for each of you. Misun?
Misun Min 24:15
Yep. So one of the most rewarding aspect will be when everything runs as expected as scale. that is on the full machine. So this includes getting timings that we expect and having a chance to visualize the full result. Additionally, it’s also great to have a world-class team. Everyone is excited to get on the call every week and to discuss the current state of the result, and to plan the course of action given the available result, and so on.
Sarah Webb 24:47
Luiz.
Luiz Aldeia Machado 24:48
I think for me is looking our results and seeing something that would be pretty much impossible to be done a few years ago. I. The computers evolve as the machines get like bigger and faster. We are now able to solve problems that we could not do it before because of the sheer size of the problems itself. So every day, every time I look to like a nice simulation that I was involved, and I think, okay, like 10 years ago we may not even be able to do this. It it makes my day a little better.
Sarah Webb 25:24
Yu-Hsiang.
Yu-Hsiang Lan 25:25
Right, so we really push our code to a very extreme scale. Like the reason why we stop at 2 billion elements is because it’s the biggest number we can run for integer 32, right? So it’s running all of the integer space we can run, and at that scale, for me, the most rewarding part is to see my early work actually pay off. Because we follow the theory, we respect the code structure, we do all the homework on our end. And sometimes, as a software developer, I just constantly feel like I’m overengineering some other things. I just try to make everything nice, but sometimes it’s really not necessary to push the limit of just a simple functionality. But it’s really satisfying when afterwards, when I find it’s actually working, and people are using my code, and because I did my homework earlier, and it works to a larger and larger problem.
Sarah Webb 26:20
And finally, Aleksander.
Aleksander Dubas 26:22
For me, it’s really satisfying seeing such a interdisciplinary team all come together and work so seamlessly. It’s it’s really brings joy to my heart. But also for me, seeing us getting so much closer to bringing fusion power to being a reality through being able to simulate something that, if you wanted to build that as an experiment, would take years of planning, lots of health and safety forms, and you still wouldn’t get the same richness of data that we get from simulation. And to see that coming out with comparatively less effort. It’s amazing.
Sarah Webb 26:58
Are there any core lessons that you’ve learned about collaboration, or advice that you would give to others who are involved in these kinds of projects?
Luiz Aldeia Machado 27:09
I would say define well the goal, define well the tasks, distribute the tasks for the people that knows better about how to do them. I think organization is fundamental, and if you are not organized and you do not know how to split the problem into small pieces, where each person involved can tackle the problem and solve it, it it will be pretty much impossible to deal with such large-scale problems.
Aleksander Dubas 27:43
Yeah, I agree. To put it in one line, it’s never too early to start planning and actually working on the tasks together.
Sarah Webb 27:49
Misun talked about the need for highly performing code and a good application, but she also emphasized the strong bond among the team from years of working together.
Misun Min 28:00
We have trust each other. We share everything between us, and everybody is growing through this team collaboration, learning a lot more in different area. So that is a really good part of building team in this way. It’s very hard to have that kind of team. So I really appreciate that we are able to have all these component together, going for long term. So we are going to continue working together.
Sarah Webb 28:28
Well, Yu-Hsiang, Aleksander, Luiz, Misun, thank you all for your time today. It was such a pleasure hearing about your project, and I wish you a lot of luck at SC in a few weeks.
Misun Min 28:42
Yeah, we’ll see you there. Thank you so much for your time.
Yu-Hsiang Lan 28:44
Yep, thank you, thank you so much, Sarah. Thank you for having me.
Sarah Webb 28:48
Some members of this team, including Yushang and Misun came to Chicago for SC26 to present this research as part of the scientific program, and at a talk at the Department of Energy’s booth in the conference exhibit hall. Other members of the team attended the American Nuclear Society meeting in Phoenix held during the same week. To read more about the team, their research, NekRS, and the Gordon Bell prizes, please check out our show notes at scienceinparallel.org.
Sarah Webb 29:21
Science in Parallel is produced by the Krell Institute and is a media project of the Department of Energy Computational Science Graduate Fellowship Program. Any opinions expressed are those of the speaker and not those of their employers, the Krell Institute, or the U.S. Department of Energy. Our music is by Steve O’Reilly. This episode was written and produced by Sarah Webb and edited by Susan Valot.
