Fellows Forge an Industry Collaboration

Transcript prepared with otter.ai followed by human copyediting.

SPEAKERS

David Krasowska, Ethan Meitz, Sarah Webb

Sarah Webb  00:03

In 2024, an engineer and a computer scientist walked up to NVIDIA’s booth at the Supercomputing conference, and two years later, they’ve turned their idea into a beta test. I’m your host, Sarah Webb, and in this episode of Science in Parallel, I’m talking with Ethan Meitz, the mechanical engineer, and David Krasowska, the computer scientist, where they’ll tell the story behind this software collaboration. These two Ph.D. students met as Department of Energy Computational Science Graduate Fellows. We’ll call that CSGF for short as we’re talking, and their conversations about Ethan’s sticky computational problem led to this side project with NVIDIA, a library called cuNumeric.jl But first, a little background.

Ethan Meitz  01:0

My name is Ethan Meitz, and I’m a Ph.D. student at Carnegie Mellon University, studying mechanical engineering.

Sarah Webb  01:13

Ethan focuses on understanding the molecular details of heat transfer in liquids and solids, and in an early project for his Ph.D., where he needed to make tensor contractions faster in his calculations, he hit a roadblock with his code. And that’s where this collaboration story begins. It’s summer 2023, right before the annual CSGF program review meeting in July. There Ethan knew he’d run into other fellows who could give him advice, people like David Krasowska. While Ethan focuses on applying computing toward physics problems, David works on making code more accessible for researchers. Here’s how David describes his research at Northwestern University:

David Krasowska  02:08

One of the things I focus on is programming systems and programming models, the ability to abstract away information from the programmer and leveraging abstraction to have a higher productivity for the scientific developer. So within the DOE, there’s a bunch of scientists that want to do science, but the problem is there’s a lot of complexities of running their code on these massive supercomputers that are heterogeneous, meaning they have CPUs, GPUs, and other types of accelerators, AI accelerators, or whatever it is, that’s a bunch of complexities that the scientific programmer should not have to worry about. And my research really dives into abstracting away those complexities, so the scientific programmer, the scientist, does not have to worry about that kind of stuff.

Sarah Webb  03:00

So Ethan and David discussed his problem, and David suggested a brand new library from NVIDIA called cuPyNumeric. David had worked with it as part of his practicum experience at Los Alamos National Laboratory, but it was in early development. David explained how he learned about the library and why he thought cuPyNumeric could be a good solution to Ethan’s problem at the time.

David Krasowska  03:27

The way I got exposed to cuPyNumeric is through also the CSGF. Basically, there’s this practicum experience where you basically have the experience to work at a national lab for summer, and that’s just part of the fellowship. And what I chose to work on was the Legion programming system. This runtime, it’s at Los Alamos National Lab with collaborators at SLAC National Laboratories and also at Stanford and NVIDIA. So working on this Legion programming system, one of the things I was doing was adding a backend support for a new emerging hardware architecture, and so I got a lot of exposure into the internal runtime and everything that was involved with it.

David Krasowska  04:18

But one of the libraries called cuPyNumeric was one of the highest abstracted interfaces that was getting relatively popular in the ecosystem, and it leveraged Python. So instead of leveraging NumPy, which is a popular numerics Python package, you could leverage cuPyNumeric, which takes your CPU-only code, it transforms it and gives it the ability to run at scale, and also on multiple GPUs. So CPUs and GPUs without any code changes, and because of the practicum experience I had at Los Alamos, I got exposure to this, and then I brought that exposure to Ethan.

Sarah Webb  05:00

Ethan saw the potential advantages right away.

Ethan Meitz  05:03

And you don’t have to understand how to write MPI code, how to do like CUDA kernel programming. It just it just all happens for you. So the scientist worries about like the algorithmic implementation, getting the physics correct, and then the cuPyNumeric library worries about getting all the communication and all the kind of high-performance computing stuff correct.

Sarah Webb  05:20

But like many promising ideas, it didn’t work on the first try. Ethan explained what happened next as he was waiting for his flight home from Washington,D.C.

Ethan Meitz  05:32

I remember sitting at the airport because I got there early, and I was trying to download this library and use it. And I was at the airport for like three or four hours before my flight, and I just could not get the library to work at all. And to NVIDIA’s credit, this was a library that was not released yet. It was in like its alpha version.

David Krasowska  05:51

Very experimental.

Ethan Meitz  05:52

Yeah, eventually I did get it to like build, and I got I was able to run the thing that I was interested in. And at that point, it didn’t end up being the solution that I was looking for, but it was my first exposure to this library.

Sarah Webb  06:04

That’s not the end of the story. In 2024, Ethan was thinking about cuPyNumeric again. He’d been using Julia as a programming language for another project, and he started to wonder if a tool like cuPyNumeric could help streamline the process of getting his Julia code to run on GPUs, but there was a problem here: cuPy Numeric was in Python, so he had to make some choices: move his code to Python, find a new tool? Ethan and David saw each other again in Atlanta in November 2024 at the annual Supercomputing meeting.

Ethan Meitz  06:46

At supercomputing. David and I kind of were were talking like, what if we could just get this thing in Julia? And David told me like, hey, you like you know these people are like at supercomputing like there’s they have a booth the people who are actually like like cuPyNumeric is being officially released like right now at supercomputing, so or before just recently before supercomputing,

David Krasowska  07:04

At this point in time at Supercomputing, they’re doing their first public beta launch, and it has more stability than ever before.

Sarah Webb  07:11

Okay, so you’re at supercomputing. NVIDIA’s at supercomputing. What happened next?

Ethan Meitz  07:18

I think David had said like he knew some of the people from his practicum, maybe one or two of them, and he had talked to him earlier in the week. And then later in the week, David and I were on the show floor together. He’s like, “You want to just go talk to the guy?” So we kind of just we went up to him and just started talking, and we ended up– like they have their normal demo screen with like the very high-level details. But by the end of the day, we were like like that we were in the source code of of cuPyNumeric. He was like pointing at us to like, hey, like if you want to like hook this up with Julia, maybe like this is the place to start. And if you can get this to work, like send us an email, and we’ll be interested to to talk more.

Ethan Meitz  07:54

And supercomputing is late November, so I remember over winter break, David and I kind of worked on it and got a a minimum working example up and running, and we emailed them. And we’ve been meeting with them weekly or so ever since for the last two years.

David Krasowska  08:09

There’s a little bit more to that story as well. In between the winter break and supercomputing, there there was another conference that I attended in between that had a familiar face, the person that we talked to at NVIDIA. And I also talked with him there at the Legion retreat at Stanford. So I was presenting my practicum work that I did at Los Alamos at the retreat. However, this was the next phase of like talking with him after we explored the library a little bit, trying to figure out an idea of how to approach this project. And like Ethan mentioned, we basically grinded it out during winter break, and then now it’s 2026, and we’re still going at it.

Sarah Webb  08:48

Their library in Julia is called cuNumeric.jl. Their primary goals are allowing researchers to use multiple GPUs seamlessly without needing to write detailed custom code and enabling them to reuse existing scientific libraries.

Sarah Webb  09:07

Tell me a little bit about the progress you’ve made over that time.

Ethan Meitz  09:11

I’ll start by explaining like what we’ve completed. So there’s like the NVIDIA library, which is in Python, and they have a layer of it which is in C and C++ and that’s the part of the library that that we interface with, and because NVIDIA is their goal is to ship this thing in Python, their C++ layer is not really targeted at like what David and I are trying to do. The Julia thing is not something they ever designed for. So the main challenge initially was like trying to figure out how to work around the fact that the API surface they had was maybe not designed for what David and I were trying to do.

Ethan Meitz  09:44

So the first couple months were just figuring out those issues, and I’d say maybe within like a couple months or so we had a framework that was working. And well, we didn’t have all of the same functions that were in their library. We had the ability to basically match what they were providing one-for-one.  And later that summer, there was JuliaCon, which was like a conference for Julia that we were targeting a presentation at. That we ultimately did do that presentation, and in that presentation, we basically showed that we have like performance parity with cuPyNumeric for some specific scientific tasks. And then we also talked about some problems that we ran into that I’ll maybe let David elaborate on since he’s he works on them a lot, a lot more than me.

Sarah Webb  10:22

All right, take it away, David.

David Krasowska  10:23

Yeah. So one of the biggest things that we felt that there there was a specific application that we were targeting is a specific benchmark that was a Gray-Scott heat diffusion. It was a 2D reaction that brought us a lot of trouble to this library, and we were matching cuPyNumeric’s performance for the most part. There was some complexities with the garbage collection interface interfacing with Julia. However, we quickly resolved those problems by just creating our own heuristics in the garbage collector. But the next thing is, there was a competing library within the Julia ecosystem called Implicit Global Grid. They had 10x better performance than both our library and cuPyNumeric for the exact same Gray Scott heat reaction, heat diffusion application. And we did a lot of analysis, and it boiled down to performance optimizations in regards to kernel fusion, meaning we’re launching too many of these operator kernels.

Sarah Webb  11:31

So David has focused on fusing these operations into a single expression to speed things up on this specific task, where Implicit Global Grid excels.

David Krasowska  11:42

We’re not at the level of Implicit Global Grid, but we’re getting closer.

Sarah Webb  11:46

I get the impression this is an important project, but a side project, for both of you. How do you feel like working on this, and working on this together, has enhanced your growth as a scientist? How is it fed into your other work?

Ethan Meitz  12:03

For me, this is a big help in the fact that it’s like more of a software project. Like my research is focused on physics and science, whereas David is in the computer science department. So while I do really like programming and making libraries that people can use that are easy to use, I’ve never really had to write a library nor interface with another team of people who are making changes at the same time. Usually, it’s just me in my own little world doing whatever I want to do, whatever I think is best is what I’m going to do in the library. But that doesn’t– it doesn’t work the same way when I’m working with David on this project, and also when NVIDIA upstream is making changes to library that they have their own goals and their own mission that is maybe not aligned with what David and I are doing.

Ethan Meitz  12:43

So I think understanding how software development works and how there’s oftentimes a lot of different competing interests going into this thing. So like the library David and I are making, we’re putting a lot of energy and effort into making sure that it is very well tested. So like there’s been some bugs that our testing suite has found that like the cuPyNumeric team had not actually found yet. So that felt pretty good that we were able to find bugs in their library that they weren’t catching with theirs, and just things like that, like best practices for how to write software and how to work in a team with other software developers, because that’s just not something I get in my normal Ph.D.

Sarah Webb  13:18

What about for you, David?

David Krasowska  13:19

I mean, there’s a similarity with my answer as well. This project has definitely improved my ability to do software engineering. Like Ethan has mentioned, I am more in the computer science domain. My research spans runtime systems and focusing on them, so I do have similar projects. This is not my main project, but they all encompass some sort of runtime. CuNumeric.jl leverages the Legion runtime programming system. However, I work on other projects that leverage different runtimes and some runtimes that I custom-built as well. But basically, this has really helped me in the best practices of software engineering, doing CI pipelines and CI integration, making sure we have robust test suites and making sure our code is really resilient, and we find bugs and stuff like that.

Ethan Meitz  14:08

I think another component of it is also the fact that, like, well, Dave and I are kind of just doing this for fun and to learn. For NVIDIA, it is like cuPyNumeric is a real project. They’re not like selling it. It is open source, but they want people to use the library, so at the end of the day, people use their GPUs. And I think with research, that’s not always something you’re super concerned about. Like you’re trying to push the envelope, do something new, and explain to the world why it’s important. But at the end of the day, like if you don’t get 1000 citations on your paper, like your career isn’t over. You just you just shift directions and you do something new.

Ethan Meitz  14:40

But at a company like NVIDIA, if no one uses your software, like okay, like we’re just not going to work on this project anymore, like it’s done. And like we want people to use this. So like Dave and I have been recently thinking about like how can we like actually like pitch this library and market it to people and get them to use it because that would not only get more buy-in from NVIDIA, but it would also allow us to commit more time and resources to this library.

David Krasowska  15:05

NVIDIA is a very large company, but it’s structured in a way of like 1000+ startups in one very large company ecosystem. So it’s all branded as NVIDIA, but every single subsidiary is its own individual startup, and that startup has multiple projects that encompasses like something like the Legate and cuPyNumeric, for example. And the more users, the more marketing, the more funding they put towards it.

Sarah Webb  15:29

Are there other either skills that you’ve picked up or things that you’ve recognized were important in this project that maybe you didn’t emphasize in doing other types of research work?

Ethan Meitz  15:48

Yeah, I mean the user experience at the end of the day, like just has to be so clean for a library like this. Because one, the goal of the library is like you don’t change how you program, but you get all this extra, like, performance. You don’t have to program for GPU. You just write your normal code. So like that contract has to be satisfied, or else our library isn’t worth using in the first place. If it is more of a pain to use our library, even if it gives you that extra performance, people probably won’t use it.

David Krasowska  16:14

So one of the big difference between cuNumeric.jl as a project versus my other projects, as a grad student perspective, there’s this notion in at least computer science and probably other fields as well as gradware, right? So you have these research objectives, and usually the software packages do just enough to get the results, but they’re not like you could run it on any arbitrary system or anything like that. They’re very hyper-specific to fulfilling the goal of getting results and just publishing the code so it’s open source and publicly available. Versus cuNumeric, like Ethan has mentioned, it has to be robust.

Sarah Webb  16:58

Is there anything else, sort of scientifically, computer-science wise, about the code itself that you think is important to mention.

Ethan Meitz  17:07

I’m not how important it is, but one thing we’ve realized is like it’s very hard to get compute resources to test these things. One, like you only can run this on NVIDIA GPUs. That’s that’s one thing, and NVIDIA has provided us some resources to do this, but of course we’re just working for free. We don’t actually work at NVIDIA. They can’t like just give us access to their servers, and we don’t have research funding for this. So like, while we have some access to compute through CSGF, it’s just been hard to kind of test this library, especially since the point of it is to scale across like multiple GPU systems and multi-node systems.

Sarah Webb  17:40

So not everything is simple. There’s always complexities, even just practical ones, in terms of getting to test some things out.

Ethan Meitz  17:48

Yeah.

Sarah Webb  17:49

I recorded this conversation with Ethan and David in July. Since then, they’ve continued to work on cuNumeric.jl, and they’ve just offered a beta version of it. Check out our show notes at scienceinparallel.org for more information and a link.

Sarah Webb  18:08

As we wrapped up our conversation, they both emphasized how the fellowship brought them together and made this project possible.

Ethan Meitz  18:16

David’s practicum working with Legion and meeting these this NVIDIA team like that wouldn’t have been able possible without CSGF. I would not have met David if it was not for the practicum, and then we would not have gone to supercomputing to have the ability to ambush these NVIDIA folks and get them to like buy into the project that we’re working on. So definitely a three-pronged thing that, none of which would have been possible without CSGF.

David Krasowska  18:40

Yeah, the CSGF provides funding for each for the entire cohort to go to Supercomputing each year, which is a great thing.

Sarah Webb  18:48

I think that sounds like a really great place to wrap up, Ethan. Thank you, David. Thank you. It’s been a pleasure talking with both of you about this.

Ethan Meitz  18:57

Thank you for having us.

David Krasowska  18:58

It’s been great meeting you.

Sarah Webb  19:04

To learn more about David Krasowska and Ethan Meitz and their computational research, please check out our show notes at scienceinparallel.org. We’ll also include a link to our recent episode with Ethan and one of his Ph.D. advisors, Jerry Wang, about the power of conversation in collaborative research.

Sarah Webb  19:28

Science in Parallel is produced by the Krell Institute and is a media project of the Department of Energy Computational Science Graduate Fellowship Program. Any opinions expressed are those of the speaker and not those of their employers, the Krell Institute, or the U.S. Department of Energy. Our music is by Steve O’Reilly. This episode was written and produced by Sarah Webb and edited by Susan Valot.