EPISODE 1754

[INTRODUCTION]

[0:00:00] ANNOUNCER: Asahi Linux is a project that aims to port Linux, to Apple Silicon chips, which use a custom-arm based architecture. The project is fundamentally important, given the popularity of Apply Silicon Macs. It's also a rouge effort, because Apple Silicon is an entirely undocumented platform.

Alyssa Rosenzweig is a well-known computer scientist who describes herself as a graphics developer, passionate about software freedom. She is currently a contractor at Valve, where she develops open-source software to improve Linux gaming. Alyssa is also a contributor to Asahi Linux, and works on reverse engineering the Apple M1 GPU, among other contributions to the project. Alyssa joins the podcast to talk about reverse engineering hardware, Asahi Linux, new advances, and gaming on Asahi, and more. 

This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

[INTERVIEW]

[0:01:08] SF: Alyssa, welcome to the show.

[0:01:09] AR: Hi, it's great to be here.

[0:01:10] SF: Yes. Thanks so much for being here. I'm very excited to talk about this project. I talked to lots of people, I think this is probably one of the most exciting things that I've gotten to talk about in a while. But before we get into that, I thought it might be helpful for listeners to cover a little bit of the history of Apple and chip designs and set the stage, so to speak. So, feel free to jump in if you have something to add or additional context. I'm sure you know way more about the space than I do. 

Essentially, the x86, which was created by Intel back in 1978 is the popular and widely used chip architecture in the world. Intel, AMD chips are based on this and Apple's been using the x86 chips since 2006. Because Apple was using x86 chips for their Mac computers, those computers could of course run OS X, but they could also run things like Windows and Linux on them because the CPU instruction set was known. The ARM chip architecture, Acorn risk machine was created in 1985. Apple started using ARM chips for iPhone back in 2008. Then, in 2010, with the first-generation iPad, they licensed the ARM architecture because it gave greater control over performance, energy efficiency, it was a huge benefit, of course, like smartphones back in the early days.

Then, in 2020, Apple transitioned their entire line of Mac computers from x86 to ARM, starting with the M1 chip. This was like from the ground up rebuild, giving them greater control over the software and hardware. But this also was completely proprietary technology, where only OS X could run on the M series of chips. Linus Torvalds even said when asked about having Linux run on these chips, said that it would probably be more work than it was worth it to do. But then along came a bunch of crazy engineers, including our guest today that said, "Not on my watch, Apple," and started working on this super ambitious project called Asahi Linux, aimed at reverse engineering the M-Series in order to get Linux to run on Apple Silicon. Alyssa, tell me about the project, how did it get started, how did you get involved, and do you have any additional context to share on the history that I've shared?

[0:03:12] AR: Sure. Well, as far as I know, that history is right. I don't know the dates by heart. Apple had announced that they were transitioning to the M1 chip and having ARM desktops, which suddenly made the ARM hardware, the Apple ARM hardware very interesting to a subset of us in the Linux community. Obviously, the M1 chip is nothing new. It's just a scaled-up version of what Apple's had in the iPhone for years prior. But with the locked boot loader on the iPhones, we can't run anything, but iOS on it. At least, not on any of the current devices. 

Even on the ones where we could because of boot ROM exploit, it's just not so interesting to us. I don't know. There are people who are very interested in putting Linux on mobile devices, and I think that's awesome. I think that's cool. It's not my jam. There are a lot more people that are running Linux on desktops if we ignore the Android space, which is of course, we can't ignore the Android space.

Anyway, when this chip came out for the M1, it suddenly put itself on our radar. marcan, Hector Martin, announced that he was going to work on this new Asahi Linux project, and he launched Patreon, and got a lot of excitement really fast. It looked like this project was actually going to happen. In his initial postings in the project, right after the chip had launched, this would have been back in December of 2020. We're all stuck at home with nothing to do, but reverse engineer hardware. 

marcan had said, publicly, that he was going to have to work on all these different upstream projects, the Linux kernel, yes, but also things like Mesa, where all of our GPU drivers live in user space. He had just put out a call for any upstream maintainers that wanted to show them the ropes for their parts of the stack. He would be very appreciative of that, I bet. I was at the time working on Panfrost at Collabora. I was a student, but that was my part-time job at least.

We started talking over IRC and giving marcan the rundown of, this is what I expect a GPU driver development project would look like for this new Apple hardware. Long story short, I ended up myself and I'm one mini for that Christmas and you know what happened next. 

[0:05:31] SF: Back in 2021, most people were making sourdough. We decided like, let's reverse engineer some Apple Silicon. How many people were actually involved or what's the scope of the core team that's working on this project?

[0:05:43] AR: Well, for the previous comment, there is a little bit of Panfrost code in the Asahi Linux driver and the Mesa driver, and there's plenty of other drivers in the Panfrost driver. In a way, I am making sourdough. Sorry. What was the question after that?

[0:05:57] SF: How many people are sort of involved in the core project?

[0:05:59] AR: Yes. It depends how you define involved. There are only a couple of us who are getting paid directly for working on Asahi. There is maybe a half dozen people who are very active in the project, if not full-time, part -time. Then, when you include the broader community of people who are working on whatever their little piece of the Asahi Linux umbrella is, I'm sure you'll get dozens there.

[0:06:24] SF: Okay. Why is a situation like porting Linux to Mac under this condition is such a hard problem?

[0:06:32] AR: There are two broad issues here. The first is technical in nature. There are a number of architectural deviations between the Apple hardware and what the rest of the industry is doing. The reason for that is simple. If you are Intel, or AMD, even Qualcomm, you need to make sure your hardware can run Windows, Linux, and Android. Not just run the OSes themselves, but also run essentially any users based on top of them. Even Qualcomm has Windows and ARMs to keep them honest in some sense in terms of following the progression of the industry.

With Apple, Apple only cares about running the Mac OS, and running iOS, and whatever other OS they have, watchOS, TVOS, they only care about the Apple ecosystem. Unlike Microsoft, which has to care a lot about backwards compatibility for various business or political reasons, I'm not a Microsoft person, I couldn't tell you. I do know Microsoft cares a lot of backwards compatibility, Apple does not. I don't think that statement is controversial. Apple has several times dropped backwards compatibility for different software, different hardware. Apple has the blessed Apple way of doing things, and they just don't care so much about what the rest of the industry is maybe doing or even what they were doing 15 years ago.

What that means is that, they can iterate very quickly on the hardware, and they can do unusual hardware designs, which from a technical perspective is very neat. But it also means that if we want to be able to just run some random Windows game that came out 10 years ago, it might not have the hardware support we need for that. So, there's a lot of challenging technical problems of trying to port standard software, standard APIs, port standards onto hardware that was really never designed for it. 

The second part of the answer is that we're doing this all with our eyes closed. Apple does not document their hardware in any meaningful way. I'm sure they have documentation internally for all their hardware. It's not like nobody at Apple is a technical writer, but certainly, there's nothing they've released publicly. We don't have any hardware documentation under NDA either. We have no reference code from Apple. All of Apple's own drivers are proprietary software with vanishingly few exceptions. 

Occasionally, there will be something in a kernel source drop that's useful. But by and large, all the interesting drivers are proprietary. So, we have no reference code on how to use the hardware. We have no documentation on how to use the hardware. So, we have to figure everything out just by either guesswork or observing what Apple's driver does from the outside. We can't get too close either, because black box techniques are generally preferred for legal reasons.

[0:09:20] SF: Okay. So, essentially, you had just base level problem of, "Hey, we have to map a bunch of stuff to hardware that wasn't designed essentially for it to work with this piece of software." Then, on top of that, you don't even have insights essentially to how the hardware functions. So, the mapping process you're mapping is between this like black box. In terms of the backwards compatibility, I don't think anybody would argue with you that Apple doesn't really care about backwards compatibility. I mean, in some ways, because they're not burdened by backwards compatibility, they probably have some advantages in terms of this like M-Series, like rebuild, they can really like optimize it to run. They don't have to worry about standards backwards compatibility. That gives them an advantage probably with optimizing energy, efficiency, compute cycles, whatever it is essentially that they particularly care about.

[0:10:07] AR: For sure.

[0:10:08] SF: In terms of with something like this that's very challenging technically, and there's a scale to it essentially, where do you even start a project like this?

[0:10:17] AR: I think marcan put it best. The hardware doesn't have any public documentation, so the first thing to do is document the hardware. The problem of writing drivers is challenging for sure, but there are lots of people who have written drivers for hardware that has documentation. So, really the start of the challenge is reverse engineering the hardware, not worrying about how we're going to drive it, but just worrying about what the hardware itself does.

I do mean hardware. I don't mean reverse engineering Apple's drivers, Apple's software, Apple's APIs. It is irrelevant what Apple does on their software team. All that matters is what the hardware is capable of and how the hardware itself works. So, how do we do that? Well, the good news is that we do have macOS running on the machines, and Apple does have drivers for their own API, which doesn't support everything we need in the standards. But it does support almost everything the hardware can do, almost. What we do at a high level is we have little test applications that will run on macOS, and we will intercept the communication between Apple's drivers and the hardware itself.

By intercepting it, we can dump out all of the interesting information, all the data structures in memory, the registers, what have you. When you do this for the very first time on unknown hardware, well, you just have a big pile of hex values that you don't know what to do anything with. Great. You save it. There's your first dump. Go back to your test application, change one tiny little thing, get a new dump, and compare the two. If you're lucky, there will be a single tiny little change in the output, and you've just reversed engineered your first bit of the hardware.

Obviously, something like a GPU is massively complicated, which means, you will have to do thousands of iterations of this before you really understand the hardware. But there aren't any leaps of faith for the most part. It's just one building block after another. When you do this thousands of times, you get a pretty good picture of how the hardware works. You document it as you go along. I don't write formal hardware documentation because I have better things to do than become a technical writer. But we do have XML descriptions of the hardware data structures, for example, and we do try to keep those as accurate and complete as possible. 

So then, once we have this understanding of what the hardware does, we can understand all these little data structures in it. We can write drivers for that. We can start out writing drivers that mimic what Apple's drivers do, monkey see monkey do, because we know that is a correct way to drive the hardware. But once we understand how the hardware actually works, we don't have to stop there. At this point, we've been reverse engineering the M1 GPU for years now. While there are certainly bits we don't understand yet, there's no giant piece of fixed function hardware that is just a big mystery blob. We understand roughly how the hardware works and roughly what's there, which means it's no longer relevant now what Apple's drivers do or how Apple approaches problems.

The reverse engineering is sort of done, and that segment of the project is going a little quiet. Now, it's an engineering problem of, we are driver developers. We have a hardware that can do these things with this constraint, we know how to drive it. We need to make and do this other thing. How do we do it? What's the best way to do it? Sometimes, that means doing something very similar to what Apple did. Sometimes that means doing things very different to what they did. Certainly, running DirectX 11 efficiently on this hardware requires quite a bit of creativity.

[0:14:03] SF: In terms of the sort of starting process of like intercepting, the messages are going from the driver to the hardware, how do you actually intercept those messages? 

[0:14:13] AR: The macOS driver in user space, whenever it wants to do anything with the hardware, it calls into kernel space with a system call. The way it does that is through Apple's I/O kit framework. So, I/O kit is an open-source Apple framework, which makes things marginally simpler, because these are public entry points, I/O Connect call method, whatever. Then, all the different methods are undocumented for the GPU, but the general signature of this I/O Connect call method is public.

What we have to do is write our own version of I/O Connect call method that calls into the real OS one. But also, right before calling in, will dump everything to standard output to a file, wherever. We then use a little bit of a dynamic linker, trickery to run our test application, but with our version of the I/O Connect call, wrapping the real one. Then, we are right in the middle. The other way we can do this is actually at a system level for doing kernel work that won't work because you can't intercept the user space talking to the kernel, if the kernel is the thing you're working on. But you can't intercept the kernel talking to the hardware. 

I believe marcan wrote a fascinating hypervisor where we have the capability of booting macOS in this thin hypervisor we control. So, in essence, the entire system is being wrapped, and then any register right, any memory access, anything is going through our custom hypervisor, and we can instrument that to dump the things that are interesting to us. I believe Asahi Linux has written a tracer that's able to dump the GPU stuff at a system level as well. If Apple changes things and we can no longer do the I/O Connect trick, we're not out of options. Although, that would certainly be inconvenient.

[0:16:08] SF: Thank you. In terms of the GPU architecture, how does that compare to other platforms that you've worked on?

[0:16:15] AR: In many respects, the Apple GPU as shipping in the M1 is one of the simplest GPU architectures I've seen. This is both a blessing and a curse. It's a blessing because it's very straightforward to drive this hardware. Drawing a triangle on this hardware is in many senses easier than on other parts. There are just fewer things going on in the hardware. It meant that we could get up to speed pretty fast.

When I first started working on this GPU, I thought this hardware architecture was amazing, because I was used to dealing with hardware that had millions of stupid sharp edges for literally no reason. It was refreshing to have an architecture that didn't include so much nonsense.

[0:17:00] SF: Do you think that's in part because they did this sort of from the ground up, rebuild, and they don't have to worry about any of the backwards compatibility?

[0:17:07] AR: Well, the hardware's not from the ground up.

[0:17:10] SF: Well, yes, sir.

[0:17:12] AR: It's very clearly a power free art derivative, but I'm not sure it's that. I do think that Apple has brilliant hardware engineers. I have a lot of respect for the engineers on Apple's hardware team, for sure. They have a set of engineering problems posed by iOS originally, and the solution that they built is very well-suited for its domain. But I said it was a blessing and a curse that it's so simple. The curse is, of course, once you get over that initial phase, you're just kind of stuck. You run out of hardware to do things, but you haven't run out of features that the drivers expected to have. So. there is piles, and piles, and piles of functionality, which would be in hardware on most other vendors, but which is just not in hardware on Apple. 

So, we have to implement that hardware ourselves in the driver, generating shader code or something. That can be nice, it's very flexible. It means that I can implement things in my driver that Apple doesn't implement in theirs because, well, it's just software. It's also very annoying because, especially something like Vulkan, strongly assumes that you have hardware that, well, implements Vulkan. 

[0:18:23] SF: Then, what were - when you were starting this project, besides some of the things that we, you know, obviously we talked about of just like, hey, you have to kind of do this step-by-step dumping of the communication to the hardware, get a sense for like how it actually works, and document that. Was there any big surprises in terms of - that you discovered about how the hardware functioned?

[0:18:46] AR: In some ways, the simplicity was a surprise. It's unheard of to have an architecture that is this simple, really. If I rattle off other GPU architectures, pretty much, you can tell me a vendor, and I'll tell you why their hardware is terrible on some axis. It turns out the axis for Apple, it's simple to a fault. It's very well suited for the design space that it was built for. It turns out that running AAA games is not what it was built for. I think Apple has realized that as well, which is why they've made some of the changes in the M3 that they have. 

[0:19:28] SF: What features of Asahi Linux are still under development?

[0:19:32] AR: The big ones, one is support for Type-C display output as well as Thunderbolt. We support HDMI on machines that have a physical HDMI port, at least some of the MacBook Pros. We don't support Type-C to HDMI adapters yet. We have patches for that. It's in progress, hopefully, by the end of the year, but there's a lot on our plate. That's the one that people complain about, anyway. So, I'm guessing that's the big key for people. Personally, I only have one monitor, so I've never had it. 

[0:20:05] SF: What about like, there's a lot of custom components in Apple hardware, like the Apple neural engine, the secure enclave. Are those things that you're also looking at in terms of the entire project? 

[0:20:19] AR: Certainly, all those components we can use if we want to, we don't have to. The Neural Engine was reverse engineered by Eileen Yoon. So, on her GitHub, there are notes for the hardware and sample code to drive it from Linux, which is super cool. It's not really clear where we go from here, because there's not an established standard-based ecosystem for machine learning acceleration on Linux. I expect it will happen given the current AI craze, but that's not the case now. This is very different from graphics, where we have a standard for graphics. It's called Vulkan, and it's called OpenGL, and it's called OpenCL for compute.

If you want to get graphics on Linux, you implement those three APIs, end of story. With something like the neural engine, there's not a standard for that. It's not clear what we should do with it yet. Asahi Linux, as I said, there are a couple of us who are getting paid for this and doing it as our job, but most people are not. For most people, it's a hobby project. Something like the Neural Engine, that's very low priority for the project as a whole, which means the very scarce full-time development resources are going to be going towards things like Type-C display out or the GPU.

As for, if somebody has an interest in working on machine learning stuff, then yeah, they can pick up and work on Neural Engine. I guess, that's what happened with Eileen, then, she went and reversed engineered and she did it. So, we do know how that hardware works. But, will that chip? I don't know. We'll see. Maybe if somebody wants to

[0:22:04] SF: In terms of the full-time people, how do you sort of organize? How do you decide which parts of the project need attention and who works on what?

[0:22:13] AR: The project is a bit of controlled anarchy. It's definitely not a corporate structure. By and large, this split is, I work on GPU user space, Lena works on the GPU kernel space, marcan works on everything else. That's the split.

[0:22:29] SF: What about in terms of like the community contributions, has the open-source community like contributed meaningfully to the project yet?

[0:22:37] AR: Absolutely. As I said, there are at least six people who are very active on the project. If we include everybody who has done significant contributions, if not in the present, then in the past. Certainly, well over a dozen people have contributed in significant ways to make this project possible. I keep mentioning the Type-C work. That was originally done by Sven Peter, is now being worked on by [inaudible 0:23:00], neither of whom are me, neither of whom are marcan, neither of whom are Lena.

[0:23:06] SF: As somebody, if I was interested in this, and I wanted to install it, and use it, what's that process like?

[0:23:14] AR: There's a script to run on macOS. So, for better or worse, it is a curl pipe SH on macOS. That asks a couple of questions on macOS to work out the partitioning mainly. Eventually, it will give you some instructions about what to do next, which will be to reboot the machine, and hold down the power button to prove to the firmware that, yes, I'm the machine owner. I'm really trying to install this.
This is not a root kit, right? Because, of course, the process to install an OS, and the process to install a root kit, from the firmware's perspective, there's no difference except for one of them the user did intentionally. Apple's firmware makes sure that you're doing this on purpose. Then, it reboots, and then you're just taken into the standard KDE or Gnome installer to finish up the process, and that's that. I don't know if it's easier than installing on an x86 PC. I haven't done that in a lot of years. It's certainly easier than installing on any other ARM system I've used.

[0:24:16] SF: As a user, today, how much functionality - would there be certain things I would notice that in terms of limited functionality, or would I be able to actually use this day to day?

[0:24:27] AR: It depends on what you're doing with the machine. If you ask 10 people if Asahi Linux is ready, you'll get 10 different answers. If you're a developer and you are building Rust packages all day and that's your whole life, it'll be great. If you are a gamer and you want to play Cyberpunk 2077, maybe it's not ready quite yet, and you should strongly consider selling your Mac and buying a Steam Deck instead. It just depends on what you're doing.

I would say. all of the standard basic functionality is there. Some people would say that the Type-C out is standard functionality. I only own one monitor, so it's not standard for me at least. There's a lot of stuff that works that might be surprising that works. There's a package you can install to get Widevine for watching DRM videos for better or worse.

For me, I've been using Asahi as my main machine for years. I have nothing to complain about. For you, I don't know, depends what you do.

[0:25:30] SF: Okay. Fair enough. How did you, personally, even before Asahi, how did you kind of build up these skills or interests in terms of being able to do these types of projects?

[0:25:41] AR: Before I worked on Asahi, I was working on the Panfrost project for Mali GPUs, which was a different landscape for sure. But the core methodology was the same. ARM had their proprietary hardware, ARM had their proprietary GPU driver, and we did the usual wrap the driver, reverse engineer the hardware, write our own driver. Which went from very humble beginnings, all the way back in 2017, started working on that. So, I was in high school at the time. Over time, built that up.

Today, there is a well-staffed, full-time team of engineers, not just at Collabora where I was later working, but also, I'm now seeing contributions to that driver from ARM employees with their @arm.com email addresses. So, it's pretty cool to see the very far from the original proprietary stack. So, no, I did not start with the Apple.

[0:26:38] SF: Prior to that project, how did you basically get your entry point into like programming
and coding? What sparked that and how old were you?

[0:26:45] AR: I started coding, I was maybe seven. I'm really showing my generation here. I was not doing impressive stuff when I was seven. That should be clear. I think I was writing websites with PHP, that was cool back then. Over time, developed interests, and compilers, and graphics, and reverse engineering. Prior to doing any graphic reverse engineering, I had a project in high school where I had a zillion different websites that I had to use for school because of - I don't understand why there's these many websites for school. I wanted a unified frontend for all of them. So, I went ahead, reverse engineered a bunch of the HTTP, REST API, set these used, wrote some Python to plumb it all into one unified place. Cool, that made my school a little bit easier so I could focus on working on Panfrost, I guess. 

Certainly, there is a progression there. There are a lot of people that I think are impressed by what I do at my age, but I've been doing this since I was seven. Which means, if I have 15 years of experience, maybe that makes more sense now.

[0:27:57] SF: Yes. When it comes to these projects, especially when you're talking about trying to figure out how the hardware works, like there's a lot of repetition where you're changing a little thing, getting the hex dump, figuring out what changed, documenting. How do you stay motivated to continue to work on something like that?

[0:28:19] AR: Everybody has their special interest. It's ironic I've been working on gaming because I just don't find games all that interesting. I will play a game for about a half an hour. I'll have fun, and then I'm just bored of it. I know there are people that will play games for 10 hours, and will only stop because it's bedtime, if that stops them. More power to you. I hope you can use my drivers to help in some way, but it's not my thing. I'm sure they would get bored trying to do what I do with the reverse engineering. I just get a thrill out of it, I find it fascinating, I'm stubborn. It might just be that I'm stubborn. You have to be stubborn to succeed in this industry.

[0:29:00] SF: Then, in terms of people who might want to get involved contributing to the open-source project for community aspect. What's your recommendation in terms of getting involved in jumping into something like this?

[0:29:12] AR: This is definitely not a good first project for things. There are lots of people that will show up and want to help, and have no relevant experience, whatsoever. It's not clear what to say because it's, even among reverse engineering projects, even among drive development, Asahi Linux is in many ways doing things on hard mode, just given all the constraints we've talked about earlier today. So, the people who have contributed successfully, I think have had significant program experience on more approachable projects first. So, yes, I would say try to build the skills. If you think you have what it takes, you won't listen to me tell you not to, because, like I said, you don't succeed in this industry unless you're stubborn. If you're stubborn, you'll ignore me saying not to, which is, of course, how I got started working on graphics was, I ignored somebody else telling me not to. So, I think that's the tradition now. Does that make sense?

[0:30:16] SF: Yes. I mean, you have to have a certain amount of determination to do something. Going back to what you said, maybe this is not the place to start. It's like, I've never done anything like this before. But if somebody was interested, this piqued their interest, like, hey, I want to do some sort of reverse engineering project. I want to learn more about hardware and how some of the lower-level parts of a system work. Where does somebody like that start? What do you recommend?

[0:30:45] AR: Find something to a reverse engineer and go at it. Even if it's already been reverse engineered by somebody else, just don't cheat, do it yourself. Hardware reverse engineering is definitely challenging because of all of the extra constraints. But the same principles, the same skills transfer directly to and from other kinds of black box reverse engineering. I think a lot of people who are successful in the hardware reverse engineering space also have experience doing something like reverse engineering web services like I mentioned, or in the game modding community, same skills. I can't speak too much to those communities, because hardware is what I've known for years.

[0:31:25] SF: Probably similar to jumping into any kind of programming type of project. Just pick a project to start with, that's like a reasonable size and in scope that you're interested in doing. Then, you'll probably develop whatever skills you need along the way, if you're stubborn enough to be successful in that project. Then, essentially rinse and repeat until you build up enough skills to tackle bigger projects.

[0:31:48] AR: For sure.

[0:31:49] SF: Is there anything else you'd like to share about the project or your personal journey with this?

[0:31:53] AR: I'm very excited about what we've just released with the Alpha for gaming on Asahi Linux, with the stack we've just dropped. This is tying together a lot of work that's been in in flight for years. Not just on the graphics driver side, but also project-wide, and across projects. A lot of open-source stack together to do something that should not be done really. We shouldn't have done it. So, this ties together the Vulkan drivers that we've written for this hardware on top of DXVK for translating Vulkan to DirectX calls, as part of Proton, which includes Wine for translating Windows system calls to Linux. All of this is x86 code being run under FEX, which is a x86 to ARM emulator, which is running inside of a virtual machine to handle page size mismatches, which is running on our bare metal. 

Miraculously, this giant pile of things means that Direct3D Windows AAA games are entirely playable and a lot of fun. It's pretty crazy to see how far we've come with this. When I started working on graphics for Asahi, people would constantly be asking, "When can we play insert AAA game title here now?" I'm like, "I don't know, I just drew a triangle, man." I'm my own worst critic, I think, and I never thought we would get to this point. So, the fact that Fallout 4 is running on my machine with drivers I wrote, I'm pretty pleased with that. I've been working on FEX as well, just working on performance work for the past year or so. So, for me, personally, it's great to see tying together all these different projects and tying together my Mac and [inaudible 0:33:41].

[0:33:43] SF: Yes, it's amazing. Thanks for sharing that, and thanks for sharing so much detail about how this project came together, how you get involved in some of the challenges along the way, and also the wins along the way. So, thanks for being here and cheers.

[0:33:54] AR: Cheers.

[END]