It will not shock anyone to hear that I’ve played a lot of computer games. One of my favourite series is Command & Conquer, and in all of the C&C games a large part of the fiction has been the Electronic Video Agent – E.V.A.
E.V.A. is an artificial intelligence tasked with assisting you (the commander) on the battlefield, keeping track of objectives and notifying you of important events. Of course, the E.V.A was really just a collection of sound files and video clips that were triggered by certain events in the game. But what if we could add an actual AI into the mix?
I’ve been playing a lot of OpenRA, which is a recreation of the classic C&C games with modern touches and multi-platform support. I recommend it to anyone with happy memories of C&C from the nineties. It is open source and written in C#/.Net, which makes it ideal for this experiment.
I used Semantic Kernel to add Large Language Model functionality to OpenRA. Semantic Kernel is useful because it allows you to swap in and out LLMs and experiment with different providers, and it also allows those models to call local functions in your code to gain information.
I hacked it together so that you could ask E.V.A. for advice about anything by typing “/eva” in the chat, and also set it to automatically call the LLM every two minutes to get the most important advice right then. I also added text to speech and made it so that OpenRA could play back the sound files without reading them from disk first.
I actually had a really interesting double bug situation to solve when the WAV files being returned by OpenAI wouldn’t play. It turns out that in WAV files there is an 32-bit unsigned int header declaring how many bytes of data will follow in the file. OpenAI just sets all the bits in that header to 1, which I speculate is because they don’t know the final size of the file when they start generating it. Essentially it means that they declare all their WAV files to be 4 GB in size and hope the program reading these files is lenient about parsing them.
OpenRA wasn’t lenient, but more than that, it had a bug where it read the unsigned int as a signed int. That mean that instead of trying to move 4,294,967,295 bytes ahead in the data stream (also not good), it was trying to move -1 bytes ahead, which obviously made it hard to play back the data correctly. I fixed that bug and made the handling of WAV files more lenient.
I created indicators for the game economy and build queues, and fed the LLM an initial prompt with how it was supposed to behave and what faction the player was. I added a bit of my own tactical wisdom and finally linked to a strategy guide for OpenRA. Getting the tone right was tricky – LLMs tend to be very verbose and friendly, and what we need is a no-nonsense robot.
After a decent amount of fiddling about, I got it all working. The sounds of the new E.V.A. collide a bit with the original E.V.A., but that could be fixed with some more effort.
One of the useful features of Semantic Kernel is the ability to switch LLMs easily, so I experimented with that. I tried Microsoft’s Phi Micro 4, OpenAI’s GPT-4o-mini, and their 4o model.
The great thing about Phi Micro 4 is that it ran locally! The not so great thing is that on my old clunker, it took a full minute of fans going maxed out to get an answer.
GPT-4o-mini ran faster, but the obvious disadvantage is that you have to be always online and pay real money every time you play. 4o was even better at being E.V.A., but costs multiple times more money every time you play.
It has to be said though, none of the LLMs were savants at playing Red Alert. Obviously having a tuned or pretrained model specialising in mid-nineties real time strategy games would help a lot here.
There’s an interesting discussion to be had here about the tradeoffs between local models and high-spec hardware versus remote models and pay-per-use billing. Especially if you were going to design a computer game around having LLMs as part of the experience, how would you split the cost of that LLM usage between you as the developer and your customer, the player?
All in all, I had a blast making this! I don’t think I’m going to spend more time perfecting it, but it sure was an interesting experiment and gave me some insights in what you have to weigh against each other when implementing LLM functionality into programs. I held a talk about the project at Sopra Steria Norways internal tech conference, Bytefest, where I feel it was well received. It might be making its way to a recording on the net or as a lightning talk other places, in which case I’ll be sure to update this blog post.