First, revivify your brain with some rad video game music, click the little triangle button to make delicious sounds spring forth from your computer´s speakers:
Recently, I pledged towards the kickstarter for the game Blasphemous
It looks very cool, I love the handmade pixel art.
More importantly, what inspired today´s post is this update the developer, The Game Kitchen posted about some of the technology behind the game:
Our goal is to create an AI system that is able to deliver powerful, challenging and meaningful opponents for you to fight against. But we’re not stopping there, we also want an AI system that is able to impersonate a human player! “Why?” You’d ask. We want to take advantage of what is know as “automated testing” which is having the game being able to play itself overnight, finding holes in the collisions of the level and other systematic bugs. That would allow us to deliver a better quality game, despite not having key resources (like a dedicated QA department in the studio).
In my day job, developing online shops, I have worked with automated tests for quite some time now.
It´s considered a best practice to do so, and there are tools that facilitate it.
However, when it comes to video games, there is not too much information about it out there.
I guess this usually happens behind closed doors. Personally, I always assumed that automating tests in video games is very complicated (with exceptions).
What is automated testing?
I will probably write a separate article on this, but the gist of it is:
You have a piece of software, the test subject, and you want to make sure that it acts in a certain way.
A very simple example would be a calculator app that performs multiplication.
It is in it´s default state, the display shows “0“
You input the sequence of keys: , [*],  and [=]
The expected output would be “54“. Any other output would be an error.
Now, for a software project, you would have hundreds, thousands or even more of these test cases, ideally set up in such a way that you can run them all on a single command and see if they all succeed.
Why do automated testing?
So why would you write all these tests? The software works already, all the tests do is say that “Yes, the software works.”, which we already knew?
Software is often subject to change. Development cycles for commercial software (such as games, apps, websites) often are 6 months or more.
During that time, many changes are made to the code base. Each of these changes could affect other parts of the program.
Automated tests help find such cases. Once such a problem has been identified, a developer can check why this is happening, if this is something that should be happening (and then adapt the test) or if this is an error (and then look for an error in the respective code, and adapt or correct that).
In our calculator example, a developer adds new functionality to the program, let´s say the ability to calculate square roots.
They might need to change or add some algorithms, and our multiplication function could be affected.
Now, multiplying six by nine results in 42, which is the wrong answer (in our universe).
How do you find out? Manually test all pre-existing functionality of the calculator? This is fine as long as there are only two features, multiplication and square roots, but look at an actual calculator, nobody actually tests all other functions when they change one, it´s just too tedious:
The developer happily finished programming the square root function, it seems to work, all is well.
In the meantime, they might have unknowingly broken the multiplication function, but didn´t notice.
Here, automatic testing steps in. After finishing their work, the developer runs all tests available on the calculator app. (Or they run themselves.
The test for the multiplication function would fail, and the developer would be notified that their changes have broken something.
A calculator is a simple application. I do not know how many lines of code reside in this monster:
Automated testing in video games
So why did I think that testing video games in an automated manner is too complicated?
Look at this scene, there are only two characters, but still, there is a lot going on:
The amount of input parameters and variables in video games are staggering, and so is the amount of output that is generated.
Let´s take a fairly simple example:
The original Super Mario Bros.
At any point in the game, there is a multitude of possible game states.
The game keeps track of the level architecture, the enemies, the player, and items. But also the score counter, live counter, coin counter, objects in the game world that can be interacted with (e.g. the blocks which Mario smashes his head against, coins or vines erupt from them… Mario is absurd.).
Each of the objects in the game world can move independently of the others, there are things like collision detection, movement speeds etc. to be taken care of.
Oh, and all of these things happen on a timer, the game world updates 60 times per second.
To test a very simple game scenario such as this one on the right, very complex input and output scenarios have to be created.The input is not too hard. You can break it down frame by frame. The expected output however?
Very difficult. You would have to specify the expected game states of, at least:
- the goomba (dead)
- the 4 question mark blocks (2 are now empty),
- the 3 bricks
- the mushroom
Now what would happen if, for example, the developers decided that the wanted a faster game? Let Mario move 10% faster. Or the Goomba is a turtle now.
Our test scenario is broken and needs to be rewritten.
Adding more complexity
Additional difficulty arises when you have to take into account that the game might have to run on different hardware and software configurations (E.g. various versions of windows, different graphics cards, CPU etc.).
The game should also not be silent, so sounds should be played at the correct time, the list goes on.
Of course, some games are better suited to automated testing than others.
Turn-based games, and classics like Chess should probably work already.
On the other hand, 3d games, where
the player has six degrees of freedom of movement, it suddenly becomes very hard.
I wouldn´t know where to start.
Calculations are very good subjects for Unit Tests.
Things like damage calculation in RPGs can be tested with Unit Tests. E.g. a character with fire immunity should never take fire based damage. Stuff like that.
Classic video game testing: Why bother?
So you might ask, why bother?
Couldn´t you just hire a bunch of students, they want to play video games don´t they?
Sure, they do, as do we. But usually, you play a video game until you have finished it, or until you stop having fun with it.
You play it at your own pace, and do the thing that is the most fun/excitement for you?
The reality is different. Game testing is hard, tedious and low paid work, that is apparently also severely “unfun”.
Part of this problem comes from the fact that they have to test stuff that could be tested automatically.
With a shift towards automatic testing, the focus of these test users could go from
“Does it work?” to “Is it fun?”, “Does it look good?”, “Do the controls feel right?” “Is the puzzle in the fourth level too abstract?”, etc.
You know, concentrate on the stuff that humans do better than machines:
Finding the fun in the game, identify bad level desing or missing clues, getting the pacing and “feel” of the game right, finding visual glitches and audio problems.
The Final Form: Tool assissted speedruns
Something I thought of, shortly before hitting the “Publish” button on this article:
This means that tool assisted speedruns (TAS)
are basically the final form of testing a game.
A test scenario for one particular version of a game, on one particular difficulty setting, with one specific path throughout the game. (An example.
If you are interested in this kind of thing, the TASBOT demo at SGDQ 2015 has a good explanation by a nice man named Dwango AC: Youtube
I think The Game Kitchen are going in an interesting direction here.
If they do this right they develop their game AI and ensure a smooth development process with the testing grounds they put up.
I´d love to have a look at this.
A game that learns to plays itself, dreaming of being played.
A simulation of a fantasy scenario, in which a fictional character – controlled by a machine that thinks it is playing a video game – fights for survival against an equally unconsciously manipulated virtual antagonist.
There is a kernel of weirdness hidden here, I just can´t quite put my finger on it.