I’ll first let you know what’s new with Freekick core development. There’s been a lot of small improvements – I’ve added some server functionality such as allowing human players to play along or goal detection, but the most important update is that the OGRE client now has the ability to control a player and kick the ball around. This actually makes Freekick a soccer game that can be played. This doesn’t mean, however, that it would actually be fun to play the game – the relatively silly AI makes sure we’re not quite there yet. Another thing that makes the game less fun is that the clubs aren’t really there; the players have no differences in skills or personality, there are no kits or club names or tournaments.

The next thing for me to do is try and eliminate these two problems: first, the AI needs to be made better. At the moment it’s not really that bad – they pass the ball and try to score – but they only have a few actions which makes the match seem repetitive, and the goalkeepers are horrible. Second, the soccer match needs to have a menu around it, including creating and viewing tournaments, lineups, league tables – the whole deal. Actually I’ve already started work on that, which brings me to Python.

You see, working with C++ the whole time with Freekick core was not bad, but after you’ve seen something like Haskell in action, sooner or later you start to wonder if everything really needs to be so complicated. I’m talking about the sheer amount of time spent typing the code in, the loops, the iterators, the classes and headers and declarations, the curly brackets…

As Freekick was designed with modularity in mind and was split to multiple processes from the beginning on, the question of using Python for the game menu arose. After poking around with PyQt for a while the first drafts of the game menu were already finished. I had had my doubts about dynamic languages and duck typing before, but now, after having written a few small applications with Python I’m slowly starting to see the advantages such a dynamic high-level language can bring. I was amazed how fast you could actually write a simple-looking Python script that does the same thing as an application you could spend weeks writing in C++.

Encouraged by my Python adventures, I played around with the idea of solving the other problem – AI – with Python. Since I don’t really fancy writing all the Freekick client side logic in Python, I thought about using Boost.Python to export the C++ classes into Python. Doing the experiment was interesting, but the unfortunate truth is that exporting the classes still seems like too much work, even with the excellent library, especially with the page-long error messages and the fact that the STL containers would apparently also be have to exported manually. The other way would be embedding Python in C++, but I’d still have to export the data structures to Python and back, which makes me think it’s probably faster just to stick with C++ after all.

The GUI part is still far from ready – the only event that starts a match at the moment is a friendly match, and the GUI-match interface is non-existent at the moment – but the base is there and will be expanded when the time is ready. At the moment I think improving the soccer match itself is more important, since there is really no good reason to start a match if playing it isn’t fun, which brings me back to C++ and the AI. I’ve been reading the book “Programming Game AI by Example” by Mat Buckland again, which really is quite an inspiring work, but I think it will still take a while for me before the AI code is in a state where it both seems halfway smart and is easily extensible.

Luckily there are still a couple of mini projects for me to do in case writing AI gets boring. First of all there’s the GUI with all the tournament and schedule creation to do, but I think the time is also slowly getting ready for another Freekick client, this time with ncurses as the “graphics” library. I’ve been wanting to get my hands dirty with ncurses for a while now, and I’m all for minimalistic software, so hey, why not. Other fun things include creating a TV style soccer camera in OGRE and taking player skills into account on the server side. I’ll keep you posted.

Valve has always been a game developer corporation close to my heart. From the early days of my childhood running scared through Half Life’s laboratory complex to the more complex Half Life 2 Saga.

Now, of course, they’ve released another game called Left 4 Dead.

Left 4 Dead, as you might tell by the picture, is a horror, a survival horror. Now, I’m not a big fan of these kind of games, because normally I play to be entertained, not play to later require a change of underclothes.

Even worse, it’s a cliché; you and three other people have to shoot down hordes of zombies in order to escape from the zombie infection.

But it isn’t as bad as it sounds, Valve has developed a system they call the ‘AI Director’. This Director controls where the zombies spawn, what kind of zombies they are and how many there are. It also controls what items spawn and where. Everything aside from the terrain and the survivors are controlled by the AI Director.

Read the paper:

On-Line Case-Based Planning

by Santi Ontañón, Neha Sugandh, Kinshuk Mishra, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-09-08.pdf
www3.interscience.wiley.com/journal/123263882/abstract

Kurukshetra 09

We wanted the event to reflect the current trends in the Game AI development industry. It is left to the participants to be the judge of to what extent the main goal is accomplished. The trends seem to be

• Player specific content
• A move away from hard-coded-rules based opponents.

We chose Lunar Lander to be the test game for both problem statements. Why? Lunar Lander was one of the earliest games to use Vector Graphics. But that’s not the reason why. Lunar Lander is simple, to code and to play. And.., it seemed to offer scope for hacking both problem statements. And now to the problem statements.

The First problem statement is a challenge to create a player that plays the Lunar Lander. You could try a rule based approach and try coding it. And you may succeed too. But I’ll bet the easier way out is to make your player learn how to handle the Lander. There are many situations where a rule based approach would fail. Consider the case of a car racing game, a rule based approach may satisfy the initial requirements of the game. Your racer may successfully out-race the human player, but it wouldn’t do well on tracks you, as its programmer would not have seen.  Because your AI doesn’t know how to ‘drive’. Well, that’s the motivation for this problem statement.

The second problem statement is with regards to content creation. Simple games such as Lunar Lander, I think deserve to engage the player for longer than 3 levels. The content of the game is very simple. It involves very little graphics. Is it possible to use modern / not-so-modern techniques to generate an infinite set of levels?? Theoretically, yes. Further, would these levels, ‘engage’ the player as much as hand-crafted levels would? It is definitely worth a try.

Well, if you are a Game AI developer PRO, and all this seems, well, ‘childish’ to you,  or in case you disagree with what we have identified as the current trend in the industry, try the third problem statement. If you have any innovative ideas on game AI, an implementation of your concept is most welcome. Although, we are not intending to accept any text-book based search algorithms. Even slightly modified but better versions are accepted.

This is the link to the event’s page. Please go through the problem statements again. This is the first time we are attempting such an event. Your comments and suggestions are welcomed. They would be most invaluable to making this a more successful event in the next editions of Kurukshetra. Have fun, Merry Christmas and Happy New Year!!!

]]> http://dankline.wordpress.com/2008/12/11/ai-architecture-anim-ai-layer-con/ Thu, 11 Dec 2008 17:35:51 +0000 Dan Kline http://dankline.wordpress.com/2008/12/11/ai-architecture-anim-ai-layer-con/ Dave Mark raises a good question:

Hmpf. God help us all in trying to get this bit squared away anytime soon, Dan.

I’ll just stick to the top 3 layers and let other people figure out the rest.

Should the Anim Coordinator be in the province of the Game Industry AI Programmer?  I define success as being used by designers and animators, shouldn’t it be their runtime support engineers?

In our cases, I would say the AI Programmer has to get this started.   While long-term I hope that there are specialists or standardized tools to handle this problem, but we aren’t there yet.  And right now most of the information originates in the AI decision maker, and the coordination is the main outlet for traditional AI data, such as locomotion planning, AI decisions and gameplay data.  It’s our bottleneck.  The animation layers, on the other hand, are used to thinking they run almost independently.   Hey, they run the player too!   So Animation programmers can’t see the problem, and Locomotion programmers are frequently too busy and specialized to see it either.  If another discipline had thought it was an important goal, we’d have a better concept to build on!

Plus, the benefits to the AI Programmer are significant.  So much of AI’s value comes from the animation results, and that means having efficient tools to convert sophisticated AIs into complex animations.  Even though I haven’t seen great success yet, what I have seen shows the value and pay-off for AI.  Anim script is a great parallel – it used to be that no animator wanted to do scripting, but many now have seen the value in being able to work faster, add variety, and add new techniques.  I want those same gains in my AI’s output!

I think we’ve got a pretty good generally consensus on most of these, but there’s that vague one in the middle that we don’t talk about much and it’s kinda a mess.  That Anim-type chooser thingy.  I’ve heard it called the Animation AI or just the “state machine” for those who’ve given up, but usually it just ends up being this vague code API that relies on some sort of horrifying combination of data-driven and hard-coded strings.  If you’re lucky.

It should really be called something like the Anim Coordinator.  An Anim Coordinator is the layer that takes your vague AI request and identifies the specifics needed for the Anim layers below.  Back in the day, we thought this was pretty simple, but animations are getting pretty complex, and there’s a lot going on in there.   For example, given the “go there” behavior, the Anim Coordinator identifies that the full body should then be walking, and serves as a central hub for fielding the related data requests: the destination from your Behavior, how much to turn from your Locomotion interface, the walk type from the game data, or whether you can even walk right now at all or have to wait for something, for example.  The Behavior above doesn’t really care about these things.  It wants to know when you’re done getting there, or maybe if you failed, but for the X seconds that it takes to negotiate the world, the upper part of the AI goes on holiday and can spend its CPU cycles getting drunk on jagermeister shots.

But just because the upper AI’s decided the best thing to do is go across a room doesn’t mean animation has enough information to pick some keyframes.  There’s signals in and signals out that need to be processed, game state requests, conditionals, transitions between sequences of anims, etc.  In particular, sequences of synchronized anims is tricky, like the Quick Time Events in God of War, and yet something that builders should be able to easily do frequently, and with substantial variation.  (Do we really want the AI to have to have  seperate Stun behavior for every sequence where the AI is stalling waiting for a button?)

So far, the best architecture I know of for this is the finite state machine.  And there’s one big advantage to using a FSM – your player is probably already using one.  Most games with lots of actions available to the player have a similar problem.  Players hit buttons, which sends signals to animations to play.  But not every animation can play after every button press, so there’s a whole FSM of valid animations from every point in time, game state interfaces and queries, conditions, etc.  In my experience, works pretty well.  Looks pretty much like what an FSM is supposed to look like.

There’s one problem – the Anim Coordinator doesn’t look like that.

In practice, most AI Behaviors want to be able to switch to every other behavior.  Consider Behavior Trees, where there are traditionally no limitations on valid transitions, just pre-conditions in the Decision Making layer.   If the Behavior says it should run, 9 times out of 10 the Anim Coordinator should try and comply.  That means every macro Anim state in your FSM is connected to every other macro Anim state (assuming some sort of hierarchical organization for sanity).  That’s the worst case for an FSM architecture imaginable.  Oops.   Some games have used planning to map through this morass, but that doesn’t make the FSM any easier to work with, it just means you can have longer independent state strings.

In the other Anim Coordinator architectures, the parts that seem to work well are these automatic transition-based approaches (insert the technical term here) – recognize the request to switch anim types, end the current type, and start the new one.   Unfortunately, the UI for these is pretty horrible, and in an ideal world, an animator/designer can go into your Anim coordinator and add custom transition states and custom game state reactions.  I’ve never once seen that happen.  I heard of it happening one time, but apparently that animator was a whiz kid coder too who half-built the FSM, so I’m not sure that counts.

We need a UI that expresses this  “send all” and “receive all” concept with good debugging to follow them, but still has all the functionality of the hFSM, for those cases where we really do need to force a certain kind of transition.

I want to see animators/designers sitting in front of these kinds of AI tools and using them.  I think this time around I’d like to try a hFSM with these 2 special kinds of jump nodes, and make my initial version of the AI only use these nodes.   Then allow the hFSM to use custom signals that different parts of the tree can set, to emulate a level of content scripting that the game state code doesn’t really need to know about right away.  Like “my posture is sitting now”.   It sounds simple.  Hopefully, this time we’ll get it right.

Developing a Drama Management Architecture for Interactive Fiction Games

by Santi Ontañón, Abhishek Jain, Manish Mehta, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-08-11.pdf

It seems there’s 2 different classes of problems that just drive me nutty, because it’s difficult to tell when I’m dealing with them.  Knowing the difference between them helps.  I call them stupid problems to solve and dumb problems to solve.  Consider a simple situation of trying to add 2 numbers.  Here’s a stupid solution:

function AddUp(int totalToAdd, int& originalValue)
     originalValue += totalToAdd

This could be called the “why bother” solution.  Here’s the dumb solution:

for(totalToAdd; totalToAdd > 0; totalToAdd--)
    totalSum -= 1;

Otherwise known as the “WTF” approach.

Now both of these are not good solutions, but the dumb solution is way worse.  It not only doesn’t solve the problem well, it makes every problem harder to deal with too, because it makes everything using it unreasonably slower.

So am I just saying don’t be dumb or stupid?  If only it was that easy.  But not every problem is this simple.  There are some genuinely hard problems out there.  And every once in a while I find myself working on something, and discover I’m actually trying to solve a dumb or stupid problem.  Stupid problems are easy to fix:  just ask someone else whose done it before.  But some problems are inherently dumb.  These problems are evil.  Think NP-complete problems.  These problems, by their very nature, will eat up your time and drag your project down.  They will seem right, but every solve you think of will bloat your schedule, or your CPU time, or something.  That’s a sign your problem could be a dumb one to solve.  But frequently, once I realize I’m tackling a dumb problem, all it takes to fix is to take a step back, say “This is a dumb problem, there has to be an easier way to achieve this.” I just change the question I’m answering to something easier, or cheat the output a bit.  Most of the time, I didn’t really need to be solving the dumb problem in the first place.

Yes,  I guess that means most dumb problems in development are stupid problems as well.  What can I say.

Okay, what’s this all about?

In this blog I’ll be writing about artificial intelligence, game design, virtual worlds, massively multiplayer online games, social games, and a variety of usually related subjects.  I welcome topical, respectful discussion.

My background: I run Online Alchemy, a small company in Austin, Texas, specializing in online social games, particularly those with a virtual world component.  We have been working for six years on advanced social AI for games and simulations (I’ll talk about that in another post), including working for several years with the Defense Advanced Research Projects Agency (DARPA). We have also created a unique social networking technology and have a variety of games in development.

Read on for more details of my past (well, not that many).

Before founding Online Alchemy in 2002, I worked for Electronic Arts for three years as a Senior Game Designer, leading game designs for several projects including SimCity Online, The Sims 2, and Ultima Online.  Prior to that I co-founded and was the Chief Creative Officer for The Big Network, an early family-oriented social networking company.  There I designed and produced several Java-based games, and in 1998 designed our primary product, MyPlace.   Before The Big Network I was one of the co-founders of Archetype Interactive (started in 1994, acquired in 1996), where I was the lead designer on Meridian 59, the first 3D massively multiplayer online game (MMOG).

Before getting into games I worked in software engineering, primarily in developing 2D and 3D CAD systems, and in user interface design and user-centered design for medical and scientific visualization systems.  In the early 1990s I designed the user interface for the first 3D visualization system used in the operating room by neurosurgeons.

I have a BS in cogntive science and have done graduate work in AI (expert systems, neural networks, genetic algorithms, etc.)

Oh, I live in Austin, Texas, with my wife, the last of our six kids (the rest are married or in college), three dogs, and a sugar glider.

Commercial and Academic Research in Games

Generally, most research done by academics is of some use commercially. And naturally, one would expect the same to be the case in game AI research also. Only, its not. Commercial games concentrate more on the graphics, and the user experience part of the game, when compared to AI. If there is a bold announcement from the industry stating that the next edition of their game is to have the ‘next-generation-human-like-AI’, they have mostly been huge let-downs, and miserable failures. The strongest AI, in commercial games, these days are those that ‘cheat’. Have you never wondered in-game, at least in the harder levels, ‘When did he get so much points?’,'Does his nitrous never empty?’. And sometimes just plain ‘WTF!!’

Instead, the NPCs ( Non Player Characters ) in the majority of games are controlled by finite state machines, lists of conditional statements, and the occasional A^* path finding algorithm. Typically, no learning is present in either development or execution of the AI. Everything is hard-coded by the human game developers.

-Julian Togelius

Academic Research into game AI hasn’t been too rewarding either. So we built the strongest Chess player in history. And we are building better ones. Has that helped us understand how people play the game? How we learn to play the game? How we interpret patterns? Also, Academic Research is mainly focussed on making AI increasingly better at the game, aiming to make it the best. Naturally, that’s not the commercial requirement. Would anyone but Kasprov want to play Deep Blue? What is required commercially is that the player’s opponents should neither be too domineering, nor too easy.

How Games interest us

According to Raph Koster, a game is fun to play because we learn the game as we play; we understand and learn the patterns underlying the game, and finally “grok” how to play it. This requires that the level of challenge always is approximately right, and that new patterns are always available to learn – games that are too simple or impossible to understand are boring.
Another theorist who has tried to nail down the essence of games and why they are fun is Thomas Malone. He claims that the factors that make games fun can be organized into three categories: challenge, fantasy, and curiosity.
The existence of some sort of goal adds to the entertainment value. Further, this goal should not be too hard or too easy to attain, and the player should not be too certain about what level of success he will achieve.
Games that include fantasy, according to Malone, show or evoke images of physical objects or social situations not actually present. The sensation of being somewhere else, being someone else, doing something else.
As for the third factor, curiosity, Malone claims that fun games have an optimal level of informational complexity in that their environments are novel and surprising but not completely incomprehensible. These are games that invite exploration, and keeps the user playing just to see what happens next.

The most interesting applications of real Computational Intelligence and Machine Learning methods

NERO, is a fantastic new concept. The player ‘teaches’ (initially stupid) bots to become ‘trained’ soldiers. In the core of the game is a learning system called NEAT. It involves using Genetic Algorithms to train Artificial Neural Networks( NeuroEvolution ). NEAT involves continually taking ANNs, from an initially random population, and then combining the best among them in the hope that it’ll result in a better ANN.
Another place where learning methods may be put to use is Automatic Content Generation. Here is an intriguing new concept. Customized racing tracks according to a player’s skill. The next Prince of Persia edition claims, custom player experience. Although it is not clear of how ‘custom’ it is to be, this sounds like the beginning of something big. So Automatic Content Generation is definitely IN.
Another instance of Content Generation, in the process of development, is a game called Galactic Arms Race, being developed by Ken Stanley, and his team at UCF, the same dude who pioneered NERO. Remember the classic, Space Invader arcade game? It seems to be something similar except, there is a large and INCREASING variety of guns you can choose from. And each gun fires differently. That is, the graphics is different. And the graphics was not coded by hand, it was EVOLVED by a custom version of NEAT.

Game competitions at GECCO

GECCO 2009, is to conduct two competitions, Salamander and Car Racing. The details of Salamander are not too clear to me, but looking at this and this, it seems, the Salamander is to learn to walk and then catch flies around the house.
The car racing controllers are tested on TORCS. (TORCS, does not have dynamite graphics, but I find it a lot more difficult than NFS, but thats just my perception. It was built mainly for testing AI conntollers). Last year’s entries were quite a variety. There was a controller based on NEAT, and there was also a hand coded controller(Although, I doubt it performed as well as the other trained/evolved controllers, score CI/ML!). The significance of a trained controller is that, provided it is trained sufficiently well on a good track, it acquires the basic driving skills. This would be a big plus for games like Trackmania Nations, which allow the gamer to design his own tracks, but does not give competing cars, leaving the gamer to look for human opponents, to race in his own track. Another aspect of trained units is the versatility. A small change in the training process and the controllers become more aggressive, trying more to push the other players off the track!!

And now..,
The crux of the matter. If you were to host a contest, which game, do you think, considering the expanding scope for computational intelligence in games illustrated by the examples above, would be a good bed for innovative application of CI/ML?
Remember, its a time bound contest. The participant should have enough to time to experiment.

(links to my Day 1 and Day 2 conference live blog)

Halo 3 Objective Trees: A Declarative Approach to Multiagent Coordination by Damian Isla (Bungie)

Damian Isla is stealing from me, I swear.  (Kidding!)  Unfortunately for me, this is totally the talk he gave at GDC 2008.  He’s sketching out an Encounter Manager – an AI that designs encounters and orders AIs around. He’s still talking about it in terms of squad tactics rather then narrative though – how groups move around, retreat, create “spice” variety.  I remember this sort of thing being awesome back in Halo.  Isla divides the encounter space up into territories, based on the player’s approach. He lays out a 3-step, 3-pronged encounter that he says is “simple”. Within each territory, he cites “tasks”: the language mission designers for telling squad AI what to do. Tasks identify an squad’s territory and their behavior – like aggressiveness, rules of engagement, and inhibitions. These aren’t individual instructions per-say, these are captain orders. In fact, each territory is a set of discrete points, like way points, that the low-level individual AI cycles through Fight->Flee->Cover inside. So it’s easy to switch territories on the AI – it just tells the low-level AI to use points somewhere else.  Very elegant. Ah, they did do it old-school style in Halo – I was right! A really neat demo of the island map where the marines springboard between rocks along the beach, switching territories. Also a really interesting demo of how the marines move and choose points – they move a lot more then I realized as a player. He draws a pretty strict line between the Individual/Squad AI (AI Engineers and designers) and the Task/Mission AI.(Mission designers and Isla). In Halo 2 they just used a FSM with script tests with testable transitions to try and solve this. But the scripts got too complicated for some missions because of non-deterministic branching that lead to almost complete state machine interconnectivity.  FSM main limitation is that transitions have to be explicit, and that can get crazy.

Imperative or Declarative?  Imperative is line-by-line straight “how”, whereas declarative, like regular expressions, just describe what they are looking for. Imperative has more flexibility, but declarative is more maintainable and simpler. Declarative is just flat-out better for designers, but Isla says we usually do imperative . The Sims did a simple declarative model with their environment based AI, with the agents so simple, climbing the “happy-scape” of the object-space. In behavior trees, tasks are declarative, and have priorities, can be made up of sub-tasks, and have a finite capacity (because the territories and goals have finite capacity). So he went back to behavior trees again. In an aside Isla points out that what make behavior trees unique is that the decisions are just a prioritized list (although there is equal priority that splits individuals evenly), which is just simpler then weighting or percentage. Plus it’s nice that behaviors are self-describing and explicitly know only about themselves, and no external knowledge. They fit “objective trees” for his tasks well. As Isla puts it, it’s a dynamic plinko machine that’s run thousands of times! Doesn’t sound like he’s evaluating all trees all frames – just the conditions of the current tree until it’s finished or failed, but that’s likely a developer call. Next, he runs through the algorithm, which is straightforward. I love how descriptive Isla is – he doesn’t shy away from the math, but he keeps it understandable and his techniques simple.  The most interesting part is the cost function that determines what task should be chosen by which squad – because positional awareness of territories is important, but that requires inter-agent “communication to avoid making other squads look stupid. Instead, he weights areas by “all groups distance covered” – the cost of the distribution is the sum of the cost of all individual’s distance, and then takes a greedy approach.

Bungie made refinements throughout the process from designers. They added task “filters” which were type-conditions to limit who can use tasks. They used infinite costs to represent a bad filter match. They could keep tasks open or closed forever, independent of whether the initial condition was still true. Death and Living count were explicit conditions for this too because they were so frequent. Bungie also explicitly supported one-time assignment to a task. In examples, Isla showed how to do leaders with leader filters and no leader filters, and then created a “squad broken” task, that would activate when a leader was killed. (This could have been a broken override for all tasks, that shuts them off explicitly for 10-20 seconds, but I think it was just a high priority task option). Second Isla showed player pickup in vehicle encounters. He shows the vehicle filter, a player that needs vehicle detection, and then a pickup task. Designers can add this task at the top of the tree at the very end and the transitions just work themselves out.

Badness summary – designers require training for declarative languages, and there was a sometimes awkward relationship between overall level script and objectives, particularly dialogue syncing with objectives changing, and tying together enemy and ally fronts.  He tried matching the two fronts in concert with a scripting version because the enemies and allies were 2 separate behavior trees. And lastly it’s not always intuitive to bucket by squad – because in Halo 3 individuals in squads couldn’t change squads when they change roles (like, grabbing a new weapon). But designers liked it, the system took requests well and it matched how designers think about encounters. It was a great quick prototyping tool. And it scaled well so everyone used it for all cases (rather then mixing with script, say). Interesting, it was also driven from the UI up, and then they chose the data and the trees. Declarative showed its chops – less direct control and flexibility by author (reliant instead on AI), but more concise representation, more manageable, and ultimately better results.

Question: How was training? Interestingly, the whole system was design requested, so design put forward the initial tree idea and owned the training, which really drove success. Question: The most hackiness was the front coordination – they just sort of globally script the major steps and gate both sides until they reach major points. Question: you’re using global knowledge, might not work for many players? Isla says true, but just design and tweak to make realistic, and give player feedback from AI for whys (animation, alert state, etc). To add, talks about awareness activation conditions on tasks. Question: Influence map for the fronts issue to do mapping? Issue would be designers like to be really specific in how things fall apart.  Might work well for allies.

Navigating Detailed Worlds with a Complex, Physically Driven Locomotion: NPC Skateboarder AI in EA’s skate by Mark Wesley from EA

After coffee, talks on paths and path following.  How do you make skating bots? Build skate paths in an editor. Unfortunately I missed some of the setup details, but there’s a path editor, and it looks like they are just creating short “skater lines” using curves and about 5 velocity points, and the skaters just direct navigate between the ends of the lines. In fact the skater tricks are encoded into the path itself.  Wesley even had path recording from player input to make it easier – creater activates, does trick, creates path. Skaters use local checking to navigate the board along the generated path. Then on top of this there are skater profiles in the AI that influence when they do the tricks on the path, use path following, and dynamic avoidance. EA also had automated testing – because AI Controller could be attached to anyone, including the player, so easy to soak test the paths. It’s a large open world – do they need a lot of paths? Yes, Wesley says, and EA relied on QA to do it. 5,000 of them, ¼ million nodes, 465 kilometers. They streamed the paths in, which was simpler then it sounds because they were small and discrete from each other. Pros: it worked and was understandable, easy to script to constrained paths, and otherwise the random path constraints gave nice emergent behavior. Cons: skaters were constrained to the path, requires a lot of data, and paths can be invalidated if underlying world moves. Wesley had a few automated tools to fix paths when art changed, but EA mostly just did paths once the levels were done.

The Rise of Potential Fields in Real Time Strategy Bots by Johan Hagelbäck from Blekinge Institute of Technology in Sweden

Next up in the whirlwind is Hagelbäck. More bot building! He’s pulling potential fields from robotics, used primarily in obstacle avoidance. It’s an attractive or repelling charge at a point, looking a bit like an influence map once summed up. You can generate paths using the map as a cost.  Looks like Hagelbäck uses it to get the destination, as well as the path.  His domain was Orts, an RTS. First, find game objects, then put driving forces on the map (both from objects and goals), and then build your fields – static ones from geometry, semi-static from goals, and the really dynamic from agents. A couple of tweaks to try and get better fields, control oscillations using the field itself. After the latest version, their bot was 99.25% effective against other bots in a tournament. Interesting how it positionally aggregates to handle oscillation and can deal with incomplete information.

A Cover-Based Approach to Multi-Agent Moving Target Pursuit from Alejandro Isaza

Isaza is looking at target interception pathing, using arbitrary, undirected, known graphs, turn based model, with one target and several pursuers. Problem with MTS and A* is that they can reduce distance, but not mobility, and there’s no team coordination.  It can work, but it’s not appropriate. In the demo you can see that A* just chases forever, never catches. When he says “cover” he means “corner”.  The AI in corner should progressively cut off territory from the target. It’s an interesting idea, particularly in squad shooter or horror games – you can definitely make them look a bit dumb now. Is it more fun though if the player has to avoid them when they’re closing in? Isaza’s trying to solve it – by maximizing his “cover” graph rather then minimizing the path distance only.  First, divide the space in two, then determine which is covered and what’s not. This gives automatic coordination if you share cover graphs and maximize the team’s cover. To get the cover graph, do a Breadth-first expansion simultaneously on each agent on both sides, prioritizing time-to-reach, which takes O(vertices) time, and stop the expansion where each agent’s graph meets. The cover graph is a measurement of what the agent can get to before the target. In a refinement, Isaza also incorporates risk using an offset –a willingness to sacrifice a certain percentage of cover to get closer. 10% seems to be optimal from his data.  I’m not sure I got all the steps here clearly, but it seems like there are some really good opportunities here, particularly with, say, large vehicles trying to block off ground forces – tasks that are less about one agent’s survival, since the algorithm assumes all agents are invulnerable. But it could also be incorporated with other techniques to be generally useful too. Something I expect to see implemented for certain goal types in all AIs in the future.

Talking with NPCs: Towards Dynamic Generation of Discourse Structures from Christina R. Strong from UCSC

The last session at the conference is on Language.  Strong’s investigating dialogue generation. She starts by defining “dramatic beats” – the smallest unit of dramatic action that changes character relationship and story state – aka a little piece of interaction.  Facade was authored with this in mind. She’s generating discourse structures using beats automatically. Not just sentences, conversation. To do this, she needs to explicitly representing social games, characters, and social relationships. So she uses planning to make a FSM for a beat. It’s important because Facade‘s beats are all hand authored. Mass Effect tried to reduce dialogue through design, but still had 28,000 lines. She wants to minimize the amount of writing the author has to do. Explicitly represent backstory, character, relationships, and higher level author goals, creating a wider variety of conversations then a human author could.  Plus, a good story should have affinity beats as in Facade – getting player to side with one character or another.  And a beginning, middle, and end.

So she uses Hierarchical Task Network Planning.  Higher level tasks are author goals, lowest level are dialogue.  She also uses forward state progression (?), which allows for functional effects like tension.  Her approach generates linear plans for specific situations, but doesn’t incorporate player interaction automatically, unfortunately (probably because they are non-deterministic).  To plan the conversations, she first chooses the characters, and then identify the steps are required to get the desired result.  In the middle parts of the conversation, she pads with support, questions, or past information nodes, to then force the player to make a choice.  She incorporates an annotation operator so that the FSM can represent all player choices. (Although, I’m not clear here why the whole conversation has to be actively mapped instead of dynamically generated to respond to the player.  I believe this is what Façade was doing).

She’s using small chunks of pre-written dialogue from a library, presumably because you can use less dialogue by matched more to goals then the old scripted situations.  In her work so far, they have generated 120 stories so far, and over 2000 unique dialogue FSMs – each different information or told in a different way.  It’s interesting that you could do this with vocals as well as text – it’s similar to what I was planning at one point to handle dialogue in the Encounter Manager as beats.  Her most interesting conclusion was that adding just one new information node to a character can double the number of conversations.  Strong wants to explore “Mixed Initiative Planning” because her current preconditions aren’t always good enough.  And she want to find ways to increase player interaction. Very interesting work.

Question: What about incorporating character actions? A great question – seems possible.  Facade plans actions and dialogue simultaneously. Question: Are you really helping the author do less? (Can we measure the savings? Hehe, I’ll bet Far Cry 2 is doing exactly this just out of necessity, but a good question) (Lines are annotated with tags in both approaches.) Strong points out the dialogue generated is more dynamic then pre-written because it can use dynamic information, so it might not matter.  I’d like to see someone actually try to compare the work; depending on the scale and linearity of the game, I’d expect this work to be easier.  But it’s questionable if it would work with vocals – whether planning lines of vocals out like this will sound natural .  It’s possible voice actors would find it difficult to act out these non-linear lines.

Learning and Playing in Wubble World by Wesley Kerr

The last lecture is a language construction proposal. Kerr’s building a Sentence-Scene Corpus. He wants to build a game for kids for free, have the kids play the game using his language, record it, and use that data to build a word identification library. Why kids? Cheap labor, successful outreach, patient with dumb agents, and intrinsic desire to teach.
Http://www.wubble-world.com
is his site, a collection of mini-games. You talk with your wubble and it learns language from you. You tell it about game objects, actions, etc. Kerr is parsing the sentence semantic construction of the kids, using specializing weights to learn common phrases. Active wubble question-asking gives the interactions impetus. Problem, what if kids aren’t accurate? Yep, this happened, particularly with things like size and color of objects. But that’s what they are looking for, concepts. And they measure the probability distribution, and they gradually get closer to the right concept. More of an issues with sample size, because it was hard to differentiate, but this will go away at scale.

Generally, the wubbles were able to get accurate language concepts. Conclusion is that not only are games entertaining and can teach the player, they can teach the AIs right too. Kids love the online learning results, and it’s massive and cheap. Biggest catch is it’s specialized to the provided environment. They have other games they are trying as well.  In one, Kerr requires team coordination chatter and they use voice->text translators to analyze the output to understand the relations between objects. Another game has an active player avatar that can demonstrate verbs directly in the world for the wubble. Question: was there “language griefing”? It didn’t help them solve the task because it made it harder, but it will be a problem.

The Past, Present, Future of Game AI by Steve Rabin of Nintendo

Looking forward to this talk by Steve Rabin.  In his career, he’s talked to over 100,000 game players at Nintendo, helping them out. He’s been at Surreal, WizBang, and Gas Powered Games.  Now at Nintendo doing development support, as well as AI Game Wisdom and an instructor at DigiPen and UW extension. Going from past to future, Rabin starts with the AI in Pac-Man, a game that took 17 months to create, and has individual ghost AI with no randomness, with wave-patterned attack, retreat cycle. And there’s the implicit enemy cooperation. Then SimCity with cellular automata and influence maps. Virtual pets in 1995, adaptive, emotional, memory. Remember Tamagotchi? Creatures, which is still complex today? And Thief, a new sensing model.  Half-Life – integrating AI into storyline using script.  In 2000 was The Sims – smart terrain and smart objects, similar to affordances.  Black and White with neural nets, empathy learning, and gesture recognition (AI Wisdom 1).  Fable – player reputations (not the first, an example). Halo 2 and behavior trees. Nintendogs and brain age with speech recognition that works in any language. 2005 – F.E.A.R. and the STRIPS system and enemies using the environment – going to more declarative languages rather then procedural. Forza also used neural nets to drive cars that players could train too. And Facade – interactive story, natural language processing, and the ignoring the player reasonably similar to Nintendogs. And last, he can’t help throwing in Gesture recognition in the Wii remote.

In the present, the living cities in GTA 4, Euphoria and animation.  Spore creature creator dynamic skinning and animating, Fable 2 dog and Family sim. Left 4 Dead AI Director to create experience, mood, and tension. Far Cry 2‘s dynamic narrative. Very dependent on guy’s in charge understanding AI and using it. Our current AI has adequate movement, sensing, Behavior trees. It need help, voice, declaratives. What holds our AIs back? Lack of attention, experience, design vision.  But it’s not a lack of CPU time or good algorithms.  Really, most important, why are AIs still allowed to suck? For Rabin, it because it’s not required as the key driver of fun (see MMOs), and designers and scripters who don’t get it.

Where are we going? Cost is getting higher, risk is greater.  Nintendo trying to ease it up, but that’s just one. How do we compete back? Middleware, build once and reuse, and procedural content.  But deeper AI needs expression, which costs money.  Procedural content?  Music, story, creatures names, dialogue?  Sure.  But can procedural be original and reliable?  SpeedTree is a success story.  Yet, we see new experiences and new interfaces are expanding the market. AI also allows for new experiences.  That could be the key.  There’s diminishing returns on graphics, and input, physics and animation are hot right now, but the future is AI and game design working together (of course, at this conference *grin*).  Taking parts of the AI and making it the Hook.  Rabin points to Portal and all of these other physics games coming out now.  We know this can works, see AI pets.

Yes, there are the core game AI problems. save time/money, CPU/RAM, ease of authoring (complexity, scalability, readability, debugability, robustness).  But there’s also gameplay – emergent, adaptive, novel. Even though we’re always fighting how cheating works so well, AI will win.  The challenge is that 50% of AI programmers feel problem is working with designers. In the near-term, the best AI programmers will get better collaboration, and agents will get larger vocabulary.  Bringing up the AI programmer bottom requires awareness, education, proven solutions, and middleware.  Clearly we’ll get to full actors, physical, biological bodies, subtleties of performances, let’s push it now.

So predictions: dynamic personality, style, and emotion, leading to synthesized motion in true performances.  Unfortunately, it’s not clear how game AI gets better by the more cores that are coming. Instead, watch for new gameplay.  Manipulate and toy with intelligent sims – organize, create, nurture, teach. Or just interacting with intelligent sims – command, conquer, persuasion, negotiation, relationships, learning.  Or get inside the intelligent simulation – be part of an evolving story.  Rabin also asks about Chris Hecker’s goal of a Photoshop of AI – the decomposition of AI into it’s core element.  What would this be?  A training tool?  F.E.A.R. scripting?

If we had perfect AI, what games could we make? For designers – the deep future can be modeled with a simple technique; If you could make a game using real humans, what would you make?  The humans could be mates, opponents, companions, mentors, or psychiatrists.  Maybe the game would be murder mysteries, reality TV show, playing God, or leading a battle.  And with a real human writer, you would have a dungeon master.  This potential to recreate us, ourselves, is what makes AI so powerful.

For the researchers, tips on getting research into commercial games.  Start with a problem, not a solution, prove the solution is better then others – more efficient, faster in all ways, player approved, new gameplay – and is designer tweakable.  Then make a great demo, give the code away for free, and get it in front of game developers.  Shoot for the stars, tackle the extremely hard problems commercial guys can’t.  Go crazy.  (And he’s absolutely right here.  Go guys!)

(Architecting Believable Characters the Spielberg way) by Borut Pfeifer from EALA

And lastly Borut Pfeifer, talking about his next title Project LMNO, and the future of believable characters.  He’s worked on it for several years, including in Radical’s spinoff research studio. Looking for these characters, with meaningful choice and emergent gameplay, what has to happen? The player has to understand the AI’s motivation – they have to have consistent rationale/motivation, and the player has to learn it, eventually. All at high fidelity. Pfeifer points to the “suicidal goblin problem”. It is dumb AI, but it’s not ever cited by players as dumb because the priorities of that character are very clear. We just need to bring this sort of clarity to smart AI.

Starting with “Spielbergness” approach, let’s run through the problems Project LMNO is struggling with. The goal is very clear expression, using technology that really creates that ultimate character.  The problems he’s seen come up, examined. First, Conflicting Motivations.  Watching a scene from Munich, the first murder, we see the actors have deep internal conflict, on-screen.  Player needs to see this to reason about NPC behavior and see them as realistic. Plus, the NPC should convey knowledge it has that the player doesn’t (there’s someone over there!). He recommends separating out “executing behavior” and “showing intent” from each other. But when running in parallel?  How do you avoid the complexity explosions?  (Good points!)

Next, Reactions. Watching a scene from Raiders of the Lost Ark golden idol scene that shows body and face reactions, comprehension leading to emotion.  Consideration, thought, giving the non-interactive feedback. Usually we do this with same decision making and executing behavior.  But many different reactions – subtle overlay going to full body, big full body shocks to falloff subtle behavior, important reactions in the middle of non-interruptible action, and you may have to stop a reaction to do something else, and then come back later.  With each Reaction, the player needs to see the connection to the world and get feedback if there is an internal change in the NPC happening.  Not only that, Reactions can cause or direct behavior change and denote the change is happening.   We can separate into pieces – initial, sustained, overlay, full body, but need lots of parameters – physical state, emotional state.  But each Reaction may only use subset of pieces or change the entire flavor of something active.  Watch out!  Complexity explosion again!

Let’s switch to Attention, looking at character focus in Catch Me If You Can, the airport pilot scene where he sneaks past the agents and escapes on the plane. Here agents are searching, distracted, and focused on objects. That all requires attitude/behavior, and player awareness of attitudes. Does each world object create distraction or not? How do we do multiple priorities for targets of current behavior, or reaction? When there’s multiple, procedural targets? Or there’s non-active behaviors that might need to know about possible attention targets. How is this communicated, priorities handled?

And then, Meaning, in ET with Elliot and ET learning to communicate in Elliot’s bedroom.  Objects in the scene have different semantic meaning to the different characters – Show, Play, Eat, Scary, Threatening.  This makes them applicable to different behaviors for different character.  And the player needs to know what meaning the NPC is applying to the target and how to change that meaning if possible.  The NPC has to perform that meaning, communicate it.  That means controlling how the NPC learns, where, and when, and then showing that back to the player.

Plus, Emotional Simulation, in the car crowd scene in War of the Worlds.  There’s a variety of short and long term clips of the crowd and characters – frightened, panic, violent, angry, intimidated.  Each has different impacts on different NPCs, based on where they started in the scene.  What’s the point of all this complexity?  New gameplay possibilities – believable, consistent, richer characters.  Player needs to know the NPCs emotional state, and the affordances for NPC behavior that come from that state that the player can manipulate.  Ideally these are as natural as possible.  Emotional state has to show and hide, based on mood – how does player track it when it’s not available?  Is it too much to keep track of?  Pfeifer’s not thinking float sliders with traits – not directable enough.  He wants emotional states that designers author rules about, and create the transitions in/out.  A FSM above a normal AI system.  It can affect everything below it – goals/actions become active/inactive, attention targets and frequency, tone of reactions and movement, and filter interpretations of objects and characters.

Lastly, Physicality – pathfinding, collision, world registration, object interaction, social nuance as in public/private, indoor/outdoor.  Player needs to know when/how this has changed and to believe it.

So there are 3 recurring problems – control and feedback between layers, mind-body problem, the complexity, explosion of assets, and interleaving authored data where necessary.  The last 2 also feedback loop each other, because procedural data requires authored knowledge to tune and case study.  With Mind-body – layers can fail to talk together, so communication needs to be good about failure, and we should have higher level layers deal with failure.  This is fundamentally software architecture problem.  But complexity and authoring, well, all these sims are intersecting at various times, and some events are more crucial to the experience then others – looking at you in tense situation versus wandering and exploring the environment when exploring. How can we Spielbergify these important moments?

Pfeifer presents the notion of the “Sparse Matrix” – an abstract authoring idea, a possibility space of n-dimensions with emotional states, physical state, history of behaviors, physical limitations represented.  Which combined factors do you need now to perform within the matrix?  Does the agent need to override behavior/performance for any new combination of factors?  We also need to easily define reasonable defaults and fallbacks.  Pfeifer proposes search, filter interpretation, and context author to help.

You need to search your set of these things.  GOAP sure – but behavior, animation planning, any content, all need to be searched.  Think youtube data tags.  Building and adding elements must be easy, ideally it’s declarative with no connections, but still requires a lot of run-time visualization.  Filtered interpretation means reduce the number of things to search on.  Build abstract sets of your things to reduce the set temporarily and get more usable results.  This mapping can be shared between characters, things, as long as the source data set remains the same.  Almost a way of hierarching people’s behaviors (or whatever), so you reduce the complexity for the higher levels when the complexity doesn’t matter to those levels.  And reduces content needs, potentially.  You aren’t removing the context, but abstracting it out, and only specializing when needed.  Using overrides when it really should be a specialized context. (aka A car is about to hit me!)  When considering authoring these context things – there are several kinds.  Global: high level, Are you in combat, in-code global detection.  Regional: one or few factors, affects a few behaviors, like what kind of space am I in,  public or private.  Regional is primarily used in script and filtering/mapping.  And lastly, Local context -  context within a behavior only. An example here is the classroom – if people are inside, it’s a different social behavior set then if the room is empty.  Unless someone is being chased by the cops, then it’s totally different and there should be a global override.  And Pfeifer believes this has to be driven through data and emergently.

So… the whole AI process… game events get tracked, translated into context in design and code, then filtered into meanings, which are turned into behaviors.  Each system passes this down, can search and modulate this, and pass back up failure just to get the intention performed.  Solved!  Well, not quite.  They are trying building an AI around it, and hope to have results to show in Project LMNO.

Question: In AI, is simulation or authored winning?  He says he’s just trying to pick his knobs, because ultimately you have to be able to override anything.  Euphoria isn’t stylistic, for example.  Games don’t do adverbs yet – that’s what he’s trying to bring.  (A nice point, good meme.)  Question: this is very much about social acts, there’s the translation to performance level, what about the reverse: player actions translating to social acts, giving the player emotional control?  Pfeifer calls for the low hanging fruit because there’s easy things players do, but player modeling is difficult, and they are trying to limit their contexts and conditions and building larger meaning slowly over time, in part because the player can only track so much.  Question: cultural emotion? Project LMNO is using non-humans to help create the separation from this.  Spielburg doesn’t mention working on it, explicitly, so it may be that emotions are universal.

(To see my thoughts on these lectures and the conference as a whole, follow my Day 1, Day 2, and Day 3 wrap-up here)

(A reminder.  I’m editing these notes a week after the fact, just making it readable, but I’ve left the original voice in this piece and roughness intact because I feel the live-blogging depth is more useful then a concise, thoughtful analysis would be. I’m happy to answer any questions people have, though, and I encourage you to contact the original presenters as well for more thorough details.)

(links to my Day 1 conference live blog)

The AI of Spore by Eric Grundstrom (EA Maxis)

The keynote speaker this morning, Grundstrom talked about the AI tech of Spore and its player-centric and asymmetrical focus. Starting with the Cell stage, Maxis used a lot of fluid and particle simulation alongside the flocking AI to create unique movement. I believe this was Jeff Gates work, a friend of mine. Behavior was part-AI driven and part-design to make interesting differences between the creatures. The key was always AI that was player-centered personality design. They started with depth before breadth – make the first AI really compelling before you generalize it to different sorts. A funny but familiar aside – when growl music was added for when AIs attacked QA reported that the AI seemed better, even though none of the AI had actually changed. The whole Cell stage asymmetric AI, in that it’s PvE – the NPCs don’t grow like the player and have different goals.

Grundstrom then moved on to the creature game. The goal was to duplicate Diablo‘s model. The key difference with the creature game is they wanted a real range of behaviors – responding to stimuli, interrupting and resuming behaviors, and scheduling some behaviors against time. He started with the “Interaction system” – just rank each action and choose the highest one. Each action runs till it stops. It sounds very similar to the Diablo AI, actually. NPCs get invited to group actions. The big drawback was that it scaled poorly – each action needed custom code to handle interrupts and stimulus to switch to other actions. So the big takeaway – they switched to behavior trees. The win was that they can now have all actions re-evaluate every frame, so interrupts are simpler. They wished they’d done an animation state machine to sequence animations to their actions. Another lesson was to keep the NPCs passive at nests and let players seek out interaction, because the chasing gameplay is harder to do then it sounds – it just comes down to who is faster. You’d get followed forever or you’d be dead. This general role of treating NPCs different from players was important – very not-multiplayer.

On to game 3 – the tribe game. The big new play was specializing and upgrading tools. It’s an economy slider balancing game. Still autonomous agents and packs, so the needs are fairly similar to creature game. Grundstrom kept behavior trees, but wanted to do group behaviors for the tribes. Thus there’s a new join behavior and a lead behavior, and the lead behavior gives orders to the members. This was helped by a group manager – because otherwise if the lead individual was killed, the group would lose track of its goal. There was still a general design focus on player-centered PvE – pace the AI attacks for drama, warn player of impeding attacks, etc. Not much new AI so far – the stages are so broad and vague that I believe individual games have gone farther in each AI type. The organization and reuse became correspondingly more important then normal, however.

The penultimate game AI is “Civ-style” – a very different kind of challenge, more RTS then action game. There are now cities, and a communication screen to communicate with city agents. The relationships between the various agents were very complex – Civ4 style attributes that aggregate to a number, using Civ4‘s the improvement of the added player feedback. Here the AI is strategic – building, upgrading, relationships, managing attack tactics, pathing (for the first time really, beyond steering!), and citizen vignettes that give appropriate feedback to the player. They used 2 kinds of scripts for strategy – domestic and expansion scripts, but the engineers unfortunately ended up doing them all in C++. These scripts formed each tribes plan. Grundstrom forced the AI to attack their lowest relationship so as to create a better story. He also used different relationship decay rates across attributes to add more dynamism.

Behavior trees and groups were again used for the individual attack units. The units routed using A* with some optimization, focusing on the path following rather then the path creation. There were a wide variety of unit types to support: planes, ships, and vehicles. Which meant a desired facing pathing solution as well as a position solution. In the end could have used better obstacle management and environmental awareness to handle routing. Reserved destinations on the map would have helped too. Maxis used emergent formations, coming from each group picking nodes in sequence (from front to back?), and group’s members around existing picks. (In practice, this had some problems, particularly in tribe where pathing was trickier and all members had to be in place before performing.) Player/NPC Asymmetry was less in civ stage, almost even, but he did give the player a head start and super-weapons to end the stage faster. Much more going on here AI-wise then in the stages so far.

And lastly, space. Relationship management here was key, and there are is a lot of new data throughout space to manage. It’s an avatar game, lite RPG upgrading rather then strategy. Key new techs were the terraforming and ecology simulation, and NPC empires with the same conversation, trade, expansion and war needs of civ. There was a neat random name generator for all those planets, which used a similar approach to Cope’s music generator – using frequencies of letters in common names to generate whole new names. In space they went for a turn-based expansion feel, so the pacing was very deliberate and relied on externally controlled events. This included “random” disasters and raids, which create an implicit limit on empire size (although design could tie “random” rate to size). They liked the plate-spinning aspect of it. Each empire also has a random personality, which affected diplomacy and relationship changes from events. Initially each personality was a separate goal and voice, which made a lot of combinations that were less predictable. But the problem was that the writer had to write for each combination, so they settled on only 8-9 types. Overall, data management was the key issue for the project, because not all of it was in memory at once. They still focused on staying player-centered and asymmetric – transparent relationship, local simulation, and simplified ecology, no real economy at all, and different technology tree for the player.

Big conclusion was to keep it simple and stick with established practices for low risk and ease of tuning. Use player-centered illusions because they are efficient and compelling, even though you lose the emergent simulation which is so AI-dreamy. If he did it again, he would have tried a more complex AI setup while focusing on a simple interface (not clear how this is not just a radically different goal with a interface requiring more work, but I think he wished they could have done more emergence). Grundstrom points to the ecology as an example, saying they kept the model simple for the ecology – carnivores, herbivores, etc – but simulate more behind the scenes. Lastly, the PvE asymmetry was important because it was significantly easier to balance and tune and adjust the NPCs when the game design changes if it doesn’t affect the player.

Grundstrom future desires from the AI community are more living worlds – an illusion of more complex worlds that allow players to go anywhere, do anything. This requires NPCs that are interesting, change over time, and react to others and the environment. He also wants to see more believable people that reacts to a mental model – personality, memory, relationships, emotion – as well as events and behavior customizable for any game. He also points to movie stories that are really “tight” – everything matters and builds progression to overall global goals. Mentions Facade as a direction, but thinks things can be simpler – probabilistic state machines that can be tuned.

Overall, the data issues sound like the most interesting challenge, here, but unfortunately there’s little detail. Question about the LOD touches on this a bit – there was local simulation of empires. Tribe LOD did something similar – different levels of processing for each distance. I asked him about the data management afterwards and he answered the senior engine programmers were able to solve it and keep the data invisible from the AI through something like 10 substantial refactoring, using a mix of data caching, time delays, and data structure optimization. It was made trickier by the need for procedural content from the network, but they pre-streamed content packages in, and used Maxis content off the disk in a pitch.

Into the sessions!

Stochastic Plan Optimization in Real-Time Strategy Games from Andrew Trusty (I believe)

Games have huge decisions spaces, huge state spaces, non-deterministic, real time – your classical approaches don’t work directly. We could abstract the game state or abstract the decision space or add domain knowledge. Trusty’s approach is to use case-based planning for RTS micro battles – use a pre-created planning library, find a plan that is similar to the goals of the current problem, and adapt it to the situation. So, where do the plans come from? How do we find them? And how do we adapt them? He calls their approach the Darmok architecture. The idea is to get plans by observing human experts. Today he’s focusing on fixing bad plans from the experts, unintentional errors. Stochastic Plan Optimization is called for! When a plan fails, the algorithm evolutionarily reevaluate the plan and finds a better plan for the failed goals the next time. Assumption though, the entire plan space can been searched and the original plan isn’t wholly bad. Phase 1 of the search – take the original plan and alter it randomly, to get a search space. Then run each plan in-game (for hours) behind the scenes and apply a fitness to each plan. Phase 2, take the top 4 plans, and then randomize/prune/mix/in-place randomize them together. They check to make sure that there aren’t duplicating a plan here, to explore further. Then run them through the generations again, evolvingly.

First issue in tests was that the experts were just not very good at this particularly game – or situations compounded inside the gameplay that forced them to make obviously bad tradeoffs. This led to a pretty bad overall database. So how well did this algorithmic fix them? Pretty well, the expert plans turned out better. The catch? The algorithm still takes a lot of time, particularly because you are running the game every time you test and evolve your plans. He needs to reduce the search space or run the game faster. And this is for only winning local RTS battles. This needs to scale up to wars for larger plans. There’s some potential here for micro in RTSs though, which is otherwise really hard AI. Trusty needs to account for terrain more, but that could probably be done in the plan actions. There would also have to be some accounting for difficulty, like APM limits – since the optimal plan here is likely just more complex and faster actions wouldn’t be perceived as fair by most players. There’s player coaching potential too – training players to make better RTS decisions in during their play or while analyzing replays.

Constructing Complex NPC Behavior via Multi-Objective Neuroevolution with Jacob Schrum from University of Texas.

Daytime television programming for kids! Nope, couldn’t fool you, that’s not what this is. It’s making tools that discover NPC emergent behavior ahead of time and provide levers to tune emergence. Or, more directly, Schrum’s concern is that reinforcement learning and evolution don’t consider multiple objectives in their fitness function. So he applied a “Prareto optimization” approach(?), which requires exceptional performance in every objective considered. Since most objectives are in conflict, this will find a “front” along the trade-off axes where the trade-offs are most optimal. This also means the trade-offs could be optimized in different ways (the levers), by moving the optimization algorithm along the front. He uses a Non-dominated Sorting Genetic Algorithm II (K. Deb 2000) algorithm to find the points. (Catch here is you have to be able to quickly generate the possible data points, which means simulating each one as they are randomly moved as the rewards for each trade-off are varied). Schrum used neural networks to be the bots for each mutations. He started with the simplest static player strategies to evolve against and then went to more aggressive strategies. He compares his tests to the standard approach using the weighted sum from individual objectives. Results show definite improvement. In examples the single objective approach can’t find a good balance of the objectives – in one case they just run away and chase forever (because all agents are the same speed). With Schrum’s multiple objectives approach, there is better teamwork – baiting the player bot by getting the player to turn away or a charging strategy that takes advantage of the player’s facing and swing direction after getting the player gets hit from another angle. Both are valid answers, but balance the objectives along the front differently. Many of these behaviors evolved were in fact unpredicted, but successful. The research definitely creates a distinctive fixed behavior model for the NPCs that players could play and learn against, because the player trainer bot is static. Haven’t noticed this feature before – makes it much more viable in games, because predictable NPC behavior is so important. Others must be doing it too. Schrum proposes learning against humans (which would remove this benefit, so I’m not sure that’s the best step). He also wants to optimize the training and sim iterations for real time learning. The big win of the research is the emergent strategies are more apparent and more integrated across multiple conflicting goals – there’s a balanced approach rather then the oscillations that comes from using a single fitness score for multiple goals. The drawbacks are just the complexity of the training algorithm – the game would have to be really focused on this need.

Effects of Communication on the Evolution of Squad Behaviours from Darren Doherty.

Doherty’s domain is squad-based shooters, an area that needs AI for situation-based tactics. He looked at PvP players and cited communication, role definition, and skill as 3 factors of team success, and tried to build them into a bot. Using goal driven agents, he evolved a ranged combat decision making tree constrained by limited senses and memory. He minimized his evolutionary algorithm against damage taken, team damage taken, and maximizes enemy health, and then tested the bot in a squad, fighting a more capable single enemy. Unfortunately his data comparisons are just against communicated data versus uncommunicated data, and shows communications improve team performance, but only 2.5%, yikes, less then sensory improvements. I’m not sure I understand the goal of the research here. Since we can fake this pretty well with shared team knowledge, I’m not sure how valuable this is. Even setting this up on individuals alone isn’t too hard (I’ve set it up myself several times).

Implementation Walkthrough of a Homegrown “Abstract State Machine” Style System in a Commercial Sports Game from Brian Schwab of SCEA.

Schwab created a situational data-driven system for the non-technical staff on SCEA’s basketball title, similar to a scripting language, and goes into the slimy guts of the language. In the tool, any resolution of game state was important, and multi-agent coordination was key. Schwab built his AI starting with a global FSM that ran a gross game state, using a custom script, for taking care of pausing, free throws, etc. Then a perception layer (variables, condition bools that are optimized, and percepts – ramped conditions) that can be tested against in agent behavior. Then there was the agent behavior animation-selection layer – a FSM with nodes representing animation selection. Next he went into the designer’s scripting language. Each agent was defined by a set of game roles in the script that were collections of expressions and conditions using AND based logic (to keep them simple to create technically). The scripting language itself was built around situations – the general decision making using expressions, conditions, and roles. Situations weren’t just states, but nodes within a decision tree, with grouped children under parents, priority based. There were many different node types: abstract nodes to partition states, persistent states – the current head of the tree for the team, and the non-physical situation nodes which are abstract but can change variables and fire events (not clear why better then abstract nodes here). I think this was to allow multiple “threads” to run at once from one script. The roles for agents supported “inheritance” that give the scripts memory. There were several types of special roles as well: non-required, non-physical roles, and exclusion roles. This language wasn’t an abstract state machine so much as a formal method for specifying algorithms for the designers to follow. A state with a set of values and rules. It’s a data structure that models code – aka a scripting system. Designers used it for cameras, cutscenes, crowd, commentary, storylines, minigames, etc. They could switch between reactive script to interactive AI, and could filter out interrupts from other places and declare how the script was allowed to exit. For debugging, Schwab used the game’s replay buffer to get debug visual and text data, including any situations running. He supported dynamic loading and a nifty “teleport” button to setup each situation immediately for debugging. His big takeaways – don’t change your AI engine and rewrite the content in less then 2 years. Also, Usability is key – steep learning curve is no fun. There were lots of pitfalls and gotchas. There’s a lot of special case situation times added. Make sure you support syntax checking and keep GUI conventions that are useful and consistent. He has a great search bar for searching every behavior, variables, and situations. This helps organize and track the situations. It even has scoping windows to see who could interrupting each situation! These kind of usability details are very handy.

Dynamic Expansion of Behaviour Trees by Gonzalo Flórez-Puga.

Florez-Puga has been studying Damian Isla’s behavior trees in the seat next to me. Ah, it was for his presentation. He goes into the backstory of what the behavior tree is – see Damian Isla’s paper if you want the details, it’s quite good. Florez-Puga recalls that within each hierarchical state the priorities can change at runtime. And users can get parameter creep in tuning trees. Since they are used across all characters trees could get wrecked by other designers at runtime unknowingly. So he’s building tools to help designers author behavior trees. In particular, there’s a search of existing trees to help designers jump start making new states (states in the tree? He’s falling into FSM language… ah, he’s using a FSM inside each node of the behavior tree, where the behaviors are more goals). He points out the trees are similar to case-based reasoning in their reuse. Maybe we can use case-based reasoning to fill our behavior tree at run-time? (Hmmm… why couldn’t the behavior tree have those cases in it already and run the reasoning inside its initial preconditions? This seems to be research that’s strayed far from it’s stated purpose. – helping designers) Now he’s got a neat demo of a simple 3d environment where each room in a dungeon is a C++ class and contains methods as quests and byte code data as prey. He’s using this as his domain. But I’m losing where he’s going. He shows the process for his research, but I’m not clear on the benefits. He’s pointing out that you could dynamically remove and add nodes to the tree based on general game state. (But the tree evaluates that inherently within each action. Are you effectively pulling the evaluation outside of each action? Since the case needs to know the action’s parameters, I’m not sure if that’s a win, over generalizing the evaluation to data for all actions. It it’s a win, it seems pretty straightforward). Florez-Puga says this could add dynamic learning and unplanned behavior mid-game. But it still requires tweaking floats and regressive testing and QA. The big question is whether it actually makes authoring easier, unanswered.

Hierarchical Petri Nets for Story Plots Featuring Virtual Humans from Daniel Balaš

Last before lunch. Balaš wants to compare dynamic Finite State Machines to “petri nets” for controlling story agents. FSMs are formal, comprehensive, graphical, and can be made hierarchical. But they can’t run in parallelism and it’s hard to represent variables and constraints within the transitions. He displays a story as a FSM with each node as a location to travel to or an action or result that can be reached. Issue is that it’s difficult to remove nodes from the machine, and then enable new story transitions simply. Instead, petri nets are directed graphs of places and transitions. But, transitions can also be from multiple input places and output places. The key is there can be multiple ins and outs from each node that both get traveled. The petri net will pause each branch until transitions join back together in a node. Petri nets are also are non-deterministic, supporting dynamic movement between nodes. In a story, Balaš represents the agents as occupying a node (“a token”), and then creates story conditions based on the location of each token. This makes the results of each transition predictable. He modifies tokens with attributes, like color, to add query for the transitions, as well as supporting hierarchical petri nets. He built this technique into Europe2045, an educational multiplayer online game for high schoolers. He demonstrates the game’s Conflict in Darfur story branch setup in a petri net. Can we use it to control agents and specify a whole story? Yes, with some changes – petri nets can’t specify “when all agents come”, but he uses a new kind of transition to guard against this.

Balaš is still working on the testing and development of the technique within the specific game to get results.

This is a representation model for stories, but I’m not clear how this is useful – it’s an algorithmic model of things that already had to be created somewhere else. It would only help if this model’s construction is faster then other methods. Question: yes, Balaš can also extend the petri nets to account for time.

Logical Agents for Language and Action in Martin Magnusson

Unfortunately, I missed the talk over lunch. Check www.logicagents.com for his work. Magnusson is exploring a theorem proving out language as a new form of gameplay,. I’ve seen the demo video before, but haven’t played with it. He really focused on constraining the gameplay options so that the logic is more communicative and more interpretable by a machine. Worth checking the demo out. How different will this feel in practice, from something, say, like Creatures?

Recomibinable game mechanics for automated design mechanics by Mark J. Nelson

Nelson is trying to give answers to game designs using formal logic for game representation, rapid prototyping, level randomizers, etc. He’s breaking down mechanics-heavy design dynamics using MDA (mechanics-dynamics-aesthetics) into a fashion that could be formalizable. So he can sim it. For example, Nelson references Spore’s rapid prototypes – each either investigating specific mechanics or building a more complete model. The idea is interesting – theoretically designers could queries their ruleset on outcomes – max # of zerglings in 3 minutes, can 2 turrets make base impregnable, can base be overrun with [units]? The goal is to answer balancing questions and provide new information-visualization features. He’s been interviewing game designers, mainly Chaim Gingold, to learn more of what designers need. The idea here is actually blindingly simple – if mechanics are rules, why can’t we analyze them mathematically and make computer queries about their structure?

Nelson’s work focused on board games, using first-order logic. He’s writing a designer tool in a declarative logic language, supports arbitrary reasoning, but requires frame axioms and has no real natural representation of time. Event calculus provides the framework within the first-order logic for reasoning about the system dynamics. The tool he’s writing has states – fluents, events, and timepoints. Fluents can holdAt() timepoints, events happen at time points and can start or end fluents. The axioms define the semantics (rules). Fluents don’t change their values, and you don’t have to specify what isn’t effected, only proactive rules. Fluents thus become game state, events encode player input and game-world events. Game mechanics are the state-evolution rules that result from these rules and states.

Momentary realization rant: part of the problem with AI for design is AI is a procedural solution to a one-off problem. If you only need one solution, one time, making a program to find that solution is a tough sell. Outsiders (read Middleware) could provide solutions to lots of people, or you could find situations (like fine tuning) where AI is needed to respond to the simulation frequently, but one-run AI solutions just won’t be used by designers. They like thinking about it themselves the first time too much.

Last, Nelson focuses on setups that are already procedural or have tunable rules. The tool’s still “assembly-ish”. The potential is good though , particularly for game-games rather then story-games. There’s also a setup time issue – whether this sort of tool is actually useful depends on how long an idea takes to setup to test against. But there seems to be a lot of good potential uses – map balancing in starcraft, for example, or drama management testing. I think it needs more formalization and design input, but while it’s a bit vague, it’s interesting, applicable work, and I’d like to stay in touch with him. Question: How to deal with non-determinism of games, given logic is deterministic? Some models don’t require it, and logic can be modified to do some of it (say with multiple simulations). True – many times you are trying to model a particular path anyways.

Into the poster sessions!

PaSSAGE: A Demonstration of Player Modelling in Interactive Storytelling from David Thue

Thue had data to show that drama management had greater then a 75% increase in surveyed enjoyment in Neverwinter Nights if you just picked events using a calculated player archetype. I’ve never really tried to model player archetype because I thought players liked to choose and change their archetype while playing, but he’s trying it. He’s doing player modeling across 5 archetypes, mapping the encounters to these axes, and then choosing the most appropriate encounter from the results. There’s feedback within each encounter to change the player’s archetype – conversation optoins within the encounters that are pre-weighted to signal a certain desire. This give, some feedback into the model. The data is a great feather for player modeling, but I’m not sure it’s a big enough finding to be significant yet – there’s still reusing all the content lost in the encounter branches, for example.

EMPath: Integrating Experience Management into an adventure game from Anne Sullivan and Sherol Chen

Sullivan and Chen tried to do Drama Management in adventure games to see if it provides a better experience specifically for quests. Correction, they call it Experience Management rather then Drama Management because they really aren’t trying to create drama, they are just trying to guide and improve the experience. Their test game uses real event sequencing to control the experience – measuring Story Density, Information Intensity, Thought Flow, and Manipulation to manage the overall experience. There’s also a differentiation of hints and other types of changes from new events that I need to remember to ask for more detail on. The Experience Manager has a set of actions it can do – move an event, stall an event, force an event on a player – and it chooses them based on those controls to push you along it’s multi-branched story path. This is somewhat more similar to what I was doing before. They use significant plot points rather then zones to demarcate when to take Experience Manager actions. The Manager really helped remind people of things they otherwise would have missed within the plot line. Most importantly , Sullivan and Chen did surveys of players that show significant experience improvement. Really interesting, familiar work. I had more of the drama element in my attributes, and a bit more larger game plot pointing and content development, but it’s definitely down the same lines and they have some data to show for it. Hopefully this will become another good argument for doing narrative management – demonstrable experience improvement – along with the reduced content reasons that I’ve used in the past. Exciting! A good take-away from the conference, and I’ll be interested if they can show the content reduction games as well.

Direction Maps for Cooperative Pathfinding from M. Renee Jansen looked like a faster simpler way to do patrol routes using vector flow fields, although this problem doesn’t seem to come up too often. I could really see using it in crowd pathfinding tech.

Automatically-generated Convex Region Decomposition for Real-time Spatial Agent Navigation in Virtual Worlds from D. Hunter Hale should give the path-planners and maybe collision detection guys a lot of fun. Hale proposes an interesting technique for simpler and faster convex mesh filling, with the downside being the merging and reduction steps are left to the user and it works only for fixed, known convex obstacles. Basically, in 2D, the technique relies on slowly growing out the mesh starting inside user-grid defined regions. Region edges stop growing if they hit something, vertices grow along anything they hit. Wait until they stop growing, then detect adjacent empty space, and repeat these steps until all regions are full. Then combine any regions that would stay convex, remove the collinear vertices and zero length edges, and you’re done.

Whew! Two down, one more to go. Another day with tons to cover. But I learned a lot!

(I’m editing these notes a week after the fact, just making it readable, but I’ve left the original voice in this piece and roughness intact because I feel the live-blogging depth is more useful then a concise, thoughtful analysis would be. I’m happy to answer any questions people have, though, and I encourage you to contact the original presenters as well for more thorough details.

I’ll be posting Thursday and Friday’s notes shortly. Enjoy!)

(links to my Day 1, Day 2, and Day 3 conference analysis posts)

I’m at the Artificial Intelligence and Interactive Digital Entertainment Conference 2008 (AIIDE) as I write this.  Here’s the program.   I’m not sure what the conference etiquette is here, but I’m planning to live blog as the event continues. The big push this week seems to be into proceduralism and “Drama Management”, what I’ve been calling Encounter Management, so I’m looking forward to that tomorrow. I cover a lot of ground in the notes here fast, particularly on the more interesting talks, so please give a more leeway on the editing, etc.

Panel Discussion: Realistic Human Characters with Borut Pfeifer (Project LMNO), Michael Mateas (UCSC), and Richard Evans (Sims 3)

I just caught the end of a panel on Realistic Characters. The questions were wide-ranging and wandering, as much about proceduralism as about AI or behavior. There was an interesting question about having 2 human reactions rather then just one. As in, if you tap someone on the shoulder, there’s 2 reactions – the “reptilian brain” jump and then the spin around “huh”. Usually we do this with one animation, but maybe it should be 2 separate states. The procedural animation discussion focused on if you could blend code as opposed to just animation data, unfortunately only raising questions. Also, some discussion of player-human interaction – once characters get real in games, shouldn’t the player’s verbs get better? Shouldn’t the AI be two-way? This was an interesting point, and one that surfaced a couple of times through the conference.

Intelligent Trading Agents for Massively Multi-player Game Economies by John Reeder from Stanford.

He’s looking at the game auctions like in World of Warcraft and Eve. Developers can control item frequency, “taxes” and fees, and ways players making money, but there aren’t great levers for developers to push, and players can easily drive the economy into the ground. He’s looking for a way to be the anonymous federal bank of the market. In low population you can provide liquidity. (I’m getting reminded of the economy crisis.) He cites TAC Classic and TAC supply chain management papers as other sources, for those interested. He focused his research on Eve because of its economic focus and the production steps of complex materials. So he pulls from real market economics to structure his AI bot – market orders (stock requests) and limit orders (sale proposals?), buy and sell orders that create a mid-spread balance sheet. Steps are Collect Data, Infer Transactions, List of Transactions, Generate an order book, and then put the item up. Just by collecting the data he can simulate what the whole of the market is doing by watching items and catching when they disappear, assuming disappearances that aren’t based on expired time are sales. This helps him identify the true price, much like a human player would. Focusing on minimizing cost and maximizing profit, he evaluates the agents on the gains over time. His process sounds good but it’s not clear how he defines success. Ah – he’s using reinforcement learning on the evaluation, and he’s trying to find the optimal algorithm. He shows that reinforcement learning does generate a better result then a traditional fixed market techniques. Seems like a straightforward and reasonable application of reinforcement learning, although he’s not taking the final step of showing how designers can control the market significantly better with this work. Interestingly, having his bot guessing prices ahead of time didn’t help, because the time cycles were so short and the markets move so fast. He also plans to compare the bot’s success to humans traders, although from a design point of view there’s not a need that it be superior. Question: is cheating if you tie the economy to real money? *chuckle* Where’s an ethicist when you need one?

These 20 minute sessions are fast. This is going to be a long set of posts!

Learning to be a Bot: Reinforcement Learning in Shooter Games by Michelle McPartland

She’s trying to reduce code through reinforcement learning, although that dodges the production issues with it (we don’t care about how much code we ship!). I missed her other reason, unfortunately. She brings up that there’s 2 versions of reinforcement learning, Tabular and Generalization, which I wasn’t aware of (shows how often I’ve built this stuff). She’s using the Tabular method, with the Sarsa-Lambda algorithm. There’s 4 parameters, Learning rate, greedy value, discount factor, and trace factor. It’s been researched in racing, RPGs, fighting, and squads in FPSs. Her research is in FPS bot training navigation and combat. It’s funny, researchers use FPS for the same reason we create them, it’s one of the simplest designs. She chooses her own discrete reward values, which seems unfortunate (is getting items or killing someone more rewarding more frequently?), but I’m probably missing something. She did run several different reward models. Basically it seems like it comes down to what you measure, rather then how much. Intuitively, seems like that means the model is too simple. The stories of how the reinforcement learning gaming the reward system are pretty funny – camping the item spots because that’s what’s rewarded. For navigation, she didn’t find she needed planning – that reactive behavior worked quite well. Good reinforcement for current obstacle avoidance models. Even given how simple this is, it’s interesting that she’s able to fix the rules and just vary the rewards to get such different results from the bot. Biggest takeaway is that the bots got very different personalities. But that’s part of what makes reinforcement learning hard to use – small changes lead to big emergent differences. It seems like the biggest potential here is the thing that I recall Unreal did, not less code but more unpredictability and different bot styles. A good question: could it be used to also help train the player. I’m also surprised they used so few trials – is that common? If I had these sims and was going for data, I’d run it for a thousand different variants and train them against each other. Ah, the learning trials were 500 iterations, but they didn’t generate more then a few bots. I’d actually love to see this research applied to a Starcraft bot, which is a notorious problem.

Agent Learning using Action-Dependent Learning Rates in Computer Role-Playing Games from Maria Cutumisu

She’s looking at improving NPCs that lack adaptability so that we can produce more captivating stories. She seems to be focusing on combat “stories” though. In Neverwinter Night, she’s looking at generating adaptive behaviors without programming. Cool! First finding was that fixed learning rates or decaying learning rates learned too slow in games – action is too fast. Actions can also be lost if they aren’t learned early. So she created ALeRT to replace Sarsa-lambda algorithm, using action-dependent learning rates, trend detection, and exploration rate decrease/increase with wins and losses. This creates a pretty interesting return for reinforcement learning – she’s essentially added an AI to her reinforcement rewards to better tune them. If it works, this is actually pretty useful – it would allow people to be much more hands off in the highly dynamic game environments. Well, her data says it’s not always better then basic reinforcement learning. Over time it’s actually less optimal. But it adapts substantially better, which in the game’s highly dynamic choice-space is more valuable. The most interesting of this set, I’d say. Against or with humans, I expect it would do much better. It’s higher complexity does make it harder to implement, though.

Lightweight Procedural Animation with Believable Physical Interactions from Ian Horswill

And we’re on to the Singular techniques section. In a switch up, Horswill hops in first. He wrote twig, a procedural animation and simple physics library, open source and runs on XNA, to be a platform for some emotion research. It’s pretty cool – “fast, AI-friendly, scriptable and authorable” and open source. He shows a great demo of parent child attachment, including some great mother-leg grabbing, then running off to play, then runs back. The library’s based on Jakobsen’s work on the Hitman engine from GDC 2001 on Advanced Character Physics. The trick is you represent character bodies as particles – point masses connected by massless rodes and represented only with position and time. Thus can easily capture state of system with only 2 frames. So if something is out of the way you just move it. Then you use rigid distance constraints as springs to keep it together. Moving hands becomes really easy. Emulates real life, constraints, physics, and IK sim just handles everything else for you. No joint angles, just c artesian coordinates. It need help to handle joint constrains, that’s a bit harder without angles, but it’s worth it. Horswill‘s got a posture control system modeling the spine. I had a friend actually propose something very similar to this idea before, but..

OK, problem one, no conservation of mass and energy. The forces are directly applied. They break things using real physics. Horswill cites Perlin’s gait generation – move the center and using balistic motion to drive legs when they get stretched to far. twig also has sensing and attention. Collisions generate “pain” and attention and a gaze reaction. He’s using the library to create a web comic – used XNA’s pipeline and Google sketchup to put the art together (although it takes C# to define collision volumes and prop actions). Interestingly, holding some object, the “something” can just drag the arm around. (How does he keep from breaking the arm? Common Sense character relative? Nope, just loose snapping, unfortunately) He’s added behaviors such as Hold and Write for props, Walk, Gesture, Speak, etc. The Hug, reach, grapple, drag stuff is usually really hard and he makes it sound trivial. Scripting is RPC interface or read from a text file. He cites some related papers, Jakcobsen, Veriet, PhysX doing “Position-based physics”. Procedural character control from Badler, Goldberg & Perlin, etc. Well, ok, I didn’t know this was this far along yet.

Problems: twig’s bad at accurate simulation, photoreal, handle complicated collision, or path planning. But twig’s good rough-and-ready character behavior, relatively expressive motion and dynamic (even though it’s inaccurate), and seems expandable. Neat to see this stuff in open source, but most of these problems are the core things that need to work well in games, so this still likely has a long way to go for complex commercial applications.

Combining Model-Based Meta-Reasoning and Reinforcement Learning for Adapting Game-Playing Agents from Patrick Ulam

As in, “How can agents act in highly complex non-deterministic limited information environments?” Ulam proposes combining existing approaches. Use symbolic reasonings with numeric machine learning, localized failures, and adaption. In model-based meta-reasoning the agent reasons about its failure, and then tries to target and correct it. Big weakness, a lot of expert knowledge is needed to use this. Reinforcement learning, on the other hand, requires little knowledge and can get optimal, but is slow and suffers “curse of dimensionality”, it doesn’t scale. Not so good for these environments. So he combines both – uses meta-reasoning to localize the failure, and then reinforcement learns against the failed solution. He tested this approach in FreeCiv – defending a city for 100 turns using the TMKL model – Tasks, Methods, and Knowledge separation. E.g. what is done by the computation, how a computation is done, and concepts used (i.e. data). The adaptations and failure database itself is first designed by an expert. The model-based reasoning tries to find the task in the database that failed. The reinforcement learning is used to help build new libraries. Model-based in general does pretty well, but here he got slightly better results with this hybrid. Hybrid does reduce the amount of knowledge engineering necessary and reduces training time, but you still need an appropriate model. It’s interesting how focused Ulam is on the gameplay side possibilities for this – but this is an interesting step along the path for smart design tool. Most of this research can’t be put into development until it’s much more developed and has better interfacing, and no one seems to be focusing on that end of it yet. Are they leaving that up to development?

TAP: An Effective Personality Representation for Inter-Agent Adaptation in Games by Chek Tien Tan.

Tactical Agent Personality – synthesizing adaptive inter-agent behavior. Oh, he just criticized the Diablo 2 Necromancer pet AI as being too blindly dumb and then said “there’s no Blizzard AI guys here, right?” (Must.. keep… from… laughing…) “Let’s fix that” he says, by trying to measure player and/or NPC motivation, extracting Agent motivation from its actions. For players he just tracks number of uses over time, for NPC he uses chance-to-use. Then for each interlude, he calculates an error value to back propagate into a neural network. Whoa. Where did that come from. OK. Think I’m fully over my game programmer reaction to unnecessarily complex AI techniques. For now. I’m here to be proven wrong. So he’s using the neural network to predict what the agent’s goal is. His latest research is to switch his weights from the actions to the action transitions, so that he can build one-step deep combo sequences. Wouldn’t this skew the neural net badly in situations like fighting games, where combos go to 2-5 steps? Plus, one-step combos are almost always designer-created, so the expert knowledge is there without needing the neural network to find it. More interesting use would be for long term strategies – aggregating big picture patterns in an RTS and then predicting tech, econ, or attack, for example. Although here, it seems reinforcement learning does pretty well to, and expert knowledge might even be the best design (we want to train rather then win). Maybe for helping designers predicting emergent behavior? A good question on why he aren’t searching for the higher personalities traits that the net could communicate back to the user. But I’m not convinced this is a fruitful solution for games yet.

Offline Planning with Hierarchical Task Networks in Video Games by John-Paul Kelly

After lunch off-site, we’re back for some tools related help. “A method that uses AI Planning to generate game scripts automatically” in Oblivion. Scripting is manual off-line planning by a human being – takes time, is costly, error prone, and doesn’t scale well (depends on the design of course, but generally true). He says we could try on-line planning, (UT, Fear) but scripts are standard, easy to implement broadly, fast, can’t be interfered with by the user, so let’s consider off-line. Can AI tools generate scripts automatically?

Hard but interesting: almost “Can you script fun with AI?” His approach – represent the daily plans of NPCs in an Oblivion town as scripts that map actions in a plan to AI behavior packages. Game programmer writes a hierarchical task network, then the planner runs through the HTN, and sends the results to the translator. The script interprets the translation and sends it to the game. The gain here is that the hierarchical task network is an abstraction of game knowledge and can be reused cross-character. Each node is an abstract task -> method-> atomic tasks. It’s hierarchical goals, basically, like a decision tree. Attributes drive the goals chosen, can be character or world based. (It’d be nice if they were not just linear decays in the planner, more realistic algorithim.) Used JSHOP2, open source HTN planner. Plan that comes out represents all NPCs, because they are just interactions. The translator then creates an individual script for a per character basis. Then when the script gets executed – cthe script hecks the plan is still valid in world state, updates NPC state, activates AI to implement actions, and send messages between individuals.

OK, in practice not as much fun driven as living world driven. Not necessarily very reactive to the player. It’s offline, so it’s really just pre-generating scripts we’d normally do by hand, and the planner can be debugged to handle the interactions correctly every time. It’s interesting because I was just reading that Oblivion had to simplify their AI script systems down because the player interactions were breaking it. So this is ignoring the player and making the scripts more complicated. Whoops. It’s still a very creative idea. Because the scripts check world states it’s more robust then pre-scripted usually is (fixes eating even though we stole food off the table for example – although this might have more to do with the backend package?). Catch is if script is checking world conditions and not AI, then script need a lot of extra “ifs” – contingent planning – and the player can still break it during play. Seems not too difficult to set up for very predictable attribute behavior, but hardly a replacement for dynamic AI. Really how well can you do the contingent planning in script? Could also be very good for quick-dying combat, just to do an initial pass. Could you just grab a new script when the current one failed? It might be more interesting to make the scripts themselves more akin to swappable AI packages. I’ll mention it to him.

Simulation-Based Story Generation with a Theory of Mind from Hsueh-Min Chang and Von-Wun Soo

Next! Can agents generate stories through their interactions? Arbitrary plans probably, but not usually story – they need some new AI. Chang believes it’s a mental-level interaction among agents. Chang separates Agent and Patient – influencer and influencee and define a social plan as a character plan whose goal or subgoal is to change the mind of others. Their characters use theory of mind to infer other’s goals and change them. The influencer creates a plan using the game’s public knowledge of character attributes (envy, fear, etc.) to find actions that manipulate the influencee’s attributes to get the desired result. Their tests modeling Othello, creating fairly manipulative characters, for sure. Visible feedback from the influencer would be regularly required, which could make the social behavior seem silly right – “I’m trying to fool you!”. And of course, the plan has to be redone if the conditions change or the actions are not verifiable by the influencer to have happened. Most interesting here, this idea of a “social villian” could create a “new” kind of information game – players competing against (known or unknown) social agents. “New” as in done by Crawford in Legacy of Siboot, but that was 20 years ago, so who remembers that? There’s definitely potential here. Question: Do you have to replan after to every action because of interfering plans? “Yes” More evidence this works best with only one active agent and the player.

Automatic Generation of Game Level Solutions as Storyboards with David Pizzi from the University of Teesside

And finally, Pizzi, who was working with Alex Whittaker, an Eidos AI programmer. Pizzi points out game designers are reluctant to use planning to procedurally generate content because of QA issues. But planning can provide all the alternative solutions that a designer might not see. His research was used in Hitman: Blood Money, where they needed particularly good level planning tools. He interprets the levels using STRIPS interpreter on game actions. Then plans out a possible level solution using heuristic search planning, and generate a storyboard to show the result. Storyboards are normally expressive, easy to produce, universal, and easy to understand. But they describe animation or possible situations, aka a plan. The issue he ran into was that each plan implies a new drawing. He solved this with dynamic generating each storyboard panel from parts. Analyze the world state, pick the next action in the plan, and load the corresponding pane template, filling in the details of the pane dynamically. Draw the result using automatic composition techniques, drawing an atmosphere, environment and then actor layers. Seems like it could be a very useful for some games. Combinatorial explosion of possible panels? They had 90 possible actions, but only 24 templates – each template can cover multiple actions. Solutions usually numbered 25 to 40 actions for an area. In the planner, iteration led to a temporal element to account for NPCs that moved around, and designers could select walkthroughs by player’s play style as well. He shows the designer’s tool. The sheer number of possible plans here just seems to make it useless as a comprehensive check – many of the plans would be nearly exactly the same. Being able to filter down the plans would help, but then you start to miss the QA aspect of it checking all aspects of your level design. Also might be cumbersome to import the level into the tool, rule by rule. They are actually looking at the final level, this is more for design planning, pre-production level planning. It’s kind of neat to see the planning tree and walk through the options at each level though. And to rate how complex the plans coming off each potential choice in the level are, so you can balance them in the design, a critical part of Hitman design.

Modeling Culturally and Emotionally Affected Behavior given by Vadim Bulitko.

It’s 3:00, time for social intelligence. My social intelligence is telling me we should have a short break first, but to shut up about it. (Yes, clearly working.) So, Bulitko asks, “why model culture?” Because, say, you’re trying to be more realistic in virtual trainers for soldiers going to Iraq. His demo shows a conversation that would normally be pre-created. Bulitko says that would be laborious and error-prone. Why not model “procedural culture”? He defines “schemas” that are a framework of cultural knowledge and “shared symbols” that are shared icons of a culture. We can build a library of schemas and create plans to show what steps a cultured NPC would support, i.e. rates highly. As an aside, in this plan we can also model how NPC buys into a player’s promise-commitment by measuring the player’s cultural commitment, “trust”, a future predictor initiator the player will follow through.

He combines this with procedural emotional behaviors. Using Smith and Lazarus’ appraisal theory, emotions are modeled by the appraisals of current state with respect to goals and beliefs. Tie your procedural culture plan output to a higher decision level that maps to desirable and undesirable emotions, and then you can get one weight to evaluate each potential plan. Combining plan layers models humanistic emotional reaction alongside culture affronts – ie disrespecting Islam.

Basically this is just a dual layer plan evaluation system, emphasizing culture and emotion, rather then just emotion as games normally handle it (behaviors weighted by emotional impact). It seems like a pretty obvious approach and could be applied to any other situation with two factors to incorporate. Qs: How do you select the plan’s weights? In this case, he used a human expert and they watch results and then say yeah or nay, and then you debug what happened. With training, he might be able to have experts tune the numbers without playing.

Otello: A Next-Generation Reputation System for Humans and NPCs from Michael Sellers.

Advantage of 20 minute conversations: we cover a lot. Downside: tech setup between presentations isn’t fast. Here’s something from within the industry. Well, the web industry. Otello’s a web site combining Facebook and Digg, because Digg doesn’t rank stories based on who your friends are. Sellers separates out opinion (what you know) from reputation (what you heard from someone). Reputation is outside of your control. It’s what someone else heard from your associates about you. So he wants you to rate relationships by value and confidence, while keeping them one-direction, transitive. These ratings could be done by humans or AI bots, and by being transitive, humans could choose to “trust” the output of these search bots. He uses a weighted recursive aggregation search using the relationship value and confidence, as well as a “social friction” value to restrict the length of the search. This search can trace the network to find early adapters, key connectors, and sharing-killers. The whole site could be used for news story, safe chat with kids, limited visibility of individual’s content (or anything that drives funding, really.) He’s looking for a game connection, but admits it’s been trouble in the past (Facebook friends aren’t great gameplay because you just end up with competing for thousands of friends you don’t know). Mostly an idea and a sales pitch, but might be some ideas in this. Interesting proposal for using this approach to relationships as a score in an MMO, to enable UO-kind of stores, taverns, or theater feedback and gameplay aka the gameplay Sims Online tried to capture that the Sims does well.

Modeling the Dynamics of Non-Player Characters’ Social Relations in Video Games by Vincent Corruble from the University of Paris.

Working with Quantic Dream (of Indigo Prophecy) and an AI middleware company, Corruble had worked in the past towards improving dialogue in adventure games through measuring Emotions, Experience, and Personality. He wanted to add another layer: Social Relations. Usually this is scripted, he’s trying to go procedural. Starting from emotional/personality/social relations models from psychology, he searching for a model that is comprehensive, manageable, computational and implementable. He settles on the same emotional appraisal as the planning above (Bulitko), as well as measuring personality attributes (Big Five, EISN? Models) and social roles (Brown and Levinson – Liking, Dominance, Solidarity, Familiarity). His AI is event-driven. Given an event, first the raw emotions the event causes are selected. Then the personality is overlaid to get the character’s true reaction. Then the character’s overall emotional state is aggregated, and the AI behavior is changed (modified by the social role). The social role reacts to your emotions and other people’s emotions (i.e. solidarity, dominance when they are same/different). Seems like they still have quite a ways to go, not much detail here to go on yet, particularly on how the social role contributes to the model. It’s interesting, though, to see a model that tries to take into account so many diverse character factors.

Experiments in Musical Instruments by David Cope

The invited talk of the afternoon is David Cope, an AI programmer/artist and professor. Cope, faced with a bad case of composer’s block, decided one day that instead of writing an opera by hand he’d try to get a computer to construct Bach’s chorales for him! The chorales had a lot of competing rules, but there are a lot of examples and the structure is generally well understood. For example, each has independent running music lines, but there’s always a repeating note because they are 3 chords for 4 parts. So he wrote a program that followed the rules and came up with something that seemed to work. But it was missing something. And he wanted to handle different rules as well. He really just wanted to analyze the rules arising from a database of music. As simply as possible. So he started over, creating a Bach database, organizing it using grouping only the chords that came from the same previous note. So as the program choose notes, it could search and know where it could go next. The rules are embedded in the data organization. And he noticed Bach doesn’t follow the rules! Well, only 5% of the time. (Take that my music theory teacher – I get to break a few too!) His program’s algorithm – Eliza – randomly samples all of Bach’s chorales. It’s never breaking the rules, because it’s only using Bach’s notes. There’s no sophisticated knowledge. Just Bach. Can’t get more expert knowledge then that. The results were better. But there were still problems.

First, his program created phrases of uncertain lengths and direction-less music. No place to take a breath, no shape or form. So the program needs a model to limit phrase length. At first he tried to program form and create rules in the code. But then he tried went back to the data sampling the same way, pulling out models of a Bach phrase. This whole thing really reminds me of Encounter Manager and story arcs – matching story beats rather then music beats from data, so to speak. To measure phrase, Eliza just matches the first and last chord and Bach’s overall phrase length and then just fills individual notes in as above. He then added overall form, modulation, key changes, balance, theme, shape, etc. He just scaled bigger and bigger. Recursively building it up. I’m sure Lisp was great for this. What’s particularly interesting here is that the singer’s breath time comes from Bach’s own sense of timing, automatically, not the program’s inherent design.. He’s put 5000 of these songs on his website. He can’t even listen to them all.

Pushing beyond Bach chorales, most music doesn’t have 4 voices without rests. Different instruments, textures, styles, volumes, form. Can we emulate all that? We’d have to track character , section, cadence, variation and repeats and more all within the beats themselves. Cope gave it a shot. The Rachmaninoff piano piece he shows here is stunningly pretty, with incredible variation given his small sample set. “It means, Rachmaninoff wasn’t dead.” Poetic. The meaning of art recreated procedurally, beautifully. He does all this data by hand, meticulously copying and checking each note.

He didn’t start composing his opera until a month and a half before the premiere, and he completed it in 3 days, to the best reviews of his life.

A question whether he’s tried to do this with text. He tried it to create a world anthem, with music. In essence, it worked, although he didn’t like the data. Close to being deemed the world anthem by the UN, but it’s a terrible piece of music. The whole program right now is about 20,000 lines of code, optimized down. It’s past the “Turing Test” with 9 out of 10 faculty at a school of music. He’s had a lot of trouble getting it performed, because people know he can always make more – there’s no sense of rarity, of special. The program isn’t rare, so it’s not valuable to perform. So he has destroyed the databases. It takes a long time and a lot of work to put the databases together because they have to match the style you are going for. And es, they perform it now a bit more, but in a sense the program is dead. We took a vote on if it is creative. I voted yes – I’ve done it before by hand, and I believe it’s little different. Cope says he believes it’s not and he’s written a book on that question. I ask him about the difficulty caused when the database sample size, the expert knowledge, is too small, and he says that it’s definitely an issue, but it’s very case-specific – some data is easier then other because it’s significantly different enough from itself, while some just falls apart. But there’s no shortcut for the sample size problems the encounter approach can run into, unfortunately.

A great talk to end a day that covered a lot of ground on a lot of different. His technique is very similar to the technique I originally drafted to do procedural interactive stories, before I knew anyone else was even attempting it. It’s still surprising to see how many people have tried it and how often its worked. Gives me some good hope. The most useful part of this approach is how straightforward it is and how no beat needs to be generated before all the ones before it have been played. Less use in music, but a significant win in game narrative. Well, off to rest and get ready for Thursday.

The industry presense at AIIDE gives the academic researchers a much-needed reality check. After almost every talk, the questions were “How does this improve the player experience?” or “How does this make the designer/developer’s job easier?”. Sadly, in many cases the presenter did not have a good answer to this question. In many cases it seemed more that the game was merely being used as a simulation for testing the AI research, and not the focus of the research itself. This is, of course, totally valid. AI researchers need simulations in which to test their algorithms, and games provide us with some of the richest simulated worlds available. However I would have expected a conference on “AI for Interactive Digital Entertainment” to contain more work which was actively trying to advance the art of games.

That said, there was some very interesting work presented. The interesting theme for me this year was that of “social games”. There seemed to be a strong interest both among academics and industry in producing better character AI, to simulate emotion or knowledge or language-use or deal-making. These are areas of AI that are beginning to become mature enough to start appearing in games, and they offer the promise of more meaningful characters and stories.

This presents us with a problem: As our NPCs become more complex, the poverty of the player’s expressive power becomes increasingly evident. As someone pointed out, Gordon Freeman never smiles. The input pipe is very thin – 2 joysticks and 12 buttons does not allow for much expression – and this, I think, is going to be our next major hurdle in gaming.

The highlights of Friday were both personal and professional.  I was honored that Steve Rabin put some of my ideas on the future of game AI up alongside those of Damian Isla, Chris Hecker, and others.  The presentation was a nice way to really bring the goal of the conference home and focused, almost such that it should have been the first speech rather then the last.  But the most insightful presentation Friday was Borut Pfeifer, who discussed the AI architecture Project LMNO at EA is creating to try to make believable “Spielburgian” human characters.  My notes were quite extensive, and his analysis of Spielburg’s films as source material for character behavior was very useful and contextualized to industry.  We do have the groundwork for such AI frameworks already, but there’s a long way to go.  His summary of how he intends to move forward was fantastic.  But the first challenge (which unfortunately he didn’t discuss) is a design one.  How can we make these characters seen?  It’s all well and good to do facial anims, create appropriate reactions, and model emotions.  But if the player can’t see enough pixels to tell, or the camera isn’t focused on the character, or if the player is incentivized to do other things, then the AI work would be poorly focused.  I believe we can solve the character priority behavior problem fairly well, given time.  The design solution, though, needs to be proven out and chosen first.  Even the 1st person emergent shooter environment that Doug Church is likely making at Project LMNO (give his love of Deus Ex, System Shock, etc), have much to grow to reach the level of manipulation and timing in film.

This was one of my biggest takeaways from the conference.  The AI technique development is good, but solving AI problems without the context of solving game design problems is tricky.  Ignoring the struggles of game design yields results that are likely poorly targeted.  Instead trying to predict the struggles of game design, particularly without the designers in the conference, is paralyzing.  AI research is still at this stage where we need the designers to be driving the problems, not the other way around, and yet the designer->programmer->academic relationship is maybe not strong enough or consistent enough to do so yet.  I’m not sure bringing game designers to the conference is that solution.  Maybe it’s time I went to a serious academic game design conference to find help bridging that gameplay/AI divide.

(links to Day 1 and Day 2)

I’m still checking on those notes.  Today the highlights were the discussions about game design formalism through formal logic.  It seems like a really neat approach to help develop the science of game design, and give game designers more demonstrable tools.  There was a lot of agreement (among AI programmers anyways) that game designers need to be able to think on a rules-based level, not truly code, but in a specific scientific  sense, something that certainly fits my experience.

Oops and feelings of dread, I just discovered I left my fiance’s laptop cord at the conference.  I hope I can get that back tomorrow.

The big topic of conversation was Steve Rabin’s talk tomorrow on the Future of AI.  I recall Chris Hecker’s similar challenge at GDC.  We covered the main obvious areas – animation, environmental interaction, more believable humans, etc.  But Steve ultimately was most excited about the next improvement not in development process but in gameplay.  As he put it: think of Portal, a successful game built around a new technology that was fantastic.  It had a new technology and a new gameplay at its core.

Steve is far more qualified to talk about this subject then I, but I’m not sure new gameplay is the most important role AI can play.  Sure, there will be new genres future AI creates.  Facade and Far Cry 2 are leading examples of this.  The difficulty is that Portal, while fantastic, isn’t fantastic because of portals.  Narbucular Drop and several other games have had portals, but were hardly impressive.  Portal was fantastic because of its narrative experience.  Portal was fantastic because it was polished.  This is the true challenge in modern game development.  You look at all the top games, and they take half a decade.  Blizzard.  Valve.  Bungie.  Maxis.  Popcap wins at casual games in large part because it’s dev cycles are 5x longer then everyone else.  Braid and Castle Crashers took somewhere around 4-5 years.  The irony is that the ideas for these games came in around the first 20% of the time.  They had to, because no team will work on a game for that long that doesn’t have something to build on.  to prove the trend, the reverse of this also holds reasonably true.  Games that spend little time on polish consistently do worse in both reviews and the marketplace, certain factors such as franchising withstanding.

This is a serious problem because there’s really no sense in creating a fantastic game today, and then spending 4 years of your life getting people to play it, all just to justify the idea you had in the first place.  Being good isn’t enough, being marketed isn’t enough – you have to be polished.   If you aren’t polished someone else will just copy your rules overnight and put the game up for free.

So what makes a game polished?  I can think of a few key things: design core iteration, design variation, content quality, content quantity, and technology.  A game has to polish each of these facets to varying degrees to turn out a great product.  How can AI help the games of the future?  By making these tasks easier, so polish can be completed in a more reasonable amount of time.

AI, like most code, is fundamentally about taking repetitive things and making them procedural.  But it’s also about modeling and predicting behavior.  For example, to improve content quality and quantity, modeling creator behavior could be very useful – making easier interfaces or automated tools.  Testing is a key part of design iteration, so automated playtesting could be very useful.  And once you’ve identified problems of a particular class, automated solutions could be very useful – things like patching collision mesh, fixing animation transitions and sequences, moving spawn points, or even elongating stream signals.  It’s tricky to say if any parts of these systems would be automatible, but each such step builds towards a major time savings and quality improvement.  Design variation (mapping design rules onto understood patterns and then selecting new longer-term games or mini-games that best build a deeply interesting game) could benefit from tools that construct and balance new rules and rule interactions built on the core experience.  And finally there’s technology, where middleware is helping keep standard functionality simple, along with helping AI research techniques be generally applicable.

Is this at all practical?  I don’t know.  I’ll be pleasantly surprised if there’s substantial progress in 10 years.  But it is important.  Polish is almost soul sucking – the cost to the industry as a whole is incalculable.  Certainly, parts of it are not only necessary, but important and even engaging.  But current project time frames are excessive in every possible way, and they are trending worse.  Just as Maya makes art more achievable, we need comprehensive tools and techniques that make the overall construction and polish of games straightword and mind engaging, rather then mind numbing, or we risk drowning our medium in a sea of repetitive grunt work.

(link to Day 1)

I was talking with Pat Olivier (I believe) and the proposal came up, as it is wont to do in AI circles – “How long will it be before AIs are replacing our humans in games?”  The designer and the AI guy in me clashed.  There was a circle of fire, spikes, chanting, side betting, the whole works.  But looking at market data always wakes me up.  The data just doesn’t support this.  We use AIs now to communicate experiences that we couldn’t do any other way.  But the reality is, given a choice, humans prefer to play against other humans.  World of Warcraft, Counter-strike, even GTA, Madden, these are games people play with other people.  Far and away.  It’s a similar problem to what David Cope spoke about today – there is little joy in playing against a machine that you know will exist, repeatably, forever.  Chess is “solved”, people train against computers at chess, but far far more people play chess against other humans.

The counter argument is that yes, this is true now, but what if the AI passes the Turing test?

Well, assuming such a thing is reasonable over the long scale, even then I don’t think it will matter.  If you could have a meaningful relationship with an AI you would by definition be engaging in it.  But would you only engage with AIs of that type?  I don’t think there’s much evidence of that.  People engage with their family, their co-workers, their spouses, their lifelong friends.  Maybe some people engage with their dogs more then they do with other humans, but that’s a rarity.  It seems likely that if AIs do start impersonating these other humans, people will just demand tests that differentiate.  Because it’s no fun to be friends with someone who deep down, you know, is just telling you you’re right because they want to be your friend.

But what if they aren’t just game entertainers or companions, they have intrinsic human motivations that don’t rely on consumers?  It’s allows possibly to say but what if they are just more human in that way.  But each one of these depths is a new scale of learning, and we still aren’t even baby stepping down the path.  I think a better question would be: Will humans ultimately create such an AI?  Or will it be an AI creating this AI?

Read the paper:

Stochastic Plan Optimization in Real-Time Strategy Games

by Andrew Trusty, Santi Ontañón, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-08-09.pdf

In this paper we present an intelligent IDE (iIDE) that has the following capabilities: it allows the author to program initial versions of the behaviors through demonstration, presents visualizations of behavior execution for revision, lets the author define failure conditions on the existing behavior set, and select appropriate fixes for the failure conditions to correct the behaviors. We describe the underlying techniques that support these capabilities inside our implemented iIDE and the future steps that need to be carried out to improve the iIDE. We also provide details on a preliminary user study showing how the new features inside the iIDE can help authors in behavior authoring and debugging in a real-time strategy game.

Read the paper:

An Intelligent IDE for Behavior Authoring in Real-Time Strategy Games

by Manish Mehta, Suhas Virmani, Yatin Kanetkar, Santi Ontañón, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-08-08.pdf

We have developed real-time adaptation techniques for case-based planning and specifically applied them to the domain of real-time strategy games. In our framework, when a plan is retrieved, a plan dependency graph is inferred to capture the relations between actions in the plan suggested by that case. The case is then adapted in real-time using its plan dependency graph. This allows the system to create and adapt plans in an efficient and effective manner while performing the task.

Our techniques have been implemented in the Darmok system, designed to play WARGUS, a well-known real-time strategy game. We analyze our approach and prove that the complexity of the plan adaptation stage is polynomial in the size of the plan. We also provide bounds on the final size of the adapted plan under certain assumptions.

Read the paper:

Real-Time Plan Adaptation for Case-Based Planning in Real-Time Strategy Games

by Neha Sugandh, Santi Ontañón, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-08-06.pdf

In this paper we introduce the concept of a situation and explain a situation assessment algorithm which improves plan retrieval for CBP. We have applied situation assessment to our previous CBP system, Darmok, in the domain of real-time strategy games. During Darmok’s execution using situation assessment, the high-level representation of the game state i.e. situation is predicted using a decision tree based Situation-Classification model. Situation predicted is further used for the selection of relevant knowledge intensive features, which are derived from the basic representation of the game state, to compute the similarity of cases with the current problem. The feature selection performed here is knowledge-based and improves the performance of similarity measurements during plan retrieval. The instantiation of the situation assessment algorithm to Darmok gave us promising results for plan retrieval within the real-time constraints.

Read the paper:

Situation Assessment for Plan Retrieval in Real-Time Strategy Games

by Kinshuk Mishra, Santi Ontañón, Ashwin Ram

www.cc.gatech.edu/faculty/ashwin/papers/er-08-07.pdf

Basically the difficult part was that I wanted the server to establish connections with many clients, send them some information about the match during connection and then broadcast the match data to all clients simultaneously. For this, of course, you need concurrency and threads, though the fact that Haskell as a purely functional language already threads everything confused me a bit. The answer is Control.Concurrent.forkIO and, for the more complicated stuff (i.e. communication between threads) Control.Concurrent.STM. For tutorials on how to program servers with Haskell, a nice, light introduction can be found at
http://metavar.blogspot.com/2007/03/simple-socket-programming.html
while a bit more difficult but certainly as interesting article is at
http://sequence.complete.org/node/258
. For a client, check out
http://www.haskell.org/haskellwiki/Roll_your_own_IRC_bot
. But I digress. Actually I wanted to write about AI.

So, after finishing my physics server (actually it’s not completely finished, it has no soccer rules yet, but it’s a server with some physics understanding after all), I went on to do the AI part. Since it can get quite complicated, I thought about doing some exercises on AI first. Now, the way how to start designing and programming AI was not really clear for me. I have the book “Programming Game AI by Example” by Mat Buckland, (a very nice book by the way) which has installed the relatively simple scheme of state machines in my head. I was not all sure, however, if a state machine would really be enough for soccer players, so I went on to do some more research.

I then stumbled upon neural networks, a topic about which I had only briefly read in the past – they seemed interesting but I never had the patience to sit down and try to grasp the theory. I finally went through the basics about them on a few sites including Wikipedia, but noticed none of those sites really showed any practical examples. I finally came upon two web sites that really explained how to use neural networks in detail. The first one is actually in German and describes the use of neural networks in a Counter-Strike bot, see
http://johannes.lampel.net/bll137.html
. It was a very interesting read, but for grasping the details you’d need to dive in the source code. The second site can be found at
http://www.ai-junkie.com/ann/evolved/nnt1.html
; this one is a very very practical introduction to neural networks, it helped me a lot and it’s also by Mr. Buckland, who keeps surprising me how simple it can be for some to make difficult things seem simple. His tutorial trains the networks with genetic algorithms, for which he also wrote a great tutorial (
http://www.ai-junkie.com/ga/intro/gat1.html
). The code, like all code from Mat, seems to be written for a less standards compliant compiler such as Visual Studio 6.0 or similar, which means you have to tweak it a bit if you want it to compile on a compiler that follows a C++ standard, but it’s doable.

Encouraged by the theory and these practical tutorials, I went on translating Mat’s Minesweepers into Haskell. During this process I also realized why almost all of the tutorials and the theory of artificial neural networks didn’t show any practical examples; my neural network code itself is merely about 50 lines of active code and contains mostly just the mathematical formulas to the networks. The applicaton of neural networks, however, is nearing 200 lines of code (huge, isn’t it?) and basically describes the whole learning and using process of the brain, err, neural network.

What I learned was that how the neural networks work is basically just a few formulas that don’t really say much about the application; in the end the network is really just a black box that you don’t really know, but it just “magically” gives you the correct values. The difficult part when applying neural networks to a problem is designing the way they’re used and figuring out how to make them learn as well and fast as possible. (I haven’t figured out any of the other ways of getting networks to learn (like simulated annealing and whatnot), i.e. providing positive feedback to them than genetic algorithms, but for now it seems good enough.) I also realized how the whole process was based on randomness, and while I thought that was quite funky, I did have my moments of despair trying to figure out if the thing had actually learnt something or not. (If in doubt, wait.)

Well, in the end it worked, though the minesweepers weren’t rolling around the screen like in the Windows/C++ counterpart, but the path of a minesweeper is drawn into a SVG file instead, as in
http://finder.homelinux.org/haskell/Mines-0.1/svg_sample/gen400.svg
. In this picture, you can actually see a minesweeper (the line) going for the mines nearby (the dots). Oh, want to take a look at the code? That can be found at
http://finder.homelinux.org/haskell/Mines-0.1/
.

That being cleared out, the next thing to do would be to actually finally implement a simple soccer AI. And, well, if I actually did it like I originally planned, with the locations of all the players and the ball and the player skills and personality as input, action including the direction and power of the action (let it be running, kicking the ball etc.) as output, it would probably end up being a pretty big neural network. The time to train it would probably also be pretty long, I guess.

The solution I’m trying to come up with includes state machines, neural networks, decising trees and maybe an occasional Markov decision process. (I also need to finally get my hands on the book “Artificial Intelligence: A Modern Approach”, which I’ve managed to miss… until now.) The basic principle is the same one as discussed in my last post – breaking everything into smaller parts, with state machines or decision trees to assist with the splitting, while using neural networks for the smaller, more limited problems. Probably my next post will still be on AI, but when that’s done (or at least the skeleton of it), and the AI players are kicking the ball, the only bigger thing missing is a client that actually shows us the action…