Hadn't looked at the old blog in a while - I just noticed the archives were broken and it was inundated with comment spam. So, bye-bye comments, and a new template just to get the archives back up. Anyhow, as it says below, I'm blogging at www.gamedevblog.com now.

New blog has launched

I am now blogging here: www.gamedevblog.com. Come one, come all.

Happy Birthday To Gamedevleague. Death To Gamedevleague.

That's right, one year ago today, the first post to Gamedevleague appeared.
Over that time, the readership went from around 10 hits a day to 150, and then hit a plateau.
I've learned a lot in the last year. I've learned that there's already a word for emergence and it's...emergence. I've learned that I'm not alone in thinking that Zelda's sailing sequences were not fun. I've learned that automated scheduling systems are no substitute for human attention, but I didn't learn that from this blog. I've learned a lot about marketing. I've learned that I like section headings in a small font better than in a big one. And I've learned that typepad seems to be a better source of blogness than blogspot. (I never learned why some of my posts have spaces between the paragraphs and others don't...)
As the hermit crab grows it must migrate to a new shell. Before I was blogging here I was blogging there. Even though only a few of my closest friends read that old one, that makes me one of the industry's oldest bloggers. And now I'll be blogging somewhere else. Details to come.
Since I'm home sick today, now would be a good time to migrate. I've been holding off, because, being a managerial type I like arbitrary milestone dates like "one year from today", and seeing the blog go a whole year seemed cool. And it was cool. It is cool.

More on cross-training

A web developer named Christian Mogensen had this to say:

I'm not a game developer, but I've seen the same problems you have. At my first job, a web design shop called vivid studios in San Francisco, we did quite a bit of cross-training because it made us more flexible, and it promoted understanding across disciplines.

Graphic artists began to understand why the programmers liked certain solutions better than others, and they could thus change their output to better suit the programmers. Vice versa, the programmers could tweak the odd broken gif if they knew their way around photoshop.

I (a programmer by training) actually ended up training my managers in the mysteries of McConnell. (They were all typical web "producers" - amateurs with experience and little training) That was training well spent! If your manager is clue-impaired, give him more clues! :-)

The efficiency/execution-time gains are a plus, but there are second-order effects that give you a further boost as people adapt their work using their extra knowledge of what the other side needs.

Oh, and of course cross-training improves your bus-coefficient (how many people can be run over by a bus before your critical path halts?)

We didn't have a formal training program for artists, it was more about having an atmosphere where cooperation and training "in the moment" came naturally. So when there was a problem, the engineers would walk the artists through an issue. So over time the artists understood what the engineers were talking about. Vice-versa, if you hang around graphic artists all the time, you're not going to avoid picking up some photoshop skills.

Engineering did have a formal training program (readings from Code Complete and such-like) and an artist would occasionally show up for these. The artists would take us along on field trips to the Moma or the Exploratorium. It rubs off on you...

One Of My Sacred Cows Is Dying

A new Manager In A Strange Land is up, after some technical difficulties.

My wife is back from Long Island. Thank God. I've gotten really used to having her around.

Put in an 80 hour week last week. Been a while since I crunched that hard - over a year, at least. I don't recommend it.

And now for the main course:

Once, I was very into "brittle" systems - Chuck Tolman's term, I'm not sure if that's the official software engineering name for it - the idea being if somebody did something wrong it would break. Have fatal asserts, and don't bother checking if a pointer is null and failing gracefully. "I say let 'em crash!" It helps Catch Bugs Early, because when somebody can't work he goes and drags a programmer to his desk to fix his problem NOW. And catching bugs early is a good thing. I indoctrinated the team with this attitude, and it seems to have paid off - we shipped our last few products on time, and our current, most ambitious, project seems pretty solid, generally soaking for an hour or two before it crashes. (Side note: I've noticed this happy thing about fixing crash bugs - as you fix them, the soak time of your game increases exponentially. Your game crashes after a minute. You fix the bug. Your game crashes after two minutes. You fix the bug. Your game crashes after four minutes. Etcetera. Then it's people - one out of two people who play the game all the way through see a crash. Then one out of four. Then one out of eight. At this point, testing is too slow to find the remaining crash bugs. You ship a product which crashes at least once for one in eight people. You live with it.)

Still, others have shipped games without this attitude. David Cook of Triple Play and Kelly Slater likes skippable, nonfatal asserts. Chris Carollo of Deus Ex likes them too. And I have to admit, this attitude has its warts. Just yesterday I was fixing a dereferenced null pointer. The programmer put in a fatal error before the dereferenced null, but did not handle the null case. The error was that the wrong kind of icon was being displayed over the head of a certain entity. This wrong icon was displayed for a single frame while the screen was black. Our code was incorrect, and the fatal assert caught us. But is it a bug if the end user never sees it? Which ties in with my bitching about test driven development. What percentage of the time do the tests catch non bugs? Although it's cheaper to fix a bug if you catch it early in the lifecycle, is it really so much cheaper that it's worth the cost of false alarms going off in the code? (I'm sorry, but I think those numbers about it being 100 times cheaper to catch a bug at design time than in beta are fallacious. I'll accept several times.)

Thing Two: a few days ago I demanded that we make our sound system robust to missing sound files, just like we made our graphics system robust to missing textures. There was a painful chicken-and-egg problem with sounds - here's the lifecycle of a sound asset on our project:

- mission designer requests sound
- sound engineer adds sound to the database, submits
- mission designer syncs, adds sound to mission script, submits
- sound engineer rebuilds all sounds in the game (this takes about an hour), submits. (this involves over a gig of data and brings Perforce to its knees)
- mission designer syncs, tests to make sure the sound is okay

If this process was done out of order the game would crash due to the missing sound asset.

And as fast as Perforce is compared to Sourcesafe, whenever we do a sync we usually have to recompile and rebuild a ton of data, which can take anywhere from a few minutes to half an hour.

Are you feeling our pain yet?

Now, once we slaughter my sacred cow, and force the system to be robust to missing sounds:

- mission designer requests sound
- sound engineer gives mission designer a filename in e-mail
- mission designer adds sound to mission script but doesn't get to hear it yet. submits
- sound engineer adds sound, rebuilds sounds, submits
- mission designer syncs, tests

We've gone from four handoffs to two.

So, after all this, I would revise my philosophy. First - don't settle for brittle. Go for Steve Maguire's "Brittle & robust." Whenever you add an assertion, also take the steps to make sure your system will survive when that assertion fails. Second - some things should be warnings, not errors. Missing textures and sounds should be warnings. Third - at the beginning of the project, fatal asserts are good: they keep your development on a leash, they keep you slow and steady, they make sure you take the time to do things right. But at the *end* of a project, when you need to fix the high priority bugs, and triage dictates that you should ship the low priority ones - let the asserts be skipped. Because those asserts may be just confusing your priorities - you have real crash bugs to fix, and these asserts are often firing on things that are cosmetic bugs...or not even bugs at all.

Case Study in Line Extension

And it's back to marketing, speaking of no focus.

I just checked out the Secret Weapons Over Normandy post-mortem in the latest GD and #5 on the What Went Wrong caught my eye: they had trouble getting the word out that this Secret Weapons was more arcadey than the last one, and apparently some people aren't happy with its arcadey-ness.

One thing Jack Trout and Al Reis would say (this from *the 22 Laws of Marketing*) is that line extensions give you a short term gain for a long term loss. In other words, by taking the Secret Weapons line of hardcore flight sim and trying to extend it into an arcadey console market, LucasArts will sell a few more copies at first but pay for it later because nobody will know what Secret Weapons means anymore. Jack & Al would probably argue that it would have been better to create a whole new product line than try to ride on the success of previous titles. In other words, the difficulty with product messaging started with the very name of their product. If they wanted to avoid misconceptions about the title, they should have simply named it something else, and their #5 What Went Wrong would have gone away.

I'm making it out to be worse than it is - now that Totally Games has jumped with both feet into arcade-flight, they can stay there and build a happy home. (As long as they stop doing hardcore sims under the Secret Weapons name.) There have been examples of titles that used to mean one thing but now mean something else - although no really good examples are coming to mind right now, there was a list of them in *Positioning*...

But if they try to keep both markets then they'll end up with a weak brand.

Getting into marketing is kind of depressing for us developers because usually all we can do is hope our publisher's marketing department doesn't screw it up...

Tired

Thanks for the compliments in the comments, guys. I haven't updated in a while because I've been crunching full time these past few days. Wake up. Get latte. Go to work. Come home. Sleep. Wife's out of town, good excuse to make...game...better. But I've definitely hit a point of diminishing returns...those extra hours just aren't all that productive.