Work-Life Balance, and Finishing a Game

It's been a month since my last post.

Traditionally, I've been pretty good about keeping a rhythm on this blog. One post, bi-weekly for over two years, save for the occasional vacation. And even then, I was usually quick to make-up for lost time.

Lately, though, it's been harder to keep up that rhythm. For one thing, it's getting harder to find topics that are both relevant and as-yet uncovered.

Two years ago, I had seemingly endless questions about becoming an indie developer. How much money does it require? What's a day in the life of an indie like? Why do developers choose to become independent? I tried to dissect each of the events and considerations I faced, in an effort to draw back the curtain for others.

The bigger issue, though, is psychological. As I make NEO Scavenger's final push to completion, I find I'm using every bit of energy I can muster to stay on-task. Despite what I can only describe as a surprising success so far (positive feedback, encouraging sales, acceptance from vendors, and even Greenlight approval), I feel emotionally drained, fearful, stressed, and distracted.

It turns out that this isn't unusual. I definitely struggle with perfectionism, often putting off finishing something because I'm not completely happy with it. And two (plus) years on the same game/engine/job/problems has dulled that once-sharp edge of excitement and novelty.

In the aforementioned article, Ms. Saunders's suggestions for dealing with those blocks are helpful, too. Itemized to-do lists are definitely a huge help, particularly when the tasks are smaller and well-defined. The days when I can't stop working are those when my direction seems clear. Conversely, those that are the most grueling are when my task is large and poorly-defined (e.g. "finish NEO Scavenger's plot").

Recognizing the effort expended vs. the effort remaining is also useful. I'll admit that starting a new game looks enticing, to the point that I could find motivation to work on it in my spare evenings and weekends. But taking a look at how much effort it took to get here, and realizing that I would be here again in two years, helps put things into perspective.

Prioritization is also key. There are several areas which I try to balance in my life. Listed in order from most to least important:

1. Maintaining relationships (spouse, family, friends)
2. Staying healthy (i.e. eating right, getting enough sleep, getting exercise, avoiding crunch)
3. Running my business, Blue Bottle Games
4. Developing my first game, NEO Scavenger
5. Running this blog

That is to say, if something higher up on the list is threatened, something lower gives. So things like vacation, weddings, and funerals usually trump everything else (even healthy sleeping and eating, as a recent funeral has demonstrated). I'm also pretty rigid about not working on evenings and weekends, and eating at regular times (with a few exceptions).

Numbers 3 and 4 have been more demanding lately, not to mention s summer thick with number 1, so blog posting has admittedly been swept aside. And even then, skipping the blog only frees up a few hours every two weeks (normally), so I shudder to think where else the squeeze has occurred (I'm looking at you, number 2).

All things considered, I think I'm surprisingly still pretty healthy in my balance. I still quit at 6:30pm each day, and don't touch work on weekends, except to deal with occasional emails (less than 1-2 hours per weekend).

It's been hard to focus, though, and stress is at a local peak. I'm gritting my teeth again, clenching my jaw unconsciously, and my mind wanders at any possible opportunity. I had a good day, Friday, when I refused any internet distractions for most of the working day. I got a lot done, and I felt better for it. But it was a slog.

I think this is the way it's going to be until I finish NEO Scavenger and launch it. And even then, the prevailing wisdom is that it just gets more stressful in the subsequent months. The road's getting tough, and the toughest is yet to come.

Fake Graphs About Real Things

I have a tendency to be pretty verbose in my blog posts. My typical topic length has inched ever higher over time, and I think it's starting to impact my comfort level in starting new topics. Each new post, it feels harder to write about something because of the growing number of things I've discussed, and the growing number of words I seem to use each time. And as readers, I'm sure the sight of a wall of text is none too enticing.

This week, I had an idea for something different: fake graphs about real things. Basically, there are some concepts in game development that seem like they are easily described in graphs. They're not always possible to quantify, but the trends are pretty easy to qualify. Like describing volume as a function of distance from a speaker, or speed as a function of time spent accelerating, we can describe certain things in our industry in relation to other values.

So, before I accidentally type another paragraph, let's start "graphing!"

Fig. 1: Amount of time spent actually working on game over length of project.

The above one's pretty easy. Particularly if you're a solo indie, this is what you should plan for. In fact, it's a bit incomplete:

Fig 1a: Everything else you'll be doing.

Oh, and one more thing:

Fig 1b: It ain't over 'til it's over.

Here's another one for those running a game-as-a-service or Minecraft-like beta funding model:

Fig 2: Anxiety as a function of build release schedule.

Those with heart conditions shouldn't wait too long between releases! Particularly if:

Fig 2a: Ohgodohgodohgodohgod!

This one, while sobering, is probably no surprise to most:

Fig 3: Game sales over time.

"That's not so bad," you might think. But we may be missing some points of reference:

Fig 3a: WTB rent-free treehouse with free electricity and internet...

There are a few things you can do to help, though:

Fig 3b: Or this might just be a NASA pulsar graph

Each of these peaks will likely have a diminishing effect on the subsequent peaks, as market saturates and relevance decreases over time.

You could also try adjusting the price of your game:

Fig 4: Sales as a function of price.

$0 is a popular price! Conversely, not many people are into mortgaging their homes to buy a game. Though, as the app store has shown us, there are 8 people who will buy your game no matter what (1 of which, accidentally).

How does this look in terms of actual money? Probably something like this:

Fig 4a: Number may not actually be magic.

This assumes some minimum price, of course. Most places aren't selling games for $0.000001. If that were the case, the vertical slopes would be more gradual.

Let's talk about another topic. Phil Fish has recently made the news (again), and you might be wondering what you'd do in his shoes. Being criticized can be unpleasant, but not all criticism is created equal:

Fig 5: Amount of attention paid to criticism vs. how much I respect the critic.

Put another way, if someone is the type of person who slings insults at complete strangers over the internet, it's hard to take them seriously. After all, with such a low barrier to lashing out, they probably do it to everyone. I might even feel left out if they didn't.

Similarly:

Fig 5a: Attention paid as a function of bias.

Overly negative (and positive!) commentary isn't very useful, so I tend to pay the most attention to balanced critique and critics. Overly biased feedback is about as helpful as a ruler with all 8s.

Speaking of usefulness:

Fig 6: Likelihood of reading feedback as a function of its length.

If I open up an email or forum thread, this is pretty much how things are going to play out. Don't get me wrong, the following is also true:

Fig 6a: Wanting and doing are different things, though.

But if it's going to take me 30 minutes to read, 15 minutes to digest, another 10 to re-skim, and then an hour to reply...well, I might not. Of course, I tend to dwell in the far right hand side of this graph when writing, so there's some irony/hypocrisy to be had here.

And while we're on the topic of weaknesses, here's a big one for me:

Fig 7: Desire to postpone decision as a function of its importance.

Decisions can be hard. Big decisions even more so. While it can be healthy to take time to consider options when faced with a decision, it can be unhealthy to postpone for too long:

Fig 7a: "Just a little more research..."

The important feature of this graph is the peak and subsequent negative slope. For every decision, there is some amount of consideration that is optimal, and beyond that, an ever-increasing opportunity cost. The real-world version of this graph may be bumpy, but the general form is the same.

There are more, of course, but this is already quite a bit. Let me know if you found this format to be helpful, entertaining, or even erroneous. And enjoy the break from my walls o' text while you can!

The Hardest Part of Making Games

I was playing Rock Band the other day. We dug the plastic instruments out of storage, and fired up the 360 to dust off our faux musician skills. And at one point, we played Nine Inch Nails's "The Hand That Feeds."

As the little, colored notes dropped off the screen, I found myself imagining how cool it must be to make music like that as a career. I love musical instruments, and to be surrounded by synths, sequencers, 1/4" audio cables, humming amps, electric guitars, and mixers would make my day.

I've dabbled in composition from time-to-time, and I've even had some relatively pleasing results. Sure, they're amateur at best, but creating them was fun. I actually long for the time when I'm simultaneously un-busy enough and motivated to get back into that.

That last thought brought me back from my daydream, even as the colored notes continued. "I'm too busy and too tired," I thought, "because I already have a cool day job like that." It was as sobering as it was uplifting.

In reality, Trent Reznor probably doesn't sit in his studio, admiring his collection. He doesn't sit with a cup of tea, trying out all the synth presets on his Korg M1. He doesn't feel the blissful absence of pressure as he surveys his laboratory.

Mr. Reznor has a business to attend to. He has music to compose, a brand to promote, investments to consider, and relationships to manage. What we picture him doing each day is blissfully ignorant of pressure.

What's So Hard About Making Games?

Like with music, I think there are many on the outside looking in at game developers. Each has their vision of what goes on in a developer's day, and some are more accurate than others. Some may picture developers coding away in the late night hours, illuminated by a cold monitor. Others might picture long conference tables, half-empty pizza boxes, and colored dry-erase diagrams.

Some people picture...other things.

So when it comes to game development in practice vs. imagination, what actually goes on? What are the types of work being done? Are any of them pure enjoyment? Tedium?

And what is the hardest part?

As with music, games are complex products, the creation of which involves a wide variety of tasks. How hard or enjoyable each of these tasks is will depend on the person(s), their abilities, and their preferences. It will also depend on the type of game being made, the tools being used, and the project constraints.

Since NEO Scavenger and Blue Bottle Games are primarily a solo project, I wear a pretty good number of hats. I do most of the dev work, content creation, business administration, and I've even collaborated with a few talented individuals.

What follows, then, is a review of the types of things I've had to do as part of developing NEO Scavenger and running Blue Bottle Games since it started. Along with each, I'll try to call out some of the more enjoyable, challenging, and tedious aspects of each.

If you're considering getting into game development, maybe this will help illustrate some of the rewards and challenges you'll face. And if you're already there, maybe it'll comfort you to know you're not alone :)

• Game Ideas - It won't surprise you to learn that this is one of the easier, more enjoyable aspects of game development. I love coming up with ideas. I can't not do it, in fact. When I get up from bed to go to the bathroom at 3am, I have new ideas by the time I crawl back into bed. I have journals stacked in a drawer with ideas, a massive folder on my hard drive, another on Google Drive. When I bought my first Palm III, it was so I could jot down ideas as I rode the subway to work. I gave up a lucrative job at BioWare so I could make my ideas into games. I've been known to turn down perfectly good ideas in favor of coming up with ideas of my own. Have I mentioned I like coming up with video game ideas?
  
  While usually not tedious, there are a few times when it can be frustrating to come up with an idea. Usually, this is because one needs an idea for a specific problem, but none are coming. Ideas pour when they rain, but they're not always what you need when you need them. They're also notorious for derailing otherwise productive work with distraction.
• Research - This one's a bit like ideas, above. I have to restrain myself from researching things if given the chance. Before I start drawing something, I research it. Writing about something? Research. Start building a new system? Research what other games do.
  
  Research can be tedious, of course. Plowing through an academic paper is rarely as instantly gratifying as Wikipedia. And many times, there are conflicting sources that need to be resolved. Vocabulary can also cause research effort to mushroom, as one is confronted with a wall of unfamiliar terminology.
  
  And then there's the question of research's value. Some of it is justifiable, as research can reveal useful strategies or opportunities. Most of the time, though, it's a guilty pleasure that gets in the way of real work. The challenge is often keeping that in check.

Not the hardest part of making games.

• Pixel Art - Drawing game graphics is moderately hard work, but I enjoy it. It was harder and less enjoyable when I started, as the results were more frustrating. However, with practice, I'm getting better at producing the level of quality that I want, and I enjoy the process more. (The process grows faster, too.) The least enjoyable part of this job is when a piece requires other "infrastructure" art to help it.
  
  For example, drawing a new rifle is lots of fun. Drawing that same rifle again and again, once for held, worn, and stored versions, and again for the combat UI, then all of the above again for the strapped version, again for the scoped version, and again for the strapped, scoped version? Tedious. I'd rather draw a new rifle. However, for it to work in the game, it needs all of its requisite art done, not just the fun part.
• VFX - This one can be a lot of fun, but also very frustrating and complex. The cool thing about it is that subtle changes in code and art can make some really spectacular and interesting things happen. It's almost like a mini game development project within a larger project. (E.g. programming the flying, blinking lights in the Detroit map within NEO Scavenger, below).
  
  The hard part comes when the VFX requires complex algorithms, wrangling multiple coordinate spaces, or bypassing limitations in the platform (e.g. alpha blending that doesn't kill performance).

• Audio Design, Recording - Audio work is something I am terribly amateur at, but still enjoy immensely. As hinted at in my musings about Trent Reznor, I'd love to sit down with some audio equipment and just noodle for days. And I've even afforded myself a few such days during the course of NEO Scavenger, field recording, cutting and looping effects, and even composing a title screen piece (mercifully replaced by Josh Culler's great soundtrack work).
  
  It's not fun all the time, though. Sometimes, it means listening carefully to hours of recorded audio for useful bits, cleaning them up, slice and package them, and labeling them with tags for reference later. And in the worst cases, it can be doing this process over and over until it sounds "right."
  
  Frankly, producing good audio is a full-time job of its own, and I haven't given it the love it deserves. It's beyond my capacity right now, as features and bugs eat the lions share of my development time. I still want to, it's just been back-burnered while fish are frying.
• Programming - This is where a lot of time is spent. If I had to guess, this probably takes the largest share of my time. For just about everything I create in NEO Scavenger, some coding needs to be done to make it appear. Even coding sometimes requires more coding to work (i.e. bugs).
  
  As I approach the end of the project, that seems to be changing, though. Most of the early art/writing/design was for new systems that needed to be built. Now, I'm mostly filling in content variety in established categories, so the coding is shifting more towards maintenance and fixes.
  
  The difficulty and enjoyment in coding varies pretty wildly. Like with VFX above, coding can sometimes mean big changes from little code. This is also often the case when adding a design feature. At such times, coding is my favorite task.
  
  At other times, coding is a tedium that is hard to endure, or impossible to sort through. Rewriting the GUI for NEO Scavenger is one process that was agonizingly boring and involved. And some of the more complex problems can involve abstract and difficult thinking, especially when the system one is building is dependent on one or more systems that also haven't been built (or defined) yet.
• Writing - Writing may actually trump artwork, in terms of time spent on tasks. NEO Scavenger is pretty heavy on text, so a lot of work goes into planning and creating that text.
  
  As with many tasks above, there is fun writing, and there is boring writing. The fun writing comes when my brain is in the right mode, and I have trouble keeping up with it. The story just seems to flow, and I always know what to write next (at least roughly).
  
  The tedious part of writing comes when one needs to handle all the peripheral infrastructure. In NEO Scavenger, players can take multiple paths in encounters, and that means those paths have to be written. The tedium is helped a bit by the fact that I try to make each path interesting. However, nothing helps that sinking feeling when I realize, "ugh, they might do that, and that means a whole new string or tree of encounters I need to conjure up."
  
  The hard part of writing is rather like what I described in programming, above. When so little has been defined, it might seem like one has lots of freedom. However, that freedom can be paralyzing, as one is constantly second-guessing whether the idea or character is plausible, interesting, or whether it might sabotage the story somehow down the line.

"...and the player will be able to choose ANY action they want!"

• Game Design - Design is a bit hard to separate from the above, since it involves almost all of them. On a good day, it means recognizing clever ways of making two systems interact, producing a complexity greater than the sum of its parts. At times like these, one can almost enjoy their own game as if they're a new player, watching systems interact in surprising ways.
  
  On other days, there is a design problem that appears to have no good solution. I've spent days on some problems: thinking; starting down a path; recognizing a new issue; erasing; repeating. It can be very disheartening and frustrating. Sometimes, the problems with a new system don't emerge until all the work is done. Then, it becomes a trade-off between spending extensive time fixing, or extensive time rebuilding.
  
  And tedium can rear its head here, too. Some systems require reams of data entry and proofreading. Other times, navigating a web of interdependencies to ensure the systems don't undermine each other.
• Build and Platform Management - This refers to the process of getting the game to run on the platform of choice, whether it be a browser, Windows, Mac, or Linux. It also refers to the process of getting that game into the hands of the person who needs it, be they oneself, a collaborator, or a customer.
  
  I'll admit, there's little here that I enjoy. I'm thankful for some of the experiences I've had in making NEO Scavenger cross-platform, but I wouldn't call the work enjoyable. Mostly, it's a mixture of tedious and challenging.
  
  The challenge is usually wrestling with the idiosyncrasies of each platform, be they resolution, user input devices, UI conventions, performance, or unexpected bugs. Flash helps with much of this, but as it turns out, causes almost as many problems.
  
  The tedium is usually the actual build process for me. Creating each of the platform builds requires a lot of PC-hopping, file uploading, changelog-writing, and form-filling.

They LOOK friendly enough...

• PR - Interacting with press and public is a job in itself, and can take as much time as you offer. There will always be more conversations out there about your game than you can handle, so you have to set some limits to leave time for game development.
  
  Without a doubt, reading press and player feedback is my favorite part of PR. Praise can be highly motivating, as can constructive criticism. Even knowing that people are simply playing the game can raise one's confidence.
  
  I also enjoy learning about the art of promotion, and creating plans for publicity (e.g. special events, giveaways, strategic link placement, game festivals, etc.).
  
  I'm terrible at executing those plans, though. I hate asking people for things, so cold-calling press outlets is hard for me to do. I also have a hard time pulling the trigger on a PR event, knowing that "just one more" fix or feature would improve the first impression, perpetually delaying my plans.
  
  When it comes to customers, some interactions are easier than others. I love helping people, so if my response can offer something useful to the player, I feel good about it. Conversely, if I don't have anything helpful to offer the player, responding stresses me out.
  
  There are critics out there, but they're actually not as stressful as you might imagine. In cases where the criticism is valid and constructive, it's actually welcomed and helpful. And in cases where it's more insult than critique, it's easy to disregard.
• Team Management - NEO Scavenger may be a solo endeavor, but I have had help along the way. And most of my experience prior to Blue Bottle Games was with large teams.
  
  At its best, teamwork is awesome. Being surrounded by talented, creative, and motivated people can make you do amazing things, and create stellar products. Some of my happiest moments (in any career) were when surrounded by a tight team of bright minds with a shared vision.
  
  Working with Josh on NEO Scavenger's music, and Cameron on NEO Scavenger's story and design, was equally inspiring. They could see things I missed, and offer ideas I never would have, helping to produce a fuller, more professional game experience.
  
  If there's any tedium I experienced, it was due to one major mistake I made: making myself the bottleneck. There's nothing more mitigating to a high-performance machine than forcing it to drive in rush-hour traffic, and that's what I did with Josh and Cameron. I never built the infrastructure for collaborators to contribute directly, and I wanted to own as much of the process as possible. As a result, I slowed down their progress. One could argue that I failed at one of the greater challenges of team management: delegation.
  
  Of course, there can also be bad teams. I've been on my share of those as well, and they can cause no end of tedium and frustration (for both the members and managers). Diverging goals and opinions, authority issues, poor communication, and lack of clarity can ruin even the best collections of talent. Bad teams are jobs in themselves, and making games becomes a side project compared to keeping things from melt-down.
• Business Analysis - As with PR, this is an area I enjoy exploring quite a bit. And that came as a surprise to me.
  
  Learning about economics, business strategies, and entrepreneurship can be really addictive. I enjoy looking at the game I want to make, the market it's trying to reach, and hashing out features, pricing, sales channels, and partnerships to make it viable.
  
  Running the numbers can be tedious, of course. As can digesting info to keep one's picture of the market up-to-date. However, there's a certain Skinner-like reward to reading news about the industry, and suddenly recognizing an opportunity your product is poised to take advantage of. It's one of those rare moments in life when your "money-making idea" can actually do what it says on the tin.
  
  The hard part, here, is often just finding useful information. Nobody is in the business of writing "A Market Analysis Tailored for Dan Fedor." There's a lot of filling in blanks with educated guesses, extrapolating data, and in some cases, blind faith. Will your next game idea be a hit? Nobody knows. Even the AAA analysts are clueless here.

Let's not forget DIY desktop support.

• Testing and Debugging - When folks say things like, "oh, you just play games all day?" this is probably of what they're thinking. And they're wrong.
  
  Just about everyone on a game dev team tests and debugs their work. Or at least, anyone worth their salt does. Fire up the game, navigate to the area where your recent work shows up, and start testing it.
  
  This can actually be a pretty fascinating area of work. It rewards good experimental method, note-taking, statistical analysis, and deduction. Sometimes, it can feel like forensics or detective work. And yes, once in a while, you get a chance to play through a section to see how it "feels."
  
  Unfortunately, this type of work can also require a lot of patience and persistence. It's not good enough that one tests entering combat and exiting again. One has to test cases where the player initiates combat with AI, AI with player, player with AI already in combat with AI, AI with player vs. AI. What if the player dies instead of leaving? The AI? What if one AI leaves and another stays? What if the player leaves, and more than one AI stays?
  
  There can be a combinatorial multitude of scenarios to test, and doing this job right means testing them all. Worse yet, once a change is made to the system, this entire suite of tests needs to be run again. And changes to the system are always happening.
  
  Some testing can be quite frustrating, too. A notorious example in NEO Scavenger are the null-pointer bugs that only show up later in a game session. They're frustrating because the cause is unclear, and one must play the game for long periods before the bug might appear. And often, the symptom of the bug varies from game-to-game, making it hard to tell if two bugs are related. If the bug cannot be caused reliably, and requires long periods of play to trigger, imagine trying to fix it. And worse, verifying that the fix works.
• Project Management - In an ideal world, each developer could just release the game they want, when they want to. In reality, the pressures of time, budget, and manpower impinge on that ideal. Project management is the process of quantifying those needs and constraints, and finding the compromise that balances those forces best.
  
  A lot of folks picture project management as Gantt charts, scrum, status meetings, and spreadsheets. And a lot of project management involves those things.
  
  However, all of us do some amount of project management subconsciously when we assess the value of a new feature against our capabilities, schedule, and budget. Whether it's a customer asking for widescreen format support, or a teammate asking for alpha channels in their sprites, we're assessing cost vs. value somehow in our decision-making process.
  
  There is little that I enjoy about this process, honestly. Gathering data can be a lot of work, it often involves telling people "no," and the results are based on a lot of guesswork.
  
  That said, the process can reveal useful questions and concerns that are missed in more casual assessments. It can also serve as a common reference point for discussions among others. Even as a solo developer, it can be very helpful to have a to-do list that I can sort by effort, value, finished/unfinished, or other criteria.
  
  So, for me, it's one of those things I'm glad to have done, but dislike doing.
• Taxes and Accounting - If you're running your own business, the CRA/IRS wants to know what you earn. And likely, you'll want to know as well.
  
  There is nothing enjoyable nor rewarding about doing taxes. It is 100% tedium and challenge, so I'll leave it at that.
  
  Accounting can be grueling, but the rewards are there if done right. It's important to know how one's business is performing, in order to make sound decisions. And it can sometimes be an ego boost if sales are good. But it's a lot of careful, detailed work, and it can involve some challenges of data-management and research.

More fun than taxes.

Some Additional Hard Parts About Making AAA Games

There are quite a few other areas that I could discuss, particularly from the point of view of a larger team. NEO Scavenger is a pretty small undertaking compared to AAA efforts, and I don't (yet) have to worry about such things as:

• Animation - Either traditional or 3D rigging, keyframing, motion capture, and the like.
• 3D Modeling - 3D models include characters, environment art, props, etc.
• Materials - The development of textures, shaders, and other surface material properties for models.
• Lighting - 3D worlds, and even some 2D worlds, require sources of light to be placed strategically.
• Cinematics - Positioning cameras, moving them around, as well as managing the content being shown, settings for playback, and related work.
• Localization - Translating the game for multiple markets.
• Physics - Assigning volumes and physical properties to items in the game world.
• Licensing - Negotiating permission to use licensed properties.
• Other - And everything else I've forgotten about.

What is the Hardest Part About Making Games?

So what is the hardest part about making games? You may have noticed a pattern in the list above, having to do with the aspects which I find tedious or difficult. "God is in the detail," as they say, and this is as true in game development as any other field.

Dabbling in an art or science is fun, but due diligence is work. The hardest part of any task above is when I have to get my hands dirty and follow through with all the details. There's a reason organizations usually have a limited number of leaders with the broad-strokes vision, and an army of workers to do the detail work.

However, even the tasks listed above are manageable on their own. Many are measured in hours, days, or weeks, and not much worse than school assignments in that respect.

The hardest, in my opinion, is finishing a game. Spending hours each day, every day, consecutively for over two years. Waking up on day 520 of NEO Scavenger, and finding the motivation to do one more. That's the hard part. I like making games, but even I get tired of making the same game after all that time.

Don't mind if I do.

The hardest part of making a game is finishing it. For all the things I can do on my project, that's the one I still haven't.

Return from Vacation, and the Road Ahead

I apologize for the previous gap in updates. Rochelle and I were at the beach with my family for a few weeks, and I made it a point not to do any more work-related stuff than was necessary.

I could get used to these.

Unsurprisingly, work still crept in, here and there. I cut down my email and web administration duties to once every other day, but it still took a few hours each time. There isn't really anyone to cover for me in these cases, so I'm on the hook for order issues, spambot surge mitigation, and customer inquiries. One alternative would be to "go dark" for several weeks, but that seems like an irresponsible thing to do when running a business, particularly where customer support is a component.

As such, vacation wasn't a complete break from the stress. Nothing critical arose during the break, thankfully, but it was still something that was in the back of my head. Work check-in was never more than a day or two away, and more significantly, a giant to-do list awaited me when I returned.

I think that while previous vacations have taught me that vacations are necessary and healthy, this vacation added the caveat that vacations can be diminished by obligation. Basically, there would either have to be no obligations to fulfill, or someone to fill those obligations for me in order to completely relax, stress-free.

The former requires that NEO Scavenger be finished and support-free. Admittedly, that goal is pretty far off, if even possible. More than likely, there will be a long tail of support and attention required, no matter how bulletproof I make the game. Platforms evolve, partnerships are formed, and other changes in the business landscape mean that a product is never truly dead (see Homeworld, Wasteland, etc.)

The latter requires that I expand the team, which is a daunting goal in itself. Questions of trust, availability, motivation, and compensation are each topics that require their own, significant, due diligence. I've long been reluctant to let anyone into the sandbox, and these are some of the big reasons why.

It's time to "start finishing."

As such, my current goal is to "level up" NEO Scavenger, and make it newsworthy again. A little more work, and I can at least consider the demo "done," making it ripe for sharing with portal sites like Kongregate and NewGrounds. That should hopefully make new players aware of the game, and if the beta looks like a worthwhile-enough upgrade, drive new sales as well.

The increased awareness would help with the Greenlight campaign, which is a bid for more PR and sales in itself. There are some contests I'm considering, as well as some "taste makers" I'd like to finally contact, now that NEO Scavenger is maturing.

And if things go smoothly, I can see either finishing the home stretch personally (albeit slowly), or maybe investing a little money to speed the remainder of the content work.

It's a scary moment, one of those crossroads where I endlessly find excuses to avoid taking the next step, but it's time to get into high gear. Whatever happens, I'm already happy with the outcome so far, and I only have more to gain from moving forward.

Plus, peace and rest waits for me over that next hill, and I never thought that would be such an enticing goal.

The Many Faces of Low Morale

And it's one, two, three,

What are we fighting for?

Don't ask me, I don't give a damn,

Next stop is Vietnam.

And it's five, six, seven,

Open up the pearly gates.

Well there ain't no time to wonder why.

Whoopee! We're all gonna die.

    -Country Joe and the Fish

I want to talk a bit about morale.

As game developers, we have good days and bad days. When it's good, it's pretty exhilarating. We're lucky in that our industry is one where many of us do what we love; what we believe in. Most of us approach each day energized, and ready to make something cool.

When that morale drops, however, even the passion we have for our work can be too weak to stir us. It becomes a chore to start or continue a task; to find inspiration; to be critical of our own work; even to approach others. In a way, much of what makes us good game developers comes and goes with our morale.

As someone who's worked continuously for the last 14 years, I've been through a few spells of low morale. And as I became more familiar with them, I started noticing certain signature behaviors in myself and others. During one particularly long bout, I decided to catalog my feelings and behaviors as I had them, with the intent of cleaning it up for presentation and review later.

That "later" has finally come, and I've put together the list below. My hope is that it does a few things:

1. Help managers recognize the tangible benefits of high morale by contrasting against what is lost to low morale.
2. Help employees to understand how low morale can cause a negative feedback loop, worsening the problem.
3. Help both managers and employees alike to recognize the symptoms when they suffer from them.

Low morale has a way of creeping in, so sometimes we're under its influence before we realize it. And in many cases, our natural tendencies are to bunker down and make it worse. Recognizing that it's a problem, and when it happens, are important first steps towards taking corrective actions. Because, as we will see, low morale can have some pretty devastating effects, to individuals, their teams, and projects alike.

Motivation to Work


Probably the most obvious (and expected) effect of low morale is the lack of motivation to actually do work. I would have difficulty starting new tasks, or getting out of idle slumps. The prospect of coming in to work in the morning, or back from lunch to a task, felt burdensome instead of inviting. 

The days quickly devolve into a series of attempts at avoiding work, whether it be checking email, "researching" something on the web, or even making a long, slow trip to get a cup of coffee. I found I'd actually relish the need to use the restroom at times, because it meant I could take a walk away from my desk, even for a few minutes.

Not surprisingly, this type of foot-dragging creates a downward spiral, as tasks get pushed back later and later, piling up and seeming even more insurmountable against the already low, and falling, motivation.

Reactive Posture

Another fairly obvious change in my work ethic involved my willingness to seek out work. When succumbed by low morale, it was tempting to just wait for tasks to come to me, rather than seek out what needed doing most. I just didn't have the energy nor motivation to make new work for myself. Instead, I would lay low and hope to be looked over when it came time to task someone with something.

Unfortunately, that tended to make things worse as well. Invariably, the tasks that did find their way to me were things that others didn't want to do, either. And since I was decidedly un-busy, I was ripe for nomination. Again, tasks would pile up almost as fast as my distaste.

Indecisiveness/Ambivalence

One change that was a bit more of a surprise to me was my lowered investment in decision-making. As someone normally pretty engaged in lively debates (and a confessed contrarian), I was surprised to realize I just didn't care one way or the other how things turned out.

For someone to be this far gone should be concerning. Possibly the only mechanism one has for making things better is to be involved in decisions as they are made. Furthermore, personal investment in a project or organization are only likely to get worse as one sits out important decisions.

Less Feedback

Related to indecisiveness, above, I found a significant drop in my willingness to volunteer feedback to decisions and announcements. Like above, this only serves to dissociate one from the work or environment to which those decisions and announcements pertain.

Defensive Scheduling

My time became a battleground when my morale was at its lowest. I found myself defending personal time against any intrusion. If meetings ran long into my lunchtime, or overtime was called for, I was way more likely to make a "thing" of it. 

Ironically, for all of the lack of motivation and decisiveness exhibited in all other aspects, this area seemed to be the exact opposite. I had fuel for endless battles when it came to defending my personal time, and it only grew stronger the more I was resisted.

Pigeonholing


As with defensive scheduling, above, I also felt a distinct lack of enthusiasm or energy to do things not directly related to my job description. "Not my job" sprung to mind whenever anyone approached me with something I could find an excuse not to do.

In practice, this rarely plays out as obvious as that. Most people would prefer not to be "that guy," who shirks duties and passes the buck. Whether out of politeness or self preservation, most will quietly, begrudgingly, accept the tasks. However, this attitude can be as damaging just by virtue of the stress and resentment it breeds.

Distraction

I also found it can be easy to become distracted by anxiety when one's future is uncertain. When one is worried about what comes next, their mind is not on the "now," and the work suffers. This is a slightly different facet of low morale, as it can infect even the most diligent employees. This sort of anxiety is often the result of leadership failing to communicate in ways that the employee responds to, failing to follow-through with promises, or not communicating at all.

To make matters worse, most game developers are able to call upon an extremely powerful imagination when trying to ascertain worst possible outcomes. And fear can be infectious between colleagues.

Introversion

Possibly one of the more self-destructive aspects of low morale is the tendency to withdraw into oneself. Some employees actually form stronger coworker bonds during times of low morale, finding unity in the shared suffering. 

However, many in the games industry, myself included, are introverts. And introverts often find it easy to abandon social ties when things get tough. Over a period of time, I found myself transitioning away from daily lunch and evenings out with coworkers towards lunches at my desk, and retreating home after work.

The key behavior to watch for isn't necessarily reclusiveness (which might be normal), but rather a sudden or gradual divestment in social ties.

Nothing to Lose

For all the negative side-effects of low morale, it turns out there was at least one benefit: courage. When things are bad, it can feel like there is little room left to fall. Sometimes this can manifest as a desire to champion change within the workplace. 

More often, however, this is the trigger point for employee turnover. Visions of greener grass are all it takes to start vetting other employers, or new careers altogether.

Psychological Impact of Low Morale

Many of the above symptoms are not unlike experiencing depression, and maybe that should be no surprise. For many in the game industry, passion is what defines their career, and having that relationship fail can be devastating.

What are some things we can do to improve morale? A satisfactory discussion of that topic is well beyond the scope of a concluding paragraph. I've heard it said that employee autonomy is a good place to start. Another area to examine is whether your workplace is fostering the right balance of intrinsic and extrinsic motivators, as having  purpose is an important motivator. Or maybe you just need to work on getting everyone on the same page.

Whatever the case, morale should be one of your organization's top concerns, right up there with finding top talent. Because if your current top talent is suffering any or all of the symptoms above, the aren't really top talent anymore. They're just unhappy warm bodies drawing top talent salaries until they can be inspired again.

A Note About Low Morale for Indies

Indies are by no means immune to the effects of low morale. It might sound as if many of the above are AAA organizational and leadership issues, but low morale can precipitate out of a one-man show as well. Faith in one's work, skill, or even platform of choice, can be shaken. Indecision can paralyze progress. Distractions become that much harder to combat with nobody nearby to keep you in check.

And the effects are just as damaging. Low motivation to work can halt progress. Energy for making decisions can come up short, causing a shift to busy work instead of meaningful analysis. And the tendency to withdraw socially can inhibit good customer relations and PR.

Indies have a fairly serious obligation to keep their personal health and morale in check. We may be blessed by a vast network of internet buddies pulling for us, but our remoteness from the network might mean we're "off the grid" for too long before someone notices.

Being sensitive to the symptoms of low morale can help alert us to things gone awry. We can recognize self-destructive behavior if we know what we're looking for. And armed with that knowledge, we can make incremental changes to help get ourselves back on track, or seek a friendly ear when it's too much to tackle alone.

If You Agree to Crunch, You're Part of the Problem

Recently, Ben Kuchera called to light a eulogy for LucasArts, drawing particular attention to the sacrifices made by employees in enduring crunch. In his words, crunch is a "monstrosity" not to be romanticized, and that's actually a pretty good way to put it. For not only is it a wholly destructive force unleashed on those who work in the industry, it also never seems to die, no matter how many times we try to kill it.

And as developers, it's our fault.

What is crunch?

Before I get into the hows and whys, though, let's quickly establish some definitions.

Overtime, which is a component of crunch, is defined as working more than 8 hours in a day, and more than 40 hours in a week. Some places vary in those numbers (e.g. Alberta, Canada uses 44 hours for their week), but it's roughly in that ballpark for most of the world.

Crunch is the "sustained" application of overtime. Definitions of what "sustained" means here vary, but I think all will agree "months" qualifies without hesitation. "Weeks" has a pretty good shot of qualifying as well. "Days" is probably arguable, and starts to depend on how often within a month it happens.

Crunch is, basically, leaning on overtime too heavily to solve problems. And as it turns out, there are some more concrete definitions of how much overtime is "too much."

Why do we crunch?

What is good about crunch? Why do we do it?

Ostensibly, crunch is meant to increase productivity of a given set of developers within a given time frame. Evan Robinson actually addresses this question in his Why Crunch Mode Doesn't Work: Six Lessons:

Management wants to achieve maximal output from employees — they want to produce a (good) product as cheaply as possible. They also want to avoid hiring extra resources that increase the cost of the finished goods unless absolutely necessary. On the surface of it, Crunch Mode looks like the most obvious and logical way to reconcile these two desires.

He also points out that this is often related to a looming project deadline. He then expresses that assumption as a simple, linear function:

O = R * t

Where O is the team's total output, R is the team's rate of output per hour, and t is the number of hours worked.

However, as Evan's extensive research illustrates, R is not constant over time. R actually varies over the course of a normal work day, peaking in the first 4-6 hours. In fact, R approaches and even drops below zero after enough hours, resulting in negative output for workers.

Effectively, an employee starts damaging the product they work on (bugs, quality issues) if left to work for too long continuously. (For reference, Evan cites 21 as the number of continuous hours of work one needs to reach a mental state equivalent to being legally drunk.)  Conversely, working 8-hour days was shown to produce a 16-20% productivity boost over working 9-hour days.

What's more, prolonged periods of long work days start to produce a cumulative fatigue, reducing the average daily rate R by more each subsequent day. In The Revay Report, experimenters found that 60-hour weeks boost productivity for a few days, but productivity noticeably dropped after a week, cancelling out the boost at 2 months, and continuing to worsen over time after that. (see figs. 2, 3a, 3b, page 2)

Evan goes on to point out that every single other industry has spent the past 100 years optimizing this formula, and has arrived at the 8-hour day, 40-hour week. In Evan's words:

Any way you look at it, Crunch Mode used as a long-term strategy is economically indefensible. Longer hours do not increase output except in the short term. Crunch does not make the product ship sooner — it makes the product ready later . Crunch does not make the product better — it makes the product worse.

I don't think I could put it any clearer than that, except to point out that this is only how crunch negatively impacts the product. For the other drawbacks, I certainly can't put it any better than those who already have. In short, crunch ruins lives, products, and sometimes companies, with no up-side.

How is this our fault?

So how is crunch our fault? I mean, developers are the victims, aren't they?

It's true that developers bear the brunt of the pain in crunch. And it might be comforting to imagine that someone "up the pay scale" is responsible for botching things enough to warrant crunch. It's that last phrase, though, that's the problem: "enough to warrant crunch."

How can a project ever warrant crunch? We've already shown that crunch has a purely negative impact except in the shortest of intervals. So how can we agree to do it? Whether for selfish reasons or love for our product, how can we say it's "ok" to begin working in such a way that damages the product, ourselves, and our coworkers?

Nothing warrants crunch. As we've just seen, industry has been wrestling with the pros and cons of crunch for a century now, and crunch has been shown to be ineffective. Crunch is not a rational option. And our guilt is enabling crunch to happen in the first place.

If you agree to crunch, you are incrementally making the whole industry worse for everyone, yourself included.

Take a moment to let that sink in. Everyone is worse off for your agreeing to crunch. You have to work more, and deal with mounting stress and diminishing rest. Your coworkers have to agree or appear unreasonable in the face of your compliance. The lives of those around you and your coworkers are affected. And your company is forced to absorb the drop in product quality, along with a workforce that's losing competence, morale, and loyalty by the day.

Crunch isn't helping anyone. And by agreeing to it, you're making it easier for crunch to exist.

What if your manager asks you to crunch?

In short, tell them "no." Be polite about it, of course, but don't let politeness give way to bad business.

Look at it this way. By asking you to crunch, your manager has simultaneously just done two things:

1. Indicated that they have failed to manage the project appropriately thus far.
2. Indicated that they have no clue how to continue going forward.

Now, if you like this manager, it might be a good time to pull them aside and direct them to this post, or Evan's piece, or any of the research sources Evan used. If they didn't know, fair enough. Now they do.

If they still insist on crunch after learning about its negative effects, you might want to say "I'll think about it" and speak to their manager. Again, arming yourself with polite concern, and some established research on the topic, hopefully you can impress your point on someone with more experience and reverence for historical data.

If you find yourself talking to more than 2-3 managers in this way without progress (or run out of managers), you might want to consider another option: finding a new employer. Because any company willing to impose crunch on their employees at that many levels of management has a deep-seated philosophical problem, and it may take more than common sense to fix it.

What if your manager doesn't ask you directly?

What if you're in a place where crunch is the elephant in the room? I.e. nobody explicitly asks you to crunch, but it's implied that overtime is expected? Perhaps that's just "the way it's always been," and sticking to normal working hours seems like an invitation to be fired.

Again, this is a topic to be broached with managers. Give them the benefit of the doubt. Bring it up, and see where things go. As above, if you can't make any progress after 2-3 rungs on the company ladder, that may be a sign you're in a place that's unhealthy.

What if they fire me?

What if they do?

Ask yourself that, seriously. If the company you work for sees your concern about unhealthy work-life balance as cause for termination, I think you can safely write them off as an employer you want a relationship with any longer.

You may have heard (or even been told at some creepier companies) that there are 100 others waiting to take your job if you leave, so it's no skin off the company's teeth if you go. That's not true, though.

Good companies recognize the true cost of finding and training good workers. It takes time and resources to find a good worker, and more to train them in your company's workflow. If your company doesn't recognize this, it's probably not a good company. They'd be doing you a favor by firing you.

What if I'm compensated for crunch?

In light of the aforementioned effects of crunch, this should be irrelevant. The focus should be on removing crunch entirely, not putting a band-aid on it.

If employees were compensated for crunch, we might see a change in crunch habits in the industry, but that's just conjecture. As it is, employees are rarely compensated fairly for their crunch, either in dollars or days off.

California may be the one place where this is addressed by law, promising anyone under $83k annual salary ($39.90/hr) time-and-a-half pay for any and all overtime (Labor Code Section 515.5(a)(3)). They also define several exceptions and exemptions, which mandate overtime for entry-level or in-training software fields, film and television artists, and writers. How this impacts software development in California remains to be seen, though.

However, whether you're compensated for extra work or not doesn't address the bigger issue of whether your management knows what they're doing or not.

The bottom line

The point to remember here is that a century of industrial research and work hours testing has resulted in the nominal working hours of 8 per day, and 40-44 per week. This combination produces the highest sustainable productivity possible.

In short bursts, overtime can produce a slight boost in productivity. But the boost wanes after only a few days, and actually cumulatively grows worse each day after that.

If your employer is suggesting (or fostering) an environment where overtime is sustained longer than a few days, start considering whether they're causing more harm than good. See if you can open a dialogue with managers or HR to address the problem in other ways. And be prepared to say "no."

If you like your employer, reward them accordingly. Give them diligence, timeliness, hard work, honesty, creativity, accountability, loyalty, and all the other things you're paid for. In exchange, expect them to provide a healthy working environment, tools and support, intelligent planning, and monetary compensation for your efforts.

Sometimes, Running a Business Stinks

The past 72 hours been a pretty wild ride. And it's highlighted some of the less glamorous aspects of running an online business. I think things are starting to stabilize this afternoon, but I'm pretty exhausted from the experience.

Things Were Running Smoothly When...

Friday was a pretty normal day. Quiet, even. Most of the day was spent working on plot development for NEO Scavenger. The day didn't go without it's hitches, but as days go, it was nothing unusual. I was happy to finish the day with some plot knots to untie over the weekend, so I wrapped-up, had some dinner, and Rochelle and I went bowling with some friends.

The next morning, I was pretty excited to see a sudden spike in traffic in my logs. NEO Scavenger was mentioned in a post apocalyptic survival reddit post, and dozens of folks were popping by the site to check out the game. Cool!

Site Maintenance

Except, my ISP was doing scheduled maintenance early Saturday morning. It wasn't until I tried loading bluebottlegames.com that I realized it was down.

5-day Site Visitor Graph: Hourly

Well crap. That stinks. Just as a bunch of new folks are drawn to check out my game, the site goes dark. And worse, dark for close to 12 hours.

"Oh well, " I said. It's a bummer, but nobody's fault, really. I mean, I guess I could blame myself for not having a redundant server or something, but I'm not that high tech (or deep-pocketed). I decided to just live with it. Besides, friendly Reddit server take-downs were nothing new. Folks probably would just figure my little server was flooded temporarily.

Site Flakiness

Later that day, bluebottlegames.com was back up and running. I did my usual email and forum check, to verify nobody was reporting any issues. And all seemed clear.

Except for one thing: every other page load seemed to turn up blank. No error message, no content, nothing. Just a blank white screen. Refreshing the page seemed to fix it, so I figured it was a temporal thing. "It'll sort itself out."

No, it won't. The following day, I was replying to a customer having issues with NEO Scavenger on their Mac, and it was still happening. Happening everywhere. Sometimes it was a content page on the site. Other times it was a forum. Even some of the site admin pages were failing. And as before, usually a reload would fix it. But the reloading was becoming less and less reliable. Sometimes, I had to reload the page 4-5 times to see anything. And if my customers were seeing the same thing, then that was unacceptable.


The White Screen of Death

I decided to dig into the issue a bit more. I started searching for related issues on the web, and was initially happy to see others reporting the same issue. Blank screens in Drupal (the content management system I use, v6.28) were pretty common. Maybe finding a solution will be easy?

However, upon further investigation, I was less happy to discover I had the same problem. This problem, as it was known to the Drupal community, was the "White Screen of Death."

The WSOD is a common issue with Drupal, but it doesn't have a common solution. In fact, there are almost as many causes of the WSOD as there are Drupal installations, and finding the right one for you can be a real quest.

The link above is probably the biggest authority on the issue, and even there, there are no less than 28 different causes listed in the article, and some 80+ comments detailing other issues customers have had. Basically, Drupal's WSOD is a symptom of practically every Drupal disease. I'm having trouble thinking of a human analogue. Headache? Fever? Common cold?

I spent hours on Sunday trying to make rhyme or reason of it. None of the remedies I saw seemed to help. I couldn't even get error or log messages. And worse, it was an intermittent problem, so I couldn't even reliably cause it to happen.

The only things I could verify about it were:

1. I only get the WSOD on my live server. Migrating the db from live to my localhost didn't duplicate the WSOD issues.
2. I only started getting the WSOD since my ISP's scheduled maintenance, when the server was (apparently) upgraded/restarted.
3. I only get a WSOD in Chrome.
4. Additionally, Chrome seemed to exhibit stylesheet issues when the page did load. Textareas would be too narrow to fit their <div>, or the page would nudge upward when I clicked a link (seemingly to adjust alignment with the Admin Menu module's topnav bar). Something was stalling the css until I clicked a link, upon which it would fix itself, then load a new page with stalled css (or js, or WSOD).
5. Firefox never exhibits any issues.
6. IE seems to work too, except for one page partially loading (later determined to be a known issue with YouTube embedding)
7. No errors appeared on the page, in Drupal's logs, nor server logs, even with error reporting hard-coded to be on in index.php.
8. Using Chrome's "Inspect Element" context menu option revealed that the page was entirely missing the <body> tag and contents. It was just an <html> and <head>, and the <head> seemed to be missing some elements. Also, Chrome usually complained of "Failed to load resource" on the page itself, but all css/js/images were loaded ok.
9. Using "View Source" on Chrome apparently reloaded the page, and showed the correct, full content source.
10. The WSOD appeared more frequently when js and css optimization was turned on (but still occurred when all Drupal optimization/compression/caching was disabled).
11. Flushing all caches had no effect.
12. Running update.php had no effect.
13. Manually truncating Drupal db tables had no effect.
14. Using Backup & Restore on Drupal's db had no effect.
15. Disabling all modules outside of Core and Core Optional had no effect.
16. Switching from BBG's custom theme to the Garland theme had no effect.
17. Rebuilding permissions had no effect.

By the time Sunday evening rolled around, bluebottlegames.com was stripped down to core modules, no theme, no caching, and had a rebuilt database. And it was still flaking out.

Worse, the node access modules I disabled caused the permissions table to get out-of-date, which caused all site content to disappear for all users, all the time. Basically, when the WSOD wasn't happening, all of the site's pages were empty blue shells, and the forums all had 0 posts in them.

Even worse still, to rebuild the permissions and fix the empty content, I had to run a script via Drupal. And that script failed with a WSOD whenever I attempted it.

I had totally messed up my site. The "Site Crash" label in the above image refers to the time when I took the site offline to avoid any more users seeing the empty shell of a site.

Fortunately, Firefox was able to run the permission rebuild script without any WSOD. And I was able to at least get the site showing content again.

But as my efforts continued past 10pm, I decided it wasn't going to be solved anytime soon. I started making changes necessary to return the site to the formerly flaky intermittent WSOD. It wasn't ideal, but the occasional user reload was a far cry better than no site at all. If nothing else, I wanted the forums and contact page online for users to report issues, if needed.

I posted a news item to the homepage alerting customers to the WSOD issue, and apologized for the inconvenience of the downtime. Then, I went to bed.

Come Monday, It'll Be Alright

I was back at the computer at 6:15am. And unsurprisingly, the site hadn't fixed itself. I fired up the usual websites, checked messages, looked for forum posts. Some users reported seeing similar WSOD issues. And, bless them, they blamed their internet connection instead of me.

I decided to try a different approach this time. Instead of grasping at straws offered by forums on the net, I decided to debug Drupal. I added print statements to Drupal's index.php, to see if I could trace the value of the content. And when that failed to reveal anything, I started adding traces to Drupal's core code (*.inc files).

I don't like doing this sort of thing, as I'm nervous about screwing things up worse than they are. Plus, doing it in a way that doesn't affect the live site's users is tricky. But in retrospect, it's the only way to really know what's going on.

I found a function in bootstrap.inc which loads various Drupal bits in phases: drupal_bootstrap($phase). Every page on a Drupal site calls this function first, doing a full bootstrap (all phases). I added a trace inside the while loop that executes for each phase as it loads, and I printed the ID of the phase.Testing on my local site, I could see that my site loads phases 0-8.

When I uploaded the modified bootstrap.inc file to the live server, I saw traces for all phases. Reload. All phases again. Repeat this another dozen times, on different pages where I most often encountered the WSOD. Everything loads normally.

Was the WSOD gone?

Tentatively, I backed out the bootstrap.inc changes, so it was back to normal. Still, the site seemed to be loading ok. I reenabled caching to normal. Still ok. Turned on page compression. Still ok. I stopped short of turning on optimized js and css. Maybe I'll muster the courage for that tomorrow.

A Wizard Did It

So what happened? Uploading that file seemed to stop the WSOD, and leave it fixed even after that file is restored. What dark magic is this?

That's actually my first theory: dark magic. But if you pin me down for a more mundane answer, I'm going to guess it was some sort of behind-the-scenes cache. I'm not sure what else would explain a site's complete performance alteration when a single file is uploaded and then un-uploaded again.

That was at about 11am today. As of 5pm, I haven't seen a WSOD. Mercifully, no players have posted in the White Screen tech support thread on my forums, either. I'm hopeful this issue is resolved.

But What About That Mac User?

Oh yeah, remember me mentioning way up there that this whole investigation started when trying to help a Mac user with NEO Scavenger? Yeah, Mac compatibility is an issue in it's own right, concurrent with the site debacle.

I'm not going to detail that issue here, as it's a pretty lengthy topic of it's own. The forum thread linked above has all the details. And what's more, I've partially touched on it in the past.

The short version is this: Flash is rapidly becoming as much a burden as a boon. For someone trying to develop a stand-alone application, I'm at the point where I am highly reluctant to recommend Flash as a viable option. Issues include:

1. "Create projector" no longer supported on Linux, as of v11.2.
2. Flash CS6 no longer supported on OSX 10.8.
3. Any projectors one does create are going to trip security on modern OSes. And in Mac's case, Gatekeeper is a sticky issue for OSX 10.7.3 and 10.7.4.
4. Digitally signing Flash projectors appears to have spotty support, unless one uses 3rd party wrappers (which are, in themselves, reportedly unreliable).
5. Adobe's recommended solution, building AIR apps, is unsupported on Linux. Also, AIR's installation process is flawed, at best. Also, AIR has the periodic "Update AIR" nagging dialog.
6. Flash is no longer officially supported on Android nor iOS.

I was a long adherent to Flash. It served me well. NEO Scavenger wouldn't be if it weren't for Flash's ease of use, and then-universal deployment options.

So it's unfortunate that this era appears to be in it's winter.

All's Well That Ends Well

The good news? At least we're back to normal. The site seems to be running again. Upgrading OSX to 10.7.5 seems to fix Flash projector Gatekeeper woes until I can find another way to certify projectors. I think I may actually be able to return to plot development tomorrow.

Let's just hope that stretch of actual game dev continues for a while!