Monday, June 17, 2013

Game Dev From the Ground Up

Recently, some of the players on NEO Scavenger's forums posted a request for advice on how to start making games. I've talked quite a bit about bootstrapping indie game development here, but a lot of my focus has been on the business aspects. I do mention tools and techniques here and there, but they've been spread across two years of posts, and are often directed at a somewhat more experienced crowd.

Today, then, I thought I might take a bit of time to walk newcomers through the field. There's a ton of info out there. And sometimes, that can be just as bad as there being no info out there. How does one know they're following advice or tutorials that will lead anywhere? What if one follows a tutorial, only to find that it dead-ends somewhere down the line?

As it turns out, there's both good and bad news here. The bad news is that false-starts are common. Even experienced devs are lured into dead-ends. Technology is an ever-shifting field, and games even more so. It's easy to commit time and resources to something that won't pan out, and that's par for the course.

The good news is that there's usually something to be gained from such failures. This is particularly the case for beginners, where every step is contributing to new skills and perspectives, if not the end product. When one is a complete beginner, there's nowhere to go but up.

Where to Go for Info

As already mentioned, there are billions of places to get info on developing games. Google searching can be a bit daunting:

Better make some coffee...
So how does one know where to start?

I considered compiling a list of indie development resources, but then I remembered that someone has already gone through the trouble. In fact, many people have:


  1. Pixel Prospector maintains a gigantic list of indie resources. It's probably one of the most comprehensive lists on indie game development tools, techniques, and wisdom, collected from around the web. http://www.pixelprospector.com/indie-resources/
  2. TIGForums is one of the more active independent developer forums. Subforums abound, covering every aspect of development, including creative, technical, business, and tutorialshttp://forums.tigsource.com/
  3. Gamasutra is a major source of all things gaming, with a focus more on the developers than consumers. There is a vast catalog of knowledge here, including news, tutorials, GDC videos, jobs, and op-ed. http://www.gamasutra.com/
  4. StackExchange is probably one of the most valuable places to know about when trying to find the answer to a technical question. The questions and answers are usually of high quality and value, owed in part to their curation and voting mechanisms. Many of my epic quests for esoteric programming or technical knowledge end here. http://stackexchange.com/
  5. Daniel Cook's blog posts are also a good place to read about game design, business, and the craft in general. Aspiring indies who struggle with art may be particularly interested in his article about sourcing game graphicshttp://www.lostgarden.com/2008/07/directory-of-posts.html

Some folks like sharing what they've learned, which might make this a good time to mention an important lesson I've learned: see if someone's already done the work for you. The internet is a pretty amazing thing, giving us near-instant access to a growing proportion of all human knowledge. When it comes to game development, a lot of the problems (most, in fact) are ones that have been tackled before. It's worth doing a quick search first, just in case someone's already saved you the trouble.

Which Tools to Use

Choosing a platform, engine, tools, or project management method is like choosing a religion: fanatics are everywhere, and will try to sell you on their favorite. But in the end, just about any of them will teach you something, even if it's that religion isn't for you.

Again, I'm going to harp on Pixel Prospector. One of the first links they list covers a vast list of tools and engines. Read what people say about them. And more importantly, look up what people are building with them. There's no greater proof of a tool's validity than the thing it was used to create.

What did I learn on? Well, this:

Computers can do ANYTHING
I also learned by copying BASIC code from Antic magazine into an Atari 800, and watching the results go. Also, trying to emulate Leisure Suit Larry as a text app in my junior high computer class, using BASIC on an Apple IIe. And taking a "programming for engineering" course at univeristy. And buying 20lb "Learn 2 Program" books from Barnes & Noble, and copying their C-code into engines like Allegro. And writing Flash apps for my web development employer. And following Microsoft's XNA banner for a while.

The point of my telling you this isn't to say that I've had a lot of training. Most of the above was better described as "flailing," or maybe just "failing." And even the things that did succeed became obsolete with time. Technology comes and goes fast, as do platforms.

The point is to learn something in the process. All of the above taught me bits about how computer programs work, how information is organized, how logic controls data in an application, and how bitmaps work. Sometimes those lessons were learned while modding Dark Reign. Other times, I wrote code that put pixels on the screen in a starfield, and I moved them about in sloppy for-loops.

What's my tool of choice? I'm a fan of Flash's Actionscript, and flixel. Actionscript is a c-style, object-oriented (OO) language, which is a good type of language to learn. Many game-related technologies are c-style, OO languages:

  • C/C++
  • C#
  • Objective C
  • Flash
  • Unreal/UnrealScript
  • Java
  • HTML5/Javascript
  • PHP
  • Perl
  • Haxe/OpenFL (a.k.a. NME)

Learning one is usually a good head-start in any of the others.

Flixel is a game engine built on Flash actionscript, and is a good balance of existing systems and free-reign to do what I want.

Would I build my next game in Flash/Flixel? Probably not. Flash is starting to limit me in some ways, and I'm looking towards Haxe and HaxeFlixel as a next step. It takes a lot of what I like about Flash and Flixel, adds some powerful abilities to it, and frees it from the clutches of Adobe (and the Apple-Adobe-Android wars).

That said, building something in Flash/Flixel now should put one in a good position to transition to HaxeFlixel later. They're similar enough that learning one teaches most of the other.

Hello, World

And once you've chosen your tools of choice, what then? Should you write an epic design doc, plan out all the facets of your game, and start building?

I'd focus on "Hello, World." If you can get code to compile and display that phrase on the screen, there's nothing stopping you from writing "Health = 100," or "You are standing in an open field west of a white house, with a boarded front door."

Or simply print some ascii art, and let the arrow keys move it around the page. Or maybe write some text about how much candy you have. Or how your dwarves feel.

The point here is that once you've got something displaying on the screen, the rest is just tweaking and watching the results. NEO Scavenger was originally a sprite of a man, and clicking a button moved that man closer to a dot. Each time he moved, his sleepiness went up. If you clicked another button, he slept, and his sleepiness went down. I kid you not.

So choose a tool, any tool. Search Google for:
tutorial "hello world" <tool name>
and don't give up!

Monday, June 3, 2013

Return from Vacation, and the Road Ahead

I apologize for the previous gap in updates. Rochelle and I were at the beach with my family for a few weeks, and I made it a point not to do any more work-related stuff than was necessary.

I could get used to these.
Unsurprisingly, work still crept in, here and there. I cut down my email and web administration duties to once every other day, but it still took a few hours each time. There isn't really anyone to cover for me in these cases, so I'm on the hook for order issues, spambot surge mitigation, and customer inquiries. One alternative would be to "go dark" for several weeks, but that seems like an irresponsible thing to do when running a business, particularly where customer support is a component.

As such, vacation wasn't a complete break from the stress. Nothing critical arose during the break, thankfully, but it was still something that was in the back of my head. Work check-in was never more than a day or two away, and more significantly, a giant to-do list awaited me when I returned.

I think that while previous vacations have taught me that vacations are necessary and healthy, this vacation added the caveat that vacations can be diminished by obligation. Basically, there would either have to be no obligations to fulfill, or someone to fill those obligations for me in order to completely relax, stress-free.

The former requires that NEO Scavenger be finished and support-free. Admittedly, that goal is pretty far off, if even possible. More than likely, there will be a long tail of support and attention required, no matter how bulletproof I make the game. Platforms evolve, partnerships are formed, and other changes in the business landscape mean that a product is never truly dead (see Homeworld, Wasteland, etc.)

The latter requires that I expand the team, which is a daunting goal in itself. Questions of trust, availability, motivation, and compensation are each topics that require their own, significant, due diligence. I've long been reluctant to let anyone into the sandbox, and these are some of the big reasons why.

It's time to "start finishing."

As such, my current goal is to "level up" NEO Scavenger, and make it newsworthy again. A little more work, and I can at least consider the demo "done," making it ripe for sharing with portal sites like Kongregate and NewGrounds. That should hopefully make new players aware of the game, and if the beta looks like a worthwhile-enough upgrade, drive new sales as well.

The increased awareness would help with the Greenlight campaign, which is a bid for more PR and sales in itself. There are some contests I'm considering, as well as some "taste makers" I'd like to finally contact, now that NEO Scavenger is maturing.

And if things go smoothly, I can see either finishing the home stretch personally (albeit slowly), or maybe investing a little money to speed the remainder of the content work.

It's a scary moment, one of those crossroads where I endlessly find excuses to avoid taking the next step, but it's time to get into high gear. Whatever happens, I'm already happy with the outcome so far, and I only have more to gain from moving forward.

Plus, peace and rest waits for me over that next hill, and I never thought that would be such an enticing goal.

Tuesday, April 30, 2013

The Many Faces of Low Morale


And it's one, two, three,
What are we fighting for?
Don't ask me, I don't give a damn,
Next stop is Vietnam.
And it's five, six, seven,
Open up the pearly gates.
Well there ain't no time to wonder why.
Whoopee! We're all gonna die.

    -Country Joe and the Fish

I want to talk a bit about morale.

As game developers, we have good days and bad days. When it's good, it's pretty exhilarating. We're lucky in that our industry is one where many of us do what we love; what we believe in. Most of us approach each day energized, and ready to make something cool.

When that morale drops, however, even the passion we have for our work can be too weak to stir us. It becomes a chore to start or continue a task; to find inspiration; to be critical of our own work; even to approach others. In a way, much of what makes us good game developers comes and goes with our morale.

As someone who's worked continuously for the last 14 years, I've been through a few spells of low morale. And as I became more familiar with them, I started noticing certain signature behaviors in myself and others. During one particularly long bout, I decided to catalog my feelings and behaviors as I had them, with the intent of cleaning it up for presentation and review later.

That "later" has finally come, and I've put together the list below. My hope is that it does a few things:
  1. Help managers recognize the tangible benefits of high morale by contrasting against what is lost to low morale.
  2. Help employees to understand how low morale can cause a negative feedback loop, worsening the problem.
  3. Help both managers and employees alike to recognize the symptoms when they suffer from them.
Low morale has a way of creeping in, so sometimes we're under its influence before we realize it. And in many cases, our natural tendencies are to bunker down and make it worse. Recognizing that it's a problem, and when it happens, are important first steps towards taking corrective actions. Because, as we will see, low morale can have some pretty devastating effects, to individuals, their teams, and projects alike.

Motivation to Work


Probably the most obvious (and expected) effect of low morale is the lack of motivation to actually do work. I would have difficulty starting new tasks, or getting out of idle slumps. The prospect of coming in to work in the morning, or back from lunch to a task, felt burdensome instead of inviting. 

The days quickly devolve into a series of attempts at avoiding work, whether it be checking email, "researching" something on the web, or even making a long, slow trip to get a cup of coffee. I found I'd actually relish the need to use the restroom at times, because it meant I could take a walk away from my desk, even for a few minutes.

Not surprisingly, this type of foot-dragging creates a downward spiral, as tasks get pushed back later and later, piling up and seeming even more insurmountable against the already low, and falling, motivation.

Reactive Posture

Another fairly obvious change in my work ethic involved my willingness to seek out work. When succumbed by low morale, it was tempting to just wait for tasks to come to me, rather than seek out what needed doing most. I just didn't have the energy nor motivation to make new work for myself. Instead, I would lay low and hope to be looked over when it came time to task someone with something.

Unfortunately, that tended to make things worse as well. Invariably, the tasks that did find their way to me were things that others didn't want to do, either. And since I was decidedly un-busy, I was ripe for nomination. Again, tasks would pile up almost as fast as my distaste.

Indecisiveness/Ambivalence

One change that was a bit more of a surprise to me was my lowered investment in decision-making. As someone normally pretty engaged in lively debates (and a confessed contrarian), I was surprised to realize I just didn't care one way or the other how things turned out.

For someone to be this far gone should be concerning. Possibly the only mechanism one has for making things better is to be involved in decisions as they are made. Furthermore, personal investment in a project or organization are only likely to get worse as one sits out important decisions.

Less Feedback

Related to indecisiveness, above, I found a significant drop in my willingness to volunteer feedback to decisions and announcements. Like above, this only serves to dissociate one from the work or environment to which those decisions and announcements pertain.

Defensive Scheduling

My time became a battleground when my morale was at its lowest. I found myself defending personal time against any intrusion. If meetings ran long into my lunchtime, or overtime was called for, I was way more likely to make a "thing" of it. 

Ironically, for all of the lack of motivation and decisiveness exhibited in all other aspects, this area seemed to be the exact opposite. I had fuel for endless battles when it came to defending my personal time, and it only grew stronger the more I was resisted.

Pigeonholing


As with defensive scheduling, above, I also felt a distinct lack of enthusiasm or energy to do things not directly related to my job description. "Not my job" sprung to mind whenever anyone approached me with something I could find an excuse not to do.

In practice, this rarely plays out as obvious as that. Most people would prefer not to be "that guy," who shirks duties and passes the buck. Whether out of politeness or self preservation, most will quietly, begrudgingly, accept the tasks. However, this attitude can be as damaging just by virtue of the stress and resentment it breeds.

Distraction

I also found it can be easy to become distracted by anxiety when one's future is uncertain. When one is worried about what comes next, their mind is not on the "now," and the work suffers. This is a slightly different facet of low morale, as it can infect even the most diligent employees. This sort of anxiety is often the result of leadership failing to communicate in ways that the employee responds to, failing to follow-through with promises, or not communicating at all.

To make matters worse, most game developers are able to call upon an extremely powerful imagination when trying to ascertain worst possible outcomes. And fear can be infectious between colleagues.

Introversion

Possibly one of the more self-destructive aspects of low morale is the tendency to withdraw into oneself. Some employees actually form stronger coworker bonds during times of low morale, finding unity in the shared suffering. 

However, many in the games industry, myself included, are introverts. And introverts often find it easy to abandon social ties when things get tough. Over a period of time, I found myself transitioning away from daily lunch and evenings out with coworkers towards lunches at my desk, and retreating home after work.

The key behavior to watch for isn't necessarily reclusiveness (which might be normal), but rather a sudden or gradual divestment in social ties.

Nothing to Lose

For all the negative side-effects of low morale, it turns out there was at least one benefit: courage. When things are bad, it can feel like there is little room left to fall. Sometimes this can manifest as a desire to champion change within the workplace. 

More often, however, this is the trigger point for employee turnover. Visions of greener grass are all it takes to start vetting other employers, or new careers altogether.

Psychological Impact of Low Morale

Many of the above symptoms are not unlike experiencing depression, and maybe that should be no surprise. For many in the game industry, passion is what defines their career, and having that relationship fail can be devastating.

What are some things we can do to improve morale? A satisfactory discussion of that topic is well beyond the scope of a concluding paragraph. I've heard it said that employee autonomy is a good place to start. Another area to examine is whether your workplace is fostering the right balance of intrinsic and extrinsic motivators, as having  purpose is an important motivator. Or maybe you just need to work on getting everyone on the same page.

Whatever the case, morale should be one of your organization's top concerns, right up there with finding top talent. Because if your current top talent is suffering any or all of the symptoms above, the aren't really top talent anymore. They're just unhappy warm bodies drawing top talent salaries until they can be inspired again.

A Note About Low Morale for Indies

Indies are by no means immune to the effects of low morale. It might sound as if many of the above are AAA organizational and leadership issues, but low morale can precipitate out of a one-man show as well. Faith in one's work, skill, or even platform of choice, can be shaken. Indecision can paralyze progress. Distractions become that much harder to combat with nobody nearby to keep you in check.

And the effects are just as damaging. Low motivation to work can halt progress. Energy for making decisions can come up short, causing a shift to busy work instead of meaningful analysis. And the tendency to withdraw socially can inhibit good customer relations and PR.

Indies have a fairly serious obligation to keep their personal health and morale in check. We may be blessed by a vast network of internet buddies pulling for us, but our remoteness from the network might mean we're "off the grid" for too long before someone notices.

Being sensitive to the symptoms of low morale can help alert us to things gone awry. We can recognize self-destructive behavior if we know what we're looking for. And armed with that knowledge, we can make incremental changes to help get ourselves back on track, or seek a friendly ear when it's too much to tackle alone.

Monday, April 15, 2013

If You Agree to Crunch, You're Part of the Problem

Recently, Ben Kuchera called to light a eulogy for LucasArts, drawing particular attention to the sacrifices made by employees in enduring crunch. In his words, crunch is a "monstrosity" not to be romanticized, and that's actually a pretty good way to put it. For not only is it a wholly destructive force unleashed on those who work in the industry, it also never seems to die, no matter how many times we try to kill it.

And as developers, it's our fault.

What is crunch?

Before I get into the hows and whys, though, let's quickly establish some definitions.

Overtime, which is a component of crunch, is defined as working more than 8 hours in a day, and more than 40 hours in a week. Some places vary in those numbers (e.g. Alberta, Canada uses 44 hours for their week), but it's roughly in that ballpark for most of the world.

Crunch is the "sustained" application of overtime. Definitions of what "sustained" means here vary, but I think all will agree "months" qualifies without hesitation. "Weeks" has a pretty good shot of qualifying as well. "Days" is probably arguable, and starts to depend on how often within a month it happens.

Crunch is, basically, leaning on overtime too heavily to solve problems. And as it turns out, there are some more concrete definitions of how much overtime is "too much."

Why do we crunch?

What is good about crunch? Why do we do it?

Ostensibly, crunch is meant to increase productivity of a given set of developers within a given time frame. Evan Robinson actually addresses this question in his Why Crunch Mode Doesn't Work: Six Lessons:
Management wants to achieve maximal output from employees — they want to produce a (good) product as cheaply as possible. They also want to avoid hiring extra resources that increase the cost of the finished goods unless absolutely necessary. On the surface of it, Crunch Mode looks like the most obvious and logical way to reconcile these two desires.
He also points out that this is often related to a looming project deadline. He then expresses that assumption as a simple, linear function:
O = R * t
Where O is the team's total output, R is the team's rate of output per hour, and t is the number of hours worked.

However, as Evan's extensive research illustrates, R is not constant over time. R actually varies over the course of a normal work day, peaking in the first 4-6 hours. In fact, R approaches and even drops below zero after enough hours, resulting in negative output for workers.

Effectively, an employee starts damaging the product they work on (bugs, quality issues) if left to work for too long continuously. (For reference, Evan cites 21 as the number of continuous hours of work one needs to reach a mental state equivalent to being legally drunk.)  Conversely, working 8-hour days was shown to produce a 16-20% productivity boost over working 9-hour days.

What's more, prolonged periods of long work days start to produce a cumulative fatigue, reducing the average daily rate R by more each subsequent day. In The Revay Report, experimenters found that 60-hour weeks boost productivity for a few days, but productivity noticeably dropped after a week, cancelling out the boost at 2 months, and continuing to worsen over time after that. (see figs. 2, 3a, 3b, page 2)

Evan goes on to point out that every single other industry has spent the past 100 years optimizing this formula, and has arrived at the 8-hour day, 40-hour week. In Evan's words:
Any way you look at it, Crunch Mode used as a long-term strategy is economically indefensible. Longer hours do not increase output except in the short term. Crunch does not make the product ship sooner — it makes the product ready later . Crunch does not make the product better — it makes the product worse.
I don't think I could put it any clearer than that, except to point out that this is only how crunch negatively impacts the product. For the other drawbacks, I certainly can't put it any better than those who already have. In short, crunch ruins lives, products, and sometimes companies, with no up-side.

How is this our fault?

So how is crunch our fault? I mean, developers are the victims, aren't they?

It's true that developers bear the brunt of the pain in crunch. And it might be comforting to imagine that someone "up the pay scale" is responsible for botching things enough to warrant crunch. It's that last phrase, though, that's the problem: "enough to warrant crunch."

How can a project ever warrant crunch? We've already shown that crunch has a purely negative impact except in the shortest of intervals. So how can we agree to do it? Whether for selfish reasons or love for our product, how can we say it's "ok" to begin working in such a way that damages the product, ourselves, and our coworkers?

Nothing warrants crunch. As we've just seen, industry has been wrestling with the pros and cons of crunch for a century now, and crunch has been shown to be ineffective. Crunch is not a rational option. And our guilt is enabling crunch to happen in the first place.

If you agree to crunch, you are incrementally making the whole industry worse for everyone, yourself included.

Take a moment to let that sink in. Everyone is worse off for your agreeing to crunch. You have to work more, and deal with mounting stress and diminishing rest. Your coworkers have to agree or appear unreasonable in the face of your compliance. The lives of those around you and your coworkers are affected. And your company is forced to absorb the drop in product quality, along with a workforce that's losing competence, morale, and loyalty by the day.

Crunch isn't helping anyone. And by agreeing to it, you're making it easier for crunch to exist.

What if your manager asks you to crunch?

In short, tell them "no." Be polite about it, of course, but don't let politeness give way to bad business.

Look at it this way. By asking you to crunch, your manager has simultaneously just done two things:
  1. Indicated that they have failed to manage the project appropriately thus far.
  2. Indicated that they have no clue how to continue going forward.
Now, if you like this manager, it might be a good time to pull them aside and direct them to this post, or Evan's piece, or any of the research sources Evan used. If they didn't know, fair enough. Now they do.

If they still insist on crunch after learning about its negative effects, you might want to say "I'll think about it" and speak to their manager. Again, arming yourself with polite concern, and some established research on the topic, hopefully you can impress your point on someone with more experience and reverence for historical data.

If you find yourself talking to more than 2-3 managers in this way without progress (or run out of managers), you might want to consider another option: finding a new employer. Because any company willing to impose crunch on their employees at that many levels of management has a deep-seated philosophical problem, and it may take more than common sense to fix it.

What if your manager doesn't ask you directly?

What if you're in a place where crunch is the elephant in the room? I.e. nobody explicitly asks you to crunch, but it's implied that overtime is expected? Perhaps that's just "the way it's always been," and sticking to normal working hours seems like an invitation to be fired.

Again, this is a topic to be broached with managers. Give them the benefit of the doubt. Bring it up, and see where things go. As above, if you can't make any progress after 2-3 rungs on the company ladder, that may be a sign you're in a place that's unhealthy.

What if they fire me?

What if they do?

Ask yourself that, seriously. If the company you work for sees your concern about unhealthy work-life balance as cause for termination, I think you can safely write them off as an employer you want a relationship with any longer.

You may have heard (or even been told at some creepier companies) that there are 100 others waiting to take your job if you leave, so it's no skin off the company's teeth if you go. That's not true, though.

Good companies recognize the true cost of finding and training good workers. It takes time and resources to find a good worker, and more to train them in your company's workflow. If your company doesn't recognize this, it's probably not a good company. They'd be doing you a favor by firing you.

What if I'm compensated for crunch?

In light of the aforementioned effects of crunch, this should be irrelevant. The focus should be on removing crunch entirely, not putting a band-aid on it.

If employees were compensated for crunch, we might see a change in crunch habits in the industry, but that's just conjecture. As it is, employees are rarely compensated fairly for their crunch, either in dollars or days off.

California may be the one place where this is addressed by law, promising anyone under $83k annual salary ($39.90/hr) time-and-a-half pay for any and all overtime (Labor Code Section 515.5(a)(3)). They also define several exceptions and exemptions, which mandate overtime for entry-level or in-training software fields, film and television artists, and writers. How this impacts software development in California remains to be seen, though.

However, whether you're compensated for extra work or not doesn't address the bigger issue of whether your management knows what they're doing or not.

The bottom line

The point to remember here is that a century of industrial research and work hours testing has resulted in the nominal working hours of 8 per day, and 40-44 per week. This combination produces the highest sustainable productivity possible.

In short bursts, overtime can produce a slight boost in productivity. But the boost wanes after only a few days, and actually cumulatively grows worse each day after that.

If your employer is suggesting (or fostering) an environment where overtime is sustained longer than a few days, start considering whether they're causing more harm than good. See if you can open a dialogue with managers or HR to address the problem in other ways. And be prepared to say "no."

If you like your employer, reward them accordingly. Give them diligence, timeliness, hard work, honesty, creativity, accountability, loyalty, and all the other things you're paid for. In exchange, expect them to provide a healthy working environment, tools and support, intelligent planning, and monetary compensation for your efforts.

Tuesday, April 2, 2013

NEO Scavenger's Humble Beginnings

I was cleaning up some old files on my laptop this weekend, and I came across some of NEO Scavenger's earliest prototypes and documentation. And let me tell you, they were pretty eye-opening. Even for me. NEO Scavenger has come a long way.

Since this blog is meant to help newcomers to the indie world, I thought maybe I'd share some of what I found. Not only is it interesting to see how an idea evolves over time, it also might encourage those who are daunted by the finished products they play.

Few games, if any, spring forth whole like Athena from Zeus. More often, they're ugly, slimy, helpless little buggers, just like you and I were when we were born :)

A Little History

In April, 2011, I left my job at BioWare to try my hand at making games of my own design. NEO Scavenger was a foggy idea I had kicking around my brain, and was actually quite a different beast back then.

In fact, NEO Scavenger even had a different name: Post Apocalyptic Oregon Trail (PAOT). Here's a look at it's original design doc (pre-dating the design doc published in June 2011):


Title
Post Apocalyptic Oregon Trail (PAOT)

Category
Survival/Strategy

What Is The One Sentence Story Line?
Exiled from MegaCity, the player must survive the wild wastes as he travels west in search of paradise.

What Is The Game About?
Salvage food, shelter, and tools from the wastes to meet daily survival needs, as you travel west to paradise beyond the mountains.

Game Overview
  • Man against nature
  • Plot your course west, one day at a time.
  • Search for food in the wilds and wastes, to stave off hunger.
  • Find shelter from the elements to stay healthy.
  • Find and improvise tools to help in your journey.
  • Balance the provisions you carry against the fatigue of carrying them.
  • Manage inventory by arranging your belongings in pockets, compartments, and other containers like puzzle pieces.
  • Find transportation to help you travel faster, at the cost of fuel and maintenance.
Differentiation Within The Category
  • Focus more on survival through resourcefulness rather than gunplay
  • Opposition is mainly environmental rather than soldiers, monsters, etc.
  • Play style is more laid back and pensive than Resident Evil type survival games.
Franchise Strategy
  • Free Flash game, licensed to game hubs
  • Possible ad revenue through MochiAds
  • Story establishes IP for POST (Note: POST was a working title for a post-apocalyptic setting)
Similar Titles
  • Oregon Trail (I, II, Yukon, Amazon, etc.)
  • Survival Kids (1, 2)
  • Lost in Blue (1, 2, 3, Shipwrecked)
The game described is actually not too far off from NEO Scavenger, at least in terms of theme and gameplay. The focus on traveling west is different, as is the origin of the character (MegaCity exile). And the transportation bit is admittedly optimistic.

The "Franchise Strategy" is a completely different story, though. NEO Scavenger as a free, ad-supported game was based on the assumption that I would work on it for a few months, then launch. Little did I know that I'd be working on the same game for years (starkly against my own advice).

Also, revel in how little I knew about the survival genre. Granted, many survival titles have cropped up in the past two years. But I'm pretty sure I was missing several important titles in that market assessment.

NEO Scavenger's First Prototype

Perhaps a more telling example of my lack of understanding is NEO Scavenger's first prototype:

Press the "I" key to see the map in the prototype.

Impressive, huh? You can move colored blocks around a grid, and switch to a map view to watch our protagonist travel west from "Elis." There's some random weather variables, tracking of time, hunger, sleep, and distance, and the ability to choose between traveling or sleeping for the next 4 hours.

The next prototype added a fancy map to the background:




Not much more to do, though. I eventually added a more representative item to the inventory screen, and made each location contain two of those items. The player could take the "soup" at a given stop, and the character would automatically eat when hungry:




It even featured an end-game! Patient players would be rewarded with green text proclaiming "You Made It!"

It wasn't very fun, though.

There were some interesting systems under the hood: weather simulation; tracking hunger and sleep; and item management in a grid. But I found myself clicking the "travel" button thoughtlessly until I needed to recharge "sleep." There wasn't much to the gameplay loop. No strategy. No risk. No fun.

I started wondering whether it might be more fun to navigate a map rather than a line. What if the player had to choose where to go, and there was no guarantee food or encounters would await them in the direction they chose? What if moving around was more or less costly depending on the terrain? What if visibility was limited by terrain?

Also, I was hooked on Civ V at the time.

So I decided to try changing the game to utilize a hex map. And this is how that looked, initially:


Still not much to look at. In fact, almost all of the systems of the above prototypes were stripped away, here, and it was just turn-based movement on an hex map, with date tracking per-turn.

Visibility, though, was a big difference. Not being able to see the rest of the map, and having to choose which way to go, felt like an improvement over having a linear, prescribed path to follow.

And within a few days, I had this:


Now I was actually starting to enjoy the prototype. Exploring the map kept my interest, as I wanted to see how the world looked, and what landscapes I could reveal. Movement was more strategic, as I weighed visibility vs. movement costs.

There wasn't any tension nor risk yet, of course, but it was starting to feel like a "toy" rather than a "tech demo."

A week later, and there was day, night, and weather!

Day
Night
Rain
Still no risk vs. reward, but I was starting to get more inspired with each iteration.

It was at this point that friends were starting to ask more frequently what the point of the game was. And they were right to ask. I needed to take a step back, and come up with a plan.

I drafted a new design doc. I started prioritizing features and tasks against that design. And from that blog post onward, you can see the iterations NEO Scavenger passed through to get to where we are today.

Rainy nights with stuff to do, and things to fear!

Baby Steps

Getting back to the original point, it's important to remember that making a game is a series of baby steps. Your game might not start out very good-looking, or even fun. And as you can see from PAOT/NEO Scavenger's history above, it might even involve a false start or two.

However, determination is key. As an amateur (programmer/artist/designer), I had to keep pushing forward, despite my own shortcomings. I made a ton of mistakes, too. Some more than once.

Over time, though, the baby steps add up. The thing you're working on gets better each time you add something that's necessary, or take away something unnecessary. Your own progress becomes a thing that motivates you, rather than just hope and optimism.

And eventually, you enter the phase where you wonder when you should call your project finished. I'll let you know once I discover the secret to learning that lesson!

Monday, March 18, 2013

Sometimes, Running a Business Stinks

The past 72 hours been a pretty wild ride. And it's highlighted some of the less glamorous aspects of running an online business. I think things are starting to stabilize this afternoon, but I'm pretty exhausted from the experience.

Things Were Running Smoothly When...

Friday was a pretty normal day. Quiet, even. Most of the day was spent working on plot development for NEO Scavenger. The day didn't go without it's hitches, but as days go, it was nothing unusual. I was happy to finish the day with some plot knots to untie over the weekend, so I wrapped-up, had some dinner, and Rochelle and I went bowling with some friends.

The next morning, I was pretty excited to see a sudden spike in traffic in my logs. NEO Scavenger was mentioned in a post apocalyptic survival reddit post, and dozens of folks were popping by the site to check out the game. Cool!

Site Maintenance

Except, my ISP was doing scheduled maintenance early Saturday morning. It wasn't until I tried loading bluebottlegames.com that I realized it was down.

5-day Site Visitor Graph: Hourly

Well crap. That stinks. Just as a bunch of new folks are drawn to check out my game, the site goes dark. And worse, dark for close to 12 hours.

"Oh well, " I said. It's a bummer, but nobody's fault, really. I mean, I guess I could blame myself for not having a redundant server or something, but I'm not that high tech (or deep-pocketed). I decided to just live with it. Besides, friendly Reddit server take-downs were nothing new. Folks probably would just figure my little server was flooded temporarily.

Site Flakiness

Later that day, bluebottlegames.com was back up and running. I did my usual email and forum check, to verify nobody was reporting any issues. And all seemed clear.

Except for one thing: every other page load seemed to turn up blank. No error message, no content, nothing. Just a blank white screen. Refreshing the page seemed to fix it, so I figured it was a temporal thing. "It'll sort itself out."

No, it won't. The following day, I was replying to a customer having issues with NEO Scavenger on their Mac, and it was still happening. Happening everywhere. Sometimes it was a content page on the site. Other times it was a forum. Even some of the site admin pages were failing. And as before, usually a reload would fix it. But the reloading was becoming less and less reliable. Sometimes, I had to reload the page 4-5 times to see anything. And if my customers were seeing the same thing, then that was unacceptable.


The White Screen of Death

I decided to dig into the issue a bit more. I started searching for related issues on the web, and was initially happy to see others reporting the same issue. Blank screens in Drupal (the content management system I use, v6.28) were pretty common. Maybe finding a solution will be easy?

However, upon further investigation, I was less happy to discover I had the same problem. This problem, as it was known to the Drupal community, was the "White Screen of Death."

The WSOD is a common issue with Drupal, but it doesn't have a common solution. In fact, there are almost as many causes of the WSOD as there are Drupal installations, and finding the right one for you can be a real quest.

The link above is probably the biggest authority on the issue, and even there, there are no less than 28 different causes listed in the article, and some 80+ comments detailing other issues customers have had. Basically, Drupal's WSOD is a symptom of practically every Drupal disease. I'm having trouble thinking of a human analogue. Headache? Fever? Common cold?

I spent hours on Sunday trying to make rhyme or reason of it. None of the remedies I saw seemed to help. I couldn't even get error or log messages. And worse, it was an intermittent problem, so I couldn't even reliably cause it to happen.

The only things I could verify about it were:

  1. I only get the WSOD on my live server. Migrating the db from live to my localhost didn't duplicate the WSOD issues.
  2. I only started getting the WSOD since my ISP's scheduled maintenance, when the server was (apparently) upgraded/restarted.
  3. I only get a WSOD in Chrome.
  4. Additionally, Chrome seemed to exhibit stylesheet issues when the page did load. Textareas would be too narrow to fit their <div>, or the page would nudge upward when I clicked a link (seemingly to adjust alignment with the Admin Menu module's topnav bar). Something was stalling the css until I clicked a link, upon which it would fix itself, then load a new page with stalled css (or js, or WSOD).
  5. Firefox never exhibits any issues.
  6. IE seems to work too, except for one page partially loading (later determined to be a known issue with YouTube embedding)
  7. No errors appeared on the page, in Drupal's logs, nor server logs, even with error reporting hard-coded to be on in index.php.
  8. Using Chrome's "Inspect Element" context menu option revealed that the page was entirely missing the <body> tag and contents. It was just an <html> and <head>, and the <head> seemed to be missing some elements. Also, Chrome usually complained of "Failed to load resource" on the page itself, but all css/js/images were loaded ok.
  9. Using "View Source" on Chrome apparently reloaded the page, and showed the correct, full content source.
  10. The WSOD appeared more frequently when js and css optimization was turned on (but still occurred when all Drupal optimization/compression/caching was disabled).
  11. Flushing all caches had no effect.
  12. Running update.php had no effect.
  13. Manually truncating Drupal db tables had no effect.
  14. Using Backup & Restore on Drupal's db had no effect.
  15. Disabling all modules outside of Core and Core Optional had no effect.
  16. Switching from BBG's custom theme to the Garland theme had no effect.
  17. Rebuilding permissions had no effect.
By the time Sunday evening rolled around, bluebottlegames.com was stripped down to core modules, no theme, no caching, and had a rebuilt database. And it was still flaking out.

Worse, the node access modules I disabled caused the permissions table to get out-of-date, which caused all site content to disappear for all users, all the time. Basically, when the WSOD wasn't happening, all of the site's pages were empty blue shells, and the forums all had 0 posts in them.

Even worse still, to rebuild the permissions and fix the empty content, I had to run a script via Drupal. And that script failed with a WSOD whenever I attempted it.

I had totally messed up my site. The "Site Crash" label in the above image refers to the time when I took the site offline to avoid any more users seeing the empty shell of a site.

Fortunately, Firefox was able to run the permission rebuild script without any WSOD. And I was able to at least get the site showing content again.

But as my efforts continued past 10pm, I decided it wasn't going to be solved anytime soon. I started making changes necessary to return the site to the formerly flaky intermittent WSOD. It wasn't ideal, but the occasional user reload was a far cry better than no site at all. If nothing else, I wanted the forums and contact page online for users to report issues, if needed.

I posted a news item to the homepage alerting customers to the WSOD issue, and apologized for the inconvenience of the downtime. Then, I went to bed.

Come Monday, It'll Be Alright

I was back at the computer at 6:15am. And unsurprisingly, the site hadn't fixed itself. I fired up the usual websites, checked messages, looked for forum posts. Some users reported seeing similar WSOD issues. And, bless them, they blamed their internet connection instead of me.

I decided to try a different approach this time. Instead of grasping at straws offered by forums on the net, I decided to debug Drupal. I added print statements to Drupal's index.php, to see if I could trace the value of the content. And when that failed to reveal anything, I started adding traces to Drupal's core code (*.inc files).

I don't like doing this sort of thing, as I'm nervous about screwing things up worse than they are. Plus, doing it in a way that doesn't affect the live site's users is tricky. But in retrospect, it's the only way to really know what's going on.

I found a function in bootstrap.inc which loads various Drupal bits in phases: drupal_bootstrap($phase). Every page on a Drupal site calls this function first, doing a full bootstrap (all phases). I added a trace inside the while loop that executes for each phase as it loads, and I printed the ID of the phase.Testing on my local site, I could see that my site loads phases 0-8.

When I uploaded the modified bootstrap.inc file to the live server, I saw traces for all phases. Reload. All phases again. Repeat this another dozen times, on different pages where I most often encountered the WSOD. Everything loads normally.

Was the WSOD gone?

Tentatively, I backed out the bootstrap.inc changes, so it was back to normal. Still, the site seemed to be loading ok. I reenabled caching to normal. Still ok. Turned on page compression. Still ok. I stopped short of turning on optimized js and css. Maybe I'll muster the courage for that tomorrow.

A Wizard Did It

So what happened? Uploading that file seemed to stop the WSOD, and leave it fixed even after that file is restored. What dark magic is this?

That's actually my first theory: dark magic. But if you pin me down for a more mundane answer, I'm going to guess it was some sort of behind-the-scenes cache. I'm not sure what else would explain a site's complete performance alteration when a single file is uploaded and then un-uploaded again.

That was at about 11am today. As of 5pm, I haven't seen a WSOD. Mercifully, no players have posted in the White Screen tech support thread on my forums, either. I'm hopeful this issue is resolved.

But What About That Mac User?

Oh yeah, remember me mentioning way up there that this whole investigation started when trying to help a Mac user with NEO Scavenger? Yeah, Mac compatibility is an issue in it's own right, concurrent with the site debacle.

I'm not going to detail that issue here, as it's a pretty lengthy topic of it's own. The forum thread linked above has all the details. And what's more, I've partially touched on it in the past.

The short version is this: Flash is rapidly becoming as much a burden as a boon. For someone trying to develop a stand-alone application, I'm at the point where I am highly reluctant to recommend Flash as a viable option. Issues include:

  1. "Create projector" no longer supported on Linux, as of v11.2.
  2. Flash CS6 no longer supported on OSX 10.8.
  3. Any projectors one does create are going to trip security on modern OSes. And in Mac's case, Gatekeeper is a sticky issue for OSX 10.7.3 and 10.7.4.
  4. Digitally signing Flash projectors appears to have spotty support, unless one uses 3rd party wrappers (which are, in themselves, reportedly unreliable).
  5. Adobe's recommended solution, building AIR apps, is unsupported on Linux. Also, AIR's installation process is flawed, at best. Also, AIR has the periodic "Update AIR" nagging dialog.
  6. Flash is no longer officially supported on Android nor iOS.
I was a long adherent to Flash. It served me well. NEO Scavenger wouldn't be if it weren't for Flash's ease of use, and then-universal deployment options.

So it's unfortunate that this era appears to be in it's winter.

All's Well That Ends Well

The good news? At least we're back to normal. The site seems to be running again. Upgrading OSX to 10.7.5 seems to fix Flash projector Gatekeeper woes until I can find another way to certify projectors. I think I may actually be able to return to plot development tomorrow.

Let's just hope that stretch of actual game dev continues for a while!

Monday, March 4, 2013

Cross-Play

Recently, I was contacted by a couple of digital games vendors (ShinyLoot and Amazon Digital Games) about the possibility of selling NEO Scavenger on their services.

First of all, that's great news! It's encouraging to have folks asking to host NEO Scavenger. And those services each represent good-but-different opportunities to serve a greater market. I look forward to exploring those, and other, options further as NEO Scavenger approaches completion.

The point of this post, however, is more about the big picture, and how to properly balance NEO Scavenger's sale across multiple vendors. Specifically, I want to muse on the topic of "cross play."

What Is Cross-Play?

When I say "cross-play," what I'm referring to is the ability to buy a game from one service, and to have that unlock access on other services. I realize there are a few definitions of "cross-play" out there, but for the purposes of this post, it's about buying the game once, and playing anywhere.

In the case of NEO Scavenger, buying beta access at bluebottlegames.com automatically grants access on Desura. Similarly, buying a copy of the game on Desura allows the user to unlock a copy at bluebottlegames.com, using the Desura Connect feature.

When NEO Scavenger was sold in the Be Mine 5 bundle, Desura keys were given out, so those customers were also eligible for cross-play. Though in retrospect, that was less than fully effective: many customers didn't realize they had Desura keys, and weren't aware how to update, or even that they could. I should probably see about doing keys-only, rather than binaries plus keys, in the future to reduce confusion.

Why Provide Cross-Play?

One of the biggest reasons to provide cross-play is simply because it makes the customer happy. Customers feel like they're part of something bigger, and dealing with a business that values them more than their dollar. It let's them know that no matter which store they bought your product in, they're your customer, and you appreciate them just the same. And customers who feel appreciated are more likely to return for future business.

It also helps keep the audience fragmentation to a minimum. If communities were to develop at each storefront, then those communities may diverge over time, making it difficult to satisfy all tastes as they become incompatible. Smoothing out inroads between communities helps everyone stay on the same page, and helps keep the audience united.

There are also marketing advantages to providing cross-play. Customers who are pleased with the arrangement are more likely to recommend it to friends. And the relative rarity of cross-play may mean the press finds it a newsworthy business practice.

It can also accelerate sales, in the case of a gradual product roll-out (as is the case with NEO Scavenger). Knowing that one's preferred service will be supported in the future sometimes causes hesitation in customers, as they'd prefer to wait until the game comes out there. However, if purchasing the game now entitles them to a free unlock on their service of choice later, then there's no reason to wait. This was particularly the case with Steam users, as noted by the comments in NEO Scavenger's Greenlight page.

Lastly, there's a neat technical benefit to having cross-play enabled for one's game: redundancy. If something fails on one service, there's a way to get affected customers up and running again on a different cross-play service.

A recent example of this was when a new build was launched on Friday to both bluebottlegames.com and Desura. The new version went live immediately on bluebottlegames.com, but the Desura version was awaiting authorization over the weekend. Customers who wanted access right away simply used the connect feature to get the latest version at bluebottlegames.com instead.

Why Not?

One of the first reasons most will think of against providing cross-play is that there's a potential hit to sales. This is probably true. Anecdotal evidence abounds for customers buying and re-buying the same game across different services as they become available. Some are accidental (e.g. losing a CD or key), others are intentional (e.g. wanting the game in their Steam library for convenience). In some cases, they're even points of pride for the customer ("I love this game so much I bought it five times!").

In the first two cases, the customer is repurchasing the game for their own benefit. In the third case, it's primarily benefiting the developer. With cross-play, the customer can still repurchase the game if they want to, so the third situation is unchanged.

The important thing to remember about the first two cases is that customers who don't want to repurchase the game no longer have to. It might cost the developer the price of a new (or discounted) copy, but it gains the developer some good will. And I'd rather have an existing customer's good will than their additional $10 or less.

There is the possibility of customers giving cross-play keys to friends, so I suppose that's a consideration. I'm not really one to police pirates, though. And I figure that if they're going to pirate the game, cross-play doesn't greatly change their ability over the DRM-free version I sell now.


Technical feasibility may be a bigger reason. So far, cross-play between Desura and bluebottlegames.com has been pretty easy to implement. Desura has a workflow already setup for this, and adding a custom one to my site was entirely up to me. I want to do the same with Steam, though I may only be able to provide one-way unlocks (e.g. bluebottlegames.com customers get Steam keys, but not the other way around) depending on whether Steam has any mechanisms in place for cross-play.

Furthermore, each new case creates an increasing amount of integration work, and it would likely grow out of hand pretty quickly. There are a lot of services out there. E.g. ShinyLoot, Amazon, Good Old Games, Impulse, Origin, Google, iTunes, Ubuntu Software Center, etc. They all likely have different capabilities.

As such, it may be the case that my cross-play plans are impossible across 100% of vendors, even if I had the manpower.


Selective Cross-Play

In all likelihood, 100% reciprocal cross-play will not be possible as I offer NEO Scavenger in more places. So I'll need to come up with an acceptable compromise. The alternative would be to ignore sales channels that didn't offer cross-play tools.

I already suspect Steam is a one-way street in this regard, and cutting out Steam from sales channels would be pretty dumb. In fact, I think most online storefronts lack Desura's "connect" feature, so I may be living in a fantasy land right now.

One thing I can still do, however, is enable cross-play from bluebottlegames.com some other services. I.e. if they purchase at my site, they can get free keys to Steam, Desura, and maybe another service to two. More than likely, Steam and Desura are all most players care about anyway.

Plus, offering the one-way cross-play radiating out from bluebottlegames.com makes it somewhat more attractive to buy it there. The customer could still get a version on the service they want, plus elsewhere in the network. And more of the purchase money goes into developing new games. That seems like a nice win-win.

I'll have to see how this plays out, though. So far, I've been pretty lucky in that my only other vendor, Desura, makes this cross-play easy to do. So I've been treating it as my preferred business practice. But as I start working with new vendors, I'm see that the cross-play option is going to be complex to support.

Hopefully, I can find a way that is both sustainable and appreciated by my customers!