Versus Software   game + free time = duck

We’ll it took a while, but I managed to finally finish reading Michael Abrash’s Graphics Programming Black Book:

In some ways I feel ashamed to say that I ‘read’ it because it’s a behemoth of knowledge spanning over 1200 pages. Of course I learned a lot from it but it’s the sort of book I’ll read again and again and again until I can finally grok it. It’s a collection of over 10 years of Abrash’s papers and I doubt one can absorve it in a matter of months.

You see to learn programming concepts in a self taught manner I think it’s crucial to not only read the code in the book, but also write it down, play around with it, to truly understand what is being taught in its finer details. With my current project I intend to do just that, since it’s a FPS and the last chapters on the book concern directly with Quakes development at id Software, where Abrash worked.

What’s more interesting is that the author doesn’t focus only in the development aspect of programming, but in the general mentality of it. Not as one solves a problem, but the mind that solves it. As you become better in development I think you ask yourself less “how to solve this problem” but more of “what is the best way to solve this problem”. Abrash shows us several ways to solve a problem in the book, be it linked lists, spatial visibility or making a faster game of life, each one consistently faster than the other with either assembly optmizations, algorithm optmizations or rethinking the whole approach to the problem. The idea is to not expect that there is only one way to handle an issue. In his own words : “Assume nothing”.

The book is also quite pleasant to read, since the author narrates the development cycle more as a journal than a tech book. It’s quite interesting to read the last chapters where he focus on making a faster rendering back-to-front polygon rendering approach to Quake. Almost goes like this:

March 14, 1941. We begin our approach to the BSP tree, were still having heavy losses on how to figure out a way to make the spatial visibility problem faster. The worlds we want with Quake  feature at least 5000 polygons and in the worse case scenario we redraw each pixel 5 times. It’s too slow, we must take a better approach.

May 22, 1941. We sucessfully managed to create a potentially visible set (pvs) that managed to break into enemy lines. We will now proceed to use it to flank their defeneses.

June 10, 1941. We have now conquered the enemy’s battlefield. I’ve reduced the inner loop of the rasterizer to 2.5 cycles per texel. We decided to use z-buffering for drawing the enemy meshes, since it’s faster and not that big of a problem as we expected. Victory is eminent.

And so on. Overall the book can be divided into 3 parts:

• General assembly optimization techniques
• 3D rendering done via software
• Common 3D engine development problems and solutions.

I recommend it to anyone that’s interested in taking game development or programming in a seriously yet elegant manner. I learned a lot from it, and still intend to learn more.

While riding the subway today I saw a man my age holding a book titled “Agile Software Development, Principles, Patterns and Practices“. This reminded me of a few opinions I had about software enginnering. I talked on a previous post that it’s important to take into consideratoin the final product that’s going to be developed in order to choose a proper software Engineering methodology.

Today I’d like to keep talking about that but focusing more on the technological nature of that which is being developed. Here’s a short story to better explain why:

While I was working on Conira, early on I decided just for the hell of it that this game would be multithreaded. Given my previous experience developing the Solis engine I saw that you could dissect the game engine into three threads.

1. One that renders all the objects in the game, be it sprites, particles or models. This thread uses the ‘get’ methods each object had.
2. Another that updates all the objects in the game, their displacement, acceleration, hit detection, etc. This thread uses the ‘set’  and ‘get’ methods each object had.
3. One thread that takes care of loading/unloading objects and handling the Operating System calls like windows and messages.

Initially I thought this would be a good design. I elaborated on how the threads would communicate, tried to figure out and reduce potential deadlock /livelocks and set off to code!

It wasn’t but one day later that I had to redo the whole design.

You see there is a technical limitation when dealing with rendering systems, threads and the Win32 specification. MSDN clearly states that “the same thread that RENDERS the objects MUST be the same thread that CREATES the WINDOW and handles its OPERATING SYSTEM calls.” In other words, thread 1 and 2 had to be same. In the end it wasn’t a big alteration, meerely trivial given the design I had but it did bring me into re-thinking certain software engeneering practices people take for granted.

When I was in college I was basically told that “code development is a final product of a series of design decisions”. You figure out your system first, its classes, structures, layers and then the code you write is merely an extension of that. In other words, take the broad view of a project and as you develop it, handle the inner details and its intricancies as they come. But that type of methodology fails to work in development environments where those same inner details and intricancies are what actually defines the system as whole.

I think Agile Software advocates say the same thing but with different words. “Work on the code first, let the inner details of a system be delt as they come, so don’t worry about documentation, design, etc. Code, code and code!”. They also usually point cases where such methodologies worked. However they fail to point out the architecture of the system in which they developed. Most, if not all, of those success stories I heard were either web applications or with an interpreted language such as Java or C#.

Well it makes sense for those projects to work on those scenarios! They were built on top of languages where it is expected to delegate the inner details of a system to whomever is handling it. So all they have proven is that those agile methods work on those environments, which is ok, but not that agile methods work as a whole, which is what they usually advocate. Honestly, I’m yet to see an long run story of a sucessfull low level driver or high optmized high-level assembly language project using Agile methodologies.

Don’t get me wrong, I’m not saying that Agile methods suck or that they don’t work. They do and can work quite well but one must understand the context in which they work, or else failing into the trap of beliving that they work in any environment.

If one doesn’t take into consideration the end product as well as it’s internal architecture while taking enginnering decisions, he’ll end up eventually just making bad or lucky engineering.

Today I’ll review another book I read, Software Engineering for Game Developers.

As a book that concerns itself with software engineering I say it does the job. It’s an 800 page beast covering topics from UML, resource management, project risks, stipulations and damage control. More interestingly enough it tries to tie that with actual game development. It goes quite in-depth in each of it’s topics and gives plenty of further references if any particular one interests you. Overall it’s a good book.

But that’s not really what I want to discuss about.

You see, the book comes with a game a small team of developers made using a variety of software engineering techniques. This serves to show how one can apply them into game development. The game is quite stable, performing well given the myriad of things attached to it (3d mesh loading, textures, events, GUI, scripts, etc). However there”s just one problem with it:

It’s not really any fun.

Now I’m sure the goal of the book  is to teach software developers techniques they can use in their own development, with the game being fun not a pre-requisite in this scenario. It had to be functional, not fun.

That’s one of the things game development as a software differs from others. If a program manager assumes that a project will take:

• w hours of development
• take y developers to develop it
• cost z dollars
• have k use cases

And it ends up taking exactly that, most likely that was a successful project.

The problem with game development is that even if you manage to make a game with all your estimates correct, if the game in the end is not fun than you still have an unsuccessful project. Sure it can still sell well, but it’ll probably take tons of marketing to make up for it.

Some could say that fun is a non-functional requirement, which is true. However it undermines it’s importance into the actual game development process. Sucesfull game companies have long realized this and build entire systems of software enginnering whose sole goal is to enhance and facilitate adding ‘fun’ to a game.

Valve software uses their CABAL system, ID software with their endless internal engine prototypes, Blizzard with their QA tests, and so on.

Which leads me to conclude:

Sometimes the most important aspect of a project cannot be expressed in a process or in a form. That is still not a justification to leave it out off processes and forms.

Today I’ll review two books I read:  Professional Assembly and Assembly Language Step-By-Step.

One interesting thing is that both approach the same topic, teaching assembly language, from different perspectives. Step-by-step takes care of explaining the history of Intel’s CPU architecture, from the segmented mode to flat mode, detailing the intricacies of segmented programming along the way. It takes over 150 pages just to get to the first line of assembly code in the book.

Professional assembly on the other hand is a rocket ship, taking no apologies and going full throttle into Intel’s assembly structures and the GNU’s assembler (gas) syntax.

Step-by-step takes care of explaining basic computer architecture so that in the end you can understand assembly and it’s logic. Professional assumes you know that and blasts off, which I think in the end makes it a better book.

You see I’ll probably go back to Professional Assembly when I need a reference or review a topic, because Step-by-step while good at explaining things, once you got them there ain’t much left to go back to. Now you may ask me, which one should you buy? My answer is:

Both.

You see, one thing that I learned is that it’s good to study a particular topic several times, even if you are already familiar with it. An authors approach, no matter how good it is, will not be universal. That’s because it’s his approach to the topic, there are other things that can either be better explained or better elaborated upon.

And the more you look and study at a particular topic, the more universal your approach to it will be.

I was having a chat with my good friend this afternoon. He was having some trouble getting the XML parser on the iPhone to work and I tried to help him out where I could. I gave him the tip to always check for the xml error parser code, when something goes wrong, even if most likely nothing will.

He said that he was doing this project solo, and that such verification wasn’t necessary. That’s when I remarked “Two months from today you will be another person working on this project.” He got the joke and we both laughed.

You see even when you are working alone, as projects evolve, change focus or simply take time, you will forget about different assumptions you had previously taken. You will also forget about previous remarks you had in your code, which functions should be called, by whom and so on.

A solo project is still being developed by a team of individuals. They’re just separated by the 4rth dimension.

I finish reading Write Great Code Vol2 yesterday. I must say it’s a good book, although with one complaint.

Let’s start with the good side. It’s very well written and throughly elaborate on explaining how your compiler turns the high-level language statements into low level assembly. It then goes on for hundreds of pages explaining how to optmize that, from function calls,  arrays, structures to small if/else jump conditions, local, static and global variables differences. Basically covers most of the issues a programmer will face or would ask himself “how will this turn into assembly”.

The only complaint I have is not in the book itself but the author over and over and over and over again telling you that if you want to look further into a particular topic that the book doesn’t cover (like line caches) he states that you should get his other book, Write Great Code vol1. I don’t really have anything against good advertisement, but it just gets tiresome after the 10th time he does that.

Still It’s a good read and I enjoyed it. My copy of WGCvol1 is already on the way from Amazon and i’m pretty sure it’ll be a good read as well.

Like I said on my previous post, I`ve been reading Write Great Code Vol2. One of the ideas behind reading it was gaining more assembly knoledge and being able to write and optmize better assembly code. That is happening, at a small pace but gradually, though something more interesting began to happen that I wasn`t really expecting.

I`m now am able to read my compilers assembly output and have a better understandment of whats going on behind the scene. Now this may not sound like a big thing, but take the following example:

I was coding a particle engine for a new game i`m working on. Now I wanted this game to be a particle fest, like Geometry Wars. You could describe the basic particle loop engine as:

1. update particle position
2. render particle
3. repeat for every particle

Steps 2 and 3 are a bit of self explanatory, but i was spending way to much time on step 1. Out of curiosity I started looking at the assembly code on that section and saw that the debug build was taking way more overhead than necessary.

You see each particle is an object, and I was accessing its position via setters and getters. That was slowing way to much the debug build because of all the function calls. On the release the compiler would optmize it away, but since I tested the debug version more than the release, it slowed down the development process as a whole.

So I made a decision that most software engineers wouldn`t recommend and made the particles position public instead of private, and accessed them directly. Alas the debug version speed up!

I thought to myself that when trying to write fast code, sometimes giving up good software practices might do the trick. I was a bit skeptic at first, but ironically I ended up reading the exact same statemnt on Write Great Code Vol2. The author even used the same example of setters/getters.

Regardless of if this is a good practice or not, I`m glad I`m able to have a better undestandment of my code, how it works and to know what kind of choices should I take as each situtation arises.

I`m defiantly going to pick up Write Great Code Vol1 after finishing this one.

I’ve always admired Michael Abrash. If you don’t know who he is the man is sort of a legend. He helped develop the original quake engine, wrote tons of articles on how to get the maximum speed out of your pc in the early 90s when cycle counting used to be something respected.

I’ve always had interest in learning from him and reading his articles so I picked up courage and started reading assembly books for the 8086. I picked up a real nice one, Assembly Language Step-by-Step 2nd edition, and started studying.

I could say I “learned” this in college, assembly for the 8086 but it was a basic course and taught mostly the beginning stuff you know. mov this, add that, inc this, int 21h that. Just because I could understand the instructions from an individual point of view for me it was not enough, I wanted to get to the level Abrash talked about.

So I finished that book and have been reading and re-reading it as I go through Zen of Code Optmization and Write Great Code vol2.

I must say it’s being a bumpy ride, going back and foward with these books. Reading a chapter from one, going back to the other, while trying to understand what I just read. No wonder it takes time to master this sort of stuff.

Personally to me also it’s very satisfying to be able to read these sorts of books and be able to at least understand them. Gives a great sense of improvement. When I finish all 3 of them let’s see how I am.

But right now I want to share with you a small victory I feel I just had. One basic memory handling function in C is

memset( dest, val, size ) ;

I went inside it in it’s assembly code and managed to understand it. The most important instruction inside this function is

rep stosd

which is what causes the memory to be set once all the registors have been setup. Inside theres a bunch of checks for redundancies and type safeties, so I wrote the following that does that a memset and only the memory settting. No type checking, register juggling, no nothing. This is what I got:

mov eax, 10
mov ecx, 16
lea edi, dword ptr a
rep stosd

Which is basically what the Assembly book step by step teaches in one of its chapters. In the end this ends being 2 cycles faster than memset.

2 cycles faster. I am proud of myself ahah.

yes a small victory, but hopefully one of many to come. Let’s see how this goes.