Micro Details

I was admiring our rifle model today and remembered a very neat trick Jeff figured out some time ago. If you look closely at the details on the rifle, you'll see it has been normal-mapped from geometry for larger details like screws and grooves. These macro-details were actually modeled into the high poly reference model, before they were baked into the normal-map for the low poly one.

The rifle also has micro-details on it's surface, such as wood grain and grit on the metal. These details are not modeled into the high resolution reference, but are instead tileable normal-maps composited with the macro-detail normal map. What Jeff figured out was how to combine these two normal-maps into one, allowing artists to add fine details to surfaces without having to model every single scratch and dent.

Here is an example of a barrel. The barrel was first modeled as a high poly reference model, then as a low poly in-game model, and a macro-detail normal-map was generated. Then the low poly barrel was textured with a tileable wood grain normal-map. The wood grain normal-map was unwrapped to match the texture coordinates of the macro-detail map, and finally combined in Jeff's tool.

macro-detail normal-map of a barrel baked from a high-resolution model

micro-detail normal-map, created from tileable normal-maps, and unwrapped to match the baked macro-detail normal-map


final, combined normal-map

The combining process is not as easy as "adding" the two normals together, but it's not too bad either. It involves using one normal-map as a reference surface and marching over it pixel by pixel, peturbing each normal some more. It's a very handy tool, and if someone wants to know more about the algorithm, I can post it. I also haven't heard anyone else mention this technique before, perhaps we invented something new?

Oh yes, they float, Georgie, they all float!

I'm just about finished with floating objects using Novodex so I figured I'd share what I came up with.

The buoyant force on an object in a fluid, according to Archimedes' Principle, is F = M * G, where M is the mass of liquid displaced by the object and G is the acceleration due to gravity. To simulate buoyancy then, we need to figure out how much water the floating object displaces at any given time. For this we take our model (of a marshmallow peep) and "voxelize" it, generating sample points throughout the peep.


Each point is assigned a fraction of the peep's total volume, and when under water, each point receives a fraction of the buoyant force. Gravity is constantly pulling the peep under water and the peep floats when the buoyant forces from the submerged sample points cancel out the gravitational force. The mass of the water displaced is found by taking the volume of dispacement times the density of water (1 gram per cubic centimeter). That times the negative gravity vector gives us the buoyant force at any given sample point.

Sounds simple and straight-forward, and it works great with enough sample points and a high framerate. Problems arise, however, when realtime physics simulation rears it's ghetto head. The biggest problem I had to battle was framerate. If the simulation framerate is too low, in one frame the peep can be given enough force to toss it straight out of the water. Once entirely out of the water in the next frame, the peep now has no submerged sample points, and gravity will pull it deep under water once again, and the cycle repeats. This causes the peep to jump up and down in the water, sometimes several feet into the air!

To account for this sort of crap (which shows up all over the place in physics simulation), Novodex uses a constant time-step. In a given frame, Novodex calls it's solver function a variable number of times. If a frame lasts 0.1 seconds and the time-step is set to 60 Hertz, the solver is run 6 times. This allows for more consistent, framerate independent results.

Again, sounds great in theory, but there are still issues with our water. Ideally, you would want to check for the number of submerged points and apply the appropriate buoyant forces at every time-step. The Novodex trigger callback will tell you all about the submerged points, but will not let you apply any forces between time-steps. Apparently there are some threading issues internal to Novodex that prevent you from changing the state of the physics simulation in any callback or between time-steps.

Someone on the Novodex forums promised a special way to create force fields in a future release which will hopefully resolve all these problems. Meanwhile you have to find a hacky stabilizing solution. The biggest fix which gets you 90% of the way there is fluid friction; there is lots of it in water. It's not even a hack really, I just forgot about it at first, and without it, things would indeed bounce in and out of water. I didn't even bother to figure out the frictional force based on water density and peep surface area, I just hacked it with the linear and angular damping constants in Novodex. It took so much tweaking to get things to stablize, I wasn't going to sweat physical accuracy. I based the amount of damping on the number of submerged points. I increased the damping logarithmically so the amount of damping climbed rapidly even with a few points in the water, this just seemed to give better results.

Even with fluid friction, there is a lot of room for the peep to jump up and down on the surface, especially with a low density at low framerates (~10 fps things got really bad). I tried several things: clamping the velocity of the peep when partially submerged works really well. But when the peep is released from deep under water, there is a noticeable sluggish jerk as it breaks the surface and it's movement gets clamped.

I ended up with the following hack: given the sample point position P, the point-velocity V at point P, length of the frame dT, some constant C>1, the buoyant force F, and some multiplier 0<m<1, if( P + V * dT * C is above water level ) F = F * M. This tries to predict if the current velocity will shoot the sample point out of the water within the next frame. If so, scale down the buoyant force applied at the point severely. I added the C and M constants in there so I could tweak the equation. I found C=2.5, M=0.1 to be good values.

And the results? Good enough for me. Things still wobble a bit more at lower framerates but it looks like the wind is picking up, no more trampolining peeps though! I'm still testing other ideas (I also want to take force into consideration in the hack equation) and I have to test more shapes than just the marshmallow peep, but this seems like a good enough solution for now.

HDR to LDR Reflection Hack

I was sitting and staring (in awe) at our flooded golf-course demo again and I noticed that even though we have fresnel falloff on the water surface, the relative brightness of the sun's reflection never changes no matter where I move to. So I thought about it some more and it turns out, assuming a flat reflection plane, an infinitely distant object will only ever be viewed under one incident eye angle. The fresnel term (the amount of light reflected off the water surface, determined by the angle of incidence) will never change for the sun's reflection, nor for any other texel in the environment map. The diagram illustrates that the eye can be placed anywhere. The reflection angle R is always equal to the angle of incidence, I, and R has to be the same for the reflected vector to sample the same texel in the environment.


This means the fresnel falloff effect can be preprocessed on the environment map and baked directly into the texture. The resulting environment would remain bright at the horizon and get darker and darker towards the top and bottom. Aside from cutting precious shader instructions, this technique also allows us to take our high dynamic range environment map, bake the fresnel into it, and save it as a low dynamic range image without loss of quality! Because the benefits of HDR images are very noticeable when the images are modulated, fresnel falloff on the sky reflection really stands out in HDR. At greater angles of incidence the sky gets relatively dim but the sun remains blown out. When fresnel faloff is baked into the environment texture, there is no longer a need for any modulation, just a straight-up texture lookup will do. This means LDR will be as good as HDR.

Now, there are some minor drawbacks. When you give up true HDR you give up hopes of doing post processing effects like exposure, etc. It's also going to be difficult integrating the fresnel-baked skybox with reflections of the rest of the world, which is why we are only going to implement this algorithm in the environment-reflection-only render path. Also, the algorithm makes the assumption that the reflecting surface is a flat plane. It does not hold true for normal-mapped water because different surface normals combined with different eye positions can lead to the same texel being looked up.


However in practice (I'm hoping as we haven't actually tried this) the difference should be negligible. The overall effect is still there. Objects close to the horizon reflect more than those directly above the viewer and localized bumps in the normal-map will also appear to have fresnel falloff, it just won't be as accurate as it could be.

In other HDR news, the Lost Coast Half Life 2 level finally came out. It's a small technology demo for HDR support in the Source engine, complete with commentary from artists and programmers at Valve. The commentary is pretty cool, though they make it seem as if they invented ... any of the HDR stuff in the demo. The visual quality is quite impressive though, the weapons and water look a lot better. Even their use of bloom, also known as the chlorine-in-the-eyes effect, is relatively tasteful. One artifact that became very apparent was the transition between reflection maps on the guns. It just snaps from one cubemap to the next and can be quite distracting at times. We had talked about this too, as we will have to transition between both diffuse and specular irradiance maps when walking into, say, a densely wooded area.

I also liked their use of dynamic tone mapping (retinal response, dynamic exposure, whatever you want to call it). It was more subtle than other demos I've seen and did not look like someone was dimming the lights randomly. It was still a little touchy. There were times where it looked like the sun was being periodically blocked by clouds rather than exposure adjustment but it was unobtrusive. I would have liked to see it change through a wider exposure range and take longer to adjust though. It was also interesting how they weighed the brightness of pixels towards the center more heavily than those around the outside of the screen. I wonder if the exposure fluctuated too randomly otherwise. Anyway, props to Valve for talking about technical and game design decisions openly, and in a neat format too.

I've soiled my demo, how embarrasing

I added Jeff's water to our terrain demo today, looks awesome. We have an animated normal map for the reflection and refraction, fresnel falloff of reflection, and underwater murk. The murk is pretty cool, it's what Half Life 2 did for their water. It's kind of like fog with a clipping plane for the water surface. The more water you are looking through, the murkier objects under it seem. Doesn't sound like much but the effect is great. It makes for very nice shore lines that gradually fade to dark water.

We are having performance problems with nVidia cards specifically though. OpenGL lets you define up to six arbitrary clip planes to limit rasterization. Apparently clip planes are not widely used however, so certain vendors (*ahem*) cut corners in their implementation.

GeForce boards have never had true support for clip planes, instead they used to be emulated through some texture units (somehow) and now it appears a part of the vertex program pipeline is used. Using both vertex programs and clip planes on even the newest GeForce boards causes a massive performance hit. We think something is falling back to software.

We're also rendering the scene to a floating-point frame buffer, so our reflections are HDR and it really shows. I am however going to rip on graphics drivers some more: stencil buffers don't work with frame buffer objects! Everyone on opengl.org is complaining about it too, so I know I'm not alone in thinking it bites.

We also have the ability to throw normal mapped, circular ripples on top of the water surface for bullet effects, floating objects, etc. I don't have it integrated with the demo so no screenshots yet, but it looks pretty cool in action.

For more screenshots of the most advanced golf course rendering system in existence, check out this.

Oh yeah and buy this book, I wrote the HDR lighting chapter in it.

DIM Shadows

Jeff implemented my idea of using two Diffuse Irradiance Maps (DIM) on objects, one for lit areas and one for shadows, and it totally owns!

First a bit about diffuse irradiance maps: A DIM is a type of environment map that stores precomputed diffuse lighting information from a reflection map (also referred to as a Light Probe in this case). Every pixel in the light probe is considered to be a separate, directional light source pointing to the center. This allows for arbitrarily complex lighting conditions to be defined by, say, a panoramic photograph such as the ones Paul Debevec makes. However every point on the surface of a model would need to consider every pixel in the light probe as a light source, or at least every pixel within the hemisphere about the normal of the surface, which is not feasible in real-time by any means (at least not by brute force). Instead a DIM is calculated through a process of Diffuse Convolution. For any normal N, lighting contributions from all pixels in the light probe (really just the ones in the hemisphere around N) are summed up and stored in a new environment map called a DIM. This way any point on the surface of a model can look up the sum of all the light sources from the DIM on the fly using it's normal.

So Josh, Mark, and I went out into a prairie, took a fairly high-resolution HDR panorama and generated two DIMs from it using Debevec's HDRShop.

Light probe of a prairie


Sunlight DIM of prairie


Shadow DIM of a prairie

The light probe had to be scaled down quite a bit because diffuse convolution is O(N^2). HDRShop will tell you just how ridiculously long a full-res convolution will take before it starts (and ten thousand seconds is a lot longer than it sounds). Our 12.6 megapixel images would get done in 229 million seconds or just under 7.25 years. We ended up generating 360 by 180 pixel DIMs which ended up taking about half an hour to compute.

The sunlight DIM was computed from the original light probe normally. The shadow DIM was computed from a light probe where the sun was edited out in Photoshop. This simulates an object blocking out all the light eminating from the pixels that define the sun in the reflection map and only leaves the lighting the object would receive from ambient sources like the sky. Finally in the program itself, we compute shadows with shadow volumes or shadow mapping based on a rough estimate of the sun direction in the light probe. Fragments that end up in shadow get drawn with the shadow DIM, fragments in the light with the sunlight DIM.

The results are amazing. Here are some screenshots of Jeff's SpeedTree shadow-mapping demo:

Tool Development

glitch-art: "A Snail in Peril"

We had a heated discussion this morning about level editing and art tools. The question was do we create our own level editing tools from scratch or write Maya plugins to do everything from material management to entity placement and scripting within the Maya interface and export it all from there. It's actually not a decision to be taken lightly.

Of course being programmers, our gut response was "Maya sucks, lets write everything from scratch", but what a daunting task when you think about it! Not only does Maya have a huge set of standardized modeling tools they've worked on for years, it also features all those little things you tend to take for granted in an application. Things like copy/paste, undo, marquee selection, the list goes on. It's all necessary for a pleasing working environment. I remember we had a level editor for a school project called Metropolis that could save but not load files; let that one sink in for a moment. We ended up having a machine constantly running with the map editor open because you couldn't save your progress and load it up later. Once you started you couldn't stop! Incomplete tools are extremely frustrating to work with and will drive your artists up the wall when deadlines loom.

On the other hand, working with an API for a 500-pound gorilla like Maya sucks hard. It sucks because Maya is not a simple tool. You get dropped into this massive code-base that has been in development for over a decade and you start sifting through it with a flashlight. You may spend days just reading documentation in hopes of stumbling on a way to get at some crucial piece of information like material IDs. Often you'll find N different ways to almost do what you need. Plugin writing is an extremely slow and daunting development process and the API will fight you every step of the way. And the result is usually a crude and inflexible hack to get the application to behave.

Your hell will be doubled if you choose to use a scripting language like MEL or MaxScript by the way. They are usually handicapped compared to C++ APIs and are way too slow for complex algorithms. Scripting APIs are rarely easier to learn, they're just different APIs to figure out while learning a new language. In my opinion scripting is rarely the answer to anything. I'm a C++ programmer and I'm very proficient in it, switching to Python isn't going to magically help me come up with algorithms faster, it's mostly just going to get in the way.

We concluded that the "right" thing to do would be to utilize Maya solely because it is a professional grade world creation tool after all. However just because it's the "right" thing doesn't mean it won't be a waste of time. In the end it's a balance of developer headache versus artist headache. We decided we are going to do as much as we can in Maya, but entity placement, scripting, and final touches to the culling portals are going to be in a separate application. We still get to wrestle Maya over some of the basics, though luckily Jeff has figured most of mesh exporting out already, and we still have to implement a world editing tool from scratch. We just lack the resources to battle Maya for the next 6 months just to integrate our character scripting and material system into it. And the biggest problem we are going to face: our Maya plugin is likely not going to be able to load the data our custom app saves out. That means everything has to be perfected in Maya before things are changed in the custom app or the custom app changes have to be discarded as we reload the exported Maya data. Now we are working on some schemes to make this less of a problem.

In conclusion, we need a Maya plugin intern. Great industry experience! The hourly rate is 3 cents, a piece of string, and a Flintstones vitamin. Please contact andres@reinot.com.

8Monkey Update

This is an update on what's going on at 8monkey Labs these days.

I've been working on a couple of things recently. I've finally got a texture loading class I like and support for TGA and PFM files. PFM (Portable Float Map) is our HDR format of choice from now on out. It consists of a weird but mostly harmless header in ASCII (ew) and a binary dump of floating-point data (complete description here http://netpbm.sourceforge.net/doc/pfm.html). Best of all, both HDRShop and Photoshop support it. I'm very glad to see Photoshop is on the ball with HDR. CS2 comes with HDR editing tools as well as exposure compositing, creating an HDR image from a set of LDR images at different exposures. I'm hoping Photoshop will eventually handle panoramic transformations too (e.g. mirror ball to vertical cross pixel transforms) so HDRShop can finally go away (no offense to Debevec but that program is no Photoshop and is just another link in an already cloogy art pipeline).

I'm also glad we found an alternative to the Radiance .hdr format. Radiance is stored as RGBE, the E being for the exponent on the red, green, and blue values. This limits the relative difference between color components, so you can't store an image extremely bright in red but dim in blue and green. Only problem is PFMs are friggin' huge, one 1024x1024x6 cubemap is 72 megs!

So far our approach to data size has been to fix the problem when it becomes one. The artists have been taking ridiculously large HDR pictures (with our badass 11 gigapixel camera) for texture creation, and eventually we'll downsample the high-res images to run in real-time. This will allow us to release the game with high-quality content suitable for the time. We can even re-release the game with higher quality content years down the road (Half-Life: Source comes to mind). We're going to try to take the same approach to modeling with subdivision surfaces in Z-Brush but that's a slightly more complicated pipeline problem.

Jeff has been working on a shadow algorithm for use with SpeedTree. We need to go with a shadow-map appropach because SpeedTree is all about sprites. So Jeff's been playing with various forms of shadow mapping. He gave up on the traditional depth compare shadow-maps because they result in very aliased, jaggy edges and all sorts of other problems. Now he's going with a different projected shadow approach that doesn't necessarily involve depth-compare. I'll post more about it once he's got it figured out.

For the last three days Jeff and I have been pulling our hair out tracking down the dumbest of bugs in the cubemap code. Turns out textures are invalid until you set the min/mag filter parameters; OpenGL just won't sample the texture until filters are set! Man that pissed me off, it doesn't even throw an error!

Once we figured that out we spent all afternoon fixing our cubemap loading code. We were trying to load cubemaps from a vertical cross format and could not figure out how OpenGL wants the individual cubemap faces oriented. Turns out when you're passing in images for the individual faces of a cubemap, the +/- X and +/- Z faces need to be rotated 180 degrees. This means if you have a horizon in your cubemap on the +X face, before it's uploaded the data needs to be rotated so that the sky is pointing down and the ground up in the image.

I still cannot figure out what frame of reference this makes sense in, so 50 points to anyone who can tell me what the ARB was smoking.