A Case Against using Blender Physics

…Not that it’s always a bad thing. Oftentimes it’s great. But, for the tasks we have in mind, it’s usually going to have more trouble than it sounds. This has nothing to do with how well designed it is, it has to do with the fact that the artist is now partially in the audience, and is under the same spell; and is likely to make improper assumptions about what it can do.

I open this website with a note about the threshold of disbelief—that the real, and imaginary, are limited only by our perception of the media. This is basically rule zero when doing any kind of multimedia work. It’s the difference between hearing a cacophony of vibrating strings, and a symphony. The same goes for a 3D model; you have to keep your eye on your end goal.

That is, an illusion, or maybe an elucidation. You are communicating with your audience. To do that, you need full articular control to do it, which in this case, means full control of the behavior of your model. This is one of the things you inherently give up when you add a physics effect.

I’m sure it sounds, on the surface, like I’m being a jerk here, or failing to appreciate the wonders that Mantaflow and FLIP are capable of. I’m not, they’re awesome; the question is, what is it that you saw simulated with them? Ponds, swimming pools, bathtubs. It is great for these. But what if you’re trying to have water coming out of a faucet? And what if you just need a little water in a glass full of (yikes) non-axis-aligned ice cubes?

If you’ve been screwing around with Mantaflow at all, you know that these are impossible situations. Yes, it is possible in theory to do them; scaling down your domain or turning up its sensitivity, as an example, can hypothetically make a faucet source work. However, physics isn’t computationally simple. It takes time, and CPU cores, to do; and the first few times you’ve consistently screwed something up with it.

So, people get so worked up about the awesome possibilities of Mantaflow, they overshoot them and overlook its costs; while nine times out of ten I’ve found that much more manageable metasurfaces and shape keys will do the job just as well. They’re easy enough to do to almost consistently be used in read-time rendering.

Let’s talk about cloth. Cloth is fun, especially if you have any exposure or background as a couturier, as it behaves exactly like real cloth and borrows a lot of familiar terminology. If you want to make a T-shirt, you literally just have to build faces that match the components of the shirt and sew them together with spring weights.

However, now we have a new concern. Collision. Unless you’re just modeling the shirt—you might be—you’re going to have to get it onto the body of a model. Now you need to worry about collision detection, which means you need to understand collision boundaries. Box, convex hull, or mesh? Should you use a duplicated and simplified collision boundary or not? You’re going to have to go all out to get cloth simulation to consistently work. And, you’re going to hate it. So much.

Once again, though, people bump into the physics tab and get so excited about the options, from rigid body to soft body to fluid (liquid, smoke, and fire) to cloth. They never even notice that there’s a cloth brush, in sculpt tools, meant to help with this exact task in an operationally manageable way.

And, speaking of fire and smoke, they’re fun, and they’re not that hard to get working. Unfortunately, they’re also extremely CPU hungry. Why am I worried about CPUs? Can’t I just “get better hardware”? In theory; but CPU equates to time, which we never, ever have enough of. Don’t fool yourself, you’re already running out of it. That has little to do with hardware, and more to do with how you’re dedicating your use of it.

If you can choose between an ultra-realistic path determination for a single “molecule” of water, or an extremely impressive material, the material will always be more beneficial.

For fire and smoke, it’s much better to get it working within acceptable parameters, bake the whole thing to a folder, and import the contents of that folder (specifically the data directory) as a volume. Once you’re done editing it, you should never allow it to change itself, as a general workflow principle. The artist must emulate God with these tasks, and volumes, while relatively new to blender, are so much faster, and often more customizable, than a physics calculation.

The thing no one is inclined to realize about Blender’s physics modifiers is that they’re basically advanced noise generators. You have as much control over them as you do over a particle effect. Much like particles, they have their place; but you can’t jump avant garde into them without expecting their effect to spill over into the rest of your animation.

At the end of the day, Blender Physics is not computational physics. With rare exceptions, your computer can’t even handle computational physics; that’s the kind of job that’s typically allocated to a cluster. Physics emulation follows the same rule as everything else—does the viewer find it believable? What do they see?

So, when attempting to emulate a physical effect, I suggest this (flexible) routine:

0. Verify the need. Are you sure you need a physics simulation to do this? Are there any sculpt tools, modifiers, constraints, or material effects, which could do the job for you? Is it something that you could literally just do in compositing? If so, these will be quicker and easier paths.

  1. Isolate it. You’re still working with pixels and samples, in the end. You don’t need your effect to spill over into other materials. I will often create an entirely new file to model it in, and simply Append the finished product into my master animation when I’m done. You most certainly do not want more than one unbaked physics simulation going at the same time.
  2. Research it. There’s Quick Smoke and Quick Fire, sure, but do you really understand what they’re doing? Do you know how to customize them to meet your vision? I strongly suggest you put some time aside—as valuable as time is—to look up what all of these controls specifically do. If you’re going to apply a physics effect on a regular basis, it’s important to recognize that it is a tool in and of itself, which requires mastery. There really aren’t any useful “wizard”-like interfaces for these.
  3. Allocate for it. Unless you’re working on something very low budget or are immediately running out of time—which is, actually, a great reason not to bother with physics at all—this is going to take a while to get right. Give yourself a day if you’re completely new to it, and if you’re really feeling the creative impulse to pursue it, don’t be surprised if it overflows into a few days.
  4. Bake it once you’re done. This applies to everything—rigid and soft bodies, fluid simulations, cloth if it’s got to be attached to an armature. Once you’ve got it behaving how you want, commit it to RAM and drive space. Having multiple physics simulations going at once can increase rendering time geometrically; so it’s always better to have it sorted to key frames. Animation key frames don’t really slow things down at all.

Physics emulation is kind of like handing off your design to a less experienced designer, like an intern, and giving them total creative freedom with it; then stepping back in a few hours later.

We don’t expect Photoshop or Quark to have physics effects. Hell, we don’t even expect After Effects to have them. So, why on earth do we fail to see them for what they are in Blender? I’ve said it once and I’ll say it again—you must know your noise.

Animating Sculpted Meshes in Blender

Or, How to Apply One of Those Weird Modifiers Which You Never Thought You Would Use


Introduction (Which may be Safely Skipped)

Blender is an extraordinarily complex and beautiful piece of software, and I’m not sure what anyone else would be expecting of it. Of course it is. Aside from all the passion of Ton Roosendaal, numerous contributors have been adding to the project over the years in an effort to create a full-featured multimedia production suite out of what once was a simple 3D model editor.

In fact, sometimes it gets overwhelming. As frequently as I use it, I’m still cracking the surface of a lot of its finer algorithms and tool kits. This article exists to address a common concern among medium-tier modelers.

You see, you can extrusion-model relatively easily with Blender. It was actually the first type of modeling I ever did with it, not long after it was made public license. Of course, this was what, 2002 or 2003, and being a humble American college kid I had only an idea of what I wanted to make, but no idea how to accomplish it. However, the software has expanded to include metaballs, various types of curves, volumes, lattices, probes and even sounds. Weight painting, vertex coloring and grouping, and armature deformation, along with shape keying, has gotten very accessible.

Moreover, it’s introduced a remarkably sophisticated sculpting kit. It’s powerful enough with a mouse, but with a graphics pad (even a cheap one) it’s almost intuitive once you understand the basics.

The actual problem only shows up after you’ve mastered applying armatures to extruded models, and after you’ve sculpted a mesh in fine detail. You have your magnum opus, your masterpiece, your triumphant paragon of a million carefully placed faces, matching precisely what you had in your mind. You understand anatomy, and the rule of thirds; the golden ratio and the laws of balance of form. You move to animate your sculpture, attaching it to a handy metarig and parenting it with automatic weights. And…

It feels like profanity, if it were being spoken in some kind of obscure occult code.

Nothing happens. You get a message, in an almost intentionally uncomfortable shade of orange, warning you that “one or more” bone envelopes could not be calculated. It’s not especially helpful, and you have no idea what went wrong; but your mesh has all ten-thousand-ish vertex groups added and every single one of them is empty.

Don’t despair just yet.

Problems with Automatic Weights

We owe automatic weights a great deal. If they didn’t exist, we would probably never start animating. I personally find that while manual weight painting has come a long way, it’s still rather clunky and something I prefer to avoid; so simply painting in the weights isn’t usually going to do the job.

The most common reasons (and I do mean common) for this bug to occur is having two separate component meshes in an object which intersect in a weird way (or really almost any way), throwing the calculation off; duplicated vertices; and in my experience, malformed normals. Separating meshes, the less common of the two, is relatively simple once you’ve found the offending geometry; but the second one is more automatable.

You can sometimes get away with switching to edit mode, selecting everything, and running the Triangulate operation. This splits every single face into component triangles. You can then select all of those triangles and run Tris to Quads, to convert them into quadrangles. You can also precede this with Merge by Distance (formerly Remove Doubles) to remove any stray duplicate vertices.

Quadrangles (or “quads”) are easily the easiest face type to get along with on an armature; they’re flexible, deformable, and sensible; and I don’t personally like mucking with anything else. However, once you have enough faces on your mesh, this can get inaccurate or incomplete. I’m not concerned about the relatively scattered nature of the quads, I’m more concerned about the fact that sometimes it can’t merge them all and you will still have a few hundred triangles to figure out.

Forget this notion I keep hearing about, that all polygons being broken down into triangles on the graphics card. That’s not how graphics cards work mathematically, it’s only how they work incidentally; and that’s not even necessarily true anymore. A triangle is easily just a truncated quad, and quads are much easier to guide into convincing organic shapes without weird seams showing up.

The other operation is the Remesh modifier, which I still often use; but it comes at a cost. Remesh will scan over your mesh as a volume, and build a new (and perfectly armature-friendly) mesh over it occupying roughly the same volume. (It’s effectively a voxel scan.) It’s very good at getting an animatable mesh; but the problem is that it screws up your topology which you’ve been working so hard on. The second problem is that you can easily exceed a million faces zooming in with it, which is not an easy operation to undo.

There’s a way to apply either of these that will not change your topology at all, and will allow your model, no matter how godlessly non-manifold your vertices may be, to properly deform—using the Transfer Data modifier and a duplicate object.

Duplicate-Remesh-Transfer

The thing to remember about bone groups is that they’re vertex groups with weight gradients; and those will work on anything regardless of topology. So, what we can do is simply duplicate (not instance, by the way—literally duplicate!) our mesh, apply a remesh (or quadrangulation) to our new mesh, and fit it to our armature; then transfer the vertex data over to our unaltered original on the basis of the nearest relative vertex.

Start by duplicating your object with Shift+D, in the 3D View-port space. This is going to be our crash-dummy object which will have its vertex data destroyed; we won’t need it forever. All of our lossy modifications are going to happen on this.

Apply Remesh, and only increase the iterations (or decrease the step size) until the new mesh covers all of the area you expect your armature to deform. Don’t worry about what’s happening to its appearance, we won’t be using it for anything else. When you’re satisfied, apply the Remesh.

Alternatively, you can switch to edit mode and Merge by Distance, Triangulate, and Tris to Quads your vertices, if it works better for you. As always, I encourage experimentation and getting to know your toolkit.

Parent your armature to it, with automatic weights, and our topology-damaged mesh will animate smoothly. Feel free to test this before you continue to the next part.

Switch to your original mesh, and add a new modifier. In the top-left corner of the Add Modifier drop-down, you will find Transfer Data; this is what we want. What Transfer Data does is map data from one object’s vertices, edges, and faces (if specified) to its host with an approximation algorithm. As far as I can tell, it can be used for almost anything.

Data Transfer Modifier

We could ask for a lot of different things from the Data Transfer modifier, including vertex colors, smoothing, even freestyle data; but for now, we’ll just want to check Vertex Data and beneath it, Vertex Groups.

The Ideal Data Transfer set-up

Remember how your vertex groups were automatically populated in your duplicate mesh, which should still (very roughly) approximate your original? What we will be doing here is copying data from it. Click Source and select the name of your duplicated object. This will be our data source.

For this next part, I believe it is very important to ensure that your origin point for your original, and deformed, objects is at exactly the same place. It will save you some time. You could in theory also do this by using local coordinates, but that isn’t the default, and most of us are, after all, on some form of deadline.

We will next want to click on Generate Data Layers. What this does is create the essential layers for our original object from our deformed object. If I’m going to be straightforward here, I have no idea exactly how essential it is, but it does allow you to preview your weight paints and ensure that everything is working well.

For Mapping, I generally recommend Nearest Vertex, and Mix Mode should be Replace, unless you’ve got something weird going on (and every artist who stays busy does, now and then). This determines how the property at the host object’s vertex should be interpolated, from the properties of the old object’s vertices. Feel free to play with it when you’ve got some patience to look at the other options.

If you have a lot of faces in your mesh, and I’m assuming that you do, this will take a moment to complete. Don’t worry, Blender isn’t freezing up, though the process may take up an entire core of your CPU to finish. For each and every vertex, it is scanning a BVH of the other mesh and interpolating between closest matches—in essence you’ve just got a lot of data for it to work through. Go make a coffee or check your mailbox, and come back; you’ll hopefully only need to do it once anyway.

Attaching to the Armature

Almost there.

You’ll want to apply the modifier, too. This will also take a moment. Once the modifier is applied, you can check on Vertex Groups to ensure that all of your bone groups have been transferred, and have been populated in weight paint mode.

Now, we just need to attach the armature; for which you’ll notice all of the bone groups are already properly named. To do this, select the original mesh, and then the armature. Press Control+P, and—and this part is important—do NOT select Automatic Weights. Select Armature Deform, the heading itself, and that’s it. If you select Automatic Weights or any other envelope calculating option, there’s a very good chance that it’s going to ruin your vertex groups, and you will have to start over. Since we’ve already configured our vertex groups, we do not need to do that, and should pointedly avoid it.

Once you’ve done this, you can check your armature in pose mode and ensure that your geometry, no matter how godlessly non-manifold it may be, deforms accordingly. All we needed to do was work around the weight painting constraints, and transfer the resultant data over to our original mesh, which Transfer Data is designed to do.

You can most certainly delete your remeshed copy afterward; you will not need it again.

Conclusion

When presented with a daunting task it’s important to step back and consider what the issue may actually be. In this case, it’s all about calculating vertex weights; and this can easily be done on a remeshed duplicate and cloned over with Transfer Data.

This makes virtually any geometric construction animatable in Blender, without compromising topology or detail.

Keeping a Texture Aligned on a Sculpt-able Mesh

UVs are awesome. They align simple 2D images—a lot of them, actually—on a material, even when the associated mesh moves, even when that mesh is animated. You can even have more than one map, if you need it, and we should all be extremely grateful that they exist.

That said, sometimes they don’t. As an example, when sculpting, a task that is particularly important for humanoid models, there is a feature called “DynTopo” in Blender (or “Dynamesh”, the near-but-not-quite equivalent, in ZBrush) that does not preserve UVs and, looking at the math, can’t necessarily be trusted to preserve the right UVs if it does. With our models, we do this a lot.

The general routine is to finish sculpting (or whichever other miscellaneous UV-destructive task you need) before unwrapping our mesh into UVs. This isn’t bad, all things considered; but from an artist’s workflow it can be extremely limiting. What if we want to add another feature later, or something we have planned doesn’t work out like it does in our heads? What do we do, just use an unrealistically small polygon size?

There are idiot ways around this, and then there are smart ways around it. The most popular among them is perhaps the multiresolution modifier, which is great, but it’s not the only one, and my favorite is much more subtle than that. I’ve rambled enough; let me show you how to select specific points on your model (in Blender, in this case) and map a texture by displacement from them, regardless of what happens to your polytope.

Adding your Control Points

Technically, you could use any object to do this. Sometimes a volume or a curve is more sensible; but, in the interest of any systems engineer, I like to do the minimum amount of damage when I add a new feature, control points notwithstanding.

In Blender, there is such an object as an “empty”. It does not render, at all. You can choose multiple ways of representing it, and the only service it provides is to remember a reference point in your image. If we were talking about GIMP or even Inkscape (which is a bit more complicated with it), we would call these guides; they exist purely for the renderer and artist’s sake.

Presuming that you’re starting with a basic 3D humanoid, place your 3D cursor over the solar plexus. If you’re rendering something else, go with a rough center point to its anatomy, from which everything else can be rendered. This is a question purely of what will work for the artist; but it is the point you will be measuring from, so keep it close to the root of your armature.

Suggested alignment on a humanoid model

From here, we can add an empty. In Blender, this occurs with Shift+A and is its own category. Subcategories are all empties but include such things as plain axes (which I will be using here), arrows, a single straight arrow, a number of simple geometric primitives, and if none of those work you can even use an image.

Plain axes are the knee-jerk choice, but for visible clarity, let’s choose “single arrow” instead. Plain axes are a little difficult to locate inside our mesh.

Start by parenting your mesh to your new empty. To do this, select the empty, then the mesh (with Shift held down), and type Ctrl+P. Whatever your mesh now does, your empty will do too.

Now, let’s discuss what the empty can tell your renderer. It’s clearly helpful as a visual guide, especially if you’re in some orthographic view or something even weirder; but what about our material?

Material Linking

Blender has a very fancy material editor, even compared to a number of pricier products. It is actually possible to link any mesh properties, including those of other meshes, to this data, if you’re careful.

We’re going to be very basic about this, for the sake of the demonstration. (As a friendly note to other tutorial writers, I will say this. As much as I love to watch a good sculpt, I’m always a bit disappointed by your generic YouTube tutorial that spends half an hour addressing tangential needs, no matter how cool it can be to watch. I’m often on a timeline, which is the only reason I don’t just take a course on the subject. I know you mean well and appreciate the work, but please be considerate.)

Create a new material for your mesh, and make it a simple diffuse material. (You could make it anything, but we won’t need much for the core concept.) We’ll begin with two nodes, a simple shader that produces a closure for diffuse lighting, and our output. We will vary our color by distance from the empty.

A basic diffuse shader

To your color input, attach a Color→Hue Saturation Value node. Set its color parameter to full-red and zero-blue and zero-green. Saturation and value can remain at one. We will be rotating the hue by distance from our solar plexus.

Classically, if we just wanted the relative displacement in local coordinates, one way to do it is to simply use the vector difference between Geometry→Position and Object Info→Location. This gives us the distance between our literal geometric coordinate, and our object origin. If you weren’t concerned about direction and are more concerned about distance, you can just take the length of this. You could do a number of other things, too, but we’ll be keeping it simple for now. Let’s start by doing this.

A basic way to rainbow-shade based on distance from object origin

You’ll see with this shader that your actor now has a rainbow hue from the hue rotation, centering around his origin.

In my case, the origin is near the feet, so the hue rotation occurs mostly by z-coordinate. This isn’t what we want, but it’s a start. We’re aiming for our external empty, aren’t we?

The only problem with the current state of Object Info is that it only provides info on our shaded object, no other object. However, we have a handy feature to the rescue, one related to the fact that Blender is, before anything else, a Python IDE by design. Drivers.

Replace your Object Info node with a Converter→Combine XYZ node. This, incidentally, is also a sort of Value node but for vectors. We’re going to tie it to our empty location.

Our Final Example Shader

Drivers

Drivers are a severely (severely) under-recognized feature of Blender. They allow you take nearly any property of your scene, and bind it to some other property. You could change glossiness by location, as an example; or material by light exposure. They’re seriously not talked about enough. We’ll be using one here.

Add Driver option

After replacing your Object Info node with a Combine XYZ node, hover over the X-coordinate and press Shift+D. (Or, if you’ve mucked with your default keyset and Shift+D no longer works, right click it and select Add Driver. This opens up an editor that provides no shortage of cool possibilities.

There are a few things I would like to highlight about it, but I do encourage you to play with it, break it, fix it, and learn how it works inside and out on your own time. But for now…

There are several types of drivers, selectable immediately under settings. They carry out understandable and frequently used expressions; but are all effectively subsets of the default, which is a scripted expression. We’ll be using that.

The expression itself takes any number of declared variables and performs an operation on it in Python script. (It is not yet possible to import an entire text-editor-space script like this, but this is literal Python, so I’m sure it’s doable. Our needs are more basic.)

The default expression is var, which is the default variable which we can link to other elements in our scene, plus zero… for some reason. That’s an identity operation, I don’t know why they did it, but I’m guessing it’s for illustrative purposes. This is, fortunately, all we need, but remember that sqrt(var), pow(var, 3.5), and var/12.0 + pi are all perfectly valid if they serve your needs…just make sure it’s deterministic and interpretable Python.

Beneath this expression, we have a list of our “input variables”, which are the facets of our scene that we want to use. Initially, var will be colored bright red in its object socket, as having that empty resolves to total nonsense for Blender. It will immediately tell you that with in the window.

That isn’t strictly true; there’s nothing wrong with the default expression. However, it doesn’t know where to pull the data from, which is just as problematic. To change this, we will click on that Object box under var‘s definition and select our empty. Type can remain “X Location” and space can be “World Space”.

You should now have a driver-bound coordinate in your Combine XYZ node. Try moving your empty around on the X-axis, and watch what happens to the node when you place it. We can do the same for the Y-axis and the Z-axis, making sure to select “Y Location” and “Z Location” when we do it, and we’ll have our model shaded relative to that point.

Take a look at your model now, and for a little fun, drag the empty around in 3-space.

Results!

Just like that, with the appropriate use of drivers, you can connect any object to a shader. Go ahead and add dyntopo sculpting to it if you would like; as there’s no UV map added, it won’t affect our shader.

That said, this is not the only way to do this or even always the best way. Once an artist is absolutely certain that sculpting is done and topology is fixed, it’s still a good idea for them to replace these with UV maps. Mapping, in general, can do a lot to make any model look more realistic.

However, in cases where you might be modeling multiple shaders over each other in the same material, this is often the best way to do it. I personally am working on a professional model of the ancient Babylonian monster-god-of-fresh-waters, Absu, and needed to add both a visible poison pulse, and a golden mud melting off of him; and I could come up with nothing better for multiple features than drivers and empties.

It’s for a syndication, but as I have limited processing power (here’s a little secret… nobody has infinite processing power or time, not even at major studios) and am working on my next book at the same time, I needed something quick and reliable.

Some day I might write a text on drivers by themselves; but I’ve still got a great deal to learn about them myself. They’re the easiest way to add a major feature to Blender without diving into the actual C++.

I hope this method becomes more popular as I don’t meet a lot of other modelers who make use of it, and I’ve yet to bump into a serious problem with having it in my model source. Perhaps it will help all you guys out too.

As a full disclosure, the particular humanoid model used in this tutorial was done with MBLab, an add-on produced for creating humanoid armatures and associated meshes in Blender. It’s open and it’s awesome. I don’t personally usually use it, as I feel it’s too all-encompassing to be any good for me when I’m doing weird stuff with my models; but it’s still great for pounding out an idea for basic testing, or getting something ready-to-go for a tutorial. Needless to say, all vertex groups, constructive modifiers, and so forth were removed before the tutorial itself.

(It is definitely not even close to my Absu mesh. I’ve been working for two weeks on that guy with nothing but virtual clay and a graphics pad. He is humongous, terrible, glorious, and monstrous, and should probably be shared here when I’m done!)

Sculpting Example, post-shader. (Sorry. I just finished playing Doom Eternal, which was weirdly therapeutic; and it’s still kind of with me.)

Thank You for your Impassioned Response!

High time, I think, that I dropped an update on here.

The reception for my first two books this year has been overwhelming—I’m feeling a lot of gratitude. It’s also helpful for morale, to see so many people grabbing them off the digital shelf. Writing a proper book is not a small task; and for those of us with a real goal, not just writing-for-the-sake-of-writing, it’s not something that should be taken lightly, either.

My next two books are, almost invariably, going to be on LV2—the open and modular sound processing API that has asserted itself as a likely future for VSTs—and OSL, or Open Shading Language, the customizable shader system built into Blender. They’re distinctly separate subjects from my last two books, on Blender plug-ins (unrelated, or at least not immediately related, to shader programming) and GIMP plug-ins.

LV2 is a step away from the last two, which arguably revolved around creating plug-ins for 3D and 2D art. This is audio. Audio is a sophisticated subject; while it’s easy enough to play back a sine wave, once you’re working with envelopes and the frequency domain you need to start considering things like Fourier transforms and complex analysis. That’s definitely not the kind of thing you can rush through explaining.

As flagrantly-industry-insider as all of that sounds, I promise these things are actually kind of fun. The point of the book is convey them in a tangible way, one that’s easy to understand. It’s new terrain, but you can do so many cool things with it!

It’s arguable that these topics will be beneficial to graphics programming, too; JPEGs actually compress the entire image as a Fourier transform. God knows there are all sorts of cool plug-ins I could make using it. However, it requires a steady pace to explain, when I first learned about Fourier transforms it was rushed at me. It took about a month before I had my Eureka moment and understood them; and looking back it’s so painfully simple. I’m not going to make the same mistake explaining it to you all; you’re going to get them introduced in the most colorful way I can manage.

Additionally, unlike with Blender and GIMP, I need to go over latency compensation. There’s almost no such thing as a latency-free effect; the closest I can get to one is maybe a simple amplifier. Processing digital signal data takes time, and whether you’re dealing with real-time audio (you totally can) or a post op, that latency needs to be taken into account. Heck, if we’re building something like a delay, latency is the intended effect.

This means that LV2s have to worry about statefulness—internally remembering what it was that they were doing—and make requests to supporting host programs to call them again, so they can finish. They need to be able to reset that state, too.

Unlike Blender plug-ins, which can get away with using Python; and GIMP plug-ins, which produce a static result, LV2s can rely on the fact that they are running in parallel with a great many others of their ilk. Efficiency is more important than ever before. Additionally, while I can clearly see that their older version (LADSPA) took direct influence from the structure of GIMP plug-ins, the new version improves on that by providing TTL (pronounced turtle) metadata, so the plug-ins don’t have to be run in order to understand what they do.

There is far less documentation available for LV2 than Blender or GIMP; but I’m experienced using them and I have a plan. To check myself, I’m even building a Hurdy-Gurdy contemporarily, just to help me visualize the audio mathematics. By the end of the book, I should have covered a wide swath of common guitar effects, and toward the end of it I may even do something original, like imprint a JPEG as a spectrogram and play the image as a sound. (You know, the usual weirdness.)

The unfortunate part of my reality is, after finishing the GIMP book I suffered from about two weeks of cognitive fatigue! I suppose it’s understandable, the new one hasn’t been much different. I’ve already edited over about half of my wording on this book so far, finding a structural or ethical problem with its presentation, and have gone well over 60,000 words in total, not counting code. This is usually how it goes, though; and two edited books written in four months is pretty extreme already. Thank God I used to be a science journalist or I would never be able to pull this off.

I know that I still need to get paperback copies of The Warrior Poet’s Guide available—that is on its way, I’m simply deferring that edit until after the fourth book. Amazon is currently offering a deal that will allow me to sell the paperback copies to people who already own the eBook for a couple of bucks each, which sounds quite reasonable. (Everyone enjoys the feel of a good paperback, for all the benefits of an eBook.) I admit I wasn’t entirely stoked about the idea at first—paper isn’t cheap and these books are built on evolving subjects—but apparently they can be printed by order. I’m still sorting it out, but they are coming, likely before 2021.

I’m debating also covering VST with it, since the two are technically inter-operable (with a little wrangling) and most DAWs can support both through the use of one. I’m not a huge fan of the details of VST’s implementation, but I do love its concept. We’ll see if that’s a good idea, or if it calls for a book of its own.

So, the site has been quiet because I’ve been working on the next book, on audio. This is likely to happen again when I get to OSL (a sort of modernized incarnation of RenderMan), but let no one say that The Warrior-Poet’s Guide is slowing down or dead! It’s all I can think about right now.

Depending on how negotiations go, I may also end up selling some T-shirt designs on Amazon. There’s also been some renewed interest in my old Patreon account, which isn’t technically attached to anything anymore, and I’m thinking about rebuilding it around the media-hacking plug-in culture. With 2019-nCoV spreading in the US, I’ll be stuck inside for a while anyway. (There have been three confirmed cases in my town as of yesterday, with who-knows-how-many unidentified cases still in the air.) So, expect me to keep busy in the studio!

Happy Media-Hacking,
—Mick

What is Going On with our Color Standards?

So, today I was in the middle of what will hopefully be the final pass of “Warrior Poet’s Guide to Writing Plug-ins for GIMP 2.10”. It’s been an interesting journey; I had to resort to demonstrating techniques in C instead of Python, as an example. I’ve also reviewed and completely rewritten about half of the chapters. Punctuality may be important, but quality and information-congruency, and while-we’re-on-the-subject longevity, are a bit more valuable to me.

I got to the point where colors were described in GEGL, in the format of an interpreted string. The means of doing so is relatively simple—you can define any color in an RGB or RGBA color space, using a range in zero to two hundred and fifty five, a percentage, or a floating point between zero and one. Random experimentation showed me that you could even mix and match them—you could have your R component as a percentage, G as an integer value, and B as a fraction of one, and GEGL would figure it out. I honestly have no idea why anyone would do this, but it was a fun discovery.

What’s more, you can use the full CSS color constant list, right in your program. Instead of this:

GeglColor *color = gegl_color_new("rgb(0.0, 1.0, 0.0)");

You can do this:

GeglColor *color = gegl_color_new("lime");

To a lot of people, that may seem like a good thing; but for me, the hair on the back of my neck was already standing on end. The more I thought about it, the more random it seemed, “standardized” or not. To begin, why is “Green” defined as only roughly 50% of the green channel; but “Lime” designated as high-intensity green? Moreover, have any of us actually seen a lime that it objectively this color? Because I definitely haven’t, of any species, and I make black-lime teas and key-lime pies that will just melt you.

Moreover, on what hardware were these colors determined? One monitor to another, even modern models, will render things slightly differently. In the old days (early-1990s) we had it relatively simple—everybody had agreed to a resolution of 640×480 for television screens, and there were only three phosphors commonly in use for them—we had yttrium oxide-sulfide with a europium activator for red, zinc sulfide and copper for green (also known as P31), and zinc sulfide with just a hint of silver for blue (P22B). These were easy to get our hands on, they consistently emitted the same color when they were sprayed with electrons from the CRT, and people could plan on them.

It’s almost 2020 now, and we now have IPS displays, LCDs, a scattered handful of CRTs still lingering around (probably), plasma, LED displays—heck, OLED and QLED complicate it even further. These all render color in decidedly different ways; in fact, Sharp even introduced its Quattron television series that had an additional channel for yellow built into the display. (And no, there was no technical reason to do this; a study at Queen Mary University in London showed any difference to be negligible. There was no data being broadcast to drive the yellow sub-pixel!)

What’s more, the human eye is only roughly capable of differentiating between about a hundred different tones per type of cone cell (color receptor), which is 100 × 100 × 100 = 1,000,000 colors. Which tones they happen to be is subject to both variation and interpretation, and it is apparently possible to train your eye to have a “refined palette” for tonal differences.

For this reason, radiologists, as an example, will buy 8K displays with as many as twelve bits of channel sensitivity—that’s 4,096 tonal variations per color element—to prevent relevant medical data, like-I-don’t-know-cancer, from being lost in the overflow. Such displays aren’t cheap. If you’re curious, a typical monitor only has eight bits of color depth, or 256 different tones, if that, and some error on which tone is reproduced; but don’t worry, unless you’ve trained your eyes for years, you will not likely notice the difference on a 10-bit monitor.

The variation in color between displays is not a new thing; I remember as a child finding a “hue” dial on an old CRT TV set, which seemed to alter the color between red and blue. I remember it being decidedly skewed toward red, which may have been an artifact of the set. What it ultimately did was alter the strength of the aligning electromagnet inside the set, spraying more electrons toward one edge of the RGB-phosphor-chain than the other. (I spent some time playing with it and learned a great deal about how video worked even then, down to the RGB sub-pixels, using water magnification through drops from a squirt gun, much to the anger of my grandfather.)

So how did we come to this strange set of CSS colors? If they don’t even represent the same thing on every monitor, why choose arbitrary names? Where did the standard come from? Pleasantly, it’s at least an old standard. It came from the X Windowing System, initially release X10R3. If you happen to be using a Unix-based operating system on the Linux or BSD kernel, you can still find a relic of the original file, at /usr/share/X11/rgb.txt.

That was February 1986, more-or-less the dawn of color computer monitors. VGA monitors didn’t even exist until 1987, so we’re really down in the mud here. The best we had was CGA/EGA, and only 69 colors were listed at the time. The file was originally written by Jim Gettys at MIT; specifically calibrated for the VT240 monitor. It had an 80 × 24 character display, and cost just short of US$1,400. (Doing the math, that’s maybe 73¢ per character.)

We didn’t get the full traditional range of colors until around 1989, when Paul Raveling, John C. Thomas, and Jim Fulton (on file maintenance) added the standard named set. Fulton made a number of stupid-but-critical changes to the file, including removing the color “transparent” (that would have been a serious problem down the road…) and a few blank lines. He calibrated everything to his specific HP monitor.

In a sense, that answers most of my questions right there. I think we can rest assured that (0, 255, 0) looks exactly like lime on Jim Fulton’s exact monitor that he was using in 1989. On that specific monitor, it is a damned fine lime green. And no, we don’t actually have any idea what model number of HP monitor that actually was.

Most of his color names purportedly came from Sinclair paint swatches. This brings me to a new concern, though; many years ago, I worked at a paint desk at a local Home Depot. We had a fancy little marketing gadget that would match the color of the paint to any sample put under it, which sounds like it works better than it actually does. Most of the time, after formulating a sample, we would get to the real concern—was this not-quite-objective photographic match precisely what the customer had in mind?

After a while, employees developed something of an artist’s touch for adjusting the color to meet customer expectations. I was well aware even then of how arbitrary the device was, but it was importantly a place to start. Right next to it we had a display allowing people to view paint samples under fluorescent, “natural”, and “incandescent” light, which of course were all actually fluorescent bulbs which may, or may not, have a light filter wrapped around them. And no, I didn’t trust it completely, but like I said, it was a start.

Interestingly, John C. Thomas eventually took inspiration from a box of Crayola crayons, while refining the color set. Almost comically, more than half of the colors used were represented. The last real update to the X11 set was in 1994; but in 2001, the W3C decided to merge the X11 set with the CSS3 standard.

There were immediate problems here, mostly involving color-nomenclature overlap. The grays, as an example, were entirely off. I honestly have no idea how it was standardized, but evidently the W3C just decided to codify what everyone was already doing.

I suppose this post has gone on for long enough; but I’ll end it with a (slightly-desperate) plea to stop using these arbitrary color names; and also note that a lot of them came from the defunct Sinclair paints, and Crayola; not that they had any idea that they were being used as a reference. I’ll leave a half-hour-long talk and CSS Conf 2014 from Alex Sexton here, which covers a lot of the same material. His email citations are quite entertaining!

Alex Sexton, CSSConf2014

If anyone ever finds what remains of that old HP monitor, it’s probably worth millions now.

The Warrior Poet’s Guide to Writing Plugins for GIMP 2.10

After publishing The Warrior Poet’s Guide to Python and Blender 2.80, I was at a deciding point. It’s clearly not meant to be the only element in the series, but I had a lot of options then. I could throw money into marketing (well, more of it), I could break for a while and get back to one of my in-house projects, or I could start on the next book.

I’m not a person to waste time, not even when I arguably should relax; so, I decided to get started on the next book. The theme of the series is professional, proven, and most importantly available tools, which are flexible enough to use to write one’s own plug-ins and add-ons. On top of that, these books are directed at people who are ambitious and driven enough to use the information, but may not always know where to begin.

The choice was obviously the GNU Image-Manipulation Program, or GIMP. GIMP has had a resounding impact on the VFX industry. Its only realistic competitor for use is Photoshop, which is saying something—GIMP was produced entirely by volunteer engineers for projects of their own. It has objectively been a spearhead for the open-source software movement. It’s spawned various other projects out of both its own original code, and its ideals, such as the GTK+ windowing toolkit.

Moreover, unlike Blender, you can easily extend GIMP with an assortment of different languages. It has native support for C, Perl, Scheme, and Python. There’s an awful lot going on under the hood in that software. However, with GIMP being software built by volunteering professionals, many of the tutorials are frequently out-of-date. (Documentation takes time too, after all!) I will frequently come across often-cited walkthroughs on the internet, which recommend techniques which have been deprecated for nearly a decade.

I’m actually glad that the walkthroughs are still there; people who aren’t used to actual source-code diving need a place to start. However, version 2.10.14 was released not long ago, the world has changed in GIMP, and no one talks about the new features. The existing material isn’t enough. Where are the saga-long YouTube videos on GeGL tricks? Where are the WordPress pages of colorspace filters and distortion stunts?

I’m now four chapters into writing The Warrior Poet’s Guide to Building Plug-ins for GIMP 2.10, focusing on Python and C. There are still worlds of material, and associated artistry-magic, to cover here. Moreover, it’s pivotal that the material is accessible to the broadest part of my target audience, so I may have a few appendices on subjects like Python, C, and GeGL. I may even append something on basic Python to Python and Blender 2.80.

If you’re interested in writing your own plug-ins for modern GIMP, then check back to this website over the next few weeks. I’m in full science-journalism mode and hope to have the initial draft done by the end of November. After that, there are a host of different products I could take on next, from LADSPA in Audacity, to OSL.

We’ll see what comes next.

Exporting Ultra-High-BPP Gray-Scale with FreeImage

Are you trying to encode data that corresponds to a specific point? Perhaps a height map, or an MRI or CAT scan? Even a density map?

If so, you may be considering storing it in an image or video format. Of course, if you’ve tried that, you may have bumped into the sharp limit of what our monitors can represent, and for that matter, what our eyes can realistically see.

Most images have a maximum of 8-bit-channel color depth, that is, eight bits per pixel color, or 256 possible shades each, of red, green, or blue. The reason for this is that the eye does have its limit on perceivable and noise-free differences in shade. According to the Young-Helmholtz trichromatic theory¹, Our eyes have three types of cone cells responsible for detecting color, distributed across low, medium, and high wavelengths. (Overlap is why we perceive a gradual blend between colors, with intensity-detecting rod cell sensitivity somewhere in the blue range, partially explaining relative “brightness” of different shades. Hence, the rainbow!)

Each cone has a resolution of roughly 100 shades. That means that our eyes, in theory, are capable of distinguishing between 100³ or 1,000,000 different colors. For the rare tetrachrome with a fourth cone cell type, it’s possibly closer to 100,000,000. In any case, 256 shades for red, green, and blue (corresponding to low, medium, and high wavelength color) gives us around 256³ or roughly 16,700,000 possible colors. All the same, for the trained eye, monitors do exist that can render high-bit-depth color, extending into the 12-bit-per-channel range; but beware that they are not cheap.

The drive for higher bit depth usually stops with scientists; however, when we’re talking about storing data (rather than appearance) in an image, eight bits only get us two hundred and fifty six possible values. For more extreme precision, 16-bit (or even higher) can look awfully appealing.

But enough introduction. How does one export a high-bit-per-pixel single-channel image? This is fairly easy to do, and generally cross-platform, with a toolkit called FreeImage. However, like everything else, it recognizes that most of its usage comes down to displays, and it isn’t immediately obvious how to do so.

There’s a specific (and very efficient) function in that toolkit which can set a pixel color according to a provided parameter. It’s usually easy to use. The specification is thus:

DLL_API BOOL DLL_CALLCONV FreeImage_SetPixelColor(FIBITMAP *dib, unsigned x, unsigned y, RGBQUAD *value);

Where FIBITMAP is a FreeImage bit mapping, x and y are unsigned values specifying coordinate from the bottom-left, and RGBQUAD is… RGBQUAD is, ah… uh oh.

RGBQUAD is 32-bits. It contains one byte for each primary color, and one byte reserved (where you could in theory store the alpha channel or any other special value). However, that limits us to, one, color; and two, no more than eight bits per channel. So, if we’re dealing with 16-bit gray-scale, we can’t use it; and if you try to pass in a pointer to a ushort you’re asking for trouble on compilation.

To modify monochromatic data, broadly speaking, we have to edit the scan lines directly. RGBQUAD has to be purged from the code for anything high-bit-def. There is precious little data on doing this available; but it comes down to another wonderful method included in FreeImage, specifically for cases like this one.

DLL_API BYTE *DLL_CALLCONV FreeImage_GetScanLine(FIBITMAP *dib, int scanline);

A scan line is the line drawn, classically by the CRT beam in older monitors, across the width of the screen. It carried over to image and texture terminology, where alignment matches a monitor’s in most cases. So, a scan line is the range of data corresponding to one “sweep” of the screen.

This method returns a BYTE *, which many will recognize as a pointer to a specific byte in memory. If you need a pointer to, say, a ushort (16 bits), you can pretty much just cast it in most languages. So, with a little pointer arithmetic, we can access the location of every quantized color value, regardless of what bit count it’s made of, in memory!

The code that ultimately ended up working for me, in D, was this.

for(int y = 0; y < height; y++) {
	ubyte *scanline = FreeImage_GetScanLine(bitmap, y);
	for(int x = 0; x < width; x++) {
		ushort v = cast(ushort)(data[cast(ulong)((y * width + x)] * 0xFFFF);
		ubyte[2] bytes = nativeToLittleEndian(v);
		scanline[x * ushort.sizeof + 0] = bytes[0];
		scanline[x * ushort.sizeof + 1] = bytes[1];
	}
}

For each scan line in the image, we iterate over the number of pixels in its width. (You must know this ahead of time.) We calculate our value and store it in v. (data was sent to the function in a parameter, as a double[]. It’s the data to be rendered in the image, as a list of values, of dimension height × width of the image.)

v is a ushort, so it’s already two bytes and interpreted as unsigned. Of course, a ushort can’t be dropped into a byte slot, it needs twice the space, but the equivalent byte array, available through std.batmanip.nativeToLittleEndian, will do just fine. My machine is working with little-endian format, so this is all that’s needed.

I suppose you could also do it manually with a little pointer math, if you enjoy that kind of thing or don’t have an appropriate package in your environment.

We can then marshal those bytes into their equivalent positions on the scanline array. We’re done! The image can be exported to a file.

Unfortunately, you may bump into an issue where the output is upside down. This isn’t as noticeable for noise arrays, but stands out on anything with actual dimensions. To change this, you need to fill from the bottom-up instead of the top down. This is fixable with a single line of change.

ushort v = cast(ushort)(data[cast(ulong)(((height - 1) - y) * width + x)] * 0xFFFF);

By decrementing from the maximum value of y, you can reverse the order of iteration and flip your image.

PNG and TIFF are two good formats for 16-bit-depth gray-scale images. It’s hypothetically also possible with higher bit depths, but the procedure should rough be the same, and there are too many possibilities to tangle with right now. I recently even heard something about 96-bit-depth monochrome, but that’s 2⁹⁶ or ~8 × 10²⁸ values and I’m not sure we can even get that kind of precision on most measurements. (Even Mössbauer spectroscopy only goes to change detection of a few parts per 10¹¹, and last I checked, that’s the finest measurement any scientist has yet made.) I suppose it’s a comfort to know that, if we ever are that certain about anything, modeling it as an image is a possibility.

Also note that higher-than-16-bit channels are very difficult to read with most image software. I start bumping into trouble around 32-bit, and others can expect the same. So, roll your own, or come up with a better method of parsing the data, like a hex editor.

Final Code

void toFreeImagePNG(string fileName, const double width, const double height, double[] data) {
     FIBITMAP *bitmap = FreeImage_AllocateT(FIT_UINT16, cast(int)width, cast(int)height);
     for(int y = 0; y < height; y++) {
         ubyte *scanline = FreeImage_GetScanLine(bitmap, y);
         for(int x = 0; x < width; x++) {
             ushort v = cast(ushort)(data[cast(ulong)(((height - 1) - y) * width + x)] * 0xFFFF);
             ubyte[2] bytes = nativeToLittleEndian(v);
             scanline[x * ushort.sizeof + 0] = bytes[0];
             scanline[x * ushort.sizeof + 1] = bytes[1];
         }
     }
     FreeImage_Save(FIF_PNG, bitmap, fileName.toStringz);
     FreeImage_Unload(bitmap);
 }

Fun fact: The image used for this page is 16-bit, though the data doesn’t really necessitate it. (One of my generated topographies; I’ve noticed plateauing on my data, and am upping the output to 16-bit, with this, to remove it. The resolution still needs to be scaled up above 256×256, though.)

You can check this, on most Linux distros, with:

$file multarraya_fi.png
 multArrayA_FI.png: PNG image data, 256 x 256, 16-bit grayscale, non-interlaced

Happy sciencing!

¹ Kalat, James W. (2001). Biological Psychology, Seventh Edition, Bellmont, CA: Wadsworth/Thompson Learning

Webcam Sampling

Did you know that this is doable from within Blender? I added a new chapter to The Warrior-Poet’s Guide on it tonight. It shows how to access OpenCV, the binding standard of computer vision, from Python; it also shows how to use ffmpeg to stream webcam data directly into Blender.

Either contains a world of possibilities for image manipulation. OpenCV and ffmpeg both support a variety of concurrent devices, in different manners. In fact, I made a note at the end about someday dedicating a Warrior-Poet’s Guide to each of them. This opens up a range of possibilities for 3D image processing.

My web camera, projected onto the surface of a beveled cube, in quasi-Predator-Vision.

Purchase the book on Amazon to gain access to this wonderful trick. I can’t wait to see what people start doing with it!

Geomorphology with Blender

Geomorphology is the science of the shape of mountains, rivers, islands, and other natural landmasses. Paleogeomorphology is the study of faults, erosion, and even tree rings which provide data on earthquakes which may have happened thousands of years ago.

The most fascinating aspect of geomorphology is the time-bracketing of events. Earthquakes can leave faulting which, relative to known processes like C₁₄ uptake and shoreline erosion, can result in windows of several centuries over which they may have occurred.

For a computational designer, it’s more traditional that mountains and terrains are generated purely from Perlin noise and Voronoi diagrams. However, while the process is remarkably efficient, they never seem to look quite right. Without the involvement of human hands, there is rarely any erosion or character on the terrain. This provides limited value in the context of a game, but what if we want to see a mountain with a history behind it? What if we want the land to tell a story, as real terrain often does?

Generated digital terrain formed in Blender and DLang
Sample terrain fragment after applying wind erosion, basic particulation, tree groves and fundamental water erosion.

An in-house project, currently going on here in the studio, uses a LAN-scoped web interface built in D, and a local server, to evolve land masses over time, from a baseline. Techniques like Söbel softening, very basic Voronoi diagrams, and Perlin distribution of density allow for erosive processes to iterate year by year. Since this is effectively a bake, time hasn’t been as much of an object; however, for 512×512 images it has been moving surprisingly fast. Depending on detail, we can currently run it in roughly two seconds; combined with rendering (including volumetrics) Eevee can have it ready in about fifteen seconds.

Currently, output is saved as a sequence of 8-bit gray scale PNG height maps and density maps, unfortunately that limits us to just 2⁸ or 256 values! An effort is being made to expand it to 16-bit gray scale PNG, bringing us to 2¹⁶ or 65,536 grades, or even 24-bit TIFF at 16,777,216 possible grades. However, most displays are only capable of 8-bit color, so not as many options remain. Most of these files are for scientific output, purely; as far as I know, no monitor is capable of accurately representing anything higher than 16-bit color, and most only 8-bit.

The FreeImage toolkit allows for, in theory, 32-bit gray scale or even double-precision floating point gray scale (64-bit), but that may do more harm to the project than good. After all, portability is important, and nothing meant for the human eye ever seems to go above 16-bits, so there is little support for this in most design software.

In theory, Blender (or any number of other 3D design programs) would truncate it at 16-bits anyway, and after a number is above the atomic unit of operation for a processor (typically 64 bits these days), there’s a noticeable slowdown. Time may be more valuable than precision here.

The remaining standing question is, what was our planet like for its baseline, before any tectonic or fluvial geomorphing? If we wanted to start from a clean-slate, with basic density voxels, would that give us a realistic result? It’s very hard to say at this point in history. We have a rough idea of how the planet formed, but between the stresses of heat and age, could any reliable evidence of that still exist? This period is generally known as the Hadean period, taking its root from “Hades”. The world was indeed very hell-like; it had an abundance of highly radioactive and short-lived elements, which have since decayed, and given that it was a mass of recently-pounded together space rocks, it was ridiculously hot. We’re talking about peak environmental erosion here, so assuming a perfect sphere with reasonable thermal radiance may still apply, whether the world ever looked like that or not.

One of the only remaining elemental minerals from that period is known as Hadean Zircon. Zircon, consisting of ZrSiO₄. It has a fusibility (melting-point) of 2550° C or 4620° F, which is ridiculously high. Its Mohs scale hardness value is around 7.6, a little short of topaz. It is also largely insoluble. Given the gemstone’s relative indestructibility, it seems reasonable that it would be one of the last surviving elements from the Hadean period; even still, only roughly 1% of zircons found globally are confirmed to be Hadean.

Hadean Zircon Fragment
Valley, John W., et al. “Hadean Age for a Post-magma-ocean Zircon Confirmed by Atom-probe Tomography.” Nature Geoscience 7.3 (2014): 219-223

The plan is to move from 2D-mapping to 3D-voxel data, and work from there.