VFD4 Prep: VMTurbo

VMTurbo Logo

VMTurbo appear to make a piece of fancy monitoring software that uses “free-market principles” to make optimisation decisions. All the resources in your data centre are modelled as sources of supply and are demanded by other parts of the data centre. CPUs are basically raw ‘compute’ commodity, as is storage. VMs need compute, RAM, and storage, so they demand it. The amount they need, and the current availability of the supply affects the price of the resources.

If RAM is tight, the price goes up, and only the VMs willing to pay the high price can have it. Same for CPU or storage. Different classes of storage, say SSDs and 10k RPM SATA, will have different prices, and at a high enough price, it might make sense for one VM on a CPU constrained cluster with SSDs to move to a 10k SATA based cluster with spare compute and give up the compute (and more expensive disk SSD in that cluster), for a VM that needs the compute more, even though the SSD isn’t really needed.

I’ve watched a bunch of videos – both on their website, and from previous Tech Field Day events – I’ve read their marketing copy, and a bunch of other blogs about the software. It looks like it could be nifty, but some key details about how this “market” works are missing.

Now, part of marketing is to get you interested enough to contact the company for more information, so the marketing is doing its job, as annoying as some of the copy is (As American as apple pie? Really?). I am intrigued, and I have many questions.

Economic Modelling

Essentially, all models are wrong, but some are useful.” – George E. P. Box

Economic systems, full of human actors, are decidedly not efficient. The Efficient Market Hypothesis is demonstrably false, but is a useful approximation in a variety of circumstances. The supply-demand curves of Econ 101 (a good summary at Kahn Academy here) are wrong, but are a useful approximation for analysis in a variety of ways.

The key assumption underneath VMTurbo’s optimisation engine appears to be the relationship between resource utilisation and price. As utilisation increases, the price increases, but it’s not a linear relationship, it’s an exponential one. At low utilisation, extra utilisation doesn’t cost much extra, but at high utilisation, getting an extra 1% is much more expensive. Supply is constrained, so the price goes up. Ok.

The behaviour of the model is very dependent this curve. It affects the price elasticity: how much the price changes based on a change in demand or supply. It’s not a straight line, because then the price change between 0-1% would be no different than 98-99%, which isn’t what we want. How curved does the curve need to be to promote the behaviour we want in the model? What happens to the price as utilisation nears 100%? What if we consider ‘full’ utilisation something lower, like maybe 95%, because of the way the infrastructure works. Does the software just know that, or do we set it ourselves? Do different resources have different price-utilisation curves? How many are there?

However, if we stop worrying about model perfection and consider that it might simply be useful, then the VMTurbo approach could be very useful indeed. You can tune the assumptions, or make the model more complex when dealing with situations that need the added complexity. Newton’s laws of motion don’t work when you’re dealing with things that are really small, or moving very fast. Then you need quantum mechanics and Relativity, but using the equations of quantum mechanics to to do ballistics calculations would be tediously complex and unnecessary. Newton’s laws are good enough for most everyday situations.

Thus VMTurbo could well be a useful enough model for the majority of today’s workload optimisation problems, and a damn sight better than trying to do it all manually. I really like the idea of a system that can automatically find equilibrium points as an emergent phenomenon.

Model Limits

The trouble with computer models is that, like all models, they’re wrong. If you use a model, and the VMTurbo software is just a model of reality, you need to know about where it doesn’t work in order to use the model effectively. If you use the model outside of where it has relevance, then you’ll get nonsense answers. Much is made of this by Nassim Nicholas Taleb in his book The Black Swan, which I recommend.

I’m curious about what the limits of the model are, and if it’s smart enough to detect where the system might be becoming unstable.

For example, does it include transaction costs in its model? Moving a VM from one place to another, much like context switching in the kernel, takes time. Constantly moving a VM from one host to another because it’s almost at equilibrium would be bad. A transaction cost – moving to another host costs $x – would take this into account, and the VM would only move if the price of resources and the transaction cost was worth it.

And what about adding new resources? Does everything madly move about if you bring a new host (and all this extra CPU, RAM and maybe storage capacity!) online, or do the resources move more slowly? Flash crash anyone? Suddenly dumping a bunch of resources into the market would cause a price shock, so does the system have a damping mechanism to stop it over-reacting?

Utility Modelling

In economics, how much you value something is modelled as something called utility. If I want a spoon more than you (say I need it to eat my soup, but you have a sandwich so don’t care about the spoon), the spoon has greater utility for me. Once I’ve finished my soup, the spoon’s utility drops to near zero, though, because I no longer need it.

How does the VMTurbo software decide how much utility a VM has for CPU, RAM, and storage? It has to be based on how much humans value the system, doesn’t it? This VM is more important because it runs the payroll system, but the one running the DevOps Minecraft server isn’t as important.

But now we don’t have rational actors making decisions any more. We have pointiest hair in the room deciding which system is the most important.

Not a Free Market

Incidentally, this isn’t really a free market with an “invisible hand”. The market is firmly guided by rules about who can trade with whom, and when, and how the prices are set by the definition of the utilisation-to-price curve(s). In natural markets, the utility of goods is driven by human desires and wants, irrational though they may be, but not centrally designed by the coders of software. Price elasticity is an emergent phenomenon, not one built in from the start.

The VMTurbo Economic Scheduling Engine is a model of a heavily regulated market exhibiting complex behaviour, using the computational power of computers and software to find points of equilibrium in a complex system. Points that are assumed to be, in some way, ‘optimal’.

I’m somewhat unnerved by the breathless prose on VMTurbo’s website, e.g. “with VMTurbo, your data center may be a perfect market. Your workloads and infrastructure are the ultimate rational actors…” Bollocks. Markets aren’t magic, and VMTurbo president Schmuel Kliger is over-stretching the analogy in this Tech Field Day video. How do “sellers”, i.e. servers or storage systems, add capacity to an environment if resources are globally constrained? Does your NetApp array email your channel provider and place an order for a new disk tray automatically? That’d be neat if it was actually happening, but I don’t think that’s what Mr Kliger actually means is going on.

Which bothers me, because it’s already a very cool concept without having to go all Free Markets! Woo! USA! USA! That sort of carry-on makes me suspicious.

An Invisible Hand

A note on VMTurbo’s use of the “invisible hand” metaphor. It’s only used once in Adam Smith’s An Inquiry into the Nature and Causes of the Wealth of Nations (Yes, I’ve read it, and I dug out my copy to check the quote), in book IV where Smith is talking about people investing their money domestically instead of in foreign markets. Here it is in context:

“By preferring the support of domestic to that of foreign industry, he intends only his own security; and by directing that industry in such a manner as its produce may be of the greatest value, he intends only his own gain, and he is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention. Nor is it always the worse for the society that it was no part of it. By pursuing his own interest he frequently promotes that of the society more effectually than when he really intends to promote it.”

Not quite the “free markets are perfect and good” idea bandied about these days, is it?

Onwards to VFD4

I will definitely be paying attention during the VMTurbo presentation. I hope they read this post and come prepared for my questions!

Bookmark the permalink.