EMC XtremIO

This is my review of XtremIO based on EMC’s presentation at Storage Field Day 5.

Getting a clear picture of what XtremIO was about from EMC’s presentation was a significant challenge. It’s taken me a few re-watches of the video, and triangulation from other sources (including Solidfire’s presentation, ironically) to figure out how it works.

XtremIO ‘X-bricks’ are dual-active controllers connected to 25 eMLC flash SSDs in a separate drive shelf. The logical path to the SSDs has two logical lookups: firstly its metadata (such as the ultimate location of the data block) and the data itself. Data (and metadata) is stored on the SSDs using a form of wide-striped 23+2 parity-RAID, which EMC call XDP because they don’t want to call it RAID for some reason. Differentiation from competitors, I assume. It looks, walks, and quacks like parity-RAID to me.

The controllers are connected to each other over a fully-meshed RDMA network running over Infiniband, which is nicely speedy. It does limit the size of the cluster to the number of ports on the largest Infiniband switch on the market, but that’s not a big deal, because the maximum cluster size is currently 4 bricks/8 controllers.

The full-mesh architecture is a big part of where the consistent latency story comes from: the path length to each data block is the same no matter how you reach it, meaning the same number of hops, which provides the same latency (assuming all hops perform the same, which they should).

Of concern, though, is that adding bricks to the cluster is disruptive. Having to take the entire storage system offline for an upgrade is far from ideal, particularly in this day and age, and particularly for something that bills itself as a “scale-out” solution.

In fact, because of its tightly-coupled architecture, XtremIO is far more like a scale-up solution than scale-out. It’s the elasticity of being able to add, and remove, components that makes a solution ‘scale-out’ more than any other feature, in my opinion. This is generally achieved through some sort of shared-nothing style architecture, while XtremIO is more a ‘shared everything’ architecture because of the RDMA fabric, and the fact that data is only stored in one place. In my opinion, XtremIO is a lot more like a Symmetrix than a scale-out system, but let’s not get too hung up on that point.

No Compression

Another missing feature is compression. As per Solidfire’s presentation, this is tricky for XtremIO to do because of their choice of fixed 4k blocks for their RAID stripes, but one assumes they’re working on it. It’s not actually clear to me how much of an advantage compression is compared to deduplication, as I’ve not dug into the details of how other vendors do this, and how it performs in real world situations. There’s marketing material out there, but vendors will always put a positive spin on their own thing.

Snapshots

Snapshots (and clones) are brand new for the platform (they were due to be announced at EMCWorld), which is a little odd, given that snapshots and clones are a core storage array feature these days. The implementation of snapshots on XtremIO also seem a bit odd, and sound a lot like the way NetApp snapshots work in ONTAP 7-mode. XtremIO have traded space efficiency for speed: snapshots are fast, because they’re just a pointer to a metadata list. Snaps of snaps are also fast, because the pointers are partial lists.

The only issue happens if you delete a snapshot in the ‘middle’ of a tree of snaps. That removes pointers from the middle of your indirected pointer list, so you have to go through all the child snaps of the snap you want to delete, and update the pointers for all unchanged blocks to point at the snap’s parent blocks. It’s like cutting a branch at the fork, removing one branch of the fork, and then gluing the rest of the branch back on. Doing this merge is a background process, and is apparently not strictly necessary, plus the metadata is all held in DRAM, so it’s pretty fast.

Replication

Array native replication is also not available on XtremIO. If you want replication, you need to use VPLEX in front of XtremIO. The XtremIO team are apparently working with the RecoverPoint people to figure out how to put RecoverPoint techniques into the XtremIO software so you can do it natively, but it’s vapourware today.

Optimisation Means Choices

I actually think Dave Wright’s summation of the choices made by the different vendors is a great way to look at how the different systems he walked through work, and why. XtremIO’s architecture looks the way it does because of the choices they made to optimise in a certain direction. That makes it different from other solutions, but not necessarily better. It might suit certain use cases better than an alternative solution, but it won’t be a good fit for all use-cases.

Unfortunately, during the presentation the marketing message of “we’re the best!” eclipsed the fact that you can’t optimise for everything simultaneously. Too much of the message from EMC was that other choices are inherently, and objectively, bad rather than different choices made to optimise in a different way.

XtremIO have optimised for predictable low-latency block storage performance. The maximum capacity of the system isn’t as large as some other offerings, but that’s not what it’s for. (Tape is much better, but the latency sucks). The choice of full-mesh RDMA fabric, and fixed 4k block parity-RAID provide consistent latency. Fast Inifiniband interconnect, DRAM storage of metadata, wide-striping, all these choices contribute to fast performance. Time to market was also important for XtremIO, so some of their choices (UPS backed DRAM, shared disk) have been made to get a product out there and selling quickly.

XtremIO look reasonable if you want predictable, low-latency performance for block storage in the 10-80 TB capacity range, and you’re pretty sure how much performance and capacity your application(s) will need, and you have the money to spend (or know how to drive a hard bargain). I suspect that it might also turn into a good choice if you’re already an EMC shop and want something that will work well with all your existing EMC products and tools. While that tight integration isn’t there today, it’ll probably arrive in relatively short order. If the integration is important to you, I’d be asking for a roadmap briefing, possibly under NDA, to understand if EMC’s roadmap aligns with yours.

For everyone else, though, XtremIO just doesn’t seem really ready yet. It feels like it was rushed to market, but has also been slowed down because they got bought partway through their product development cycle and had to suddenly integrate into the Greater EMC Federation.

Actual Customer Experience

Chris Gurley wrote up his experience with using XtremIO in a comprehensive and fair way. It looks like the impressions I got from SFD5 bear out in reality.

To be honest, Chris’s review makes me even more critical of XtremIO. The operational hassles of managing the gear make it unsuitable for anything but a workload that desperately needs the stable performance. The business value would have to seriously outweigh the costs of working around XtremIO’s limitations, and I’m not convinced it’s the only option.

XtremIO is now firmly in my “wait and see” bucket until they get the base level features up to parity with virtually every other modern array out there.