Using VMware I/O Analyser To Test Storage Performance

Today i've been experimenting with the VMware I/O Analyser as a useful tool to drive storage performance from a fairly even baseline of tests.

The tool essentially is an Ubuntu VM with IOMeter and a web front end (which is OK, but doesn't help much), wrapped up in a nice OVF package.

First step is to download the package, which is available from the VMware Flings website (http://labs.vmware.com/flings/io-analyzer).

Next we need to push the OVF into the environment. As this is the controller VM we are not wanting to monitor the backend performance stats of the VM itself, so it should NOT be published on the datastore that we wish to test. Instead, publish the VM onto a local datastore or another shared volume. This is mostly due to the workload being generated may saturate the disks, SAN controllers or network which could affect this VM and may be detrimental to your test and not provide a fair result. 

In my case "Nimble-IOAnalyser" was my local datastore, and VMFS-01 was my datastore to be tested:

Select a datastore which is NOT to be tested by the I/O Analyser for your OVF deployment.

VMware's best practice for deploying Virtual Machines is to deploy the disks using "Thick Provision, Eager Zero" as a profile, so go ahead and use that.

Once the VM has been deployed it's key to take a look at the settings and at the disks it's created. What you'll notice is that the test disk (Hard Disk 2) has been created with only 100MB as a size; which means 100% of the testing will reside in memory or in controller cache. This is something we need to avoid as it does not provide a fair and real-world result. This Hard Disk is also provisioned into the incorrect datastore by default.

The default testing drive needs to be deleted (Hard Disk 2)

Delete this disk and create a new one with at least 100GB as it's size. This disk should also be placed on the datastore for which you wish to test the performance of, and again use "Thick Provision, Eager Zero".

Once deployed, power the VM on. You should be greeted with the familiar VMware appliance screen. Change your timezone to what's relevant to you (GMT for me). You could also change the network settings here, but I left mine as DHCP as it's not really needed. Login to the VM with the default credentials of root/vmware to continue.

Note: we're not going to use the web client provided for this tool, as whilst it's ok it's not possible to change any of the default values in the I/O testing phase of this workload.

Once logged in you'll see an Ubuntu screen with a terminal window open. Right-Click on the desktop and open another (Terminals->xterm).

In the first terminal, you want to type "/usr/bin/dynamo". This starts the backend IOMeter worker and thread engine.

In the second window, type "wine /usr/bin/Iometer.exe". This will open the IOMeter application, and should tie into the dynamo engine started.

Note: ignore the comment underneath of "Fail to open kstat...", as long as you opened the engine before IOMeter it'll be OK.

From here onwards it's down to how you use IOMeter. There are lots of incorrect methods to using this tool to provide expected results, so here are my settings:

• Two workers, both mapped to the new 100GB volume (sdb).
• Maximum disk size of 200000000 sectors (translates to 95GB).
• 32 Outstanding I/Os per target (if this is left to 1 the test is not adequately driving the storage array!).

• A new workload policy is created for 4K blocks. 100% random workload, and 100% write. (100% write should ALWAYS be the very first workload to the volume, otherwise there is no data to actually read back which does not provide real-world results).
• Align the I/Os to the block size of your volume to remove disk misalignment performance discrepancies. In my case it's 4K.
• Assign this workload to each of your workers to ensure you're consistent with your tests.

Some people may ask why 100% random; if you wish to test for IOPS performance in your storage, random IO is the key to generate these statistics. If you wish to test network performance (rather than IOPS) then Sequential I/O should be used. You should NOT mix these workloads together as you will be given inconsistent and inconclusive results for both disk and network stats.

• Before running the test ensure you set a ramp up period (I use 15 seconds) and set a standard run-time (I use 10 minutes). Ensure all your workers are selected.
• Click the Green Flag to start the test!

I hope the above has been useful to you. Please feel free to run this test and let me know your stats in the comments box on this page, along with the make/model of your storage array - would be a fun survey!

PS - if you wish to see my results:

Nimble Storage 2.0 Part 1 - Scale-To-Fit

This is the first in a series of blog posts focusing on the new Nimble OS 2.0 which has been made available in Release Candidate status to our customer & partner base.

Nimble Storage has hit a milestone in it's product development and feature set by hitting 2.0 of it's awarding winning Operating System based on the CASL file system (Cache Accelerated Sequential Layout). 2.0 introduces some nice features which i'll cover over a series of blog posts in the following weeks.

Scale-To-Fit - Nimble OS 1.4

Approximately 12-14 months ago Nimble launched a concept of "Scale-To-Fit" storage technology. The idea being that traditional storage systems had limited ways for expansion once the customer had deployed the environment; these typically being:

1. Add lots more slow disk for capacity or fast disk for performance (very common, not much risk involved).
2. Forklift replace controller headers for new generation controllers to allow for further disk expansion (very expensive, high risk due to potential data downtime or loss, lots of Professional Services involved, rarely done).

Scale-To-Fit was designed to break this mould and give the customer a choice in how to upgrade their investment as their environment evolves. This was released in firmware version 1.4 of the Nimble OS, and allowed any Nimble customer to do the following:

1. Add additional shelves of high-capacity drives for extra storage capacity.
2. Add bigger SSD drives into the array to allow for bigger working sets for random-read cache.
3. Upgrade Nimble controllers from CS200 to CS400 series to add more IOPS in the array.

All of the above upgrades were (and still are) designed to be performed online, without any maintenance windows or downtime of data - and it worked without any hiccups, which is very impressive (especially number 2 & 3 - see here for how our very first production customer with a 3 year old system did it this year: http://www.nimblestorage.com/blog/customers/non-disruptive-upgrade-to-nimble-storages-first-deployed-array/).

So what does 2.0 bring to Scale-To-Fit?

In Nimble OS 2.0 we complete the Scale-to-Fit strategy by bringing Scale-Out technology into the Nimble family!

Scale-Out is a technology which was mostly championed by EqualLogic (now Dell, my former employers) and Lefthand Networks (now HP) and is designed to allow customers to place multiple arrays into a "group" which allows volumes and/or datasets to be distributed, balanced and "live migrated" across different storage platforms without downtime. 

The downside to the two legacy technologies mentioned above was that scale-out is/was their only way of scaling. If a customer required just more capacity for their investment they had to buy a whole new array with controllers, networking and rack space just for a few more additional TB's of space - and these boxes were typically more expensive than their initial purchase due to less aggressive discounts offered (the age old sales ploy of discount high to win business then discount low to increase margin - seen it time and time again). One of my old EQL customers ended up with TWENTY THREE 3 or 5u arrays in their production site!

What Nimble offers is a full suite of scaling technologies without any gotchas; so if a customer just needs capacity they can buy additional shelves, but if they want to Scale-Out to group performance and capacity of their arrays together, they can do that too! All of this can be done live, without any downtime or any professional services work! Nice!

The beauty about scale-out is it does not limit customers to generations of gear; meaning that older arrays can be grouped together with newer, faster generations for seamless migration of data before evacuating the older array to repurpose for UAT, Disaster Recovery or other means (as one example).

On first release we are supporting up to FOUR Nimble arrays in a scale-out group. These can be of any capacity or generation, have any size of SSD, HDD or controller!

Note: This does NOT mean we are clustering arrays together - we do not need a special backend cluster fabric to handle array data which you may have seen with out vendors implementation of scale-out or cluster-mode. We also do not require downtime or any swing-kit to move data off an array to enable this new feature.

The new "Arrays/Groups" section of the Nimble GUI

Scale-Out is the last piece of the Nimble Scale-to-Fit strategy, meaning customers who started with a single 3u, 500w array can now add more capacity (an additional 96TB usable using 3TB drives), add bigger MLC SSD drives for bigger cache working sets (up to 2.4TB usable), add bigger controllers to take their array from 20K IOPS to 70K IOPS, and now add additional Nimble arrays for a single management point & more performance, capacity and scale!

Nimble OS 2.0 is currently in Release Candidate stage and is available for customers to upgrade via Support. The code & technology is fully production-ready and has been through extensive beta and QA testing. It is the same process to upgrade the firmware as previous updates; a software package is downloaded from Nimble Support then applied to the array controllers one at a time (so no downtime!). If you'd like to be a part of the RC rollout please contact Nimble Support for more information.

Alongside Scale-Out we are launching some new tools for VMware and Microsoft Windows platforms to simplify the overall integration of these solutions; stay tuned for my blog posts on these features!

Cutting Through Storage Marketing & Spin

This is a very lengthy post; for that I apologise in advance! But this is something I feel very passionately about...

It's been a very rowdy few days in the world of Enterprise Storage and technology announcements! 

Typically July and August are very quiet times in IT mostly due to Summer Holidays/Vacations inside IT departments, resellers, ISVs and vendors alike (certain countries in mainland Europe even go so far as to shut down completely for 4-6 weeks during this time!). 

However this year something strange has happened; a wave of mystical product launches, PR statements and marketing/blogpost overdrive has been visible from lots of the start-up or hyper-growth storage companies eyeing your business and storage projects.

The theme seems to be consistent across the board; 

1. We are far cheaper than everyone else
2. We bring MILLIONZ more IOPS/performance than everyone else
3. Why would you buy anything else, when we're around?

From reading these PR statements and blog posts from multiple vendors it suddenly dawned on me that if I was slightly bemused by the statements, how on earth would potential customers and partners be able to differentiate and sift through the marketing statements to relate the product to their environment or requirements?

So, without further ado - here are some of my top tips to help navigate through the realms of marketing claims for end-user environments:

1. Ensure your requirements & needs are documented and ranked in importance from 1-10.

When I meet customers they typically have an immediate need for a prospective storage solution (i.e. storage for VMware vSphere and Microsoft SQL, for example). However other nice-to-have requirements may be to use application consistent snapshots for backups, or replication for disaster recovery, but perhaps a feature such as NAS connectivity is not something on the agenda right now.

Recently there are a few products coming to the table with products that literally have EVERY feature you could ever want or think of with no additional cost. Whilst this sounds great on the datasheet, in the PR statement or blogposts, often these solutions have a poor implementation of these features as development may not allow for deep engineering and full Quality Assurance testing. It truly is the idea of perhaps offering too much (and thus rarely excelling at not a lot).

By ranking by importance your requirements and features you need in a prospective solution it will allow you to move on to number 2:

2. Always trial, test (and score!) your prospective vendors before you make a purchase decision.

Once you've been able to rank your requirements and feature sets, this then allows you to form your testing criteria for any Proof of Concept evaluation you decide to run.

Time and time again I'm asked to run vendor bake-offs / POCs against competitors, but the end-user has not been able to draw up any set pass/fail criteria for the bake-off testing; opting to trust the vendor with their generic (and more often bias) test plan and criteria for any POC testing.

This is a very dangerous situation as it allows the hardware/software provider to shape and manipulate your tests to suit their best features whilst hiding all problematic or half-baked feature sets which may be a show-stopper if it were to be exposed.

Being able to rank and understand requirements from a potential storage environment upfront will allow the creation of a non-bias POC test-plan document when the time comes, which ultimately ensures that the test is 100% fair, and does not allow any vendor to influence and manipulate said testing in a way that they are almost guaranteed to win hands-down in most tests.

This also allows you to move on to number 3:

3. Never, EVER 100% trust performance tests run by vendors - run your own!

When running performance bake-offs every vendor will be cunning with how the performance test is derived. 

Most typically a tool such as IOMeter or SQLIO will be used to demonstrate performance of the product as the first thing after the storage array is installed in the environment. Whilst these tools are great (and will almost always report great results) it's often run when the array has NO data on it whatsoever, so the array is 100% optimised to deliver the best possible result as a performance test (for example, ZFS based arrays will dramatically degrade in performance once data capacity starts to be used but will show mind-blowing IO results when empty).

Another thing to consider is that a production environment will require a mixture of IO block sizes for various applications, whilst IO profiles (reads or writes, random or sequential) are often a variable mix every single day. It's a true cliche, but there's something called the IO Blender (especially with virtualised workloads) where one cannot guarantee whether the environment may drive random or sequential reads or writes, and the profile of the IO may change without warning. 

99% of all storage array vendors will NOT take the above into account as part of a POC, opting to show purely random reads or writes using small (4k) blocks as a single request to the array to give the best result. 

So it's very important to mandate that you run performance tests with YOUR data and yours alone. This ensures that you can understand the performance of the test today (on a current solution) and would be able to quantify the increase/decrease in performance once it's on a new solution. This is a far better test than being shown an IOMeter screen pushing 100,000 IOPS and running 20,000 more than the competitor's box.

4. Flash storage for $/GB less than Spinning Disk? Yeah Right.

Another interesting statistic that's been observed recently from some of the All Flash Array (AFA) vendors is the claim of having lower cost per Gigabyte ($/GB) than spinning disk media.

Everyone knows one day the above will be true, however i'm afraid right now this statement is another pure-marketing claim.

When digging more into this claim i've noticed that these claims are only true when taking into account large compression and de-duplication savings of the data set vs uncompressed & un-deduplicated data on the spinning media. 

This is a) assuming that the current spinning disk array does not have any compression or de-dupe running on it today, but more naively b) assumes that the dataset is highly compressible and would be susceptible to large de-reduplication reductions - which a large amount of production environments are not.

For example; I recently encountered the above claim in a recent POC bake-off between a series of storage vendors. The customer had their head turned by the claims of major cost reduction savings of an All-Flash Array, promising that it would be the same cost (or even less than) the spinning disk or hybrid storage guys. However the customer was wise in their POC criteria; who mandated they would copy THEIR production data and working set to the array and determine exactly the saving in primary space we will see for de-dupe and compression on the array. 

It was observed that whilst the AFA vendor promised reductions of 6-10:1, the real-world testing only delivered 2-3:1 reduction of data (with compression bringing almost 2x more reduction in space than the de-dupe on the system, which is not uncommon to see) - meaning the end result was savings were not near the marketing claims and thus being almost 2x more expensive than the competition.

Storage array vendors call the reduction in primary data needs as "Effective Capacity", and are starting to size solutions based on these figures. Whilst it sounds great in a presentation (who wouldn't want to get 300TB from 30TB of disk!?) it's very often the case that in true environments the figures are wildly off.

The moral of this point is simple; Never, EVER size to Effective Capacity (regardless of what your storage array vendor tells you) as it's never a guaranteed figure that you'll get. And so ALWAYS use Usable Capacity, and any primary storage reductions seen from compression and/or de-duplication is an additional bonus. 

I feel there could be a future blog post in this topic...

I hope some of the above tips are useful and have provided food for thought. I have more recommendations on how to navigate through the minefields of Enterprise storage today, but hopefully the above is a good starting point.

Nimble Storage - Two Weeks In

As promised, I figured that my first "proper" blog post would be about something useful; so why not start with my thoughts & experiences with my new employer Nimble Storage!

If you're not aware of who Nimble Storage are, the following paragraph from  Marketing sets the scene:

"Nimble Storage has created the industry's first flash-optimised storage engineered from the ground up for maximum efficiency. Nimble Storage solutions integrate the exceptional performance of flash with the cost advantages of high-capacity hard disk drives—making it ideal for storing and protecting data for critical applications such as Microsoft Exchange, Microsoft SQL Server, server virtualisation, and VDI."

I had heard of Nimble from quite a few avenues both on and offline over the last couple of years (huge unsubtle nod to Tech Field Day). The organisation came out of stealth in 2010 (founded by some of the clever fellows behind Data Domain, Netapp and others) with the intention of breaking the link between buying lots of spinning disk and the performance associated with said amounts of spinning disk. That subject is another blog post altogether…

At my previous employer (Xsigo Systems) we had hired quite a few ex-Data Domain teams in the US in 2009/2010. Within a period of 3-4 months in 2010 they had all left to join this new organisation called Nimble Storage. These were some of the most successful and technically apt people that I had worked with in my IT career and so the company was immediately put on my radar from that point on. Of course, being an ex EqualLogic/Storage bod also meant that it was something I could get excited about.

Nimble Storage has been in the EMEA market now for just over a year (operations started in Nov 2011), and there has already been 100 arrays purchased and shipped into the market during that time. That is SOME start. And the ambitions for the company meant that the team needed to scale quickly with expertise from sales & techies that had been there & done that with iSCSI storage, VMware & the channel.

November 19th was my official start date for the company, and I was the 3rd "SE" (Systems Engineer) in the UK. Exciting stuff!

So - as promised, my experiences and thoughts about the company & technology after two weeks. This is in no specific order exactly:

1. The technology is something truly special. I'm a geek at heart - and I need to be excited and enthused about the technology that I'm promoting. I loved EqualLogic, and too a degree I loved Xsigo. This is another product which makes you sit back and think "wow, how have they done that", rather than working for HP/Dell/IBM/EMC where it's mostly bolt on's to the same old technology which has been around for donkey's years.

2. The team I work with and report into are some VERY clever guys. I'm talking about the original management & sales structure that took Netapp to it's success in the early 2000's, and then Data Domain likewise in the late 2000's. Some very well respected and intelligent people are here in sales, marketing, engineering and at management/board level - as apposed to your traditional employer where there is a mix of good guys and coasters/lifers.

3. It's all about the channel. This is something I feel very strongly about, especially as a small startup trying to address the frankly massive UK market. The channel partners are the key to whether said organisations succeed or fail, and it amazes me how many companies believe they can run either a poor channel program or run a Direct Touch model with the market. It doesn't scale, it causes friction with the partner market & ultimately it makes your job 100x harder. So I was delighted to hear that Nimble is 100% channel driven, with the focus on investing in said partners to make them self sufficient to be able to present, demo & install for customers. Doing this rewards the channel & makes them more valuable to the business and customers - who see's value in a middleman pushing a PO, shifting some tin and making margin from that?

4. It's winning. A LOT. I have been here for approximately 13 working days (at time of writing) and the amount of internal 'win alerts' i've seen are staggering - 88 separate customer wins in 13 working days. The business has amassed over 800 customers now in just over two years of selling. I think that says it all really. From looking through those wins, 60-70% of them are where we've directly competed with either EMC or Dell EqualLogic/Compellent. The reasons are around performance, cost, functionality, or snapshots/backups. Excellent!

5. First deal about to close. I don't think i've ever worked at a place where the technology is so compelling (and at the right price) that it can win business in days rather than weeks or months. However it *could* happen here (big thank you to Dell Compellent for being as crap as it's always been for making the customer look elsewhere). Obviously there will be times when we will lose deals to the competition on either features or price, or even politics/hidden agendas. But if it's a straight up POC on functionality, performance, snapshots or replication against the mid-tier competition, I can't see it losing much of the time, and that fills me with confidence. Right product, right time.

6. This IS hard work. Make no mistake about it - working at startups is NOT easy. There are all sorts of internal headaches as well as external perception/marketing/pricing/channel subjects that are tough to navigate. HP, Dell and especially EMC sell so much tin purely because of a badge and name in the market. Sad but true.

So there we go. My first blog post, about why I joined Nimble, and what excites me about the opportunity of being here after the first couple of weeks on the job.

If you have questions, comments, abuse or recommendations for future topics please feel free to leave them below.

Cheers!

Nick

Hello World

So here we are. December 2012 and I decide to open up my own blog site.

I genuinely thought that this day would never come; I never truly believed that I would become one of those people that write about their experiences on a self-indulgent website. Not that there's anything WRONG with that, at all.

I have huge respect for people that write their experiences & gotchas online for all to see. I have used this information for years to discover the latest and greatest releases from my competition (vendor Press Releases are seldom useful and actually contain the information needed about what they're writing about!), for example. Or new releases from my partner vendors who are releasing new features or gotchas about a product I was interested in. I just figured that I would never have time (or the patience) to create postings (with diagrams) about my experiences in the field.

However, I have bitten the bullet, and I'm going to give it a go. Please bear with me. I'm learning on the job here. But I intend to fill this space with useful findings from my day job and interests.

My first "proper" post will be about my first two weeks at my new employer "Nimble Storage". Expect it imminently!