IOPS, Spinning Disk, and Performance – What’s the catch?

Black Diamond
For a quick introduction – IOPS means Input Output (operations) per second.  Every hard drive has certain IO performance.  So, forgive the oversimplification, add additional disks, one gets additional IOPS which means one gets better performance.

Now, generally speaking, I hate IOPS as a performance characteristic.  I hate them, because, IOPS can be read or write and sequential or random and of different IO sizes.  Unless one is trying to tune for a specific application and is dedicating specific disk drives to the application, the measurement breaks down as the description of the assumed utilization lacks accuracy.  For instance, assume that it has random reads & writes, but then the backups kick off and that ends up being a huge sequential read for long durations.

But, I digress.

20130625-211646.jpg Every hard drive has an IOPS rating whether SAS, SATA, or FibreChannel or 7200, 10000 or 15000 RPM.  (see wikipedia for a sample).  When a RAID set is established, drives of the same geometry (speed & size) are put together to stripe the data across the drives. For simplicity sake, lets say one uses a RAID5 set with 6 drives:  that is the capacity of 1 drive is used for error (parity) checking and 5 for data.  And continuing the example, assume that these are 1TB (terabyte) drives with 100 IOPS per drive.  So, one has 5 TB of capacity and 500 IOPS.  [Let’s imagine these are read IOPS and not write, so I don’t have to get into parity calculations, etc. etc.].    If I could add a drive to the RAID set, then I get another TB and another 100 IOPS.  Nice and linear.

And, my IOPS per TB are constant.  [Again, to simplify, I’m going to assume that it falls in the same RAID set and so I don’t have to consider more parity drive space].  So, none of this should be earth shaking.

The huge implication here is:  To increase performance, add more disks.   The more disks, the more IOPS, everyone’s happy.  However, that assumes that consumption (and more importantly IOPS demand) has not increased.  The graph on the right looks consistent with the graphs that we saw earlier.

20130625-211653.jpgThe problem here is that if one adds disks, which adds capacity, and then that capacity is consumed at the same IO rate as the original disk space, the performance curve looks like the graph on the left.  If I’m consuming 100 IOPS per TB and I have 5 TB, that is 500 IOPS of demand.  So, I add a 1TB disk and now I have 600 IOPS w/5TB of used capacity on 6TB of disk.  So, I can spread that out and yippie, those 5TBs can get 120IOPS.  But, if I also say, “hey, I got another TB of disk space” and then consume it, then I’m back to where I started and am still constrained at 100IOPS/TB.  So, what good is this?

20130625-211659.jpgThe assumption is that one is adding to a heterogenous array i.e. multi-purpose (maybe multi-user or multi-system).  So, by being multi-purpose, the usage curve should hopefully become more normalized.  If the usage is more homogenous, e.g. everyone who needs fast performance, so we move them from the slow array to the fast array – well that just means that the fast users are competing with other fast users.

Just like on the NASCAR track for time trials, if I have one race car start and then send another race car when the 1st is half way around the track, I’m probably not going to have contention.  If one customer wants high performance in the evening and the other in the business day, I probably have no contention.

However, on race day after the start, all the cars are congested and some can’t go as fast they want because someone is slow in front of them – gee, and we moved them off the freeway onto the race track for just this reason.   Well, on the storage array, this is like everyone running end-of-the-month reports, well, at the end-of-the month.

I need another analogy for the heterogenous use.  Imagine a road that one guy uses daily, but his neighbor only uses it monthly.  However, the neighbor still needs use of a road, so he pays for the consumption as well.  Overall, there may not be conflict for the road resource – as opposed to, if both used it daily.

So, yes, overall – adding disks does add performance capacity.  And without knowing usage characteristics, the generality of adding disks still holds.  Why?  Because no one complains that the disks are going too fast, they only complain when it is too slow.  There is still the mindset that one buys disk for capacity and not for performance.  And then once performance is an issue, the complaints start.  So, adding disks, to a random workload means that the bell curve should get smoother over all.  This won’t end all the headaches, but should minimize them by minimizing the number of potential conflicts.

Let me know what you think

Replication Methodologies

Blue SquareFrequently, when one tries to determine a disaster recovery transfer methodology, one gets stuck on the different methods available as they each have their strengths and weaknesses.  I’ll ignore backups for the time being – backup images can be tapes shipped by truck from primary site to recovery site, or backups replicated by backup appliances.  The focus will be on replication of “immediately” usable data sent over the network from primary site to recovery site.

Types of replication:

  • Application Replication – Using an application to move data.  This could be Active Directory, Microsoft SQL Server, Oracle DataGuard, or other application specific methods
  • OS Replication – This typically uses some application within the Operating system and used to ship files … this could be something as simple as rsync
  • Hypervisor Replication – For those that run virtualized environments, replication occurs at the hypervisor layer.  This could be VMware vSphere Replication
  • Storage Replication – This is when an array replicates the data.


With Application Replication – one can be confident that the application is replicated in a usable form.  Without quiescing the applications, applications may not be entirely recoverable.  (quiescing is the process of placing the application in a state that is usable – usually, this consists of flushing all data in RAM to disk).  However, the question that inevitably comes up is:  “What about all the other systems I will need at the recovery site?”

With OS Replication – the application that moves the data, tends to be fairly simple to understand:  copy this directory structure (or folder) to the receiving system and place in such and such folder.  However, with this methodology, the question that arises is:  “What about the registry and open files?”  If the process just sweeps the filesystem for changes, what happens to files that are open by some other application (i.e. not quiesced — could be a problem with database servers)?

With Hypervisor Replication (for virtualized environments) – The hypervisor sits underneath the guest operating system(s).  This has the advantage of catching the writes from the guest.  To write to disk, the guest has to write to the hypervisor and then the hypervisor writes to disk.  The data can then be duplicated at this level.  For this method, the inevitable question is:  “What about by physical machines that haven’t been virtualized?”  One can still have issues with non-quiesced apps.

With Storage Array Replication – The array sends to its peer at the remote site.  The advantage of this method is that it can handle virtual or physical environments.  The disadvantage is that an array on the remote side must match the protected site.  This can be problematic, as frequently, sophisticated arrays with this technology tend to be costly.

Regardless of the methodology, there has to be coordination between the two sides.  The coordination tends to be handled by having whichever facility at each end provided by the same vendor:  e.g. SQL server to SQL server, VMware vSphere Replication appliances to vSphere Replication appliances, Storage vendor like NetApp to NetApp.  [I’m ignoring Storage Hypervisors and just bundling those into the Storage Array layer.]

Every method has its own set of rules.  Imagine I had a document where I wanted a copy across town.  I could:

  1. walk it over myself – but I would have to follow the rules of the road and not cross against red lights
  2. e-mail it – but I would need to know the e-mail address and the recipient would need to have e-mail to receive it.
  3. fax it – but I would need a fax machine and the recipient would need a fax machine and I would need to dial the number of the recipient’s fax machine.
  4. send it with US postal – but I would need to fill out the envelope appropriately, with name, address, city, state.
  5. call the recipient and dictate it – but I would need the phone number and the recipient needs a phone.  I would also need to speak the same language as the person on the other end of the phone.

These simplistic examples that we use every day, are used here to illustrate that there are certain norms that we are so familiar with that we ignore that they are norms.  But, without these norms, the communication could not be done.  Likewise, users should not be surprised that each data replication method has its own requirements and that the vendors are picky as they wish to define the method of communication.


Rev 1.0 06/25/13

What’s all this “cloud” stuff? What does it mean?

Green BallIn the old days (I don’t really recall when exactly), techies used the term “cloud” – and back then it meant the network, specifically the WAN (wide area network). So, we would draw a picture on the board showing site A and site B and in between was this cloud (the network cloud). We didn’t know how we got from A to B and didn’t care — or if we did, we knew that the vendor, e.g. AT&T, could change it.


Now, “cloud” has a new meaning, stemming from cloud computing. One can now have application-as-a-service, platform-as-a-service, or infrastructure-as-a-service. An example of application-as-a-service cloud computing before such terms existed: webmail. Once, we stopped downloading e-mail from e-mail servers and just used some web interface from Yahoo or Hotmail, and the like, we moved into Cloud Computing. Those web servers are somewhere and can be moved without us knowing it.


So, “cloud” is just a way of saying, the technical resources/service is out there and the service is what is important not the “where”.

I’ll leave the different types of cloud computing – application-as-a-service, platfrom-as-a-service, and infrastructure-as-a-service for another day… Comment if there is specific interest…

Jim Rev 1.3


What’s the point?

I draw pictures. I draw a loooottttt of pictures. As an architect, I’ve learned that I can hold the audience’s attention longer with a one page drawing than I can with a 4 page (or 20 page) document. Also, I’m a simple guy. If I can draw it, I can understand it.

Whether I’m dealing with fellow technical architects or non-technical business folks, I find pictures and analogies help aid the understanding.

This blog is an attempt to simplify concepts for my colleagues at work, former colleagues from school (either UC Irvine where I got my undergrad or Leeds School of Business where I got my MBA) or whomever.

So, if you have ideas or questions, send them my way…