Notes from 2018 VMWorld

Day 1 kicked off with the announcement of AWS’ DBs running on vCenter.  This is to dovetail with VMware in AWS.  Trying to figure out how this makes sense, allowing AWS to make their move to on-prem.  Makes me wonder if the same thinking here was the same thinking that led the wizard Saruman to make a deal with the dark lord Sauron.

VMware clearly is trying to prepare for a day where there are fewer on-prem workloads.  There is more management with vRealize Cloud in the vRealize Suite.  There is more growth in the replicate to VMC in AWS.

Meanwhile on the WAN front, there is the big announcement of the Velocloud purchase and pulling that into the NSX family.  CEO Pat Gelsinger has that as a major announcement in his keynote following the AWS portion.  I thought that NSX was going to be omitted from mention, but there they did get their spot before the end of the keynote – so it is still moving along.

On the vendor front, they occasionally threw an obvious line to Dell/EMC their major shareholders.  This was especially true when it came to discussion around vSAN.  vSAN seems to have a positive direction as it was positioned as a relevant technology as opposed to the bumps it had in years past where they were touting features over adoption.  When talking about HyperConverged, it was vSAN, and it was Dell/EMC.  And there was the subtle exclusion of Cisco on the slides.  Dell/EMC, HPE, Lenovo were on the compute slides, but not Cisco.

There was a huge portion of Day 2 keynote to social action which Gelsinger mentioned a bit in Day 1.  He specifically quoted and rejected Milton Friedman’s premise that a company should maximize shareholder value. I’m not sure if (1) he really believes this (2) he’s following Michael Dell’s lead being the majority shareholder (3) he’s trying to appease the Millennial crowd who want to see companies doing this – and these are the folks he wants to hire, or (4) some combination.  It is probably some combination that leads to this errant view.  But, there was a Nobel prize winner on Day 2 – though in the overflow room, the audio went out for minutes as they introduced her.  A lot of time was spent trying to come up to speed afterward.

There were two announcements also around vmotion/svmotion on ARM chips and NVIDIA GPUs (not together).

The vendor hall was filled with the vendors you’d expect.  Apparently, t-shirts are back.  At my conferences attending a year ago, it seemed customer dress socks were the rage.  This was my first multi-day conference at Mandalay Bay.  The catered food is better at the Venetian/Sands convention complex than Mandalay’s.  Personally, I spent a fair amount talking to different vendors that are involved in one of my projects to get advice on supportability of an odd use case that I’m dealing with.

The lab environment was done well – held on the bottom floor away from the top two levels which are much noisier (live music, etc.).

The 25,000 attendees made for congested hallways and escalators.  But, I saw more former Denver colleagues there than I do when I’m around town here.

Notes from #NTAPInsight 2014

After a partial week at NetApp 2014 Insight US, here are my thoughts:

(full disclosure:  I was a presenter of one session at the conference)

1. Keynote thought
2. OnTap 8.3 announcement
3. Hybrid Cloud
   1. Data is state-ful, unlike (cloud) computing
4. Data locality
5. Different UNIX variants – Different Cloud
6. Laundry services similar to cloud computing (Jay Kidd / NA CTO)

Tom Mendoza (NetApp Vice Chairman) was fantastic in his keynote.  He focused on culture and wanting to build a culture of trust & candor.  CIOs understand every company is going to have issues, the question will be does the CIO of the customer trust the vendor to be there when there is a problem.

Lots of talk about OnTap 8.3 – though the fact that it is RC1 and not GA is disappointing.   Didn’t hear anyone reference that the 8.3 is a Release Candidate.  8.3 provides full feature parity with 7-mode.  There was little discussion about 7-mode, except for how to move off 7-mode (7-mode transition tool).  7-mode transition still appears to be a large effort.  For, 7MTT, the key term is “tool”.

The key focus in the keynotes was “Hybrid Cloud”.  One of the key takeaways is the need for data locality.  The data is ‘state-ful’ as opposed to cloud computing which is ‘stateless’ — in the sense that the resource need can be metered, but data is not.  So, when moving from on-prem to cloud, data would have to be replicated completely between 2.   Or more so, if you are working between clouds, or maybe between clouds in different countries, the full data set has to be replicated.  The concern is that government entities (Snowden effect) will require data to be housed in respective countries.  This now becomes the digital equivalent of import/export laws and regulations.

With the notion of different clouds, it reminds me of all the different UNIX variants.  We had Solaris boxes and we had HP-UX boxes and we had DEC boxes and we struggled moving data between.  Some were big endian, some little endian.  So, binaries were incompatible.

Finally and irreverently during Jay Kidd’s (NetApp CTO) presentation, my mind wandered when thinking about cloud computing analogies.  Never noticed before how metered cloud computing is so much like washing machines at the laundry mat – pay per use.

 

Jim – 10/30/14 @itbycrayon  (I don’t accept general LinkedIn invites – but if you say you read my blog, it will change my mind)

Notes from #EMCWorld 2014

After days at EMC World, here’s what sticks out in my mind:

1. 3rd Platform Paradigm
2. ViPR (& VMAXce)
3. Electricity model
4. Just like school
5. The Venetian
6. VMware
7. Backups
8. Other folks

Side note:  There was talk of SAP HANA, Hadoop, & Pivotal.  The place that I typically play in is Storage and not that space, so I’m going to ignore their emphasis there.

How to deal with exponential growth rates? And how does this relate to cloud computing?

What happens when demand exceed the resources? Ah, raise prices. But, sometimes that is a not available as a solution. And sometimes demand spikes far more than expected.

Example: Back in the early 2000s, NetFlix allowed renters to have 3 DVDs at a time, but some customers churned those 3 DVDs more frequently than average and more frequently than Netflix expected. So, they throttled those customers and put them at the back the line. (dug up this reference). This also appears to have happened in their streaming business.

Another example: Your web site gets linked on a site that generates a ton of traffic (I should be so lucky). This piece says that the Drudge Report sent 30-50,000 hits per hour bringing down the US Senate’s web site. At 36,000, that is an average of 10 per second.

Network Bandwidth tends to be the resource. Another example from AT&T: As a service provider, this piece says that 2% of their customers consume 20% of their network.

There are non-technical examples as well. The all-you-can-eat buffet is one. Some customers will consume significantly more than the average. (Unfortunately, I can’t find a youtube link to a commercial that VISA ran during the Olympics in the 80s or 90s where a sumo wrestler walks into a buffet – if you can find it for me, please reply).

Insurance customers deal with this as well. They try to spread out the risk so that if an event were to occur (e.g. a hurricane), they don’t want all their customers in a single area. Economists call this “adverse selection”. “How do we diversify the risk so that those that file claims, aren’t the only ones paying in?”

How does this deal with computing? Well, quotas are an example. I used to run systems with home directory quotas. If I had 100GB, but 1000 users, I couldn’t divide this up evenly. I had about 500 users who didn’t need 1MB, but I had 5 that needed 10GB. For the 500 users that did need more than 1MB, they needed more than an even slice.

So, the disk space had to be “oversubscribed”. I then could have a situation where everyone stayed under quota, but I could still run out of disk space.

Banks do this all the time. They have far less cash on-hand in the bank, than they have deposits. Banks compensate by having insurance through the Fed which should prevent a run on the bank.

In computing, this happens on network bandwidth, disk space, and compute power. At deeper levels, this deals with IO. As CPUs get faster, disks become the bottleneck and not everyone can afford solid state disks to keep up with the IO demand.

The demand in a cloud computing environment would hopefully follow a normal distribution (bell curve) for demand. But, that is not what always occurs. Demand tends to follow an exponential curve.

As a result, if the demand cannot be quenched by price increases, then throttling must be implemented to prevent full consumption of the resources. There are many algorithms to choose from when looking at the network, likewise there are algorithms for the compute.

Given cloud architecture which is VM on a host connected to a switch connected to storage which has a disk pool of some sort, there are many places to introduce throttles. In the image below which is uses a VMware & NetApp vFiler environment (could be SVM aka vServer as well) serving, there is VM on ESX host, connected to Ethernet switch, connected to Filer, split between disk aggregate and a vFiler which then pulls from the volume sitting on the aggregate, and then has the file.

Throttling at the switch may not do much good. As this would throttle all VMs on an ESX host or if not filtering by IP, all ESX hosts. Throttling at the ESX server layer again, affects multiple VMs. Imagine a single customer on 1 or many VMs. Likewise, filtering at the storage layer, specifically, the vFiler may impact multiple VMs. The logical thing to do for greatest granularity would be to throttle at the VM or vmdk level. Basically, throttle at the end-points. Since a VM could have multiple vmdks, it is probably best to throttle at the VM level. (NetApp Clustered OnTap 8.2 would allow for throttles at the file level). Not to favor NetApp, other vendors (e.g EMC, SolidFire) who are introducing QoS are doing these at the LUN layer (they tend to be block vendors).

For manual throttling, some isolate the workloads to specific equipment – this could be compute, network, or disk. When I used to work at the University of CA, Irvine and we saw the dorms coming online with Ethernet to the rooms, I joked that we should drive their traffic through our slowest routers as we feared they would bury the core network.

The question would be what type of throttle algorithm would be best? Since starving the main consumers to zero throughput is not acceptable, following a network model may be preferred. Something like a weighted fair queueing algorithm may be the most reasonable, though a simple proposition would be to revert back to the quota models for disk space – just set higher thresholds for many which will not eliminate every problem, but a majority. For extra credit (and maybe a headache) read this option which was a network solution to also maximize throughput

Jim – 11/03/13
@itbycrayon

(I don’t accept general LinkedIn invites – but if you say you read my blog, I’ll accept)

 

 

VMware Linked Clones & NetApp Partial writes fun

NetApp OnTap writes data in 4k-blocks. As long as the writes to NetApp are in 4k increments, all is good.

Let me step back. Given all the fun that I’ve experienced recently, I am going to alter my topic schedule to bring this topic forward while it is hot.

One more step back: The environment consists of a VMware ESX hosts getting their storage via NFS from NetApp storage. When data is not written to a storage system in increments matching its block size, then misalignment occurs. For NetApp, that block size is 4k. If I write 32k, that breaks nicely into 8 4k blocks. If I write 10k, that doesn’t as it ends up being 2 and a half.

The misalignment problems has been well documented. VMware has a doc. NetApp has a doc. Other vendors (e.g. HP/3PAR and EMC) reference alignment problems in their docs. The problem is well known – and easily googled. With misalignment, more read & write operations are required because the underlying block is not aligned with the block that is going out to storage.

And yay! VMware addresses it in their VMFS-5 file system by making the blocks 1MB in size. That will divide up nicely. And Microsoft, with Windows 2008, they changed the starting block which helped alignment.

So, all our problems are gone, right??

NO.

VMware introduced linked clones which have a grain size of 512 (see Cormac Hogan’s blog)

Once this issue is discovered, you end up reading more of Cormac’s blog, and then maybe some of Duncan Epping‘s, and maybe some of Jack McLeod, not to mention Knowledge Base articles from both VMware & NetApp. The recommended solution is to use VAAI and let the NetApp handle clones on the backend. And these 512-byte writes are technically “partial” writes and not “misaligned”.

If everything is aligned, then the partial writes require 1 disk read operation (of 4k), an instruction to now wedge the 512 packet in appropriately to the 4k, and 1 write back out. If misalignment exists, then it requires twice the IO operations.

However, if you look at nfsstat -d, you’ll notice that there are a whole bunch of packets in the 0-511 range. Wait! I have partial writes, those show up in the 512-1k. What are all these?

At this point, I don’t entirely know (gee Jim, great piece – no answers), but according to VMware KB 1007909 VMware NFS is doing 84-byte (84!?) writes for NFS locking. Given the count in my 1-511 bytes, NFS locking can’t account for all of those – but what does this do to NetApp’s 4K byte blocks?

Jim – 08/30/13
@itbycrayon

 (I don’t accept general LinkedIn invites – but if you say you read my blog, it will change my mind)