Anglia Ruskin University - HP Server Capacity Planning

Server Capacity, VMWare Sphere and HP blades

A rather specialist post!  A snapshot of our server capacity planning.

Like pretty much everyone else, we have a heavily virtualised server estate.  At the minute, we're looking at about 95%+ virtualised.

We have a combination of HP G5, G6, G7, Gen 7 and 8s.

Our on-going strategy is to retire servers that are more than 4+ years old.  Better energy efficiency, capacity improvements and increasing support / maintenance costs are our justifications for replacing old kit.  

We currently have 12 G5s between 5 and 7 years old and provide 384GB of memory and 12 x 2 x 4 cores = 96 processors (hyperthreading is not active).   That makes 32GB per server.    These days we're buying servers with 256GB of memory.

So, what are we going to do going forward?

Blades or rack mount? 

We're now firmly in the blade server camp at this point - because of the connectivity and management advantages. 

Four broad server workload requirements

1) VDI environment - which isn't covered here
2) General server capacity - retiring old HP G5s
3) Specific server capacity for our Tribal SITS Student Information System and data warehouse
- This should free up two existing HP DL580s for general purpose use
4) Test and development

Our Cambridge primary datacentre - Tribal SITS platform

One of our largest projects at the moment is the re-implementation of our student information system.  Server performance is probably the least risky aspect of the new SITS implementation.  That being said, we're moving from Oracle to SQL Server 2012 and we have no production experience of the workload under SQL Server.  
  
So, we plan to purchase four new servers for our SITS implementation.

We've historically purchased larger DL570 4-way boxes for our database loads - 12 cores per processor @ 2Ghz.

Just for info., the table below outlines the HP Gen 8 vs Gen 9 processor options.

For this new purchase we've elected to go for fewer processors and cores and a higher CPU frequency.

Specifically, the 2 processor Gen 9 BL460c 2 processor, 8 core E5-2667 v3 @ 3.2Ghz.  Which should give us a reasonable boost in raw process performance.  I know this is a small part of the overall configuration and we're also adding tiered SSD in our HDS HUS150 SAN to provide disk performance.  That should theoretically give us 16 processors per server for VMWare - 32 if we go for hyperthreading.   This seems ample to support our planned SQL install.

Anglia Ruskin University - Our VDI - what we want to do next

As I said in my earlier blog, we have VDI aspirations.  (Hosted Virtual Desktop is the term Gartner tend to use)

This particular blog talks about our intention to push our VDI use cases further.  This means we need more hardware and software - which is discussed in outline below.

Our current VDI delivers a large range of applications but..   What we do not do terribly well at the moment is provide heavy graphics and processor dependant applications and content.

As a consequence, there are a number of niche and not so niche applications that we can't provide to our students (or staff) on our VDI environment.  Of course, this means we are stuck with thick clients for these applications.  So, we're left with students needing to visit specific physical locations to access these more specialist applications.  We want to fix this if we can.   Being able to access applications from almost any device at any time is a significant gain. 

We're also the entrepreneurial university of the year and maybe we can put a package of software together that might help budding business entrepreneurs - web site software, maybe some free hosting? what else?

Specifically, we're talking about applications like the following (I'll add a more complete list at the end later):

Google Earth, Cry Engine, 3D Studio Max, Autodesk Maya, Adobe After Effects CS6, Adobe Photoshop CS6, Epic, Adobe Premiere Pro CC, AutoDesk AutoCAD Architecture 2014, Autodesk Structural Detailing 2015, AutoDesk Inventor, ArcGIS 10, World Wide Telescope, Adobe Audition CC, Adobe Edge Animate CC 2014.1, Adobe Fireworks CS6 (included with CC), Adobe Flash Professional CC 2014, Adobe Illustrator CC 2014.

We probably need more fast storage to run these new applications too but that'll be the topic of a further blog post. 

So, we think we need to complement our environment with NVidia graphics processors.  With the new VMWare vGPU profiles we can hopefully use both Grid K1 and K2 processors.  Particular users will be assigned particular vGPU profiles - meaning we can support a terrific range of graphics heavy applications.  However, the bulk of our users will run on the lesser K1 chips - with the K2 processors being dedicated to heavy use and specialist users. 

 The relevant NVidia graphic boards look like:

A list of certified applications for the Grid cards is here.

Some capacity planning is obviously vital. 

Should we host everything in our main Cambridge data centre?

If we did this then we'd cut down on hardware and complexity.   The disadvantage is clearly around disaster recovery and perhaps end user performance.  But, we do have 10Gbit links with about 10ms latency.  It might be fine.

We would engineer our environment in Cambridge to have no single point of failure.  But, if we lost both our resilient 10Gbit comms links then we'd be in trouble.  But, for a limited number of users at Chelmsford.  This might be a reasonable risk to accept.

The Capacity Requirements

We think the total user community who will benefit from this new graphics capable technology is pretty small.  In Cambridge - probably about 100 concurrent users.   Chelmsford - probably 50-60 concurrent users.  This is going to be a high estimate.

Of the total number who could use the new graphic capability perhaps 20 users in Cambridge and 10 users in Chelmsford would make use of the more top end 'Designer/Power User' capability - i.e. K2 Grid.

What do we need to do to verify these estimates? - clearly identify the applications that need the capacity.  I imagine we will not really be able to do this until we create a working proof of concept.  But, applications like ArcGIS are certified with shared K2 but not K1.

What would the NVidia Grid capacity look like?

Each K2 K220Q supports 16 users (with 512MB graphics memory).  Two = 32.  We can probably get by with Two or three K2s as long as we have at least two separate servers to support them.

Each K1 K140Q supports 32 users (with 512MB graphics memory).  If we need to support about 120 concurrent users then we need four K1s - with at least three being in separate servers.

You'll notice I've assumed we only need 512MB of graphics memory.  This is a guess.

That means we need at least five extra servers to support the new cards.

Our ideal architecture (if it works) would be to also insert an Apex 2800 card into each server - to get extra PCOIP acceleration.

What about the servers?   

The question is - can we still use our existing chassis based HP Gen 8 (perhaps Gen 9) blades? 

Our belief is that we at least need a PCI expansion blade to fit in the K1/K2s and Apex 2800 for our existing Gen 8 BL460c blades.  However, the BL460s are not supported by NVidia.   More VMWare View K1/K2 information here.

The likelihood is the BL460c is a dead end.  We ought to plan on that basis - running an unsupported VDI environment is not a terribly good idea.

This is where we need help from HP and nVidia - can we do this?  or do we need to move to another server architecture?

The nVidia server compatibility list (click here) suggests only a chassis based or specialist workstation WS460c (I'll discount the SL250 as a HPC focused solution).  VMware certification for WS460c here.  This would be a pity - as it would undermine our 'standard blade' approach.  But, they're still blades and we could still utilise our c7000s - so perhaps not so bad.  The Apex 2800s are also supported in the WS460s - so looking feasible.

But, will a K2 (and or K1) and Apex fit in the same server?

So, if it all works out this would mean five new WS460s.

So, HP and NVidia (and helpful solution providers) what's the best approach?

What about our VMWare Desktop Virtualisation Software?   

All of this will be dependant on an upgrade to our existing VMWare View Horizon 5.2 environment - to the latest version 6.


What have we discovered?

13/3/15 - Good news

Well, happily, we have had the above approach informally validated by a helpful supplier.  Even better, they have a hosted POC environment we can use to validate that some of our applications work well.  Testing will take place in the next couple of weeks.

Also, there looks to be some good news on the graphics front and more improvements.

So, firstly, the Apex 2800 card can fit with a K1 or K2 in the WS460c.  Hurrah.  We get the benefits of PCOIP acceleration and graphics from the Ks.  But, we appear to need vSphere ESXi v6 (due out soon).  Apparently the associated new pGPU driver claims to double the density of users on the K1 and K2 Grid cards.  Which would be good news.

Anglia Ruskin University - Onwards and Upwards for our VDI - What we do now

So, Virtual Desktop Infrastructure.  We've been a pretty early adopter with an initial implementation for our students and staff starting about four years ago - 2011.

This blog is intended to provide a basis for Anglia Ruskin IT Services discussion (and wider) but also information for suppliers who might be able to help us with our VDI aspirations - more of this to follow.

I've outlined, at a high level, our technology in use.  The next blog entry will detail what we want to do next with VDI - future use cases.

Our technology is centred around VMWare View (Horizon), HP blade servers, Violin Storage and zero clients.   Augmented by AppV for application virtualisation and Profile Unity for easy profile and application setting management.

It's been tremendously successful with an appreciable uplift in our student experience.  Essentially, it means that students (and our staff) get a good consistent experience across our various locations and devices.  Our total combined staff and student user population is about 35,000.  

To get a bit more specific:

End devices.

Our end devices are mainly zero client with about 950 at our Cambridge Campus and 840 in Chelmsford - roughly 1800 total.   They're a mixture of Tera 1 and Tera 2 supporting our main PCOIP protocol.

We've recently been looking at generation three LG 23" All-In-One V Series as a possible replacement for some of our older end devices

Servers

Our Cambridge Campus

We're using a combination of HP BL460c G7 and Gen 8 servers spread over two HP c7000 blade chassis.  18 blades (all with 192GB memory) - 6 Gen eight and 12  G7.

Our Gen 8 blades also have a Teradici Apex 2800 Tera 2 card installed - to provide better performance.

We used a rough and ready capacity planning rule of thumb of 50 VMs per host.  That means we ought to have capacity for about 900 concurrent VMs in Cambridge.  This is putting aside any performance uplift me might get from our newest Gen 8 blades.  We've also installed Apex 2800 PCOIP acceleration cards in all the Gen 8s.  This ought to mean 50 per blade is a very safe performance bet and this has mostly been true.   We do get the occasional slow down in specific VMs - but it has been hard to establish where the bottleneck might be (despite investigation).  Saying that, the other 99% of the time we have great performance.

On top of that, we have four servers in our VDI management cluster - all G7s with 192GB of memory.

Our two c7000 Cambridge chassis have 5 spare slots in each - 10 total.

Our Chelmsford Campus

Chelmsford is pretty similar - but with a slightly lower capacity.

14 servers in total - 5 Gen 8 with TeraDici Apex 2800 offload card and 9 G7s all HP BL460c and 192GB of memory.

Our VDI management cluster comprises three G7s.

Theoretically, using our rough rule of thumb explained earlier, this means we can support 700 concurrent users in Chelmsford.

Our two c7000 Chelmsford chassis also have 5 spare slots in each - 10 in total.

Our client VMs.

We run Windows 7 Enterprise SP1 with 4GB of memory.  Over 100 applications mostly streamed using AppV.

Other software versions.

VMWare Sphere 5.1, VMWare View Horizon 5.2.

A note on end user experience.

It's worth saying upfront that our emphasis is on providing a PC like end user experience.  Our rationale isn't about stuffing our servers with as many VMs as possible to maximise value for money.  Our objectives are far more focused on delivering the best possible experience for our students and staff.  Of course, there are limits to this.  But, if we compromise end user experience too much then we'd be better off providing PCs.  

The Cloud and Software as a Service (SaaS) – a new disruptive model for IT

The Cloud and Software as a Service (SaaS) – a new disruptive model for IT

The Cloud or Software as a Service – a new disruptive model for IT.  A second topic covered by Paul Strong, VMWare CTO and a few others at the recent 2014 Gartner conference.

Obviously, cloud infrastructure is a big topic and I don’t propose to cover it much here but a couple of points were made at the Gartner Symposium that are worth thinking about.  I’ve lumped in SaaS in with the discussion. 

The first point?  That Cloud based infrastructure and SaaS are here to stay and will be disruptive to central IT

Cloud, some argue, is more of a marketing umbrella term than a product or technology and more of an evolution from what we had before i.e. external hosting.  Peter Thiel (founder of Paypal, first external investor Facebook) made that specific point during the Gartner conference.  Well, that might be right but I’m not sure the fact that it is more evolutionary than revolutionary will stop Cloud services being massively disruptive to traditional IT – like us.  Paul Strong asserts, like many others, that cloud is a new consumption model for IT.  People are and will be enticed by the consumption model – the experience of buying commodity and relatively cost effective IT infrastructure and SaaS will be attractive.   If central IT don’t get on board then business units (Faculties, Services) and everyone else will go straight to cloud providers.  The central IT barrier, whether it’s security or some other obstacle will be moved around.  So, this is a challenge that central IT teams need to respond to.  Again, remember, that's us.  Actually, we’re seeing this last bit already.  In the last few months there have been at least two proposals to purchase Cloud based systems made by our customers.  Suppliers, because it can be in their interest, tend to gloss over and simplify issues that a Cloud based solution might have.  And, we can think of quite a few potential issues like integration with our other systems, data exchange (it’s really not as easy as they say), security, reliability and lots more.

The second point?  That Cloud based infrastructure and SaaS will be beneficial and allow us to do more with less

The cheap(ish) and scalable resources offered by Cloud services could be transformational in providing effective IT for our University – so, it’s about getting the approach right with Cloud adoption rather than really questioning the eventual destination; at least for some of our IT services.

Dave Aron (Gartner Fellow) made the following point.  We need to watch out for disruptive change like Cloud.  Remember Encyclopaedia Britannica? $600m revenue in 1990.  Down to $195m revenue in 1995.  Now, they are effectively gone.  Obviously, Wikipedia pretty much took over most of what they did.

So, central IT – we don’t want to be another Encyclopaedia Britannica.

Amazon, as you are probably are aware, kick-started the Cloud agenda – after disrupting and transforming book selling and supply chain.  Paul argues that the same disruption and transformation is happening in Cloud, again driven by Amazon and Amazon Web Services.  Perhaps, looking back in five years – we will only really understand how disruptive cloud has been.   It is worth noting that not everyone entirely agrees with this assessment.     Most analysts seem to be a bit more nuanced.

So, Central / Enterprise IT, the challenge is to engage positively and work with SaaS and the Cloud to provide value.  Soon.

Gregor Waddell, Assistant Director

IT Agility, what on earth that means and what the CTO of VMWare thought.

Some of us attend various conferences and I had the opportunity to get to a couple recently – the Gartner Symposium and Educause (https://www.educause.edu/).  There was lots of good stuff and my intention is to cover a few topics in our newsletters going forward.  Hopefully, we can stimulate a bit of positive debate and discussion.  In part, what we learn at these conferences will help make our new 2015/16 strategic plan as good as it can be - incidentally, this is due by the end of December 2014.  So, we need to get cracking.

We also need to update our current three year IT strategy to take into account our new corporate plan fairly soon.  Our new strategy needs to be cognisant of any technology advances or anything else on our 'landscape' that might have changed.

So, first up on the topic list is IT agility.

How can we cultivate IT agility? In other words, how can we respond quickly to customer demand without getting stuck in bureaucratic treacle?  Or being hamstrung by an overflowing and demanding project portfolio? But, it's a bit more than that.  How can we get ahead of the game and be a supportive and positive partner to our University?  Our knowledge and expertise ought to put us in a great place to understand how technology can be leveraged to push us (our University) collectively forward.  So, what's stopping us?  Is anything stopping us?

Well, I attended an interesting presentation by Paul Strong, Chief Technology Officer at VMWare.  I also managed to speak to Paul after his Gartner presentation.  While he didn't address the whole IT agility question, he did address a small part - which mostly revolves around IT process automation.  And, it did contain a call to action for central / enterprise IT - that's us by the way.  For those of you who don't know - and apologies to those who do - VMWare is the virtualisation technology underpinning most of what hums and creaks in our data centres adeptly managed and soothed by Rob Spalding and his team.

So, what did Paul have to say?  Paul reckons we’ve got it a bit wrong over the last 30 years.  Concentrating on infrastructure and ‘spinning rust’* rather than ‘Innovation Technology’ – as he says IT should really represent.  (*IT infrastructure joke for spinning magnetic disk.  No, it doesn't get any better in infrastructure comedy land)

What does he mean by this?  Organisations typically use the application lifecycle – with an operational process framework to help.  The framework we use, of course, is ITIL and Joe McIntyre and his team have put a lot of time into its adoption - with some of us getting certified to different levels.

ITIL is a common approach for operational management of services and provides the necessary processes to run our IT service.  We all have supporting tools such as a service desk system - we use Supportworks.  Almost everyone uses ITIL in some form.  So far, so good.  But, ITIL and other frameworks obfuscate and get in the way of agility.  People in these processes often act as gatekeepers – preventing change.  Security specialists are often particularly obstructive to speedy adoption of anything new.  Sorry guys - this isn't to say security and processes aren't necessary.  Far from it. But…this is not necessarily the way to agility.   As with everything, it's a balance Paul argues - we probably all agree.

Paul suggests that organisations have too many applications and therefore complexity.  This is the primary reason why IT has not brought the expected economies of scale and actually holds back innovation.  It’s far too complicated.  We’re far too busy managing too much technical complexity to have much time to think innovative thoughts.  Cloud providers are much better at this.  Why should this be the case?  Cloud providers only provide a small number of architectural blueprints that they refine and run efficiently.  With fewer blueprints, you can automate like crazy and economies of scale are possible.  It’s all about simplification.

How do we get better at this?  Standardise, standardise, standardise.  Not a great mystery.  The complexity is not about the number of servers*; it’s the number of server variants.  *Insert any technology for server.  This is one of the primary principles we use when looking at technical architectures at our IT Architecture Board.  But, the emphasis on a limited number of simple technical choices perhaps needs to be greater.  Compromise functionality to gain greater simplicity.  I do agree, and it’s hard to argue really, that we need to simplify.  Action point for us here is to think harder about how we can do this successfully.

I asked Paul afterwards about the “we’ve got it a bit wrong for the last 30 years” suggestion.  The main issue I had with his presentation was probably that it was pretty infrastructure-centric.  I guess that’s what you might expect from an infrastructure vendor – but, did he really believe the biggest thing around IT agility was about automation?  Turns out his opinion is a bit more nuanced.  Yes, VMWare (and other virtualisation technology) could help us to automate – once we simplify first.  But, yes, all the other stuff around Enterprise Service Buses, Service Orientated Architecture and much more besides is pretty important too.

So, as I said before, the key action point here is about introducing simplicity wherever we can.  It’s fine to appropriately compromise on functionality.  A second key action is to look at the specifics of our server automation – which might include automatic provisioning of servers for our production environment but for development and test too.

More here:

http://gregoredu.blogspot.co.uk/2014/10/gartner-2014-how-does-vmware-cto-paul.html

Gregor Waddell, Assistant Director