VMware vCAC IaaS Optimization Guide

Update 04/22/15: After further investigation around the effectiveness of these optimization tips on a vRA 6.2.1 environment, I am convinced that several of the tweaks do in fact provide some level of perceived IaaS UI performance improvements. I’m very interested in hearing your feedback on these findings (i.e. give it a try and let me know!).


Update 12/10/14: I have been advised that the optimization tweaks highlighted in this article will not provide any added benefits to vCAC/vRA 6.1 or 6.2. This is due to the way the IaaS interface is now presented back to the user (via the vCAC appliance vs. directly to the user session). The good news is VMware dev’s are hard at work at baking optimization right into the products, starting with a significant boost in the recently released vRA 6.2.

VMware’s vCloud Automation Center (vCAC) can transform how an enterprise delivers IT. It’s out-of-the-box functionality will help IT deliver Infrastructure-as-a-Service (IaaS) along with X-as-a-Service (XaaS / Everything-a-a-S) in a matter of clicks. Once extended into the datacenter’s ecosystem with vCAC’s extensibility engine, it will help integrate, orchestrate, and automate native and 3rd-party tools, services, and infrastructure, thrusting the enterprise into a new level of self-serviced IT efficiency. Whether empowering the sys admins or enabling end-users and tenants, vCAC has undoubtedly changed how enterprises deliver and manage applications and services.

With all that considered, there are always opportunities for improvement (as is the case with all software). I’ve had the opportunity to speak to many VMware customers, partners, and peers about the overall user experience after deploying vCAC. I get to hear how vCAC has helped improve application provisioning time from weeks to less than a day, how providing new services using the Advanced Service Designer (ASD) has paved the way for a Software-Defined Enterprise, and how relatively easy vCAC was to roll out (there’s a guide for that!). When it comes to opportunities for improvement, there tends to be a common theme: improve UI response time. That’s a fair request and precisely what this post is about. But before we jump in to the “how-to”, I’d like to cover a little background for review (plus, I like to drag things out a bit).

vCAC High-Level Components

The vCAC we know and love today (6.0.x) has had it’s own transformation since the acquisition of DynamicOps and subsequent VMware-branded releases of versions 5.1 and 5.2. In December 2013, vCAC 6.0 was released to deliver the next generation of cloud management and automation capabilities. The 6.0(.x) release was a re-platform, which converged a new SUSE-based virtual appliance (core services, primary UI, ASD, governance and approval engine, etc) with the Windows-based .net “legacy” code (IaaS engine). In the mix is an optional stand-alone Identity Manager to provide multi-tenanted SSO (or, optionally, customers can choose to use an existing vCenter 5.5b+ SSO instance).
When admins log into the vCAC portal to build out the logic and overall management, the majority of the interfaces are local to the VA. However, this isn’t the case for the “Infrastructure” tab thanks to the hybrid UI. While managing IaaS services from this tab, vCAC reaches out to the .net codebase (located on the Windows / IaaS machine) and delivers that UI from within an iframe. To the user, this looks mostly seamless, except for the noticeable UI response impact whenever the IaaS components are queried and brought forward. For ordinary users and consumers of IaaS, the slowed response times can occur when requesting IaaS services from the Service Catalog(s), as this is also delivered via the Windows machine. Although the overall UI can use a boost here and there, basic testing showed the greatest performance impact was associated with the Windows / IaaS machine. This was also the case for previous versions of vCAC, where the entire solution was based on similar .net codebase.

vCAC Hybrid UI

There are many moving parts to the IaaS services, but from a UI perspective, possible bottlenecks may include Windows itself, IIS (plus all the associated services and application pools), and/or the SQL database (local or remote). Being a .net app also means the IaaS engine heavily leverages Windows IIS to run. Since vCAC’s IaaS installer does very little tuning of Windows services during the installation process, I figured I’d focus my attention primarily on optimizing IIS while testing various tweaks. I haven’t totally dismissed the other possible bottlenecks, but following deployment best practices, my VM’s are all appropriately sized (cpu/mem) and my SQL server seems to be running optimally. Plus, I know that IIS is using all default settings at this point and therefor has not been optimized for the installed services.

The following section steps through several tweaks that are aimed at optimizing the performance of the Windows IIS services for delivering vCAC’s IaaS virtual .net applications. In testing, I have found the response time of IaaS components to significantly improve once these changes are made, positively improving the overall user experience. But of course, your results may vary.

Optimizing IIS for vCAC

NOTE: these tweaks are to be done at your own risk and are not officially vetted / supported by VMware. My recommendation is to test extensively in a non-production environment to understand their effect on your vCAC implementation. You should hold off on making any changes to a production environment until VMware provides an official support statement for IIS optimization. Also, make sure you have a backup or a snapshot of the IaaS machine before continuing. These instructions assume a Windows 2008 R2 OS is used for the IaaS components.

Dynamic Content Compression

Effectively using IIS compression provides the greatest immediate performance boost to static and dynamic web content. vCAC’s IaaS service is mostly comprised of .aspx files, providing a combination of static and dynamic HTLM content to the browser. And since HTLM content is highly compressible, this makes for a quick-n-easy adjustment for performance…at the cost of some additional CPU overhead.

Ensure the Dynamic Content Compression module is installed:

1. Open Server Manager
2. In the left pane, expand Server Manager -> Roles -> Web Server (IIS)
3. Find “Role Services” section then click on “Add Role Services”
4. Scroll down to “Performance (Installed)” section and select “Dynamic Content Compression” (note: IIS installs the Static Content Compression module by default, so this should already be checked. If not, select it).
5. Click Install

To Configure IIS Compression:

1. Open IIS Manager
2. Select the IIS Server in the left pane (note: compression can be configured at the Server, Web Site, or Folder level…since this is a dedicated vCAC server, I’m configuring it at the Server level)
3. Double-click on the “Compression” icon in the IIS section
4. Ensure “Enable dynamic content compression” and “Enable static content compression” are both checked
5. In the “Static Compression” section, you can fine-tune some additional settings. I chose to only compress files larger than 1KB (1024 bytes), kept default directory, and set a per-application pool disk space limit 1GB (1024MB).
6. Click on your IIS server in the left pane to exit Compression settings
7. Restart the IIS Service to apply

Output Caching

Output Caching is another feature that can greatly increase performance of dynamic web content by caching the content into memory.  Under normal operations, users accessing .aspx content invoke a script that generates the dynamic web/HTML content for the web browser.  The goal of output caching is to prevent the scripts from firing each time the same .aspx is accessed.  Instead, it caches the script into memory and prevents IIS from having to regenerate all the same content repeatedly.  IIS caches static content by default, but not dynamic content.  vCAC’s heavy use of .aspx and semi-dynamic content is a great candidate for output caching.  The trick to properly configuring output caching is targeting the correct content (in this case, .aspx) and properly flushing the cache to ensure content doesn’t become stale.  IIS supports two cache policies: User-mode (worker process) and Kernel-mode (kernel-mode driver), which provides the greatest performance increase.  In testing, enabling User-mode output cache policy resulted in some weird behaviors, so I quickly disabled it.  When configuring output caching for vCAC content, we will only utilize a Kernel-Mode policy.  DO NOT enable User-mode output cache!!!

To Enable Output Cache:

1. Open IIS Manager
2. Select the IIS Server in the left pane
3. Double-click “Output Caching” icon in the IIS section
4. In the Actions pane, click “Edit Feature Settings…”
5. Click on ‘Enable kernel cache” to enable it
6. In the “Maximum cached response size (in bytes)” field, enter 1024000 (1MB).  This sets the maximum cached response – I chose 1MB in testing as to not overwhelm IIS, but this should cover most of vCAC’s file sizes anyway.
7. Uncheck “Cache size limit (in MB)” or enter a “0” in the field to disable cache size limits. By default, IIS will use 50% of the available memory.
8. Click OK
9. Restart the IIS Service to apply (or wait until all settings are done)

To Configure Output Caching Policies:

1. Open IIS Manager
2. Select the IIS Server in the left pane
3. Double-click “Output Caching” icon in the IIS section
4. In the Actions pane, click “Add…”
5. In the “Add Cache Rule” dialog box, enter “.aspx” in the “File name extension” box
6. Select the Kernel-mode caching
7. In the “File Cache Monitoring” section, select “At time intervals” and enter “00:30:00” (note: I’m still experimenting with this setting. The time entered here determines how often cache is flushed and will help prevent stale content.  Another option is to flush cache anytime the underlying resource is changed using the “file change notifications” setting, but so far a 30-minute time interval has worked well so far)
8. Click OK
9. Restart the IIS Service to apply

Cache Worthiness

Now that caching is enabled, we need to determine what is considered “cache worthy”.  In other words, when should IIS determine that a file should be cached.  There are two properties that determine this: frequentHitThreshold and frequentHitTimePeriod, found in a local file called “applicationHost.config” in “%windir%system32inetsrvconfig”.  By default, content is cached only if it is accessed more than the frequentHitThreshold (default = 2 hits) within the frequentHitTimePeriod (default = 10 seconds).  So, based on the default settings, content needs to be accessed more than 2 times within a 10-second window in order for it to be cached.  As a result, the busier the site, the better it will be cached overall.  The default settings might be optimal for a production vCAC instance with a decent load, but not so much for a non-production lab environment or POC.  In order to ensure vCAC’s IaaS content is getting cached, I changed frequentHitThreshold to 1 and frequentHitTimePeriod to 00:02:00 (2 minutes).To Configure Cache Worthiness:

1. Browse to “%windir%system32inetsrvconfig” on the IIS machine
2. Locate and open (edit) “applicationHost.config”
3. Locate the “serverRuntime” element,
4. Enter the following syntax:

(note: these settings are applied as soon as the document is saved)

Tip: To get some visibility into which content / pages are being cached (or if caching is even doing it’s thing), you can run the following command in cmd:  netsh http show cachestate


Application Pool “Idle Time-out” and “Regular Time Intervals”

vCAC’s IaaS services use three IIS Application Pools, which are created during installation.  These are “RepositoryAppPool“, “vCACAppPool“, and “WapiAppPool“.  There are a couple of tweaks that can be done for each of these application pools that can improve application response time.  These are “Idle Time-out” (the amount of time a worker process remains idle before it is shutdown) and “Regular Time Interval” (the period of time after which the application pool will recycle).  In testing, I didn’t notice a significant increase in performance once these were configured, so use at your own discretion.  Theoretically, if these process are locked-n-loaded, response time should improve.  One word of caution on this: my buddy, @virtualwinorg, notified me that setting “Idle Time-out” to 0 may violate DISA-STIG security requirement VulnID V-13708.To Configure Application Pools:

1. Open IIS Manager
2. Expand the IIS Server in the left pane
3. Select “Application Pools”
4. In the Application Pools window, click to highlight “vCACAppPool”
5. Click “Advanced Settings…” in the right pane
6. Locate “Idle Time-out (minutes)” and enter a “0” to prevent time-outs
7. Locate “Regular Time Interval (minutes)” and enter a “0” to prevent recycling
8. Click OK
9. Repeat this for the other pools
10. Restart IIS Service to apply

Once all is said and done, it’s a good idea to reboot your IaaS machine – this ensures all the appropriate IIS and vCAC services are cycled and all tweaks are in effect.  Give IaaS services some time to come up and register with the vCAC VA (up to 10 minutes) before attempting to log back in.  If you prefer not to reboot the IaaS machine, you should at least restart all IIS and vCAC associated services.

You might not notice an immediate improvement at first logon, at least not where caching will benefit.  Remember that caching works by loading commonly used content into memory, so you’ll have to actively use it before any noticeable performance and response improvements.  Also keep in mind that none of these settings were targeted at the vCAC VA itself, just the Windows / IaaS / IIS components…that will be my next project.  But as I mentioned earlier, focussing on the IaaS components first has made a significant impact to the overall user experience in vCAC.

I’m interested in getting feedback from folks who use these tweaks, so please comment as you see fit.

++++

@virtualjad