Running Fusion Middleware in Solaris Containers

28 May 2010 by Simon Haslam (in Virtualization)

A couple of months ago (yes, I know it's taken me a while to blog about it!) I worked with a customer to build a production Oracle Fusion Middleware 11g environment on Sun T5000 hardware. Given that these servers are pretty chunky I recommended that we used Solaris Containers for partitioning the various middleware tiers, and this post describes some of the considerations.

Modern web infrastructures tend to have many "moving parts." If you take a traditional Oracle Application Server cluster running java apps, for example, you would often have separate servers for OHS, OC4J, SSO, OID and then the back-end databases. Of course in a High Availability (HA) arrangement you would always have 2 or more of each of the tiers, so most customers would have 8 or so servers just for the middleware! Typically several of these machines, in particular SSO and OID ones, would be very lightly loaded. The reason for separating the tiers is primarily for security - each server has a very clear role and the network traffic can be very tightly controlled. However having so many servers makes the middleware tier a prime candidate for virtualisation, arguably far more so than the database tier. Enter Solaris Containers...

 

In case you're not very familiar with Solaris Containers, they were introduced in Solaris10 and are a method of virtualisation running on top of the operating system (see Wikipedia). Solaris Containers runs on any regular Solaris10 installation, without having to re-install the OS (though you may choose to do so, depending on what packages you had previously installed). In the illustration above Solaris Containers is represented in the 3rd server from the left (which looks like a T2000).

Each container (or "zone" - the terms are used often interchangeably though I suspect "zones" came first and "containers" was dreamt up by marketing to represent both zones and resource management) you build has its own configuration (/etc files for example) and looks to all intents and purposes like a separate server - you can start it up, set up users, install software into its own filesystems and so on. What's more, creating a zone takes just a handful of commands so is quick and easy.

The key features of Solaris Containers for me are:

• straightforward management & low runtime overhead,
• shared operating system install & kernel for simplicity (optionally different packages for different containers),
  
• no additional Solaris licensing,
• ability to partition hardware for Oracle licensing purposes.

I'd like to expand a little on that last point as it is very significant. As you may be aware, Oracle generally does not recognise software virtualisation partitioning (see Oracle Licensing doc on Hard/Sort Partitioning). Therefore if you have a physical server requiring, say, 4 Oracle Processor licences, and want to run an Oracle product such as WebLogic Server on a virtual machine using 2 virtual CPUs (even if they were tied to processor cores) then you still have to license the whole server for WebLogic. This isn't so much of a problem if you have the entire server dedicated to the middle tier anyway (since every virtual machine might be running WebLogic or OHS etc), but it makes it expensive if you're trying to consolidate servers and want to run a database on there too. As servers become very powerful (even so-called "commodity" ones) it is quite feasible that many organisations (at least the mid-sized ones I often work for) will be able to run their entire web infrastructure, including supporting databases, on just two(!) physical servers and a small SAN or NAS - in this case Oracle licensing flexibility becomes very important.

The exceptions to the above rule are Oracle VM Server and Solaris Containers where Oracle does consider certain configurations to be examples of hard partitions. This allows for logical partitioning of the licences, improving hardware utilisation thus potentially improving both return on capital and energy/carbon usage.

So, you think Solaris Containers are a good idea, but what are the configuration considerations for the Oracle middleware administrator?

Firstly networking (as this was the primary reason for separating your servers): you can allocate specific NICs, bonded or not, to a certain container, in what Sun calls "exclusive-IP" mode (see the certification note about this below). We had a web tier and an application server (WebLogic Server) tier on the same box so gave each their own NICs - this means that they can be connected to separate zones within the DMZ if you wish. For instance, let's say you have a 4 ports onboard (fairly standard these days) plus a quad-port NIC, one scheme might be:

• (2 ports) a DMZ1 interface for web servers (and reverse proxies if required),
  
• (2 ports) a DMZ2 interface for the application servers,
• (2 ports) corporate zone for OID, database servers etc,
• a management/backup network (or iSCSI SAN etc).
  

That's a lot of cables, but fewer than you would have had for separate physical servers once you've included the LOM ports. You could of course achieve the same result by sharing the NICs between zones (and this seems to be the future trend for obvious reasons) but then the isolation currently comes software firewall control which may not be so pleasing for your security auditor (plus there are some functional restrictions, mentioned below). The conclusion, for the time being, is that you'll need plenty of network ports on your server!

So, now that we have discussed networking, let's look at the OS installation: with Solaris Containers you have the choice of either running exactly the same OS packages as in the global zone (the "main" OS, if you like) or whether to take a copy of them. If your zone has links to the global zone for all the OS packages it is called "sparse-root" or if it has copies in its local filesystem it is called "whole-root". I took the view in this case that to aid simplicity we would use sparse roots, meaning that if you patch the physical machine all containers are then patched (and have to observe any downtime requirements during the process). A very experienced Solaris administrator I know takes the contrary view: the advantage of being able to patch containers individually (at a time to suit the application) far outweighs the downside of having to patch all the containers individually. Note: I've ignored the additional disk space requirement for whole-root zones as that usually isn't too significant. More generally my opinion is that if you think you need the OS flexibility of whole-root zones you should probably be looking at a hypervisor-based virtualisation solution, e.g. LDoms (Oracle VM Server for SPARC as it's now called). I suppose this will really end up as a decision based on required uptime as compared to resources available to both schedule and carry out the patching.

Not to forget certification, this is what Oracle has to say about Solaris Containers: http://www.oracle.com/technology/products/ias/hi_av/oracleas_supported_virtualization.html#supportforsunsolaris10inmultizoneoperation
In particular: "All WebLogic Server features are supported in Solaris non-global zones when using Exclusive-IP in the non-global zones." There are some restrictions on Whole Server Migration which you would expect given the way that it works, but that's beyond the scope of this article (if you're interested see MOS Doc ID 976795.1).

Those really are the main considerations - the rest is now down to implementation details (network subnets, meaningful host names, shared filesystems for Oracle media, etc). For another customer I'd been on the receiving end, as it were, for Solaris zones already built by sys admins but at this site we had to set them up from scratch. As there are plenty of good references for the configuration of Solaris Containers I don't intend to repeat the details so here are some sites I found useful:

• Sun Blueprints Guide to Solaris Containers - a good overview
• Solaris Sys Admin Guide
• Sun BigAdmin Sys Admin Portal - practical tips
• OpenSolaris Zones & Containers FAQ - x86 but mostly still applicable to SPARC
• Gleb Reys Solaris Blog - some samples

Here are a few things I discovered relevant to Oracle Fusion Middleware installations:

• Just like Solaris Jumpstart you can enter system information automatically from a sysidcfg file - just put the file in the zone's /etc directory after you have created the zone but before you have boooted it for the first time. This will save you some time, particularly if you have quite a few zones to create.
  
• Older ethernet drivers may not support segregation - whilst this doesn't affect the new servers you are likely to be running production on, it may be relevant to test environments. For example, I have 3 Sun Netra t1 servers and I found the old "eri" driver wouldn't let me configure a NIC as exclusive-IP.
• Whilst in most respects each Solaris non-global zone looks like its own server, the root user doesn't exactly behave the same as root on the physical machine. This gave us an issue starting up OHS (based on Apache 2 in OFM 11g) - if you're running on a privileged port (below 1024) you need to set "Group oinstall" (or whatever) in httpd.conf.
  

In conclusion, for Oracle-Sun customers, especially ones without an enterprise-wide virtualisation policy in place and running on new SPARC hardware, Solaris Containers is a good candidate for virtualisation of Fusion Middleware installations. It allows you to achieve a good degree of hardware partitioning, offers a reasonably high level of environment segregation, is very robust and is relatively easy to implement.