Simon Haslam's Oracle Fusion Middleware blog

Middleware on the Oracle Database Appliance

This evening I attended a webinar arranged by the Oracle RAC SIG called "Oracle Database Appliance (ODA) - Introducing the Virtualized Platform". The presenter was Tammy Bednar from Oracle production managment. I've taken the liberty of calling this article "Middleware on the ODA" since, as you'll see later, this is the most interesting potential use for me (and is probably why you're reading this blog in the first place).

In summary, Oracle will soon be offering a version of their ODA product as a self-contained, pre-packaged server pair running virtualisation and with a database "built-in". This won't come as a surprise to many as Oracle has been quietly discussing this with some customers and partners since OpenWorld last year, but it is the first public information I've seen. Apparently the software should be officially available next week via My Oracle Support, so if you have a spare ODA lying around you'll be able to try it out!

 Sunrise (photo by Chris Gin)
Photo by Chris Gin (see below)

So, now to some technical detail. I've recently written about ODA, and there's plenty of other material on the web and in the Oracle documentation, so I'll focus on the changes in this new development and will try to cover the highlights of the webinar along with some initial thoughts of my own.

Virtualisation Approach

The high level approach is fairly conventional: you have an OVM Server installation on each server, i.e. a Dom0 for managing the hardware and the DomU virtual machines themselves. OVM is being used to sub-divide each server and provide isolation (and licence segregation) between workloads, rather than as a consolidation of resources into one compute pool like you could do with OVM Manager. This makes sense - for such a small cluster you're unlikely to be able to usefully move a VM between servers, and high availability is provided further up the stack (by clustering at the database tier with RAC, and at the application tier with WebLogic/Coherence).

Oracle Appliance Manager - the tool used to both initially set up the ODA as well as patch it (including the oakcli command line) - is still there but has been enhanced to include the creation and management of virtual machines (presumably including web service calls to the hypervisor, rather like the WebLogic admin server does with virtual "node managers" for the defunct WebLogic VE).

ODA Base Domain

The big news is how the Grid Infrastructure and Database products will be run on this ODA version, namely in what Oracle is calling the "ODA Base" "privileged domain" which has direct access to the hardware "to provide native disk performance".  Actually what I think this may be is a special DomU VM where the HBA/controllers connected to the shared storage are passed through directly to the VM (something possible with processor support, in this case Intel VT-d extensions). If so then actually it will be comparable to running RAC on Oracle VM, albeit with direct access to the storage. It wasn't mentioned but, if it were me, I would probably want to pass the 1GbE interconnect network cards directly through to the ODA Base domain too for best performance and to minimise risk of any contention/possible node evictions (and as there doesn't appear to be any clustering of the OVS respository so no interconnect is needed for OCFS2).

You then pin the ODA base domain to a certain number of cores on each host so that the OVM hypervisor limits its access to CPU resources accordingly. The resulting performance, apparently, should be similar to running a physical ODA with the same number of cores enabled in the BIOS (disabling cores is how the processor capacity is controlled in the original/physical ODA). This provides the "pay as you grow" pricing capability for the database. Note, unlike the physcial ODA, it will apparently be possible to shrink your database licence too (which in practice probably means allocate some processor licences to a different system).

Initial Set-up

Set-up sounds fairly straightforward:

  • Download and apply a special virtualised image to the ODA (i.e. overwrites everything)
  • Download and deploy the ODA_Base domain
  • Run the ODA Configurator and then apply to the ODA_Base (as you would to a physical ODA)
  • Download OVM templates and run oakcli in the ODA Base to load them and create VMs

In other words most of the management is done within the ODA Base and so that's a mandatory component (in case you were wondering whether you could just run middleware VMs on one of these machines).

Deploying Virtual Machines

So now that we have an OVM environment we can deploy virtual machines. Each server has 250GB carved out of the local RAID1 disk pair (the other half being used for the boot disk/swap) for the OVS repository (templates and virtual machine disks), however OVM can also use NFS attached storage instead. Whilst 250GB might just about be enough space for a typical application tier I'm not sure that a couple of SATA disks (hot swappable?) will really provide sufficient IOPS for many applications, especially if they're (rightly or wrongly) doing a lot of logging. Therefore I suspect the later approach, or something in the future based on ACFS, might be more common. You also need to think how you might do backup and snapshots of the VM images so again a central NFS filer might be preferable.

Even though ODA has two separate OVS repositories (one on each node) according to the presentation you can use the oakcli commands from either node to deploy your template and create VMs on either or both (presumably it copies the template over accordingly). There was no mention of Oracle Virtual Assembly Builder, though perhaps this first release is focussed purely on delivering the OVM platform with database.


The same bundled patching will still be available (for firmware, OS, GI, database) but will now include Oracle VM too. Patching of what's installed the VMs (other than ODA Base) will be the resposibility of the administrator (since Oracle doesn't know what you will be running at the moment).

Any product that is certified to run on OVM will be supported by Oracle on ODA.

Partitioning of CPU and Memory

So you've pinned the ODA Base domain to a certain number of cores, now you need to pin the VMs to cores too (or more likely sets of VMs to cores). You would do this partly for resource management reasons, and partly if you wanted to hard partition licences.

One thing that has only sunk in after the webcast is that, whilst each node has 12 physical cores, with hyper-threading you will see a total of 24 "CPU"s so you'll probably need to be very careful when pinning cores to avoid sub-optimal combinations (or maybe the appliance software does this for you?).

It's also very important to consider memory too. Oracle is recommending you leave 8GB out the 96GB for Dom0, so then you need to think carefully about how much memory to allocate to the ODA Base domain (bearing in mind what the SGA/PGA footprint of database instances you will be running) - I suspect changing this later might need an outage so this is probably the most important thing to get right. Finally you then need to allocate memory to the rest of the VMs you plan to run. On a more general note memory over allocation is often used to improve the density of virtual servers but I wouldn't normally recommend it on production Oracle servers.

Final Thoughts

Most of the middleware installations I do are on servers newly bought for the project (indeed I recommend this for x86-64 since the hardware cost is generally insignificant) so it's quite feasible that many organisations would consider an ODA at the same time as a middleware upgrade. Of course this approach would tie you to OVM, which could be a stumbling block, but the point is that with ODA you're using virtualisation in a fairly limited fashion so maybe can treat it as a black box.

So ODA can provide a ready-to-go virtualisation platform... what is more interesting is what you'd do with it. This wasn't discussed at all on the webinar, but let's speculate! The obvious first use would be WebLogic clusters for Java EE applications, but then vanilla Identity Management would fit on it fairly easily, as would probably many WebCenter installations. However the pièce de résistance, and perhaps this will only really be feasible with Intel Sandy Bridge-EP or even Haswell processors, would be SOA/BPM along with the dehydration store/metadata repository, all in one HA system. If that could be pre-packaged into a easily deployable unit that could save a lot of work and headaches for customers.

I hope you've found this write-up useful. If you attended the webinar and think I missed something vital, or you have other thoughts on the subject, please feel free to add a comment below.


I'm assuming the slides from the session will be easily available soon enough but thought it might be worth including some of the questions and answers asked by myself and others at the end of the session:

Q. Is ODA Base just a virtual machine?
A. (Bob Thorne) ODA Base is a VM.

Q. What was the reason to select the hex core Xeon versus an octo core?
A. (Bob Thorne) The current version of the ODA uses the hex core Xeon. Future versons will use the 8-core version.

Q. Can you run other databases on DomU or just in the ODA container ?
A. (Bob Thorne) You can run anything you want (and supported by OVM) on DomU. If you run something like a DB on DomU, it will need to use NFS-attached storage, as only ODA Base can access the internal disks.

Q. Can you run multiple ODA base instances on a single system?
A. You only run one ODA Base per server, but each ODA Base can contain multiple databases.

Q. (jumble of questions)
A. (Bob Thorne) In the initial release, OVM images will be stored in the unallocated 250GB system disks on each server. In the future, we will allow you to store in the shared ACFS file system, but not in the initial release.

Q. Can ACFS filesystem in ODA Base be NFS exported to a DomU?
A. (Bob Thorne) Most application domains will not need a lot of space.  If they do need space, they will either need to eat out of the 250GB space on the system drive, or will need to use NFS attached storage.  ACFS internal storage exported via NFS will be a future option.

Q. Can you use a third party storage solution?
A. (Bob Thorne) You can use third-party NFS attached storage.

Q. During an ODA workshop, I was told I could increase DB licensing on ODA, but I could not go back down in licensing.  I think I heard you say you could?
A. (Bob Thorne) This {release} will relax the "going back down" restriction.

Q. What versions of OVM Server will ODA support?
A. (Tammy Bednar) OVM 3.1.1

Q. Has the BIOS disabling of cores gone altogether then?
A. (Bob Thorne) BIOS disabling of cores is still there for physical deployments, but it's not needed for virtual.

Q. How will CPU usage be recorded for auditing/compliance - through MOS like Oracle Trusted Domains?
A. (Tammy Bednar) Appliance Manager has a command to show the cpu distribution between the VMs

Sunrise photo credit: Chris Gin via photopin cc (makes a change from a server!)


Update (thanks to tweet from Fuad Arshad @fuadar): yesterday Oracle released ODA 2.5.0.
The image comes in two flavours - the physical one and the new virtual one discussed above. You can download the images from MOS - for virtual it is:
* Patch 16186163 (290MB) for the base image you apply to the servers (i.e. hypervisor/Dom0)
* Patch 16186172 (4.4GB) is the ODA Base domain (i.e. the first VM you create)
MOS 1520579.1 has step by step installation instructions including screenshots and, as ODA people will know, 888888.1 is the central ODA alert/status note. When I last checked there have been 15 downloads of the virtual image already so some people out there must be interested in trying it out on a spare ODA...

Posted by Simon Haslam on February 05, 2013 at 10:46 AM GMT #

Post a Comment:
Comments are closed for this entry.

Search this blog

About me
Oracle ACE Director (Middleware and SOA)
Presentation downloads

I'm speaking at Oracle OpenWorld

UKOUG Ambassador Partner
Oracle WebLogic Server 12c Certified Specialist
Oracle WebLogic Server 12c Certified Specialist