Simon Haslam's Oracle Fusion Middleware blog

Oracle Exalogic - initial analysis

The biggest middleware announcement last week at Oracle OpenWorld (OOW) was arguably the new "Exalogic" server. Rumours had suggested that there was a middleware machine of some sort in the pipeline, though its name was kept extremely well under wraps by Oracle. Here is my analysis of the Exalogic product based on the OOW keynotes and a product presentation, discussions with an Oracle representative on the Exalogic stand and my understanding of Oracle's existing Fusion Middleware 11g and virtualisation software. Note: given that the product was just announced last week, and only one has been seen publically outside of Oracle's engineering labs, inevitably this article contains assumptions and quite a lot of speculation!

Oracle Exalogic front branding
Photo courtesy of  Markus Eisele (c)2010

Exalogic Hardware

The Exalogic (EL X2-2) hardware is a collection of x86-64 servers and a shared storage appliance, all linked together using InfiniBand (IB) QDR interconnects running at 40Gb/s. Up to 8 Exalogic/Exadata machines can be connected to the same IB 'fabric'; if further machines are required another switch (pair, presumably) can be added to enable very large clusters indeed to be built.

Let's look at the hardware that makes up this machine - its key components are:
  • Sun X4170 M2 Servers (1U), each with
    • two Intel X5670 6 core 2.93GHz processors
    • 96GB 1333MHz RAM
    • one dual-port InfiniBand HCA*
    • two 16GB SSDs
    • dual (redundant) power supplies
  • Sun 7320 ZFS Storage Appliance (4U) with two "head units" (active/passive, each 1U high), SAS disks with total raw space of 40TB (apparently about 18TB usable after mirroring and striping) and 4TB read/72 GB write cache.
  • Two InfiniBand (IB) switches, each providing 32 IB ports and 8 10GbE ports (a new variant of the Sun Datacenter InfiniBand Switch 36 with a 10GbE bridge perhaps?)
  • One Cisco ethernet switch for the management network
  • A 42U 19" rack (39U used in the full rack configuration)
  • About 200 small, green lights (don't quote me on that!)

* HCA = Host Channel Adapter - equivalent to the HBA in Fibre Channel implementations

All the servers are connected to the two active-active IB switches for inter-node data and storage access. InfiniBand has the concept of "virtual lanes" which is used to segregate data on the interconnect, for both performance and security reasons. The data centre ethernet (e.g. ultimately connected to the internet) connects via a bridge to the IB network, and the other ethernet network is used for the management ports (such as the ILOM on the servers).

Exalogic inside

The mirrored SSDs on the servers are "used for OS images and high-performance local swap."

Like Oracle's Exadata, this product uses standard rack mount servers (1U in this case) rather than the blade servers and chassis familiar to consolidation products from some other vendors.

The Exalogic comes in 3 sizes: full rack (30 servers), half rack (16 servers) or quarter rack (8 servers). The remaining hardware is the same for all configurations (though I'm not sure about the number of IB ports - it may be simpler for Oracle to ship identical IB switches, as well as easing the upgrade path). There will be upgrade kits available to enable customers to move from quarter to half to full rack variants at a later time.

Regular readers will notice that the processors are the rather tasty 32nm Intel Westmere-EP (Gulftown) 56xx series chips which I reckoned were the x86-64 middleware sweetspot earlier this summer. However where I had anticipated WebLogic clusters of maybe 2, 4, 8 or so servers, the Exalogic has 30 (before you start daisy-chaining racks that is!)... one heck of a lot of computing power.

Operating Systems

Now things start getting really interesting! You will be able to choose to run Oracle Enterprise Linux, Solaris x86, or an Oracle virtualisation product on your Exalogic. Mike Palmeter outlined three planned versions:

  • V1 This is the release announced at OOW which will have the operating systems running on physical servers, either Oracle Linux 5.5 or Solaris 11 Express (due soon),
  • V1.1 The subsequent release will be able to use OVM 3.0 to run virtual machines (either bare metal hypervisors or Solaris containers),
  • V2 A future release will include advances such as Java RDMA to provide even higher performance access to the storage (and presumably other nodes?) etc.

Oracle say that the upgrade from each version will be software only and be fully automated.

Clearly V1.1 is the one I am most interested in - this is the first version where you will be able to use virtualisation in a fully supported manner. On a machine of this size I expect the use of virtual machines will be critical to be enable you to carve up the compute resources efficiently, essential for the "private cloud" or "utility" model of running servers. Oracle's Virtual Assembly Builder software should allow you to provision new middleware infrastructures much more easily and duplicate them very reliably. For example you might have a complete WebLogic/Coherence, SOA Suite, BPM, Web Center and Web Tier assembly built, tested and ready. You should then be able to just "drop" the entire assembly into a new environment on the Exalogic in one go... middleware nirvana in my book!

'Exalogic Elastic Cloud' Software

This is where you have to work hardest to establish what's under the covers. The "elastic cloud" software moniker is being used for a host of optimisations Oracle are building into WebLogic Server, JRockit (and presumably Hotspot), JRVE and the Oracle's optimized Linux kernel, JDBC and OVM - much of it appears to revolve around network I/O, and in particular, making best use of InfiniBand (e.g. by circumventing the TCP stack).

Having had a detailed discussion with Frits Hoogland in the OTN Mason Street Tent about Exadata it seems like Oracle are really onto something here, though I haven't really got all the layers and protocols straight enough in my own mind yet to feel confident enough to write about it. The Elastic Cloud term probably also covers extensions to the management software (monitoring, deploying and upgrading environments in bulk).

 Exadata and Exalogic servers next to "Ironmans" (Ironmen?) at Oracle OpenWorld 2010

In the same way that the Exadata Storage Server has special features reserved only for use by the Database Machine (and vice-versa), the "Exalogic Elastic Cloud" software may only be available on Exalogic hardware.

Target Applications

As its name suggests, Exalogic is designed for WebLogic Server application servers, but of course will support other applications certified for the operating systems running on the x86-64 servers. WebLogic Server provides the foundation Java EE platform for a plethora of products from the SOA Suite through, ultimately, to Oracle's new enterprise applications. Therefore I think we can expect to see Exalogic and Exadata being sold as the best hardware on which to run your Fusion Applications.

Performance

I'm not in a position to comment on Exalogic performance other than, at the very least, the combination of so many Intel Westmere-EP chips (a full rack is 60 processors, 360 cores or 720 hyperthreads) and 40Gb/s InfiniBand means that it will have a collosal amount of processing throughput.

The Complete Exalogic Picture

If you've persevered enough to reach this point you may have been surprised to find that I decomposed Exalogic into its constituent parts before describing the big picture, in contrast to how it has generally been presented so far.

Oracle sees the appeal of Exalogic is as an integrated system, from the hardware upwards. Most of the technical decisions have been made by Oracle's engineering teams, saving the customer work and risk of sub-optimal configurations. I'm sure for CIOs and IT Directors of many companies this simplification will make Exalogic very attractive. An Exalogic server becomes a single line item on a Purchase Order, it is maintained by an Oracle field service department and it is operated by a single team in the customer's organisation (rather than a combination of storage, network, OS, and middleware administrators). Indeed such non-technical reasons may well in part account for the success so far of Exadata; if that's the case Oracle ought to be able to replicate that success with Exalogic.

The list price for a full Oracle Exalogic rack is expected to be US $1,075,000.

Incidentally, Oracle are comparing Exalogic with the IBM Power 795 server. Looking at its datasheet it too is a big beast, e.g. with up to 256 POWER7 processors running at 4GHz each with either single, 2 or 4 simultaneous threads per core. In my opinion I would say that Exalogic is a horizontally-scaled commodity server based design, whereas the Power 795 is a vertically-scaled mainframe-style design. Comparing the two is rather like comparing Godzilla to King Kong and asking which is best! In most respects a more similar comparison would be Exalogic versus a rack of Cisco UCS or HP BladeSystem Matrix, albeit both Cisco and HP solutions are rather more "assemble it yourself" than Oracle's pre-built solution.

Conclusions

Exalogic is a significant announcement, and will probably keep industry analysts happy as it shows Oracle has derived benefit and developed significant new products as a result of their hardware engineering assets received from the Sun acquisition. It also demonstrates that Oracle is very serious about high-end pre-configured appliances - between Exadata and Exalogic Oracle can now supply a significant proportion of the compute power for many enterprise IT facilities (i.e. the mainframe as was).

Of course, the biggest drawback for both Exalogic and Exadata is that, for today's workloads, they may simply be too powerful for many organisations, particularly those who haven't yet started aggressively consolidating applications. Given that they both rely on balanced stacks of x86-64 processing power with InfiniBand and chunks of fast storage, perhaps the two could be being combined... half a rack of Exadatalogic anyone?

For me, the most interesting aspect of Exalogic will be how Oracle blends its various pieces of software to work as a cohesive and easily manageable whole - this will help engender best practices for us to configure Oracle Fusion Middleware in more modest infrastructures.

 

References:

Exalogic Elastic Cloud - A Brief Introduction (Mike Piech and Mike Palmeter) http://www.oracle.com/us/products/middleware/exalogic-wp-173449.pdf

Oracle OpenWorld 2010: (S317409) Oracle Exalogic: Introducing an Optimized Compute Platform for Oracle Fusion Middleware (Mike Palmeter) http://www.eventreg.com/sb250/loginOD.jsp

For further information see: http://www.oracle.com/us/products/middleware/exalogic/index.html

Comments:

Hi Simon:

There are 5 IB switches in a full rack Exalogic:
  • 1 36-port switch (same as those used in the Exadata rack)
  • 4 32-port switches that each have a 10GbE port on them as well.
The half and quarter racks each have 2 of the 32-port switches (no 36-port switch in the partial racks).

Posted by Dan Norris on September 29, 2010 at 11:25 PM BST #

Thanks for the update Dan! I've had a separate chat with Dan and it appears the networking infrastructure is rather more involved than I described above to allow for redundancy as well as plenty of connectivity to additional Exalogic and/or Exadata racks.

Posted by Simon Haslam on October 05, 2010 at 10:15 AM BST #

Post a Comment:
Comments are closed for this entry.

Search this blog

About me
Oracle ACE Director (Middleware and SOA)
Presentation downloads



I'm speaking at Oracle OpenWorld


UKOUG Ambassador Partner
Oracle WebLogic Server 12c Certified Specialist
Oracle WebLogic Server 12c Certified Specialist

Links

Feeds