Virtualised Oracle Database Appliance POC - #3 Physical Networking
30 Jun 2013 by Simon Haslam (in Hardware)
The Oracle Database Appliance (ODA) is one of Oracle's "Engineered Systems" - an implication of which is that you can't change the hardware components at all. Therefore it is important that its specification provided meets your requirements, and will do so for the expected lifespan of the system.
I briefly described the networking for ODA X3-2 in an earlier earlier post. In this one I will discuss the ODA's physical networking in more detail, how you configure it, and its suitability for my customer in this particular Proof of Concept (POC).
ODA X3-2 Physical NetworkingEach of the servers in the ODA X3-2 has the following ethernet connections:
- 4 on-board 10GbE (10Gbase-T copper) ports - for connection to your own networks
- a dual port PCI card (10Gbase-T copper) - for the internal connection between virtual machines (e.g. RAC interconnect, WebLogic administration channel)
- a Fast Ethernet (100base-T) port for the ILOM management controller
These are configured within the ODA (virtualised ODA 2.6.0 image) as shown in the diagram below (which is taken from p14 of the Oracle X3-2 and X3-2L System Architecture white paper but with my own annotations added):
This diagram pretty much describes all you need to know about the physical network connections to the ODA X3-2.
Bonding: as I mentioned previously I was a little surprised the bonds for the on-board ports didn't span controllers, which would have been easy enough to arrange.You can also see that the on-board ports are served via 2 devices by looking at this lspci output from one of the nodes:
20:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
20:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
88:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
88:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
a0:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
a0:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
Perhaps this bonding arrangement was done to make cabling a little simpler, though they could also attach labels on the chassis to help. If you were specifying your own server you would normally bond the on-board ports with a dual or quad port NIC (though you would usually specify a 2U server to give you sufficient PCI slots). However, whilst not perfect, this approach probably doesn't matter for the ODA's target market.
By the way, it's nice to see the Ethernet controllers for the on-board and plug-in card have the same Intel X540-AT2 chipset - this has to make sense from a maintenance perspective.
Configuring the External Network
In theory you use the ODA configuration tools to set up the network. We found that the virtualised ODA tools were somewhat less mature than they appear to be in the physical one.
On the virtualised ODA once you have booted the servers you run the oakcli configure firstnet command to set up the network. Below is an example of the procedure, with default responses repeated for clarity. Note that these prompts will probably change slightly between versions (this is 2.6.0).
[root@oda1 veriton]# /opt/oracle/oak/bin/oakcli configure firstnet
Configure the network for the node(s)(local, global) [global]: global
The network configuration for both nodes:
Domain Name: veriton.co.uk
DNS Server(s): Primary Dns Server: 10.0.10.1
Secondary Dns Server: 10.0.10.2
Tertiary Dns Server:
Node Name Host Name
Choose the network interface to configure (net1 , net2) [net1]: net1
Configure DHCP on net1 (yes/no) [no]: no
You have chosen static configuration on net1
Enter the IP address for net1 on Node 0: 10.5.5.169
Enter the IP address for net1 on Node 1: 10.5.5.170
Netmask for net1: 255.255.255.0
Gateway Address for net1 [10.5.5.1]: 10.5.5.1
Plumbing the IPs now on Node 0 ...
Restarting the network now on Node 0 ...
Plumbing the IPs now on Node 1 ...
Restarting the network now on Node 1 ...
Handily this command (when you accept the default of global) applies the changes automatically to both ODA nodes for you. It also sets up jumbo frames for the private network (MTU 9000 in this case), though doesn't have an option for changing the default 1500 bytes of the external ones. What's curious is you don't have chance to set up a second network (unlike in the Oracle Appliance Manager when you're setting up a physical ODA, or ODA base). The networking configuration ends up in a file called /opt/oracle/oak/conf/dom0.xml.
From experimentation I have found that you can later re-run this command to configure net2, but it affects the default gateway. For a more detailed debate see my posting on the My Oracle Support ODA community.
How the ODA would fit into the Customer's Environment?
Of course the purpose of this POC is to find out how well the ODA meets the customer's individual requirements. So, firstly let's outline their existing networks. They have a core GbE network to which everything directly, or indirectly, connects. In addition they have a more recent 10GbE network which supports jumbo frames and is primarily used for storage (NAS filers). Finally there's an isolated backup network which, as its name suggests, is used for backing up the servers and, more importantly, for sending Data Guard traffic to a second site.
As I have described the ODA only has 2 external networks. If it were just acting as a database server it could connect one to the core network and one to the backup network. What makes it trickier is now that we are considering middleware too where we need to connect to the storage network. This is before you start thinking about DMZ's and the like.
To deploy ODAs it's quite feasible (see diagram below) that we would be able to change the network to run either the core, or the backup, over the (10GbE) storage network (as in the diagram below) but either option would need some re-design work, and so have cost implications. In an organisation with strong segregation between teams that may also mean political challenges too ("why should I change our network architecture for your Oracle machine?!").
One thing immediately apparent from this diagram though is that the ODA's limited interfaces also enforce a degree of simplification; this is rarely a bad thing!
In an ideal world, or greenfield site, you'd probably have a single, converged 10GbE network that everything would run over and sufficient network controls would be in place to manage the traffic prioritisation etc. However no-one I know yet is in that position (and if you've ever been on the receiving end of, for example, a Cisco Nexus quote, you'll know why) so inevitably you replace components as the traffic warrants it and equipment goes end-of-life. For me it did highlight one adjustment that you may have to make in order to accommodate fixed specification systems like the ODA, even if in the long run it is more beneficial.
That's most of what needs to be said about the ODA physical networking - my next blog post will be on provisioning the virtual machines.