Simon Haslam's Oracle Fusion Middleware blog

Admin Server Separation - Pros and Cons

The majority of my Fusion Middleware work is focussed around designing application infrastructure providing appropriate levels of performance, resilience, flexibility and cost; one topic that comes up a lot is how to deal with WebLogic admin servers.

Personally I feel quite strongly that, for all but the simplest installations (or those where you have a tight installation package, such as Forms or EM 12c), the admin server should be run on a separate host (typically a virtual machine) from the managed servers. Curiously, this was one of the first conversations I had with Jacco Landlust maybe 4 or 5 year ago, and it turned out he had, independently, come to the same conclusion... maybe this was a good omen for our various colaborations since! Between us we have built perhaps a dozen Fusion Middleware infrastructures using this approach and my opinion hasn't changed.

So, let me outline the considerations for separation (though this also strays into the debate over admin server high availability - HA - which deserves a post of its own):
Jacco Landlust & Simon Haslam co-presenting at UKOUG 2012
Jacco Landlust & Simon Haslam co-presenting at UKOUG 2012

Firstly, the Admin Server for HA has to run active/passive. This is different from most services on managed servers which run active/active. Even when you look at singletons like JMS queues, or some services within OSB, you still have a relatively straightforward method within WebLogic to handle that. So architecturally I would say that the admin server is different to the managed servers.

I appreciate that some will dispute that Admin Server should be highly available. Yes it's true that the WebLogic management part of the admin server, strictly speaking, only needs to be running to make changes to the domain; in fact some purist Java EE sites only run the admin server for the duration of changes, shutting it down afterwards - the rest of the time all monitoring, restarts and so on, are handled by hand-written WLST scripts and/or JMX utilities. However in Oracle environments the OEM Fusion Middleware Control (FMC) is becoming increasingly important  - you only have to see the functionality that Oracle adds to it in each release. The "old school" view for admin server protection is to copy round the domain somehow (re-mount the LUN, use a clustered filesystem, rsync or just plain FTP) and if the server hosting the admin server fails, along with the set of other more business critical managed servers, you "just" have to shuffle the VIP around, mount the LUN or copy files, and then start the admin server in its new location. Don't forget all this tinkering about is on one of your production managed server hosts and so very close to live services - if you only have two servers, on the last one. My view is that modern systems shouldn't need this kind of maintenance - even following a single hardware failure they should continue to "just work" without any expert intervention. For me this means high availability for the admin server too.

Then some people say the admin server footprint is very small so it doesn't matter where it runs. Well - yes and no. Firstly, anywhere you expect the admin server to run you need to leave spare resources for it - in memory that's currently around 1GB for 11g (WLS and FMC), but could well be closer to 1.5GB for 12c, plus a processor core or two (especially if you have SOA and don't want instance tracking via FMC to have an impact). Production Oracle licences are normally determined by cores, so if you are reserving a core or two for it that has an implied cost.

Another objection to the idea of admin server separation is that Oracle doesn't mention it in the documentation. In the EDGs for example they just say set up an administration VIP and refer you to the general CFC section of the HA Guide to think about what to do with it. Also Oracle employees don't talk about separating the admin server - possibly because no-one's standing back and questionning the conventional wisdom (developed maybe 5-10 years ago before widespread virtualisation), or maybe because they'd be accused of trying to increase the licensable footprint, I don't know. Anyway, there is a glimmer of hope - the newly architected "WebLogic running on Oracle Database Appliance" has the Administration Server running in its own VM.

Finally admin server separation offers a possible extra degree of security isolation: this is an argument I personally haven't used but one that could be relevant in very secure environments (e.g. those sites running products like Tripwire to lock down hosts). The admin server is pretty powerful - it can shut down any server in its domain, change the configuration, change the security providers and so on. A compromised host running a Managed Server could only really have an affect on itself which, whilst bad enough, could be self-contained. You could put separate administration hosts into a more highly trusted security zone than the managed servers, and restrict access to administrators. You can imagine an array of management products in such a zone: administrations servers, identity management consoles, enterprise manager OMSes etc.

Now let's look at the apparently biggest downside to separating the admin server onto its own server/Virtual Machine: licensing. You have to pay for a licence and support for the admin server (unless it's in a VMware VM in which case you're stuck paying for the whole host anyway), and this licence needs to cover the same products used in the domain (e.g. a SOA admin server will need WebLogic Suite plus SOA Suite). However you should be able to license the admin server by Named User Plus (NUP) minima, plus whatever you need for fail-over. On this basis separating the admin server for a cluster of 2 quad core x86-64 production servers adds 1/5 of a processor licence or 5%. Obviously for larger clusters this becomes even less significant. If you have a precisely sized system you could even argue that it saves you money as those cores used by the few administrators and licensed by NUP cost only 20% of the cores used by the managed servers. Note I have ignored hardware cost here because it's not usually very significant compared to the Oracle licence.

So, in summary, this is why I think you should separate the Admin Server for well-engineered, high availability OFM infrastructures:

  • Architectural cleanliness
  • Ease of managing admin server failover
  • Isolation from, and risk reduction for, transactional servers
  • Improved layer of security possible
  • More efficient licensing of the managed servers

Jacco and I have been talking about separated admin servers at Oracle OpenWorld and elsewhere for at least a couple of years but very few Fusion Middleware administrators seem to consider it. I hope this post presents some convincing arguments and I'd welcome any comments, for or against!

Comments:

Hi. I'm a total noob at all this, but I have to agree. It makes me very nervous having the AdminServer on one node of a 2 node cluster. Also, when you have multiple domains on a server, having a bunch of AdminServers lurking around is a significant overhead. My boss was sitting with me in your joint session at UKOUG where you mentioned this, but I will be forwarding this post to him as a reminder. :) Cheers Tim...

Posted by Tim Hall on June 17, 2013 at 08:41 AM BST #

Yes, you're right Tim - I should have mentioned that too - if you have several domains, especially if they pertain to one "system", why not have them all together and only have to deal with failover once? There are some trade-offs in configuration complexity versus flexibility (and licence management of course!) but I have 3 domains/admin servers on a host at one site and it works fine. Thanks for dropping by :)

Posted by Simon Haslam on June 17, 2013 at 11:03 AM BST #

Hi Simon, I don't really agree with you, most of your points can be solved by taken some steps. Architectural cleanliness * when you use different domains for the admin and managed server -> follow the EDG doc ( mserver and aservers domains then you have them separated) Ease of managing admin server failover * we also do this with a virtual IP and add this to a nic of a server so we can move this to every server Isolation from, and risk reduction for, transactional servers: * The admin server does not need much also on a soa domain ( it is always the soa infra database) * you need off course to tune the soa EM by using lazy loading and never show more then 1 hours of instances. * Off course it uses lots of memory but nowadays the minimal is 24GB or more when you buy a new blade * We also have dedicated channels/lan for admin and cluster traffic so the admin and the other servers are always reachable. * nowadays everything is on vmware or oracle vm so it can still run on the same box/blade. Improved layer of security possible * when using virtual ip and ssl this should not be a problem More efficient licensing of the managed servers * with vmware, oracle vm or cloud this is already taken care of I think this is too much ( too admin & perfect for me) and nowadays with all those cores and memory plus the great options you have in weblogic I find it ok when you do the above steps. You can better tune soa composites/ soa infra db, there is much more too gain then separating an admin server. Thanks Edwin

Posted by Edwin Biemond on June 17, 2013 at 09:04 PM BST #

Personally I feel that EDG's make FMW environments way complicated. All these VIP's for managed servers make the environment so complicated that the only name I can give it is "consultant friendly". Especially since all you are looking for is the JTA transaction logs, you don't need whole server migration for that! And you sure don't need a service like nodemanager to run as root (or with very strong sudo rules) on you systems. On the other hand you do need a VIP for the administration server because this singleton service cannot be failover to some other piece of hardware if it fails. So I disagree with most of the EDG's :)

Nobody can disagree that tuning SOA instances and databases should be done properly, but performance tuning is all about managing expectations. Whenever performance fluctuates from time to time people get nervous. In my day to day experience I see "functional" administrators tracking soa instances, and those guys are sure as hell no FMW technical experts. You can't blame them for tracking multiple instances, or for running stupid reports, or for choosing less intelligent tuning settings. They have the tools so you can expect them to do something "unexpected" with these tools. By separating the administration server you can at least limit the impact of that kind of administration actions and therefore get a more stable performance. And IMHO stable and predictable is what every production admin wants.

It is very funny that people keep on focussing on pushing down development costs, most money in the lifecycle of an application is actually spend while running production (I read some gartner note saying over 90% ! ). So whatever can push down the maintenance costs is real value. None of my customers run FMW on physical, all use some sort of virtualization. When you run multiple domains and you share the "hardware" for the administration servers the license overhead is very small. Therefore cost of license and hardware is negligible. This is fully in line with that gartner investigation. So I conclude making environments more simple really saves money.

* Having all administration servers in one subnet that is firewalled from the rest of the company (and world) so only designated administrations can access the administration server is a very simple to implement security measure (therefore saves costs in maintenance)
* Having administration servers separated from managed servers makes performance more predictable and therefore administrators are less often send on wild goose chases (and therefore saves costs in maintenance)
* Having an administration server setup as a cluster resource (with whatever cluster software you prefer) makes that the technical administrator can fix issues during the day instead of in the middle of the night when some functional guy needs to investigate some SOA instance (and therefore saves costs in maintenance)
* Having less VIP's in your environment saves in administration overhead and makes it easier to debug issues in production (and therefore saves costs in maintenance)

I see a pattern :)

Anyway, good discussion. I love to hear more comments!

Posted by Jacco H. Landlust on June 17, 2013 at 10:38 PM BST #

Hi, I think reducing complexity is always a great plan. nowadays we even think to avoid clusters and just scale up inside the box by adding cpu's in vm or physical. I have seen more downtime cause of database leasing , shared storage or load balancers. a domain with some managed servers and a oracle database with dataguard works perfectly. for most companies this is enough. the only reason to move an adminserver is cpu utilization. the rest is feeling/architecture or is nice to have . maybe security is a good reason but a managed server can also be hacked , probably it is on the same OS patch level, you can also invoke mbeans on a managed server or put a program on this server by some buffer overloading. off course we use reverse proxies , protocol switching to reduce the risk. but when you have a high cpu utilization > 40% then I would move this admin server to its own server or put it at the osb server ( stateless) continuos high cpu is almost impossible with soa suite ( there is always some waiting going on cause of a slow reference service , a soa infra db , too many invoker threads etc). cheers

Posted by Edwin Biemond on June 17, 2013 at 11:46 PM BST #

I agree with Jacco: "EDG's make FMW environments way complicated". It is great to have the option to do it the EDG way, but being able to remove complexity if you do not need it is important. Having admin servers set up on different servers than the managed servers makes it easier to size the managed servers - for example then you do not need to count for admin server memory. You could also put admin servers on a virtual server, and managed server on physical servers (and not care about the #processors on the virtual servers). I feel that the licencing of admin server nodes makes it more difficult to really have a free choice on the architecture.

Posted by Jon Petter Hjulstad on June 18, 2013 at 10:04 AM BST #

Thanks for all your excellent comments, especially to Edwin for taking the opposing view! Let me tackle a few of Edwin's points:
* Architecture: yes, I know the EDGs separate out the admin server domain home from that of the managed servers - it still doesn't make it much cleaner IMO as you have a "lump" of Admin Server on one node in your cluster.
* "we do this as a virtual IP": Quite... but how exactly do you implement that? You can't use whole server migration but have a few choices: installing Oracle CRS and setting up a resource under ASCRS, installing some open source thing (Jacco wrote an excellent blog post about CoroSync and PaceMaker), manually (as root) running ifconfig (or scripts for it) and sending out a promiscuous ARP, or just worrying about failure after it happens? I'm not saying it can't be done - I'm just saying it's messy and messy usually means unreliable ;-) Plus, for me, a component failure should not need an operational reconfiguration (i.e. approved change request) of an HA system.
* I agree that servers have lots of memory these days - 48GB or 96GB is common, and I only really see virtual servers now for middleware installations. So you could say all VMs in a cluster can have enough spare memory to support an admin server. Alternatively virtualisation is also a reason why it should be easier to have the admin server separately on its own VM - it's easy to do and you don't need to justify buying extra hardware.

I think this is a really interesting debate though - are there any other opinions out there?

Posted by Simon Haslam on June 24, 2013 at 11:24 AM BST #

Post a Comment:
Comments are closed for this entry.

Search this blog

About me
Oracle ACE Director (Middleware and SOA)
Presentation downloads



I'm speaking at Oracle OpenWorld


UKOUG Ambassador Partner
Oracle WebLogic Server 12c Certified Specialist
Oracle WebLogic Server 12c Certified Specialist

Links

Feeds