cPanel 11.44 introduces a new feature fresh from their feature request system that now allows server administrators to deploy the configuration of one server to multiple servers in what the they call a “configuration cluster”.  It will start by offering this capability to just the cPanel update preferences, but will eventually be expanded to work with other components as well.  The addition of the feature was announced in early April on the cPanel blog, which has a much more thorough explanation of what you can expect in 11.44.

I visited the cPanel headquarters on June 3rd and was shown a live demo of how this works and what it entails.  Overall, the concept is pretty solid in terms of how it integrates with the existing systems, and is an excellent step in the right direction as far as cPanel recognizing the need to make their software more portable for their larger customers.  However, I do have some serious security concerns about its implementation.  These concerns were expressed to and received well by cPanel, but if and how they plan to address them is up in the air at this point.

One thing I’ll mention that’s slightly off-topic from the point of this post, is that this system is not real configuration management.  If you plan to grow as a professional hosting company, use big boy tools like Puppet, Chef, or Ansible. cPanel developed the Configuration Cluster feature to appeal to novice customers who may not have the time, knowledge, or experience to use the tools that the industry has accepted.

The configuration cluster essentially consists of a remote server’s access hash being added to the server you’re using to store your ‘master’ configuration.  When you make a change to your master server, you can then elect to deploy that config to other servers in the cluster.  You’re not limited to one master server in this context.  Sound slightly familiar? It’s because the DNS clustering system uses the same concept to replicate DNS zones.  I’m not saying that’s any more secure, but we’ll get into that later.

If you’re not familiar with the access hash and what it does, let me give you the lowdown: it’s a 960-character plaintext string that is used to authenticate exclusively with WHM.  Root and every reseller on a cPanel server can have one, and once it is created, you can pass the access hash in the header of any request to WHM in place of the root or reseller password.  This is the most common and recommended method of authenticating with the WHM APIs, and is considered much more secure than using password authentication.  The access hash’s ability to authenticate isn’t just limited to APIs – technically you can use it to authenticate and do anything that you could normally do in the WHM interface itself.  With the root access hash, this includes:

• Adding SSH keys
• Changing the root password
• Changing user shells
• Installing packages

Because of this, the root access hash should be guarded with the same level of security that you would apply to a root password or private SSH key.  It ultimately grants root access to your server via WHM.

This is where the security concerns with the Configuration Cluster and DNS Cluster features come into play.  Both of these systems involve servers housing the root access hashes of other servers.  With DNS clustering among systems you manage directly, this isn’t so much of a problem as long as you absolutely do not check the Reverse Trust Relationship box, and the remote cluster server is a DNSONLY or other cPanel instance solely used to operate as a nameserver.  In the worst-case scenario of the setup I just described, if any of the servers in the DNS cluster get compromised, the most you’re looking at is someone also owning the DNS server itself.  Recovering from this is ‘just’ a matter of replacing the compromised servers and re-syncing your DNS zones.

With the Configuration Cluster feature, a determined hacker can obtain the root access hashes of all other hosting servers in the configuration cluster if the ‘master’ server is compromised.  From there, you’re looking at an inestimable amount of damage, considering that someone now essentially has root access to every server in the cluster.

With that in mind, the server that stores these access hashes needs to have an elevated level of security.  Going back to what I said about cPanel catering to the non-experienced sysadmin, the idea that this server should be secured beyond the norm of any other hosting server in your fleet is likely not a concept that would automatically register to the target audience.  But it needs to register with you, if you plan to use this feature and not have your entire infrastructure owned.

During the demo in which I expressed this concern, I was asked for suggestions on how to better implement a feature like this, and to that I verbally stated to the presenter that I was not able to immediately come up with a productive solution to this problem that didn’t negate all the hard work that cPanel’s dev team had already put into this.  And really, it’s the cPanel community that begged for this feature to exist in the first place, unfortunately without regard for the fact that a hosting server acting as a deployment agent for other servers is not something that should ever exist.  It’s a hard fact that a lot of users still do not consider the security ramifications of their actions.

So now that I’ve hopefully driven this point home, if you do plan to incorporate the Configurate Cluster feature into your infrastructure, here are some ideas for how you can make the master server (the one storing all your hashes) more secure:

• House it outside the DMZ or within an internal network with limited outside access
• Use the server to house and define configuration only – do not use it to host customer cPanel accounts, websites, email, etc
• Lock the server down with a firewall to only allow access to what is needed for it to perform its functions.  This starts with port 2087 from the server, and limiting port 2087 TO the server from only authorized IPs
• Rotate your access hashes on a frequent schedule, just as you (hopefully) do with your root passwords and SSH keys