Monday, July 16, 2012

Recreational XenAPI, or The New Adventures of Citrix XenServer

Today, I would like to speak about certain aspects of using Citrix XenServer 5.6. The problem I had to deal with seemed to be rather solvable: command execution in dom0 without using SSH. While searching methods to fix the issue, I found some funny features of HTTP API of the operating system: ways to get /etc/passwd, remote execution of rsync and XenSource thin CLI protocol. Now I will tell you a kind of a story of a research.

First, let's consider the origin of the object. Recently, I have released a public beta version of a security guide for XenServer, which I'm doing in order to write a clear manual. One of the recommendations (on the analogy of Security Hardening Guide for VMware ESXi) is to disable SSH daemon. The motivation is that the corporative version of Xen has an option to use the RBAC system with authentication through Active Directory. According to the vendor's recommendations, this method is preferable from the safety point of view. After certain modifications of the console running scenarios in dom0, specified in my guide (, it becomes impossible to access it through the system without entering password. Not only a password of a user with pool administrator privileges are needed to access dom0, but also root account data.

ОK. Now our task is to carry out a remote audit of the operating system using automated means. What we got at our disposal is XML-RPC leading to XenAPI, its documentation and Xen-org source code in OCaml. However, we do want to execute commands in bash and get their output for further processing. How shall we do that?

First, we should understand why we cannot do this by regular means (through the console that is provided in API). Let's recall the process of call of the console from the client: you connect to the console (https:///console?ref=OpaqueRef:console_id) using valid session_id and get to the RFB terminal vncterm. Of course, the protocol allows sending mouse activities and key pressing to the remote server and receiving raster images. Further steps are clear: modern versions of RFB protocol also allow transferring files. It takes only to study command execution and the problem is solved. But it would be too easy. Citrix uses the RFB protocol version 003.003 in its terminals vncterm: This version does not support file transfer.

Considering this unfortunate news, our developers started to analyze possible methods of transferring via RFB, version of the year 1998. Here are two ideas they came up with. First, integration with ABBYY FineReader (supporting recognition of text in raster images received from dom0). Second, emulation of mouse movements, which allows selecting text on the display and sending it to the exchange buffer available in the protocol. On a closer examination, both methods turn out to be absurd.

Gloomy prospects made me return to the XenAPI documentation reading. This time there was something that draw my attention. Plugin architecture. That is, a possibility to call your own executable file via RPC call_plugin. Modules are in the directory /etc/xapi.d/plugins/.

Now it's simple. The plugin we created is called via XML-RPC and runs the appropriate script in Python, which executes commands through subprocess. Great! Methods of command execution in dom0 and receiving a reply are clear.

Suddenly, a problem appeared. How should our plugin get to the server? While fixing the problem, we found certain hidden rocks in XenAPI.

Of course, I got interested in a function that you can access via a xe.exe tool — patch-upload. It allows you to load files remotely to XenServer and to install them to the whole server pool. Data representation format is rather plain: shar which is zipped and signed (!) by Citrix. When loading the patch, the signature is verified with a set of corresponding keys in gpg keyring. So just add your signature to the set and the problem of the plugin uploading stands no longer. It's not hard to create a similar structure, but to add your key you need access to the console. It's a vicious circle. That's why I started to search for other methods to upload the plugin.

While using the call I noticed that the official description of API does not provide such call ashttps:///pool_patch_upload. Explanation is that it is not a part of API. The question imposed by natural curiosity is — what is it then? You can find the answer easily with the help of Wireshark.

You may criticize me for straightness, but I would say that HTTP interface of XenServer API is not described at all. Moreover, I didn't know OCaml at such a level to be able to analyze source code efficiently, when I faced with this problem.

I used a splendid method for TLS decryption provided by Wireshark and a certificate in /etc/xensource/ left carefully where it can be easily found, and got a dump of communication between the xe.exe tool (from XenCenter) and the server.

I expected XML-RPC communication, which is described in the official documentation. No such luck! "POST /cli HTTP/1.0" was displayed instead. The tool sent a command and its attributes to https:///cli. There's something missing. According to the protocol decryption, the tool used a XenSource thin CLI protocol. All roads lead to Github, namely to XenAPI source code.

After some period of time (which I spent reading the source code of this wonderful component), I found out that XenSource thin CLI protocol 0.2 exists and executes commands of the xe.exe tool on the remote host.

It is described in xapi/ It's worth mentioning that this is an "API of the future" designed to make the xe.exe tool able to forward commands and to build the handler into XenAPI.

Basically, we just had to discover the CLI API. It indicates that not only XML-RPC receiver and switch /console are presented in port 80\443. Other modules that are available via such call were discovered by accident in one of the source code files (xen-api/ocaml/idl/ It's pretty easy to guess that a great number of calls provided rather interesting pieces of information. There was a remarkable call https:///syns_config_files: if you have pool administrator privileges you obtain /etc/passwd (I've already mentioned in the previous articles, it is here where XenServer stores passwords hash).

Another interesting call is made via “CONNECT /remotecmd?cmd=rsync&arg=some_nice_arg &pool_secret=your_pool_secret”. It allows remote execution of rsync on the server with root privileges, if you know the value of /etc/xensource/ptoken. In fact, it gives unrestricted access to the file system. You may ask, how should I get ptoken?

It's even easier. The Xensource developers made it possible to remotely get the pool contents in XML file. If you execute the command such as "GET /pool/xmldbdump?session_id=", you will get a set of key-value pairs, among which you can easily find the necessary pool_token.

Remote patch uploading is actually performed via "PUT /pool_patch_upload?session_id=". The server will answer, 200, OK. And will wait until you upload the information. As soon as you upload the file, the patch validity check will launch. But there's one feature: while you're holding the connection, API thinks that you're still uploading the file and doesn't use it (though the file has already been created in /var/patch). File length check hasn't been discovered. Since /var/patch is in the server's root partition, DoS is unavoidable if /dev/urandom is sent there.

Of course, it is only half the story. You can get more information on calls and necessary privileges here. The code description is accurate and I'm sure it won't be difficult to find the answer to a well-stated question there.

Actually, the said methods were enough to upload a plugin to the system without signature verification. I'm not going to provide a detailed methodology, cause it borders on "vulnerability exploiting". I'm sure you got the point.

Author: Kirill Ermakov, Positive Research


  1. There are actually a lot of new aspects of Citrix XenServer 5.6 and a lot of them are described at this site, so if you would like to figure out more you may check it out.

  2. This comment has been removed by the author.