Elasticsearch shield: First steps using java

29 Jan 2015 - Jettro Coenradie

Elasticsearch does not come with security out of the box. Just a few days a go they launched the product called Shield. This is the way to enable security within your elasticsearch cluster. Shield is launched as a commercial product that you get for free when buying a support contract with them. In this blog post I am sharing the first steps I took with Shield. I’ll give pointers to the available documentation where possible.

Introduction

Shield is the way to implement security in elasticsearch. There are a number of things it provides: authentication, authorisation, but also encryption of data between nodes. In this blog I am focussing on getting users autheticated using curl and the java driver. To track if things are working I use the auditing options available through shield and finally I will look at creating your own roles for using your indices. If you want to do this yourself you should start with the getting started guide. The last time I am going to mention this, Shield is not free to use. You need to have a license to keep working with it. How this works is explained in the next section.

Installation of Shield

Since I am not going to use secured connections between the nodes, installation is faily easy. We need to install two plugins, the license plugin and the shield plugin. Before we install the plugin I want to explain a little bit about my development environment. There are some small things you need to take care of when using the esusers script.

Since I work on different projects using elasticsearch I have a setup with different config/data/logs folders for each project. I do not want to reinstall the plugins for every project, therefore I have a plugins folder that I use for all projects. Below is my directory layout.

• elasticsearch-1.4.2
• projects
  • playground
    • logs
    • data
    • config
    • startup.sh
• plugins
  • gui
  • marvel

Installing plugins is usually done from the elasticsearch-1.4.2 folder. Now the script needs to know where to install the plugins. This is easiest done by providing an environment variable ES_JAVA_OPTS. Shield also needs to know the location of the config folder. That is a little bit harder with this setup. I have multiple folders with configs. Depending on the project I run. For now I just export these variables in the window I am going to install the plugins from as well as creating shield users.

export ES_JAVA_OPTS="-Des.path.conf=/elasticsearch/projects/playground/config -Des.path.plugins=/elasticsearch/plugins"

We are almost there, at the moment installing the plugins now works fine. Using the esusers still has an issue. It does not use the -Des.path.plugins. Therefore I have created a virtual linke from the elasticsearch-1.4.2 folder to the plugins folder and names this link plugins. Now everything is going to work fine.

First install the license plugin then we install the shield plugin.

bin/plugin -i elasticsearch/license/latest
bin/plugin -i elasticsearch/shield/latest

What just happened? For the license we just installed a plugin in the plugins folder. Shield did more. For shield we installed a part in the plguins folder, but we also added some tools to the bin folder of elasticsearch and we installed a number of files in the config folder. For now the most important file in the bin folder is esusers. The config folder contains a file with the users, users_roles, role_mappings, logging, and roles. For this blog we are going to interact with the users file using the esusers script, we are going to make some changes to the roles.yml file.

Now you can start the node, if you check the logs you should get see the following lines about a license. By default a license is created for a month. This is an evaluation license. After a month your cluster will have limited functionality. Check this page for more information about licensing.

[2015-01-29 15:45:50,355][INFO ][shield.license           ] [Node-jc] enabling license for [shield]
[2015-01-29 15:45:50,355][INFO ][license.plugin.core      ] [Node-jc] license for [shield] - valid
[2015-01-29 15:45:50,357][ERROR][shield.license           ] [Node-jc]
#
# Shield license will expire on [Saturday, February 28, 2015]. Cluster health, cluster stats and indices stats operations are
# blocked on Shield license expiration. All data operations (read and write) continue to work. If you
# have a new license, please update it. Otherwise, please reach out to your support contact.
#

Now our cluster is secured, well at least it has some security enabled. Before we try it out, let us first enable auditing. Than we can trace what is happening.

Enable auditing

Enabeling auditing is as easy as adding the following line to your elasticsearch.yml

shield.audit.enabled: true

Now a file will be created with in the logs folder with the name of your cluster and -access.logs, in my case jc-play-access.logs. In the next section I’ll show you what is going into the file.

Interacting with the cluster

Without any user, do a curl request to find the status of the cluster. Than check the response as well as the audit log.

Request
curl -XGET "http://localhost:9200/_cluster/health?pretty=true"
Response
{
  "error" : "AuthenticationException[missing authentication token for REST request [/_cluster/health?pretty=true]]",
  "status" : 401
}
Access log
[2015-01-29 17:05:17,033] [Node-jc] [rest] [anonymous_access_denied]	origin_address=[/0:0:0:0:0:0:0:1:51087], uri=[/_cluster/health?pretty=true], request_body=[]

Before we can fix this we need to add a user. This is done using the esusers script installed in the bin folder. The following command creates a user with name jettro and password nopiforme. You can omit the password, than the password is asked for. We give this user full access by using the admin role.

bin/shield/esusers useradd jettro -p nopiforme -r admin

Now we need to provide the username and the password using the curl request.

Request
curl -XGET --user jettro:nopiforme "http://localhost:9200/_cluster/health?pretty=true"
Response
{
  "cluster_name" : "jc-play",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 11,
  "active_shards" : 11,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2
}
Access log
[2015-01-29 17:03:50,590] [Node-jc] [transport] [access_granted]	origin_type=[rest], origin_address=[/0:0:0:0:0:0:0:1:51079], principal=[jettro], action=[cluster:monitor/health], indices=[], request=[ClusterHealthRequest]

Roles

Out of the box some roles are provided, I am going to discuss a few. For more information check the documentation. We assigned the admin role to the user jettro. This role can do everything. There are two groups of authorizations you can give a user. Cluster based and index based authorizations.

admin:
  cluster: all
  indices:
    '*': all

The next role I want to discuss is the one used by the Transport Client. This java client connects to the cluster and needs access to the node information. This role is also provided by default

transport_client:
  cluster:
    - cluster:monitor/nodes/info

As you can see, the transport_client role has nog rights for indices. It has only the right to call the url monitor/nodes/info. When discussing the java client it will become clear why you might need this role.

Custom role

Next we are creating our own role. We want users that have the rights to search the gridshore index and obtain an item by id. The role is called gridreader.

gridreader:
  indices:
    'gridshore': indices:data/read/search, indices:data/read/get

In my cluster gridshore is an alias to an index with the name gridshore-20141224151357, as you can see it is easy to use the alias for your security configuration. More information about creating indexes and why you should use an alias can be found in another blog post of mine: Maintain elasticsearch: the indexes. To use this role we are going to create a new user called reader that has this created gridreader role.

bin/shield/esusers useradd reader -p nopiforme -r gridreader

Using the same tool we can also verify the available users

bin/shield/esusers list
reader         : gridreader
transporter    : transport_client
justuser       : user
jettro         : admin

We are going to use this user in the next section when explaining the java client.

Authentication and the java client

More detailed steps are described in the manual for the java client. There you can find the shield dependency that you need and the maven repository to add for maven and gradle projects. We start with the java code to execute a query without security headers, of course this will fail.

Settings settings = ImmutableSettings.settingsBuilder()
        .put("cluster.name", "jc-play")
        .build();
Client client = new TransportClient(settings)
        .addTransportAddress(new InetSocketTransportAddress("localhost", 9300));
SearchResponse searchResponse = client.prepareSearch("gridshore").get();
long totalHits = searchResponse.getHits().totalHits();
System.out.println(totalHits);

AuthorizationException: missing authentication token for action [cluster:monitor/nodes/info]

The response contains a message that an authentication token is missing. Check the action that it is trying to perform. This is the same url we saw in the role for transport_client. This call is made by the transport client and does not have anything to do with the search request we want to do. To overcome this problem we need to add a security header to the client. The following code contains the changed settings object with the new header containing the shield.user.

Settings settings = ImmutableSettings.settingsBuilder()
        .put("cluster.name", "jc-play")
        .put("shield.user", "transporter:nopiforme")
        .build();

Now think about what would happen when we execute this code. Will we get back the number of documents as a result? Below you’ll see the actual response.

AuthorizationException: action [indices:data/read/search] is unauthorized for user [transporter]

Ah ok, so the transporter is not allowed to search the gridshore index. We know that. There are two things that we can do. We can create a user that is allowed to request the nodes info url and search the gridshore index. This would be the normal way. When you have multiple users accessing your application with different rights, you can also add a header to each request that you do. That is what we are showing in the next code block. In this code block we use a different user for the transport client and the request that we execute.

Settings settings = ImmutableSettings.settingsBuilder()
        .put("cluster.name", "jc-play")
        .put("shield.user", "transporter:nopiforme")
        .build();
Client client = new TransportClient(settings)
        .addTransportAddress(new InetSocketTransportAddress("localhost", 9300));
String token = basicAuthHeaderValue("reader", new SecuredString("nopiforme".toCharArray()));
SearchResponse searchResponse = client.prepareSearch("gridshore").putHeader("Authorization", token).get();
long totalHits = searchResponse.getHits().totalHits();
System.out.println(totalHits);

In my case the answer is 259, which means I have 259 documents in my gridshore index. The following block shows the output of the auditlog

[2015-01-29 20:29:39,808] [Node-jc] [transport] [access_granted]	origin_type=[transport], origin_address=[/127.0.0.1:52874], principal=[reader], action=[indices:data/read/search], indices=[gridshore]