In article by Alberto Paro, the author of ElasticSearch Cookbook Second Edition, we will cover about the following recipes:

(For more resources related to this topic, see here.)

• Installing additional script plugins
• Managing scripts
• Sorting data using scripts
• Computing return fields with scripting
• Filtering a search via scripting

Introduction

ElasticSearch has a powerful way of extending its capabilities with custom scripts, which can be written in several programming languages. The most common ones are Groovy, MVEL, JavaScript, and Python.

In this article, we will see how it's possible to create custom scoring algorithms, special processed return fields, custom sorting, and complex update operations on records.

The scripting concept of ElasticSearch can be seen as an advanced stored procedures system in the NoSQL world; so, for an advanced usage of ElasticSearch, it is very important to master it.

Installing additional script plugins

ElasticSearch provides native scripting (a Java code compiled in JAR) and Groovy, but a lot of interesting languages are also available, such as JavaScript and Python. In older ElasticSearch releases, prior to version 1.4, the official scripting language was MVEL, but due to the fact that it was not well-maintained by MVEL developers, in addition to the impossibility to sandbox it and prevent security issues, MVEL was replaced with Groovy. Groovy scripting is now provided by default in ElasticSearch. The other scripting languages can be installed as plugins.

Getting ready

You will need a working ElasticSearch cluster.

How to do it...

In order to install JavaScript language support for ElasticSearch (1.3.x), perform the following steps:

1. From the command line, simply enter the following command:

   bin/plugin --install elasticsearch/elasticsearch-lang-javascript/2.3.0

2. This will print the following result:

   -> Installing elasticsearch/elasticsearch-lang-javascript/2.3.0...
   Trying http://download.elasticsearch.org/elasticsearch/elasticsearch-lang-javascript/
   elasticsearch-lang-javascript-2.3.0.zip...
   Downloading ....DONE
   Installed lang-javascript

   If the installation is successful, the output will end with Installed; otherwise, an error is returned.

3. To install Python language support for ElasticSearch, just enter the following command:

   bin/plugin -install elasticsearch/elasticsearch-lang-python/2.3.0

   The version number depends on the ElasticSearch version. Take a look at the plugin's web page to choose the correct version.

How it works...

Language plugins allow you to extend the number of supported languages to be used in scripting.

During the ElasticSearch startup, an internal ElasticSearch service called PluginService loads all the installed language plugins.

In order to install or upgrade a plugin, you need to restart the node.

The ElasticSearch community provides common scripting languages (a list of the supported scripting languages is available on the ElasticSearch site plugin page at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html), and others are available in GitHub repositories (a simple search on GitHub allows you to find them).

The following are the most commonly used languages for scripting:

• Groovy (http://groovy.codehaus.org/): This language is embedded in ElasticSearch by default. It is a simple language that provides scripting functionalities. This is one of the fastest available language extensions. Groovy is a dynamic, object-oriented programming language with features similar to those of Python, Ruby, Perl, and Smalltalk. It also provides support to write a functional code.
• JavaScript (https://github.com/elasticsearch/elasticsearch-lang-javascript): This is available as an external plugin. The JavaScript implementation is based on Java Rhino (https://developer.mozilla.org/en-US/docs/Rhino) and is really fast.
• Python (https://github.com/elasticsearch/elasticsearch-lang-python): This is available as an external plugin, based on Jython (http://jython.org). It allows Python to be used as a script engine. Considering several benchmark results, it's slower than other languages.

There's more...

Groovy is preferred if the script is not too complex; otherwise, a native plugin provides a better environment to implement complex logic and data management.

The performance of every language is different; the fastest one is the native Java. In the case of dynamic scripting languages, Groovy is faster, as compared to JavaScript and Python.

In order to access document properties in Groovy scripts, the same approach will work as in other scripting languages:

• doc.score: This stores the document's score.
• doc['field_name'].value: This extracts the value of the field_name field from the document. If the value is an array or if you want to extract the value as an array, you can use doc['field_name'].values.
• doc['field_name'].empty: This returns true if the field_name field has no value in the document.
• doc['field_name'].multivalue: This returns true if the field_name field contains multiple values.

If the field contains a geopoint value, additional methods are available, as follows:

• doc['field_name'].lat: This returns the latitude of a geopoint. If you need the value as an array, you can use the doc['field_name'].lats method.
• doc['field_name'].lon: This returns the longitude of a geopoint. If you need the value as an array, you can use the doc['field_name'].lons method.
• doc['field_name'].distance(lat,lon): This returns the plane distance, in miles, from a latitude/longitude point. If you need to calculate the distance in kilometers, you should use the doc['field_name'].distanceInKm(lat,lon) method.
• doc['field_name'].arcDistance(lat,lon): This returns the arc distance, in miles, from a latitude/longitude point. If you need to calculate the distance in kilometers, you should use the doc['field_name'].arcDistanceInKm(lat,lon) method.
• doc['field_name'].geohashDistance(geohash): This returns the distance, in miles, from a geohash value. If you need to calculate the same distance in kilometers, you should use doc['field_name'] and the geohashDistanceInKm(lat,lon) method.

By using these helper methods, it is possible to create advanced scripts in order to boost a document by a distance that can be very handy in developing geolocalized centered applications.

Managing scripts

Depending on your scripting usage, there are several ways to customize ElasticSearch to use your script extensions.

In this recipe, we will see how to provide scripts to ElasticSearch via files, indexes, or inline.

Getting ready

You will need a working ElasticSearch cluster populated with the populate script (chapter_06/populate_aggregations.sh), available at https://github.com/aparo/ elasticsearch-cookbook-second-edition.

How to do it...

To manage scripting, perform the following steps:

1. Dynamic scripting is disabled by default for security reasons; we need to activate it in order to use dynamic scripting languages such as JavaScript or Python. To do this, we need to turn off the disable flag (script.disable_dynamic: false) in the ElasticSearch configuration file (config/elasticseach.yml) and restart the cluster.
2. To increase security, ElasticSearch does not allow you to specify scripts for non-sandbox languages. Scripts can be placed in the scripts directory inside the configuration directory. To provide a script in a file, we'll put a my_script.groovy script in the config/scripts location with the following code content:

   doc["price"].value * factor

3. If the dynamic script is enabled (as done in the first step), ElasticSearch allows you to store the scripts in a special index, .scripts. To put my_script in the index, execute the following command in the command terminal:

   curl -XPOST localhost:9200/_scripts/groovy/my_script -d '{
   "script":"doc["price"].value * factor"
   }'

4. The script can be used by simply referencing it in the script_id field; use the following command:

   curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
   "query": {
      "match_all": {}
   },
   "sort": {
      "_script" : {
        "script_id" : "my_script",
        "lang" : "groovy",
        "type" : "number",
        "ignore_unmapped" : true,
        "params" : {
          "factor" : 1.1
        },
        "order" : "asc"
      }
   }
   }'

How it works...

ElasticSearch allows you to load your script in different ways; each one of these methods has their pros and cons.

The most secure way to load or import scripts is to provide them as files in the config/scripts directory. This directory is continuously scanned for new files (by default, every 60 seconds). The scripting language is automatically detected by the file extension, and the script name depends on the filename.

If the file is put in subdirectories, the directory path becomes part of the filename; for example, if it is config/scripts/mysub1/mysub2/my_script.groovy, the script name will be mysub1_mysub2_my_script. If the script is provided via a filesystem, it can be referenced in the code via the "script": "script_name" parameter.

Scripts can also be available in the special .script index. These are the REST end points:

• To retrieve a script, use the following code:

  GET http://<server>/_scripts/<language>/<id">

• To store a script use the following code:

  PUT http://<server>/_scripts/<language>/<id>

• To delete a script use the following code:

  DELETE http://<server>/_scripts/<language>/<id>

The indexed script can be referenced in the code via the "script_id": "id_of_the_script" parameter. The recipes that follow will use inline scripting because it's easier to use it during the development and testing phases.

Generally, a good practice is to develop using the inline dynamic scripting in a request, because it's faster to prototype. Once the script is ready and no changes are needed, it can be stored in the index since it is simpler to call and manage. In production, a best practice is to disable dynamic scripting and store the script on the disk (generally, dumping the indexed script to disk).

See also

• The scripting page on the ElasticSearch website at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

Sorting data using script

ElasticSearch provides scripting support for the sorting functionality. In real world applications, there is often a need to modify the default sort by the match score using an algorithm that depends on the context and some external variables. Some common scenarios are given as follows:

• Sorting places near a point
• Sorting by most-read articles
• Sorting items by custom user logic
• Sorting items by revenue

Getting ready

You will need a working ElasticSearch cluster and an index populated with the script, which is available at https://github.com/aparo/ elasticsearch-cookbook-second-edition.

How to do it...

In order to sort using scripting, perform the following steps:

1. If you want to order your documents by the price field multiplied by a factor parameter (that is, sales tax), the search will be as shown in the following code:

   curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
   "query": {
      "match_all": {}
   },
   "sort": {
      "_script" : {
        "script" : "doc["price"].value * factor",
        "lang" : "groovy",
        "type" : "number",
        "ignore_unmapped" : true,
      "params" : {
          "factor" : 1.1
        },
              "order" : "asc"
          }
      }
   }'

   In this case, we have used a match_all query and a sort script.

2. If everything is correct, the result returned by ElasticSearch should be as shown in the following code:

   {
   "took" : 7,
   "timed_out" : false,
   "_shards" : {
      "total" : 5,
      "successful" : 5,
      "failed" : 0
   },
   "hits" : {
      "total" : 1000,
      "max_score" : null,
      "hits" : [ {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "161",
        "_score" : null, "_source" : … truncated …,
        "sort" : [ 0.0278578661440021 ]
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "634",
        "_score" : null, "_source" : … truncated …,
       "sort" : [ 0.08131364254827411 ]
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "465",
        "_score" : null, "_source" : … truncated …,
        "sort" : [ 0.1094966959069832 ]
      } ]
   }
   }

How it works...

The sort scripting allows you to define several parameters, as follows:

• order (default "asc") ("asc" or "desc"): This determines whether the order must be ascending or descending.
• script: This contains the code to be executed.
• type: This defines the type to convert the value.
• params (optional, a JSON object): This defines the parameters that need to be passed.
• lang (by default, groovy): This defines the scripting language to be used.
• ignore_unmapped (optional): This ignores unmapped fields in a sort. This flag allows you to avoid errors due to missing fields in shards.

Extending the sort with scripting allows the use of a broader approach to score your hits.

ElasticSearch scripting permits the use of every code that you want. You can create custom complex algorithms to score your documents.

There's more...

Groovy provides a lot of built-in functions (mainly taken from Java's Math class) that can be used in scripts, as shown in the following table:

If you want to retrieve records in a random order, you can use a script with a random method, as shown in the following code:

curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
 "query": {
   "match_all": {}
 },
 "sort": {
   "_script" : {
     "script" : "Math.random()",
     "lang" : "groovy",
     "type" : "number",
     "params" : {}
   }
 }
}'

In this example, for every hit, the new sort value is computed by executing the Math.random() scripting function.

See also

• The official ElasticSearch documentation at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

Computing return fields with scripting

ElasticSearch allows you to define complex expressions that can be used to return a new calculated field value. These special fields are called script_fields, and they can be expressed with a script in every available ElasticSearch scripting language.

Getting ready

You will need a working ElasticSearch cluster and an index populated with the script (chapter_06/populate_aggregations.sh), which is available at https://github.com/aparo/ elasticsearch-cookbook-second-edition.

How to do it...

In order to compute return fields with scripting, perform the following steps:

1. Return the following script fields:
   • "my_calc_field": This concatenates the text of the "name" and "description" fields
   • "my_calc_field2": This multiplies the "price" value by the "discount" parameter

2. From the command line, execute the following code:

   curl -XGET 'http://127.0.0.1:9200/test-index/test-type/
   _search?&pretty=true&size=3' -d '{
   "query": {
      "match_all": {}
   },
   "script_fields" : {
      "my_calc_field" : {
        "script" : "doc["name"].value + " -- " + doc["description"].value"
      },
      "my_calc_field2" : {
        "script" : "doc["price"].value * discount",
        "params" : {
         "discount" : 0.8
        }
      }
   }
   }'

3. If everything works all right, this is how the result returned by ElasticSearch should be:

   {
   "took" : 4,
   "timed_out" : false,
   "_shards" : {
      "total" : 5,
      "successful" : 5,
      "failed" : 0
   },
   "hits" : {
      "total" : 1000,
      "max_score" : 1.0,
      "hits" : [ {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "4",
        "_score" : 1.0,
        "fields" : {
          "my_calc_field" : "entropic -- accusantium",
          "my_calc_field2" : 5.480038242170081
        }
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "9",
        "_score" : 1.0,
        "fields" : {
          "my_calc_field" : "frankie -- accusantium",
          "my_calc_field2" : 34.79852410178313
        }
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "11",
        "_score" : 1.0,
        "fields" : {
          "my_calc_field" : "johansson -- accusamus",
          "my_calc_field2" : 11.824173084636591
        }
      } ]
   }
   }

How it works...

The scripting fields are similar to executing an SQL function on a field during a select operation.

In ElasticSearch, after a search phase is executed and the hits to be returned are calculated, if some fields (standard or script) are defined, they are calculated and returned.

The script field, which can be defined with all the supported languages, is processed by passing a value to the source of the document and, if some other parameters are defined in the script (in the discount factor example), they are passed to the script function.

The script function is a code snippet; it can contain everything that the language allows you to write, but it must be evaluated to a value (or a list of values).

See also

• The Installing additional script plugins recipe in this article to install additional languages for scripting
• The Sorting using script recipe to have a reference of the extra built-in functions in Groovy scripts

Filtering a search via scripting

ElasticSearch scripting allows you to extend the traditional filter with custom scripts. Using scripting to create a custom filter is a convenient way to write scripting rules that are not provided by Lucene or ElasticSearch, and to implement business logic that is not available in the query DSL.

Getting ready

You will need a working ElasticSearch cluster and an index populated with the (chapter_06/populate_aggregations.sh) script, which is available at https://github.com/aparo/ elasticsearch-cookbook-second-edition.

How to do it...

In order to filter a search using a script, perform the following steps:

1. Write a search with a filter that filters out a document with the value of age less than the parameter value:

   curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
   "query": {
      "filtered": {
        "filter": {
          "script": {
            "script": "doc["age"].value > param1",
            "params" : {
              "param1" : 80
            }
          }
        },
        "query": {
          "match_all": {}
        }
      }
   }
   }'

   In this example, all the documents in which the value of age is greater than param1 are qualified to be returned.

2. If everything works correctly, the result returned by ElasticSearch should be as shown here:

   {
   "took" : 30,
   "timed_out" : false,
   "_shards" : {
      "total" : 5,
      "successful" : 5,
      "failed" : 0
   },
   "hits" : {
      "total" : 237,
      "max_score" : 1.0,
      "hits" : [ {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "9",
        "_score" : 1.0, "_source" :{ … "age": 83, … }
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "23",
        "_score" : 1.0, "_source" : { … "age": 87, … }
      }, {
        "_index" : "test-index",
        "_type" : "test-type",
        "_id" : "47",
        "_score" : 1.0, "_source" : {…. "age": 98, …}
      } ]
   }
   }

How it works...

The script filter is a language script that returns a Boolean value (true/false). For every hit, the script is evaluated, and if it returns true, the hit passes the filter. This type of scripting can only be used as Lucene filters, not as queries, because it doesn't affect the search (the exceptions are constant_score and custom_filters_score).

These are the scripting fields:

• script: This contains the code to be executed
• params: These are optional parameters to be passed to the script
• lang (defaults to groovy): This defines the language of the script

The script code can be any code in your preferred and supported scripting language that returns a Boolean value.

There's more...

Other languages are used in the same way as Groovy.

For the current example, I have chosen a standard comparison that works in several languages. To execute the same script using the JavaScript language, use the following code:

curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
 "query": {
   "filtered": {
     "filter": {
       "script": {
         "script": "doc["age"].value > param1",
         "lang":"javascript",
         "params" : {
           "param1" : 80
         }
       }
     },
     "query": {
       "match_all": {}
     }
   }
 }
}'

For Python, use the following code:

curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?&pretty=true&size=3' -d '{
 "query": {
   "filtered": {
     "filter": {
       "script": {
         "script": "doc["age"].value > param1",
         "lang":"python",
         "params" : {
           "param1" : 80
         }
       }
     },
     "query": {
       "match_all": {}
     }
   }
 }
}'

See also

• The Installing additional script plugins recipe in this article to install additional languages for scripting
• The Sorting data using script recipe in this article to get a reference of the extra built-in functions in Groovy scripts

Summary

In this article you have learnt the ways you can use scripting to extend the ElasticSearch functional capabilities using different programming languages.

Resources for Article:

Further resources on this subject:

• Indexing the Data [Article]
• Low-Level Index Control [Article]
• Designing Puppet Architectures [Article]