Update Kuery Syntax

docs/discover/kuery.asciidoc

@@ -3,22 +3,32 @@
 
 experimental[This functionality is experimental and may be changed or removed completely in a future release.]
 
+[NOTE]
+============
+Breaking changes were made to Kuery's experimental syntax in 6.3. Read on for details of the new syntax.
+============
+
 Kuery is a new query language built specifically for Kibana. It aims to simplify the search experience in Kibana
 and enable the creation of helpful features like auto-complete, seamless migration of saved searches, additional
 query types, and more. Kuery is a basic experience today but we're hard at work building these additional features on
 top of the foundation Kuery provides.
 
-Kueries are built with functions. Many functions take a field name as their first argument. Extremely common functions have shorthand notations.
+If you're familiar with Kibana's old lucene query syntax, you should feel right at home with Kuery. Both languages 
+are very similar, but there are some differences we'll note along the way.
 
-`is("response", 200)` will match documents where the response field matches the value 200.
-`response:200` does the same thing. `:` is an alias for the `is` function.
+`response:200` will match documents where the response field matches the value 200.
 
-Multiple search terms are separated by whitespace.
+Quotes around a search term will initiate a phrase search. For example, `message:"Quick brown fox"` will search
+for the phrase "quick brown fox" in the message field. Without the quotes, your query will get broken down into tokens via
+the message field's configured analyzer and will match documents that contain those tokens, regardless of the order in which
+they appear. This means documents with "quick brown fox" will match, but so will "quick fox brown". Remember to use quotes if you want
+to search for a phrase.
 
-`response:200 extension:php` will match documents where response matches 200 and extension matches php.
+Unlike lucene, Kuery will not split on whitespace. Multiple search terms must be separated by explicit
+boolean operators. Note that boolean operators in Kuery are not case sensitive.
 
-*All terms must match by default*. The language supports boolean logic with and/or operators. The above query is equivalent to `response:200 and extension:php`.
-This is a departure from the Lucene query syntax where all terms are optional by default.
+`response:200 extension:php` in lucene would become `response:200 and extension:php`.
+ This will match documents where response matches 200 and extension matches php.
 
 We can make terms optional by using `or`.
 
@@ -32,85 +42,40 @@ We can override the default precedence with grouping.
 
 `response:200 and (extension:php or extension:css)` will match documents where response is 200 and extension is either php or css.
 
-Terms can be inverted by prefixing them with `!`.
+A shorthand exists that allows us to easily search a single field for multiple values.
+
+`response:(200 or 404)` searches for docs where the `response` field matches 200 or 404. We can also search for docs
+with multi-value fields that contain a list of terms, for example: `tags:(success and info and security)`
 
-`!response:200` will match all documents where response is not 200.
+Terms can be inverted by prefixing them with `not`.
+
+`not response:200` will match all documents where response is not 200.
 
 Entire groups can also be inverted.
 
-`response:200 and !(extension:php or extension:css)`
+`response:200 and not (extension:php or extension:css)`
+
+Ranges in Kuery are similar to lucene with a small syntactical difference. 
+
+Instead of `bytes:>1000`, Kuery omits the colon: `bytes > 1000`. 
+
+`>, >=, <, <=` are all valid range operators.
+
+Exist queries are simple and do not require a special operator. `response:*` will find all docs where the response
+field exists.
 
-Some query functions have named arguments.
+Wildcard queries are available. `machine.os:win*` would match docs where the machine.os field starts with "win", which
+would match values like "windows 7" and "windows 10".
 
-`range("bytes", gt=1000, lt=8000)` will match documents where the bytes field is greater than 1000 and less than 8000.
+Wildcards also allow us to search multiple fields at once. This can come in handy when you have both `text` and `keyword`
+versions of a field. Let's say we have `machine.os` and `machine.os.keyword` fields and we want to check both for the term
+"windows 10". We can do it like this: `machine.os*:windows 10".
 
-Quotes are generally optional if your terms don't have whitespace or special characters. `range(bytes, gt=1000, lt=8000)`
-would also be a valid query.
 
 [NOTE]
 ============
-Terms without fields will be matched against all fields. For example, a query for `response:200` will search for the value 200
+Terms without fields will be matched against the default field in your index settings. If a default field is not
+set these terms will be matched against all fields. For example, a query for `response:200` will search for the value 200
 in the response field, but a query for just `200` will search for 200 across all fields in your index.
 ============
 
-==== Function Reference
-
-[horizontal]
-Function Name::  Description
-
-and:: 
-Purpose::: Match all given sub-queries
-Alias::: `and` as a binary operator
-Examples::: 
-* `and(response:200, extension:php)`
-* `response:200 and extension:php`
-
-or::
-Purpose::: Match one or more sub-queries
-Alias::: `or` as a binary operator
-Examples::: 
-* `or(extension:css, extension:php)`
-* `extension:css or extension:php`
-
-not::
-Purpose::: Negates a sub-query
-Alias::: `!` as a prefix operator
-Examples:::
-* `not(response:200)` 
-* `!response:200`
-
-is::
-Purpose::: Matches a field with a given term
-Alias::: `:`
-Examples::: 
-* `is("response", 200)`
-* `response:200`
-
-range::
-Purpose::: Match a field against a range of values.
-Alias::: `:[]`
-Examples::: 
-* `range("bytes", gt=1000, lt=8000)`
-* `bytes:[1000 to 8000]`
-Named arguments:::
-* `gt` - greater than
-* `gte` - greater than or equal to
-* `lt` - less than
-* `lte` - less than or equal to
-
-exists::
-Purpose::: Match documents where a given field exists
-Examples::: `exists("response")`
-
-geoBoundingBox::
-Purpose::: Creates a geo_bounding_box query
-Examples:::
-* `geoBoundingBox("coordinates", topLeft="40.73, -74.1", bottomRight="40.01, -71.12")` (whitespace between lat and lon is ignored)
-Named arguments:::
-* `topLeft` - the top left corner of the bounding box as a "lat, lon" string
-* `bottomRight` - the bottom right corner of the bounding box as a "lat, lon" string
-
-geoPolygon::
-Purpose::: Creates a geo_polygon query given 3 or more points as "lat, lon"
-Examples::: 
-* `geoPolygon("geo.coordinates", "40.97, -127.26", "24.20, -84.375", "40.44, -66.09")`

src/core_plugins/kibana/public/dashboard/dashboard_app.html

@@ -39,7 +39,6 @@
     ng-show="showFilterBar()"
     state="state"
     index-patterns="indexPatterns"
-    ng-if="['lucene', 'kql'].includes(model.query.language)"
   ></filter-bar>
 
   <div

src/core_plugins/kibana/public/discover/index.html

@@ -39,7 +39,6 @@ <h1 tabindex="0" id="kui_local_breadcrumb" class="kuiLocalBreadcrumb">
       <filter-bar
         state="state"
         index-patterns="[indexPattern]"
-        ng-if="['lucene', 'kql'].includes(state.query.language)"
       ></filter-bar>
     </div>
     <div class="row">

src/core_plugins/kibana/public/visualize/editor/editor.html

@@ -50,7 +50,7 @@
 
   <!-- Filters. -->
   <filter-bar
-    ng-if="vis.type.options.showFilterBar && ['lucene', 'kql'].includes(state.query.language)"
+    ng-if="vis.type.options.showFilterBar"
     state="state"
     index-patterns="[indexPattern]"
   ></filter-bar>

src/core_plugins/tile_map/public/coordinate_maps_visualization.js

@@ -117,39 +117,11 @@ export function CoordinateMapsVisualizationProvider(Notifier, Private) {
 
       const indexPatternName = agg.vis.indexPattern.id;
       const field = agg.fieldName();
-      const query = this.vis.API.queryManager.getQuery();
-      const language = query.language;
+      const filter = { meta: { negate: false, index: indexPatternName } };
+      filter[filterName] = { ignore_unmapped: true };
+      filter[filterName][field] = filterData;
 
-      if (['lucene', 'kql'].includes(language)) {
-        const filter = { meta: { negate: false, index: indexPatternName } };
-        filter[filterName] = { ignore_unmapped: true };
-        filter[filterName][field] = filterData;
-
-        this.vis.API.queryFilter.addFilters([filter]);
-      }
-      else if (language === 'kuery') {
-        const { fromKueryExpression, toKueryExpression, nodeTypes } = this.vis.API.kuery;
-        let newQuery;
-
-        if (filterName === 'geo_bounding_box') {
-          newQuery = nodeTypes.function.buildNode('geoBoundingBox', field, _.mapKeys(filterData, (value, key) => _.camelCase(key)));
-        }
-        else if (filterName === 'geo_polygon') {
-          newQuery = nodeTypes.function.buildNode('geoPolygon', field, filterData.points);
-        }
-        else {
-          throw new Error(`Kuery does not support ${filterName} queries`);
-        }
-
-        const allQueries = _.isEmpty(query.query)
-          ? [newQuery]
-          : [fromKueryExpression(query.query), newQuery];
-
-        this.vis.API.queryManager.setQuery({
-          query: toKueryExpression(nodeTypes.function.buildNode('and', allQueries, 'implicit')),
-          language: 'kuery'
-        });
-      }
+      this.vis.API.queryFilter.addFilters([filter]);
 
       this.vis.updateState();
     }

src/test_utils/public/stub_index_pattern.js

@@ -16,6 +16,7 @@ export default function (Private) {
 
   function StubIndexPattern(pattern, timeField, fields) {
     this.id = pattern;
+    this.title = pattern;
     this.popularizeField = sinon.stub();
     this.timeFieldName = timeField;
     this.getNonScriptedFields = sinon.spy(IndexPattern.prototype.getNonScriptedFields);

src/ui/public/courier/data_source/build_query/__tests__/build_es_query.js

@@ -36,7 +36,7 @@ describe('build query', function () {
 
     it('should combine queries and filters from multiple query languages into a single ES bool query', function () {
       const queries = [
-        { query: 'foo:bar', language: 'kuery' },
+        { query: 'extension:jpg', language: 'kuery' },
         { query: 'bar:baz', language: 'lucene' },
       ];
       const filters = [
@@ -53,7 +53,7 @@ describe('build query', function () {
             { match_all: {} },
           ],
           filter: [
-            toElasticsearchQuery(fromKueryExpression('foo:bar'), indexPattern),
+            toElasticsearchQuery(fromKueryExpression('extension:jpg'), indexPattern),
           ],
           should: [],
           must_not: [],

src/ui/public/courier/data_source/build_query/__tests__/from_kuery.js

@@ -2,6 +2,7 @@ import { buildQueryFromKuery } from '../from_kuery';
 import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
 import ngMock from 'ng_mock';
 import { expectDeepEqual } from '../../../../../../test_utils/expect_deep_equal.js';
+import expect from 'expect.js';
 import { fromKueryExpression, toElasticsearchQuery } from '../../../../kuery';
 
 let indexPattern;
@@ -28,8 +29,8 @@ describe('build query', function () {
 
     it('should transform an array of kuery queries into ES queries combined in the bool\'s filter clause', function () {
       const queries = [
-        { query: 'foo:bar', language: 'kuery' },
-        { query: 'bar:baz', language: 'kuery' },
+        { query: 'extension:jpg', language: 'kuery' },
+        { query: 'machine.os:osx', language: 'kuery' },
       ];
 
       const expectedESQueries = queries.map(
@@ -43,6 +44,14 @@ describe('build query', function () {
       expectDeepEqual(result.filter, expectedESQueries);
     });
 
+    it('should throw a useful error if it looks like query is using an old, unsupported syntax', function () {
+      const oldQuery = { query: 'is(foo, bar)', language: 'kuery' };
+
+      expect(buildQueryFromKuery).withArgs(indexPattern, [oldQuery]).to.throwError(
+        /It looks like you're using an outdated Kuery syntax./
+      );
+    });
+
   });
 
 });

src/ui/public/courier/data_source/build_query/build_es_query.js

@@ -1,6 +1,6 @@
 import { groupBy, has } from 'lodash';
 import { DecorateQueryProvider } from '../_decorate_query';
-import { buildQueryFromKuery, buildQueryFromKql } from './from_kuery';
+import { buildQueryFromKuery } from './from_kuery';
 import { buildQueryFromFilters } from './from_filters';
 import { buildQueryFromLucene } from './from_lucene';
 
@@ -17,16 +17,15 @@ export function BuildESQueryProvider(Private) {
     const queriesByLanguage = groupBy(validQueries, 'language');
 
     const kueryQuery = buildQueryFromKuery(indexPattern, queriesByLanguage.kuery);
-    const kqlQuery = buildQueryFromKql(indexPattern, queriesByLanguage.kql);
     const luceneQuery = buildQueryFromLucene(queriesByLanguage.lucene, decorateQuery);
     const filterQuery = buildQueryFromFilters(filters, decorateQuery, indexPattern);
 
     return {
       bool: {
-        must: [].concat(kueryQuery.must, kqlQuery.must, luceneQuery.must, filterQuery.must),
-        filter: [].concat(kueryQuery.filter, kqlQuery.filter, luceneQuery.filter, filterQuery.filter),
-        should: [].concat(kueryQuery.should, kqlQuery.should, luceneQuery.should, filterQuery.should),
-        must_not: [].concat(kueryQuery.must_not, kqlQuery.must_not, luceneQuery.must_not, filterQuery.must_not),
+        must: [].concat(kueryQuery.must, luceneQuery.must, filterQuery.must),
+        filter: [].concat(kueryQuery.filter, luceneQuery.filter, filterQuery.filter),
+        should: [].concat(kueryQuery.should, luceneQuery.should, filterQuery.should),
+        must_not: [].concat(kueryQuery.must_not, luceneQuery.must_not, filterQuery.must_not),
       }
     };
   }

src/ui/public/courier/data_source/build_query/from_kuery.js

@@ -1,13 +1,25 @@
-import _ from 'lodash';
-import { fromKueryExpression, fromKqlExpression, toElasticsearchQuery, nodeTypes } from '../../../kuery';
+import { fromLegacyKueryExpression, fromKueryExpression, toElasticsearchQuery, nodeTypes } from '../../../kuery';
+import { documentationLinks } from '../../../documentation_links';
 
-export function buildQueryFromKuery(indexPattern, queries) {
-  const queryASTs = _.map(queries, query => fromKueryExpression(query.query));
-  return buildQuery(indexPattern, queryASTs);
-}
+const queryDocs = documentationLinks.query;
 
-export function buildQueryFromKql(indexPattern, queries) {
-  const queryASTs = _.map(queries, query => fromKqlExpression(query.query));
+export function buildQueryFromKuery(indexPattern, queries = []) {
+  const queryASTs = queries.map((query) => {
+    try {
+      return fromKueryExpression(query.query);
+    }
+    catch (parseError) {
+      try {
+        fromLegacyKueryExpression(query.query);
+      }
+      catch (legacyParseError) {
+        throw parseError;
+      }
+      throw new Error(
+        `It looks like you're using an outdated Kuery syntax. See what changed in the [docs](${queryDocs.kueryQuerySyntax})!`
+      );
+    }
+  });
   return buildQuery(indexPattern, queryASTs);
 }
 

src/ui/public/doc_table/__tests__/actions/filter.js

@@ -38,38 +38,6 @@ describe('doc table filter actions', function () {
       expect(filterManager.add.calledWith(...args)).to.be(true);
     });
 
-    it('should add an operator style "is" function to kuery queries', function () {
-      const state = {
-        query: { query: '', language: 'kuery' }
-      };
-      addFilter('foo', 'bar', '+', indexPattern, state, filterManager);
-      expect(state.query.query).to.be('"foo":"bar"');
-    });
-
-    it('should combine the new clause with any existing query clauses using an implicit "and"', function () {
-      const state = {
-        query: { query: 'foo', language: 'kuery' }
-      };
-      addFilter('foo', 'bar', '+', indexPattern, state, filterManager);
-      expect(state.query.query).to.be('foo "foo":"bar"');
-    });
-
-    it('should support creation of negated clauses', function () {
-      const state = {
-        query: { query: 'foo', language: 'kuery' }
-      };
-      addFilter('foo', 'bar', '-', indexPattern, state, filterManager);
-      expect(state.query.query).to.be('foo !"foo":"bar"');
-    });
-
-    it('should add an exists query when the provided field name is "_exists_"', function () {
-      const state = {
-        query: { query: 'foo', language: 'kuery' }
-      };
-      addFilter('_exists_', 'baz', '+', indexPattern, state, filterManager);
-      expect(state.query.query).to.be('foo exists("baz")');
-    });
-
   });