Building Filters

Filters are JSON strings and can be useful to narrow down a query in a complex way.

 

Overview

Must be of the format filter_root, where:

  • filter_root: {LOGIC_OPERATOR: [filter_structure,...]}
  • filter_structurefilter_rootor [operand,operator,operand
  • or [operand,operator_range,operand,operand]
  • operand: a clean tag name (eg. "raw.test") or a tag value (eg. 1 or "hello")
  • operator: one of '=', '!=', '<', '<=', '>', '>=', 'IN', 'STARTS_WITH', 'ENDS_WITH', 'CASE_INSENSITIVE_CONTAINS', 'CONTAINS', 'GEO_WITHIN_POLYS', 'EQUALS', 'NOT EQUALS', 'IS', 'IS NOT'
  • operator_range: one of 'BETWEEN', 'BETWEEN_DATES', 'WITHIN'
  • LOGIC_OPERATOR: one of 'AND', 'OR'

 

Step-by-step guide

The basic structure of a filter is:

{<logic operator>: <list of filters>}

IMPORTANT - Note that filters only have one (key, value) pair, meaning that you cannot have a filter of the format {<logic operator 1>: <list of filters>, <logic operator 2>: <list of filters>}.

 

The available logic operators are: 

  • "AND" - all of the listed filters must be true
  • "OR" - at least one of the listed filters must be true

 

Each of the filters in the list must have one of the following structures:

Filter Structure 1

[<tag>, <filter operator>, <value>]

 The tag must be in the format "<stream>.<tag>", ie "raw.temperature".

The available filter operators are:

  • "="
  • "!="
  • "<"
  • "<="
  • ">"
  • ">="
  • "STARTS_WITH"
  • "ENDS_WITH"
  • "CASE_INSENSITIVE_CONTAINS"
  • "CONTAINS"
  • "GEO_WITHIN_POLYS"
  • "EQUALS"
  • "NOT EQUALS"
  • "IS"
  • "IS NOT"
  • "IN" - currently only works with string + number tags

Filter Structure 2

[<tag>, <range filter operator>, <value 1>, <value 2>]

The available range filter operators are:

  • "BETWEEN"
  • "BETWEEN_DATES"
  • "WITHIN"

Filter Structure 3

{<logic operator>: <list of filters>}

A filter can have sub-filters.

 

Example

As an example, let's translate the following sentence into a filter:

"For my query on the courses stream, the location tag should contain the string 'california' (case insensitive) and the program tag should either be equal to '12', '15', or '18.'"

 

First, we'll start with an empty dictionary: {}

 

 

STEP 1: Figure out base logic operator

"For my query on the courses stream, the location tag should contain the string 'california' (case insensitive) and the program tag should either be equal to '12', '15', or '18.'"

  • Since both statements have to be true, our logic operator is "AND".

 

Current dictionary:

{"AND": []}

 

STEP 2: Figure out first filter

"the location tag should contain the string 'california' (case insensitive)"

Filter structure

  • No mention of "and" or "or", so it is not structure 3
  • Only 1 value is mentioned ('california'), so it is structure 1

Tag

  • "For my query on the courses stream, the location tag should..." -> "courses.location"

Filter operator

  • "contain the string 'california' (case insensitive)" -> "CASE_INSENSITIVE_CONTAINS"

Value

  • "contain the string 'california' (case insensitive)" -> "california"

Putting it together

  • Filter: ["courses.location", "CASE_INSENSITIVE_CONTAINS", "california"]

 

Current dictionary: 

{"AND": [["courses.location", "CASE_INSENSITIVE_CONTAINS", "california"]]}

 

STEP 3: Figure out second filter

"the program tag should either be equal to '12', '15', or '18.'"

Filter structure

  • Mention of "or" -> filter structure 3

Figure out sub-filters

  • They will all be the same structure, so let's focus on 1
  • "equal to '12'"

Filter structure

  • No mention of "and" or "or", so it is not structure 3
  • Only 1 value is mentioned ('12'), so it is structure 1

Tag

    • "For my query on the courses stream...the program tag should..." -> "courses.program"

Filter operator

  • "equal to '12'" -> "EQUALS"

Value

  • "equal to '12'" -> "12"

Putting it together

  • Sub-filter: ["courses.program", "EQUALS", "12"]

Putting it together

  • Filter: {"OR": [["courses.program", "EQUALS", "12"],["courses.program", "EQUALS", "15"],["courses.program", "EQUALS", "18"]]}

 

Current dictionary: 

{"AND": [["courses.location", "CASE_INSENSITIVE_CONTAINS", "california"], {"OR": [["courses.program", "EQUALS", "12"],["courses.program", "EQUALS", "15"],["courses.program", "EQUALS", "18"]]}]}

 

Final Dictionary

{"AND": [["courses.location", "CASE_INSENSITIVE_CONTAINS", "california"], {"OR": [["courses.program", "EQUALS", "12"],["courses.program", "EQUALS", "15"],["courses.program", "EQUALS", "18"]]}]}