Parser Documentation¶

This document describes the basic function and implementation of a parser on the ZENNER-IoT platform.

General¶

On the ZENNER-IoT platform there is the possibility to have packet data of a device parsed by provided source code. This allows binary or already structured data to be converted into a desired format.

The source code used is to be written in the language "Elixir" (https://elixir-lang.org/), which is characterized by its expressiveness and the ability to deal elegantly with binary data. A documentation about the language can be found here: https://hexdocs.pm/elixir/Kernel.html. However, it should be noted that only functions and modules enabled by the platform itself can be used.

The source code of the parser must contain a function Parser.parse/2, which is called for each packet. The result of this function is then stored as a reading from the platform.

Source Code¶

This section describes the path from the template to a working parser.

Template¶

The following template is suggested by the system when creating a parser.

defmodule Parser do
  use Platform.Parsing.Behaviour

  def parse(event, meta) do
  end

  def fields do
    [
      # document the fields that are returned by parse/2
    ]
  end

end

The module Parser is specified with the functions parse(event, _meta) and fields().

The parse/2 function expects two arguments, where event contains the binary / structured data and meta other metadata of the package.

The fields/0 function returns a list of field definitions. See more

Return Values¶

The purpose of the Parser.parse/2 function is to create readings from a package. There are several ways to allow this with only one flexible return value.

No Reading¶

If no reading is to be created, simply return the empty list: [].

One Reading¶

To create a single reading, this can be returned as a map with key-value pairs.

Example:

%{
  temp: 42,
  status: :ok,
  battery: 7
}

This return value is the short form of a list with only one element:

[
  %{
    temp: 42,
    status: :ok,
    battery: 7,
  }
]

Both spellings can be used, the return value is effectively processed the same way.

Multiple Readings¶

If multiple readings are to be generated from a packet, a list of maps can be returned.

Excample:

[
  %{
    temp: 42,
    status: :ok,
    battery: 7,
  },
  %{
    temp: 9,
    status: :warn,
    battery: 1,
  },
]

If multiple readings need to be returned, this is the only way because the parser.parse/2 is executed only once per packet.

Reading Parametrization¶

In addition to a reading, additional values can be optionally returned, which can enrich the reading both by means of additional information as well as resulting functions.

This is done by returning the map of the readings along with a keyword list in a tuple. The structure then looks as follows: {Map, KeywordList}.

Example:

{
  %{
    temp: 42,
    status: :ok,
    battery: 7,
  },
  [
    arg1: value1,
    arg1: value2,
    # ...
  ]

}

Example of a List:

[
  {
    %{
      temp: 42,
      status: :ok,
      battery: 7,
    },
    [
      arg1: value1,
      arg1: value2,
    # ...
    ]
  },
]

Various arguments can also be combined. If an argument is included multiple times in the keyword list, only the first occurrence is considered.

The following sections describe which arguments can be used.

Backdating Readings¶

If a reading is to be valid at a different time, it can be specified as follows.

The time is given as a DateTime, which can easily be generated from a Unix timestamp: DateTime.from_unix!(1432109876).

You can use the measured_at argument to return the time as follows:

{
  %{
    temp: 42,
    status: :ok,
    battery: 7,
  },
  [
    measured_at: DateTime.from_unix!(1432109876),
  ]
}

Readings Device Position¶

It is possible for a reading to link a location based on its GPS coordinates. This location can then be used for tracking or dynamic geolocation.

The GPS coordinates can be specified in several formats as an argument. The order is always longitude then latitude.

• As a Tuple: {9.995381, 53.555035}.

• As a Geo.Point: %Geo.Point{coordinates: {9.995381, 53.555035}}.

The location argument can be used to return the position for a reading as follows:

{
  %{
    temp: 42,
    status: :ok,
    battery: 7,
  },
  [
    location: {9.995381, 53.555035},
  ]
}

Parsing¶

The following sections describe what types of data to expect and how to process them.

Parsing Structured Data¶

For example, if the "HTTP REST" driver is used to regularly request a weather API that responds to the following JSON:

{
  "country": "Germany",
  "city": "Hamburg",
  "weather": {
    "temperature": 23,
    "humidity": 55,
    "rain": 1.2
  }
}

Then the parameter event for theParser.parse/2 function will have the following value:

%{
  "country" => "Germany",
  "city" => "Hamburg",
  "weather" => %{
    "temperature" => 23,
    "humidity" => 55,
    "rain" => 1.2
  }
}

The implementation of the parser is now supposed to convert the temperature from this structure into a desired reading.

def parse(event, meta) do
  %{
  temp_celsius: get_in(event, ["weather", "temperature"])
  }
end

In the resulting reading the value 23 is now stored under the key temp_celsius and can be visualized.

Further explanations of pattern matching with Elixir can be found in this guide.

Parsing Binary Data¶

If a packet of e.g. A LoRaWAN device is received, the binary payload can be processed via the event parameter.

To process binary data, it must be known which bits and bytes are at which position and "endianness". To demonstrate this, a simple binary example format is used.

• The sample payload is exactly 16 bits, that is 2 bytes long.

• The first 2 bits must have the binary value 00 and describe the packet version.

• There are 2 bits describing the various states of the device:

• Whether the device is self-testing successfully.

• Whether the device is connected to a voltage supply.

• A further 3 bits represent a battery level between 0..7.

• The remaining 9 bits contain a measured value between 0..511 in reverse bit sequence (little endian).

Representation of the bits with the corresponding index and identifier:

Index: 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16

Bits: 0 0 = Packet Version

1 = Selftest: OK

0 = Voltage Supply: NO

1 0 1 = Battery Level: 5 (big endian)

0 0 1 0 1 0 1 0 0 = Meter Value: 84 (little endian)

The Payload itself is displayed on the Platform in HEX notation, which in this case is "2A54".

The function for parsing this binary data is as follows:

def parse(<<0::1, 0::1, selftest::1, powersupply::1, battery_level::3, meter_value::little-9>>, meta) do
  %{
    packet_version: 0,
    selftest_ok: (selftest == 1),
    powersupply: (powersupply == 1),
    battery_level: battery_level,
    meter_value: meter_value,
  }
end

It is easy to see how the schematic representation of the bits can be translated directly into code.

A few hints for understanding:

• The notation << >> describes a binary value in which there may be placeholders for individual bits / bytes which are described next.

• A 0 :: 1 expects the value0 with a length of one bit.

• A selftest :: 1 will assign the value to theselftest variable. The value has a length of one bit.

• A battery_level :: 3 will store the value of three bits, ie the value range of0..7, in battery_level.

• A meter_value :: little-9 will store the value of nine bits as little-endian in meter_value.

• f the payload does not match the length or fixed bits, the function body is not executed and no measured value is generated.

A complete documentation on the possible placeholders can be found here: https://hexdocs.pm/elixir/Kernel.SpecialForms.html#%3C%3C%3E%3E/1

Other Formats¶

It is easily possible to program a parser which can support multiple packet versions.

For example, if a version 01 of the packet version is added without the battery but with an increased measuring range, a further complete functional body can be simply added, which is selected by the first two bits without further intervention. The first matching function head is then selected and executed.

def parse(<<0::1, 1::1, selftest::1, meter_value::little-13>>, meta) do
  %{
    packet_version: 1,
    selftest_ok: (selftest == 1),
    meter_value: meter_value,
  }
end

Defining Fields¶

You can tell ELEMENT how to display the fields your parser returns. To do that, define a function fields/0 in your parser module:

def fields do
  [
    %{
      field: "temp",
      display: "Temperature",
      unit: "°C"
    }, %{
      field: "status",
      display: "Status"
    }, %{
      field: "battery",
      display: "Battery",
      unit: "%"
    }
  ]
end

The fields/0 function returns a list of maps, which have the following keys:

Hints¶

Parser Testing Functionality¶

On the platform itself the "parser testing" function can be used to check the source code of the parser for functionality. This should also be carried out before commissioning, as otherwise no measured values may be generated.

Local Development¶

It is possible to install the Elixir compiler locally on its computer and to use the included iex-shell to develop and test the code in the familiar environment. For complex parsers, it is recommended to store them locally.

Complex Payloads¶

The flexibility of the parser also allows very complex payloads to be processed. If the specification and the test data are sufficient, this can also be done by ZENNER IoT Solutions GmbH.