Overview
Use Data Engine to turn device readings into data your application can use. A DataStream identifies the data you want to read or process; an Analyzer derives new values; a Data Stream Bridge forwards selected values to your MQTT broker.
Start with the data you need. For example, a temperature reading can be displayed directly, calibrated before use, or combined with other readings for a more complex calculation.
How Data Flows Through WEDA
Both device-originated DataStreams and Analyzer output DataStreams can supply downstream features. Choose the path that fits your task:
- Use the reading as received: Query the DataStream, forward it through a Bridge, or attach it to an Alert Policy.
- Derive a value first: Configure an Analyzer to read existing DataStreams and publish its result to an output DataStream.
- Process in stages: Use an output DataStream as the input to another JavaScript Analyzer.
Understand DataStreams
A DataStream is a named stream of values associated with a device. Each value includes its timestamp. For example, a temperature DataStream identifies which device reading you are querying, while the stream's data type describes the value format.
Keep three things separate:
- The device identifies the source or owner of the data. One device can have several DataStreams.
- The DataStream has its own ID, name, data type and source information. Use its ID to select the data for a query, Analyzer or Bridge.
- The values are the readings received over time. Creating a DataStream defines a destination; a source still needs to supply its values.
How the DataStream is prepared depends on the source:
- WEDA Node and WEDA SubNode: WEDA synchronizes reported sensor capabilities into system-managed DataStreams.
- Custom Device: Create the required DataStreams and configure the Data Source and Data Parser to map incoming data to them.
- Analyzer output: Prepare a user-managed DataStream and bind it as the Analyzer's output.
See DataStreams for discovery, creation, source relationships and value queries.
Process Data with Analyzers
An Analyzer runs in WEDA Cloud. It reads input DataStreams, applies the configured expression or script, and publishes derived values to output DataStreams. Your application can query or forward those outputs in the same way it uses other DataStreams.
WEDA v1.2 adds JavaScript Analyzers and multi-layer processing to the existing Transfer Function approach. Choose the computation according to the work:
| Approach | When to use it | Temperature example |
|---|---|---|
| Transfer Function | Apply a mathematical expression to one input and produce one output. | Convert Celsius to Fahrenheit. |
| JavaScript Analyzer | Apply script logic to one to ten inputs and produce one or more outputs. | Apply conditional calibration or calculate several derived values. |
| Multi-layer processing | Chain JavaScript Analyzers through their input and output DataStreams. | Calibrate a reading, then classify the calibrated temperature. |
Multi-layer describes the connection between Analyzers. Configure a downstream JavaScript Analyzer to use an upstream Analyzer's output DataStream as its input:
The second Analyzer can publish its own result to another output DataStream. The intermediate DataStream remains available for queries and other consumers. A chain can be at most five Analyzers deep and cannot loop back to an earlier stage.
For multiple inputs, values may arrive at different times. Data Assembly settings control how those inputs are collected into a set before evaluation, including how long to wait for missing inputs. Configure these settings on the Analyzer. A single-input chain does not need a multi-input assembly step.
JavaScript evaluations have limits on script length, execution time, memory and statement count. Scripts cannot make network requests. Each evaluation starts with fresh script state. Review the runtime limits and input/output rules in the Analyzers guide when preparing your script.
The Analyzers guide also covers Data Assembly policies, enabling an Analyzer, chaining constraints and troubleshooting.
Query and Forward Your Data
Choose how your application will use the DataStream:
| Your task | Use | Continue with |
|---|---|---|
| Display the most recent reading received within the freshness window, 60 minutes by default | A latest-value query | DataStreams |
| Inspect readings over a past interval | A historical-value query | DataStreams |
| Calculate a summary over a time range, such as average temperature | An aggregate query | DataStreams |
| Send selected values to your own system over MQTT | A Data Stream Bridge | Data Stream Bridge |
| Evaluate values against trigger and recovery conditions | An Alert Policy | Alert Policies |
An aggregate query calculates a result when you request it. An Analyzer publishes derived values to output DataStreams as it processes incoming data. Choose an aggregate query for a time-range summary and an Analyzer when you need a derived stream for further processing or consumption.
Historical and aggregate queries return only recorded values. Before expecting a past time range, check that the stream's recordToDataLake setting is enabled and that the range falls within the data retention policy.