sensor: add support to track generic single value sensors

This feature implements a sensor component that can be used to
track/log generic sensors from multiple sources. Each sensor
can have properties like unit of measurement, accuracy and a
display name that help frontends display the tracked measurements.


Signed-off-by: Morton Jonuschat <mjonuschat+moonraker@gmail.com>

docs/api_changes.md

@@ -1,6 +1,20 @@
 ##
 This document keeps a record of all changes to Moonraker's web APIs.
 
+### February 20th 2023
+- The following new endpoints are available when at least one `[sensor]`
+  section has been configured:
+  - `GET /server/sensors/list`
+  - `GET /server/sensors/sensor`
+  - `GET /server/sensors/measurements`
+
+  See [web_api.md](web_api.md) for details on these new endpoints.
+- A `sensors:sensor_update` notification has been added.  When at least one
+  monitored sensor is reporting a changed value Moonraker will broadcast this
+  notification.
+
+  See [web_api.md](web_api.md) for details on this new notification.
+
 ### February 17 2023
 - Moonraker API Version 1.2.1
 - An error in the return value for some file manager endpoints has

docs/configuration.md

@@ -2187,6 +2187,93 @@ ambient_sensor:
 More on how your data is used in the SimplyPrint privacy policy here;
 [https://simplyprint.io/legal/privacy](https://simplyprint.io/legal/privacy)
 
+### `[sensor]`
+
+Enables data collection from additional sensor sources.  Multiple "sensor"
+sources may be configured, each with their own section, ie: `[sensor current]`,
+`[sensor voltage]`.
+
+#### Options common to all sensor devices
+
+The following configuration options are available for all sensor types:
+
+```ini
+# moonraker.conf
+
+[sensor my_sensor]
+type:
+#   The type of device.  Supported types: mqtt
+#   This parameter must be provided.
+name:
+#   The friendly display name of the sensor.
+#   The default is the sensor source name.
+```
+
+#### MQTT Sensor Configuration
+
+The following options are available for `mqtt` sensor types:
+
+```ini
+# moonraker.conf
+
+qos:
+#  The MQTT QOS level to use when publishing and subscribing to topics.
+#  The default is to use the setting supplied in the [mqtt] section.
+state_topic:
+#  The mqtt topic to subscribe to for sensor state updates.  This parameter
+#  must be provided.
+state_response_template:
+#  A template used to parse the payload received with the state topic.  A
+#  "payload" variable is provided the template's context. This template must
+#  call the provided set_result() method to pass sensor values to Moonraker.
+#  `set_result()` expects two parameters, the name of the measurement (as
+#  string) and the value of the measurement (either integer or float number).
+#
+#  This allows for sensor that can return multiple readings (e.g. temperature/
+#  humidity sensors or powermeters).
+#  For example:
+#    {% set notification = payload|fromjson %}
+#    {set_result("temperature", notification["temperature"]|float)}
+#    {set_result("humidity", notification["humidity"]|float)}
+#    {set_result("pressure", notification["pressure"]|float)}
+#
+#  The above example assumes a json response with multiple fields in a struct
+#  is received. Individual measurements are extracted from that struct, coerced
+#  to a numeric format and passed to Moonraker. The default is the payload.
+```
+
+!!! Note
+    Moonraker's MQTT client must be properly configured to add a MQTT sensor.
+    See the [mqtt](#mqtt) section for details.
+
+!!! Tip
+    MQTT is the most robust way of collecting sensor data from networked
+    devices through Moonraker.  A well implemented MQTT sensor will publish all
+    changes in state to the `state_topic`.  Moonraker receives these changes,
+    updates its internal state, and notifies connected clients.
+
+Example:
+
+```ini
+# moonraker.conf
+
+# Example configuration for a Shelly Pro 1PM (Gen2) switch with
+# integrated power meter running the Shelly firmware over MQTT.
+[sensor mqtt_powermeter]
+type: mqtt
+name: Powermeter
+# Use a different display name
+state_topic: shellypro1pm-8cb113caba09/status/switch:0
+# The response is a JSON object with a multiple fields that we convert to
+# float values before passing them to Moonraker.
+state_response_template:
+  {% set notification = payload|fromjson %}
+  {set_result("power", notification["apower"]|float)}
+  {set_result("voltage", notification["voltage"]|float)}
+  {set_result("current", notification["current"]|float)}
+  {set_result("energy", notification["aenergy"]["by_minute"][0]|float * 0.000001)}
+```
+
 ## Include directives
 
 It is possible to include configuration from other files via include

docs/web_api.md

@@ -4915,6 +4915,154 @@ State of the strip.
 }
 ```
 
+### Sensor APIs
+The APIs below are available when the `[sensor]` component has been configured.
+
+#### Get Sensor List
+HTTP request:
+```http
+GET /server/sensors/list
+```
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method":"server.sensors.list",
+    "id": 5646
+}
+```
+Returns:
+
+An array of objects containing info for each configured sensor.
+```json
+{
+    "sensors": {
+        "sensor1": {
+            "id": "sensor1",
+            "friendly_name": "Sensor 1",
+            "type": "mqtt",
+            "values": {
+                "value1": 0,
+                "value2": 119.8
+            }
+        }
+    }
+}
+```
+
+#### Get Sensor Information
+Returns the status for a single configured sensor.
+
+HTTP request:
+```http
+GET /server/sensors/info?sensor=sensor1
+```
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "/server/sensors/info?sensor=sensor1",
+    "params": {
+        "sensor": "sensor1"
+    },
+    "id": 4564
+}
+```
+Returns:
+
+An object containing sensor information for the requested sensor:
+```json
+{
+    "id": "sensor1",
+    "friendly_name": "Sensor 1",
+    "type": "mqtt",
+    "values": {
+        "value1": 0.0,
+        "value2": 120.0
+    }
+}
+```
+
+#### Get Sensor Measurements
+Returns all recorded measurements for a configured sensor.
+
+HTTP request:
+```http
+GET /server/sensors/measurements?sensor=sensor1
+```
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.sensors.measurements",
+    "params": {
+        "sensor": "sensor1"
+    },
+    "id": 4564
+}
+```
+Returns:
+
+An object containing all recorded measurements for the requested sensor:
+```json
+{
+    "sensor1": {
+        "value1": [
+            3.1,
+            3.2,
+            3.0
+        ],
+        "value2": [
+            120.0,
+            120.0,
+            119.9
+        ]
+    }
+}
+```
+
+#### Get Batch Sensor Measurements
+Returns recorded measurements for all sensors.
+
+HTTP request:
+```http
+GET /server/sensors/measurements
+```
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.sensors.measurements",
+    "id": 4564
+}
+```
+Returns:
+
+An object containing all measurements for every configured sensor:
+```json
+{
+    "sensor1": {
+        "value1": [
+            3.1,
+            3.2,
+            3.0
+        ],
+        "value2": [
+            120.0,
+            120.0,
+            119.9
+        ]
+    },
+    "sensor2": {
+        "value_a": [
+            1,
+            1,
+            0
+        ]
+    }
+}
+```
+
 ### OctoPrint API emulation
 Partial support of OctoPrint API is implemented with the purpose of
 allowing uploading of sliced prints to a moonraker instance.
@@ -6290,6 +6438,30 @@ disconnects clients will receive a `disconnected` event with the data field
 omitted.  All other events are determined by the agent, where each event may
 or may not include optional `data`.
 
+#### Sensor Events
+
+Moonraker will emit a `sensors:sensor_update` notification when a measurement
+from at least one monitored sensor changes.
+
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "sensors:sensor_update",
+    "params": [
+        {
+            "sensor1": {
+                "humidity": 28.9,
+                "temperature": 22.4
+            }
+        }
+    ]
+```
+
+When a sensor reading changes, all connections will receive a
+`sensors:sensor_update` event where the params contains a data struct
+with the sensor id as the key and the sensors letest measurements as value
+struct.
+
 ### Appendix
 
 #### Websocket setup

moonraker/components/sensor.py

@@ -0,0 +1,309 @@
+# Generic sensor support
+#
+# Copyright (C) 2022 Morton Jonuschat <mjonuschat+moonraker@gmail.com>
+#
+# This file may be distributed under the terms of the GNU GPLv3 license.
+
+# Component to read additional generic sensor data and make it
+# available to clients
+from __future__ import annotations
+
+import logging
+from collections import defaultdict, deque
+from dataclasses import dataclass, replace
+from functools import partial
+
+# Annotation imports
+from typing import (
+    Any,
+    DefaultDict,
+    Deque,
+    Dict,
+    List,
+    Optional,
+    Type,
+    TYPE_CHECKING,
+    Union,
+)
+
+if TYPE_CHECKING:
+    from confighelper import ConfigHelper
+    from websockets import WebRequest
+
+SENSOR_UPDATE_TIME = 1.0
+SENSOR_EVENT_NAME = "sensors:sensor_update"
+
+
+@dataclass(frozen=True)
+class SensorConfiguration:
+    id: str
+    name: str
+    type: str
+    source: str = ""
+
+
+if TYPE_CHECKING:
+    from confighelper import ConfigHelper
+
+    from .mqtt import MQTTClient
+
+
+def _set_result(
+    name: str, value: Union[int, float], store: Dict[str, Union[int, float]]
+) -> None:
+    if not isinstance(value, (int, float)):
+        store[name] = float(value)
+    else:
+        store[name] = value
+
+
+@dataclass(frozen=True)
+class Sensor:
+    config: SensorConfiguration
+    values: Dict[str, Deque[Union[int, float]]]
+
+
+class BaseSensor:
+    def __init__(self, name: str, cfg: ConfigHelper, store_size: int = 1200) -> None:
+        self.server = cfg.get_server()
+        self.error_state: Optional[str] = None
+
+        self.config = SensorConfiguration(
+            id=name,
+            type=cfg.get("type"),
+            name=cfg.get("name", name),
+        )
+        self.last_measurements: Dict[str, Union[int, float]] = {}
+        self.last_value: Dict[str, Union[int, float]] = {}
+        self.values: DefaultDict[str, Deque[Union[int, float]]] = defaultdict(
+            lambda: deque(maxlen=store_size)
+        )
+
+    def _update_sensor_value(self, eventtime: float) -> None:
+        """
+        Append the last updated value to the store.
+        """
+        for key, value in self.last_measurements.items():
+            self.values[key].append(value)
+
+        # Copy the last measurements data
+        self.last_value = {**self.last_measurements}
+
+    async def initialize(self) -> bool:
+        """
+        Sensor initialization executed on Moonraker startup.
+        """
+        logging.info("Registered sensor '%s'", self.config.name)
+        return True
+
+    def get_sensor_info(self) -> Dict[str, Any]:
+        return {
+            "id": self.config.id,
+            "friendly_name": self.config.name,
+            "type": self.config.type,
+            "values": self.last_measurements,
+        }
+
+    def get_sensor_measurements(self) -> Dict[str, List[Union[int, float]]]:
+        return {key: list(values) for key, values in self.values.items()}
+
+    def get_name(self) -> str:
+        return self.config.name
+
+    def close(self) -> None:
+        pass
+
+
+class MQTTSensor(BaseSensor):
+    def __init__(self, name: str, cfg: ConfigHelper, store_size: int = 1200):
+        super().__init__(name=name, cfg=cfg)
+        self.mqtt: MQTTClient = self.server.load_component(cfg, "mqtt")
+
+        self.state_topic: str = cfg.get("state_topic")
+        self.state_response = cfg.load_template("state_response_template", "{payload}")
+        self.config = replace(self.config, source=self.state_topic)
+        self.qos: Optional[int] = cfg.getint("qos", None, minval=0, maxval=2)
+
+        self.server.register_event_handler(
+            "mqtt:disconnected", self._on_mqtt_disconnected
+        )
+
+    def _on_state_update(self, payload: bytes) -> None:
+        measurements: Dict[str, Union[int, float]] = {}
+        context = {
+            "payload": payload.decode(),
+            "set_result": partial(_set_result, store=measurements),
+        }
+
+        try:
+            self.state_response.render(context)
+        except Exception as e:
+            logging.error("Error updating sensor results: %s", e)
+            self.error_state = str(e)
+        else:
+            self.error_state = None
+            self.last_measurements = measurements
+            logging.debug(
+                "Received updated sensor value for %s: %s",
+                self.config.name,
+                self.last_measurements,
+            )
+
+    async def _on_mqtt_disconnected(self):
+        self.error_state = "MQTT Disconnected"
+        self.last_measurements = {}
+
+    async def initialize(self) -> bool:
+        await super().initialize()
+        try:
+            self.mqtt.subscribe_topic(
+                self.state_topic,
+                self._on_state_update,
+                self.qos,
+            )
+            self.error_state = None
+            return True
+        except Exception as e:
+            self.error_state = str(e)
+            return False
+
+
+class Sensors:
+    __sensor_types: Dict[str, Type[BaseSensor]] = {"MQTT": MQTTSensor}
+
+    def __init__(self, config: ConfigHelper) -> None:
+        self.server = config.get_server()
+        self.store_size = config.getint("sensor_store_size", 1200)
+        prefix_sections = config.get_prefix_sections("sensor")
+        self.sensors: Dict[str, BaseSensor] = {}
+
+        # Register timer to update sensor values in store
+        self.sensors_update_timer = self.server.get_event_loop().register_timer(
+            self._update_sensor_values
+        )
+
+        # Register endpoints
+        self.server.register_endpoint(
+            "/server/sensors/list",
+            ["GET"],
+            self._handle_sensor_list_request,
+        )
+        self.server.register_endpoint(
+            "/server/sensors/info",
+            ["GET"],
+            self._handle_sensor_info_request,
+        )
+        self.server.register_endpoint(
+            "/server/sensors/measurements",
+            ["GET"],
+            self._handle_sensor_measurements_request,
+        )
+
+        # Register notifications
+        self.server.register_notification(SENSOR_EVENT_NAME)
+
+        for section in prefix_sections:
+            cfg = config[section]
+
+            try:
+                try:
+                    _, name = cfg.get_name().split(maxsplit=1)
+                except ValueError:
+                    raise cfg.error(f"Invalid section name: {cfg.get_name()}")
+
+                logging.info(f"Configuring sensor: {name}")
+
+                sensor_type: str = cfg.get("type")
+                sensor_class: Optional[Type[BaseSensor]] = self.__sensor_types.get(
+                    sensor_type.upper(), None
+                )
+                if sensor_class is None:
+                    raise config.error(f"Unsupported sensor type: {sensor_type}")
+
+                self.sensors[name] = sensor_class(
+                    name=name,
+                    cfg=cfg,
+                    store_size=self.store_size,
+                )
+            except Exception as e:
+                # Ensures that configuration errors are shown to the user
+                self.server.add_warning(
+                    f"Failed to configure sensor [{cfg.get_name()}]\n{e}"
+                )
+                continue
+
+    def _update_sensor_values(self, eventtime: float) -> float:
+        """
+        Iterate through the sensors and store the last updated value.
+        """
+        changed_data: Dict[str, Dict[str, Union[int, float]]] = {}
+        for sensor_name, sensor in self.sensors.items():
+            base_value = sensor.last_value
+            sensor._update_sensor_value(eventtime=eventtime)
+
+            # Notify if a change in sensor values was detected
+            if base_value != sensor.last_value:
+                changed_data[sensor_name] = sensor.last_value
+        if changed_data:
+            self.server.send_event(SENSOR_EVENT_NAME, changed_data)
+
+        return eventtime + SENSOR_UPDATE_TIME
+
+    async def component_init(self) -> None:
+        try:
+            logging.debug("Initializing sensor component")
+            for sensor in self.sensors.values():
+                if not await sensor.initialize():
+                    self.server.add_warning(
+                        f"Sensor '{sensor.get_name()}' failed to initialize"
+                    )
+
+            self.sensors_update_timer.start()
+
+        except Exception as e:
+            logging.exception(e)
+
+    async def _handle_sensor_list_request(
+        self, web_request: WebRequest
+    ) -> Dict[str, Dict[str, Any]]:
+        output = {
+            "sensors": {
+                key: sensor.get_sensor_info() for key, sensor in self.sensors.items()
+            }
+        }
+        return output
+
+    async def _handle_sensor_info_request(
+        self, web_request: WebRequest
+    ) -> Dict[str, Any]:
+        sensor_name: str = web_request.get_str("sensor")
+        if sensor_name not in self.sensors:
+            raise self.server.error(f"No valid sensor named {sensor_name}")
+
+        sensor = self.sensors[sensor_name]
+
+        return sensor.get_sensor_info()
+
+    async def _handle_sensor_measurements_request(
+        self, web_request: WebRequest
+    ) -> Dict[str, Dict[str, Any]]:
+        sensor_name: str = web_request.get_str("sensor", "")
+        if sensor_name and sensor_name not in self.sensors:
+            raise self.server.error(f"No valid sensor named {sensor_name}")
+
+        output = {
+            key: sensor.get_sensor_measurements()
+            for key, sensor in self.sensors.items()
+            if sensor_name is "" or key == sensor_name
+        }
+
+        return output
+
+    def close(self) -> None:
+        self.sensors_update_timer.stop()
+        for sensor in self.sensors.values():
+            sensor.close()
+
+
+def load_component(config: ConfigHelper) -> Sensors:
+    return Sensors(config)