The device data and data storage solution connected to the platform in the Internet of Things project have the following characteristics:
- The dimensions, frequency, and number of devices for data collection are relatively large, and the amount of data collected is relatively large, which puts great pressure on the access throughput of the message server and the storage space consumption of the back-end database.
- Data is reported, transmitted, and stored according to the collection cycle, generally in time series.
Therefore, using time series databases in IoT projects is a wise choice. Time series databases can bring significant performance improvements, including higher storage capacity, faster large-scale query speeds, and better data compression rates.
Introduction
CnosDB is an open source distributed time series database with high performance, high compression rate and high ease of use. The main application scenarios are the Internet of Things, Industrial Internet, Internet of Vehicles and IT operation and maintenance. All code is open source on GitHub. This article will introduce how to use EMQX, an MQTT server + CnosDB, to build an IoT data platform to achieve real-time stream processing of IoT data. MQTT is based on the publish-subscribe model, which decouples the sender (publisher) and receiver (subscriber) of the message, and introduces the role of an intermediate agent to complete the routing and distribution of messages. Publishers and subscribers do not need to know the existence of each other. The only connection between them is the consistent agreement on the message, such as what topic the message will use, what fields the message will contain, and so on. This makes MQTT communication more flexible because we can dynamically add or remove subscribers and publishers at any time. Through publish and subscribe, we can easily implement broadcast, multicast and unicast of messages.
EMQX is an open source large-scale distributed MQTT message server with rich functions and designed for IoT and real-time communication applications. EMQX 5.0 supports up to 100 million concurrent MQTT connections in a single cluster. The transmission and processing throughput of a single server can reach millions of MQTT messages per second, and the latency is guaranteed to be at the sub-millisecond level. EMQX supports multiple protocols, including MQTT (3.1, 3.1.1 and 5.0), HTTP, QUIC and WebSocket, etc., ensuring the accessibility of various network environments and hardware devices. EMQX also provides comprehensive SSL/TLS functional support, such as two-way authentication and multiple authentication mechanisms, providing reliable and efficient communication infrastructure for IoT devices and applications.
The following is a detailed tutorial on building a real-time data stream processing application using EMQX and CnosDB.
Install
You can follow the official website tutorial [ https://www.emqx.io/docs/zh/v5.2/getting-started/getting-started.html ] to download and install EMQX. It is recommended to use the Docker installation image method to install. After installing EMQX, access http://localhost:18083/ through the browser .
The initial username is: admin, and the password is: public. After logging in, you can access the EMQX Dashboard and query the corresponding IP address and port information.
After completing the configuration of EMQX, you can refer to the relevant documents [ https://mqttx.app/zh/downloads ] to download the MQTT client. After entering the client, click the corresponding + sign to create a new connection.
Additionally, You can refer to the configuration in the figure to configure your connection. After the connection is successful, you can click the Send Message button to send the corresponding information.
Storage
If you want to store the corresponding information in CnosDB, you can first start CnosDB by referring to the document [ https://docs cnosdb com/zh/latest/start/install html ] and create a new database in CnosDB. Go to Dashboard Data Integration -> Data Bridging page. Click Create in the upper right corner of the page. Select HTTP service in the data bridge type.
Configure your HTTP service connection according to the example format in the figure below. In the URL part, replace the IP with the IP of your local computer. The key value of the request header is its corresponding base64 value based on the user and password information settings of your local CnosDB configuration. For example, if the username and password this time are “root:”, then the value of Authorization is echo “Basic $(echo “root:”|base64)”, which is “Basic cm9vdDoK”. It should be noted that you should configure the request body according to your data format so that the data can be correctly written to CnosDB.
After configuring the corresponding information, you can test the connection. If the connection is successful, it means that you can configure the corresponding SQL rules and store the data in CnosDB. The SQL rules need to be consistent with the information sent by MQTT and the request body settings in the previous step. The example used this time is as follows. You can refer to the examples we provide to set your data format and SQL rules.
After setting up the SQL rules, you can run the sample data to check whether the SQL rules are valid. For example, the sample data that needs to be sent by MQTT this time can be parsed into the following form by the currently set SQL rules, which is also consistent with the request body in the HTTP service settings.
After setting the SQL rules and sending the corresponding data in MQTT, you can find that the data is sent successfully and stored in CnosDB.
The above are all the tutorials on using EMQX and CnosDB to build realtime data stream processing applications. I hope this blog helps.