Home » Blog » Optimize CnosDB Performance and Latency Using Jaeger Distributed Tracking System

Optimize CnosDB Performance and Latency Using Jaeger Distributed Tracking System

In a formal production environment, database performance and latency are critical to ensuring the stable and efficient operation of the system. Especially when interacting with the CnosDB database, it becomes important to gain a deeper understanding of its performance. At this time, the Jaeger distributed tracing system plays a huge role. In this blog, we will delve into how to trace and monitor the CnosDB database using Jaeger to gain a more complete understanding of its performance, latency, and potential issues.

About Jaeger and CnosDB

Jaeger: Jaeger is a powerful open source distributed tracing system, compatible with OpenTracing API, and designed for monitoring and debugging the request process in software service architecture. By capturing the flow of requests between different microservices, Jaeger helps developers identify performance bottlenecks and potential issues.

CnosDB: CnosDB is a high-performance open source database commonly used to store and manage large amounts of time series data. In a microservices architecture, CnosDB often acts as a backend for data storage and retrieval.

Enable Jaeger Support in CnosDB

Cancel  the [trace] [ https://docs.cnosdb.com/zh/latest/reference/config.html#trace ]

configuration comment to enable the Jaeger tracing function.

| Tip: To make the configuration take effect, you need to restart the service.

auto_generate_span = false
path = '/tmp/cnosdb'
jaeger_agent_endpoint = ''
max_concurrent_exports = 2
max_queue_size = 4096

Install and Start Jaeger

| For other deployment methods, please refer to Jaeger Deployment [ https://www.jaegertracing.io/docs/deployment/ ]

docker run -d --name jaeger \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 16686:16686 \
-p 14268:14268 \

After successful startup, use your browser to access

Track Event in CnosDB

1. Add span context to the request.

| You can set auto_generate_span = true in the configuration file to automatically generate it. If you need to analyze a specific statement, please customize the uber-trace-id value in the request. The format is as follows (for detailed format instructions, please refer to: Propagation Format [ https : / /www.jaegertracing.io/docs/1.46/client-libraries/#propagation-format ])

uber-trace-id: 3a3a43:432e345:0:1


| For the data source in the example, please refer to: https://docs.cnosdb.com/zh/latest/start/quick_start.html

Query the data in the air table in the database oceanic_station, sort it in reverse chronological order, and return the first 5 pieces of data.

curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "" -d "select * from air order by time desc limit 5;"

Use Dashboards for Analysis

  1. Record span:

When a client application sends a query or write request to the CnosDB database, CnosDB sends the resulting Span record to Jaeger. Each span represents a stage of the request, including processing time, operation name and other related information.

  1. Select Service:

In the Service drop-down box of the Jaeger user interface, select the service related to CnosDB (for example: cnosdb_singleton_1001).

  1. Find Traces:

On the interface, click the “Find Traces” button and the system will retrieve all traces related to the selected service. This will display a series of requests and corresponding spans.

  1. Analyze Trace details:

Click on the trace of interest to enter the detailed view. In this view, you will see the entire request process and the execution time of each span. This timing information will help you understand how long each step of your query takes to process.

  1. Optimize queries and systems:

With detailed time logging, you can accurately analyze query performance. In a formal production environment, this will become a valuable tool for optimizing query statements and improving system performance. By analyzing the execution time of each span, you can find steps that may cause delays and take targeted optimization measures.

In addition, Jaeger can also track other events of CnosDB, as follows:

  1. Query component
  3. Arrow Flight SQL API 


By using Jaeger to track the performance and latency of the CnosDB database, we can gain a more comprehensive understanding of the database's performance. By inserting appropriate instrumented code in critical code segments, we can capture the flow of requests and database interactions, easily identify performance bottlenecks, and resolve potential issues promptly. Integrating Jaeger and CnosDB allows us to gain deeper insights and improve overall performance and reliability. 

Extended example

REST API request
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "" -d "select * from user_queries order by time desc limit 5;"
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "" -d "CREATE TABLE air (visibility DOUBLE,temperature DOUBLE,presssure DOUBLE,TAGS(station));"
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1"  -XPOST "" -d "ma,station=XiaoMaiDao visibility=50,temperature=63,pressure=52"

Using Prometheus for remote reading and writing

remote_write:- url: "" headers: uber-trace-id: '3a3a43:432e345:0:1' basic_auth: username: 'root' password: ''remote_read cnosdbremote_read:- url: ""headers: uber-trace-id: '3a3a43:432e345:0:1'basic_auth: username: 'root' password:


Write using OpenTSDB

curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "" -d 'opentsdb 1689054412124000000 1 tag1=t1 field1="f1" value1=1'
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "" -d '[{"metric":"tsdbput","timestamp": 1689069407572000000,"value": 9,"tags":{"tag1":"t1","tag2":"t2"}}]'