Exporting data

Influx data

You can use lti-get-influx-data.sh on your analyzer host to export data in CSV format.

Getting the script

Open a terminal session on your analyzer host.

Download the script:

wget https://api.latence.ca/software/lti-get-influx-data.sh

Make it executable:

chmod +x lti-get-influx-data.sh

Usage

Call ./lti-get-influx-data.sh with the following parameters:

  1. Agent ID

  2. Protocol name -- one of twamp, icmp, tcp, http, https

  3. Seconds of data to extract -- up to 3 months

  4. You can add --drops at the end of the line to dump drops (if there are) instead of latency values

For instance, running ./lti-get-influx-data.sh 100 tcp 10 will extract the last 10 seconds of tcp latency data for agent id 100. With a sampling rate set to 2 seconds, the result will be something similar to the block below:

,result,table,_start,_stop,_time,_value,_field,_measurement,agentID
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:20.009Z,29.45599937438965,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:22.07Z,29.535999298095703,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:24.131Z,28.83099937438965,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:26.191Z,28.9060001373291,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:28.251Z,29.325000762939453,delay,tcp_result,100

You have access to 4 fields :
_time : The exact time the measurement has been done
 _value : The latency measured (in ms) or the number of paquet drops
_measurement : Indicate the type of protocol and what kind of value is displayed in the value field : latency (protocolname_result) or drops (protocolname_drop)
 agentID : The agent ID used to do the measurement

You may redirect the data into a file using the > redirection.

For example, the command ./lti-get-influx-data.sh 100 tcp 10 > tcp-data.csv will create a text file named tcp-data.csv, containing the dumped tcp data.

Kafka data

You can use kafka-reader program to export Kafka data in CSV format.

Getting the program

Open a terminal session.

Download the files:

wget https://api.latence.ca/software/kafka-reader-files.zip

Extract it:

unzip kafka-reader-files.zip

Usage

  1. Go into the kafka-reader directory

    cd kafka-reader-files

  2. Configure the environment variables. Change the kafka.env file and modify the last three variables according to your need. The available topics are twamp_influx, icmp_influx, iperf_influx, traceroute_influx, tcp_influx, udp_influx, http_influx, https_influx.

    BOOTSTRAP_SERVERS=msgbus.latence.ca:12092 SECURITY_PROTOCOL=SASL_SSL SSL_CAFILE=./auth/certificate.txt SASL_PLAIN_USERNAME= SASL_PLAIN_PASSWORD= SASL_MECHANISM=PLAIN TOPIC_OFFSET=100 NUM_ROWS=200 TOPIC=https_influx

    Run the following command to load the variables.

    export $(cat kafka.env | xargs)

  3. Install the requirements.dev.txt packages

    pip3 install -r requirements.dev.txt

  4. Change the /etc/hosts to point msgbus.latence.ca to the analyzer host by adding the following line:

    <Replace with analyzer IP> msgbus.latence.ca

  5. Run the program to generate CSV

    python3 run.py

Troubleshooting data extraction

Influx data fetch script

Permission denied

If you get the following output:

-bash: ./lti-get-influx-data.sh: Permission denied**

Make sure you have run chmod +x ./lti-get-influx-data.sh

No data

  • Make sure you are using the right agent id

  • Make sure you have requested a sufficient number of seconds of data