Exporting data

Influx data

You can use lti-get-influx-data.sh on your analyzer host to export data in CSV format.

Getting the script

Open a terminal session on your analyzer host.

Download the script:

wget https://api.latence.ca/software/lti-get-influx-data.sh

Make it executable:

chmod +x lti-get-influx-data.sh

Usage

Call ./lti-get-influx-data.sh with the following parameters:

  1. Agent ID

  2. Protocol name -- one of twamp, icmp, tcp, http, https

  3. Seconds of data to extract -- up to 3 months

  4. You can add --drops at the end of the line to dump drops (if there are) instead of latency values

For instance, running ./lti-get-influx-data.sh 100 tcp 10 will extract the last 10 seconds of tcp latency data for agent id 100. With a sampling rate set to 2 seconds, the result will be something similar to the block below:

,result,table,_start,_stop,_time,_value,_field,_measurement,agentID
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:20.009Z,29.45599937438965,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:22.07Z,29.535999298095703,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:24.131Z,28.83099937438965,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:26.191Z,28.9060001373291,delay,tcp_result,100
,_result,0,2023-01-16T14:55:18.99613757Z,2023-01-16T14:55:28.99613757Z,2023-01-16T14:55:28.251Z,29.325000762939453,delay,tcp_result,100

You have access to 4 fields :
_time : The exact time the measurement has been done
 _value : The latency measured (in ms) or the number of paquet drops
_measurement : Indicate the type of protocol and what kind of value is displayed in the value field : latency (protocolname_result) or drops (protocolname_drop)
 agentID : The agent ID used to do the measurement

You may redirect the data into a file using the > redirection.

For example, the command ./lti-get-influx-data.sh 100 tcp 10 > tcp-data.csv will create a text file named tcp-data.csv, containing the dumped tcp data.

You can also call ./lti-get-influx-data.sh with the following parameters to fetch metadata of an agent:

  1. Agent ID
  2. Add --metadata-only at the end of the line

For example, the command ./lti-get-influx-data.sh 100 --metadata-only > 100-metadata.csv will create a text file named 100-metadata.csv, containing the metadata for the agent 100.

Kafka data

You can use kafka-reader program to export Kafka data in CSV format.

Getting the program

Open a terminal session.

Download the files:

wget https://api.latence.ca/software/kafka-reader-files.zip

Extract it:

unzip kafka-reader-files.zip

Usage

  1. Go into the kafka-reader directory

    cd kafka-reader-files

  2. Configure the environment variables. Change the kafka.env file and modify the last three variables according to your need. The available topics are twamp_influx, icmp_influx, iperf_influx, traceroute_influx, tcp_influx, udp_influx, http_influx, https_influx.

    BOOTSTRAP_SERVERS=msgbus.latence.ca:12092 SECURITY_PROTOCOL=SASL_SSL SSL_CAFILE=./auth/certificate.txt SASL_PLAIN_USERNAME= SASL_PLAIN_PASSWORD= SASL_MECHANISM=PLAIN TOPIC_OFFSET=100 NUM_ROWS=200 TOPIC=https_influx

    Run the following command to load the variables.

    export $(cat kafka.env | xargs)

  3. Install the requirements.dev.txt packages

    pip3 install -r requirements.dev.txt

  4. Change the /etc/hosts to point msgbus.latence.ca to the analyzer host by adding the following line:

    <Replace with analyzer IP> msgbus.latence.ca

  5. Run the program to generate CSV

    python3 run.py

Latency and Geolocation data

You can use the following program to export Latency and Geolocation data in CSV format.

Getting the program

Open a terminal session.

Download the files:

wget https://api.latence.ca/software/latency-geoloc-csv-generator.zip

Extract it:

unzip latency-geoloc-csv-generator.zip

Usage

  1. Go into the latency-geoloc-csv-generator directory

    cd latency-geoloc-csv-generator

  2. Install the requirements.dev.txt packages

    pip3 install -r requirements.dev.txt

  3. Use the following command to extract the data.

    python3 csv-writing-script.py <Influx IP> <start time> <end time> <AgentID>

    • Influx IP - IP address of InfluxDB
    • start time - Start time in GMT using the format 2023-06-21T22:00:00Z
    • end time - End time in GMT in the format 2023-06-23T22:00:00Z
    • AgentID - ID of the agent

    Example

    python3 csv-writing-script.py 34.83.172.98 2023-06-21T22:00:00Z 2023-06-23T22:00:00Z 1

Troubleshooting data extraction

Influx data fetch script

Permission denied

If you get the following output:

-bash: ./lti-get-influx-data.sh: Permission denied**

Make sure you have run chmod +x ./lti-get-influx-data.sh

No data

  • Make sure you are using the right agent id

  • Make sure you have requested a sufficient number of seconds of data