Transmission to Level 0 processing

pypromice’s tx module contains functionality for processing data transmissions from an AWS to a Level 0 data product:

  • Fetching payload messages from the Iridium SBD service

  • Payload message decoding from binary

  • Level 0 data compiling

Payload handling

SbdMessage handles the SBD message, either taken from an email.message.Message object or a .sbd file.

EmailMessage handles the email message (that the SBD message is attached to) to parse information such as sender, subject, date, and to check for attachments.

Payload decoder

PayloadFormat handles the message types and decoding templates. These can be imported from file, with two default CSV files provided with pypromice - payload_formatter.csv and payload_type.csv.

Payload processing

L0tx handles the processing and output of the L0 transmission dataline. This object inherits from EmailMessage and PayloadFormat to read and decode messages

To reprocess old messages, these can be retrieved from the mailbox by rolling back the uid counter or by reading from .sbd file.

The following function can be executed from a CLI to fetch L0 transmission messages from all valid stations:

$ get_l0tx -a accounts.ini -p credentials.ini -c tx/config
-u last_aws_uid.ini -o tx

Note

Credentials are needed to access our AWS transmissions. We do not provide these to external users, so purely include this workflow in pypromice to demonstrate transparency in our processing.

Level 0 to Level 3 processing

The process module is for processing AWS observations from Level 0 to Level 3 (end-user product) data products.

To process from L0>>L3, the following function can be executed in a CLI.

$ get_l3 -c config/KPC_L.toml -i . -o ../../aws-l3/tx"

And in parallel through all configuration .toml files $imei_list

$ parallel --bar "get_l3 -c ./{} -i . -o ../../aws-l3/tx" ::: $(ls $imei_list)

Station configuration

Each Level 0 file that will be processed must have an entry in the TOML-formatted configuration file. The config file can be located anywhere, and the processing script receives the config file and the location of the Level 0 data.

# Station configuration example, NSE

station_id = 'NSE'
logger_type = 'CR1000X'
number_of_booms = 2
nodata = ['-999', 'NAN']
modem = [['300534062416350', '2021-06-19 12:00:00']]  #Formatting [[modem,start,end],[modem,start,end]]

['NSE_300534062416350_1.txt']
format     = 'TX'
skiprows = 0
latitude =  66.48
longitude = 42.49

dsr_eng_coef = 12.66
usr_eng_coef = 13.90
dlr_eng_coef = 8.55
ulr_eng_coef = 11.26
tilt_y_factor = -1

columns = ['time','rec','p_l','p_u','t_l','t_u','rh_l','rh_u',
          'wspd_l','wdir_l','wspd_u','wdir_u','dsr',
          'usr','dlr','ulr','t_rad','z_boom_l','z_boom_u',
          't_i_1','t_i_2','t_i_3','t_i_4','t_i_5','t_i_6','t_i_7',
          't_i_8','t_i_9','t_i_10','t_i_11','tilt_x','tilt_y','rot',
          'precip_l','precip_u','gps_time','gps_lat','gps_lon',
          'gps_alt','gps_hdop','fan_dc_l','fan_dc_u','batt_v', 'p_i',
          't_i','rh_i','wspd_i','wdir_i','msg_i']

The TOML config file has the following expectations and behaviors:

  • Properties can be defined at the top level or under a section

  • Each file that will be processed gets its own section

  • Properties at the top level are copied to each section (assumed to apply to all files)

  • Top-level properties are overridden by file-level properties if they exist in both locations

Note

Be aware the column names should follow those defined in pypromice’s variables look-up table. Any column names provided that are not in this look-up table will be passed through the processing untouched.