process

process.aws

process.L0toL1

process.L1toL2

process.L2toL3

postprocess

postprocess.csv2bufr

qc

qc.persistence

get

get.get

tx

tx.tx

AWS Level 0 (L0) data transmission fetching module

class tx.tx.EmailMessage(email_msg, sender_name)

Bases: SbdMessage

Email message object

checkEmail(email_msg): Check if email is Message object

checkSender(sender_name): Check email message from field matches sender name or names

getEmailBody(): Get email message body

getEmailInfo(): Parse message in email object

getIMEI(): Get modem identifier from email subject string

tx.tx.GFP2toDEC(Bytes)

Two-bit decoder

Parameters: Bytes (list) – List of two values
Returns: Decoded value
Return type: float

tx.tx.GLI4toDEC(Bytes)

Four-bit decoder

Parameters: Bytes (list) – List of four values
Returns: Decoded value
Return type: float

class tx.tx.L0tx(email_msg, format_file=None, type_file=None, sender_name=['sbdservice', 'ice@geus.dk', 'emailrelay@konectgds.com'], UnixEpochOffset=0, CRbasicEpochOffset=631152000)

Bases: EmailMessage, PayloadFormat

L0 tranmission data object

check2BitNAN(msg, type_letter, letter_flag=['g', 'n', 'e'], nan_value=8191): Check if byte is a 2-bit NAN. This occurs when the GPS data is not available and the logger sends a 2-bytes NAN instead of a 4-bytes value

checkByte(b): Check byte format against payload formatter object

checkLength()

checkPayload(): Check message payload

getByteValue(ValueBytesCount, BinaryMessage, idx): Get values from byte range in binary message

getDataLine()

Get data line from transmission message

Returns: Dataline string if found
Return type: str or None

getFirstByte(): Get first byte in payload

getFormat()

Get binary format type from first byte in payload

Returns

bval (int or None) – Format value
bfor (str or None) – Format string characters
bname (str or None) – Format name
blength (int or None) – Expected format length
bidx (int) – Format index
bool – Valid format flag

isDiagnostics(DataLine): Flag if message is diagnostics

isObservations(DataLine): Flag if message is observations

isSummer(DataLine): Flag if message is summer message

isWatsonObservation(DataLine): Flag if message is Watson River measurement

isWithInstance(DataLine): Flag if message is with instance

updateByteCounter(value): Update byte counter for decoding message

writeEntry(entry, i): Write out comma-formatted data entry from message

class tx.tx.PayloadFormat(format_file=None, type_file=None)

Bases: object

Payload formatter object

readFile(in_file)

Read lines from file

Parameters: in_file (str) – Input file path
Returns: lines – List of file line contents
Return type: list

readFormatter(in_file, delimiter=',')

Read payload formatter from file. Outputted dictionary set as key[number]: [expected_length, format_characters, description]. Flag column (info[4]) used to signify if entry should be written to output

Parameters

in_file (str) – Input file path
delimiter (str, optional) – File delimiter. The default is “,”

Returns

payload_fmt – Payload format information

Return type

dict

readPkgFile(file_path)

Read lines from internal package file

Parameters: file_path (str) – Package file name
Returns: lines – List of file line contents
Return type: list

readType(in_file, delimiter=',')

Read payload type setter from file. Outputted dictionary set as key[type_letter]: number_of_bytes

Parameters

in_file (str) – Input file path
delimiter (str, optional) – File delimiter. The default is “,”

Returns

payload_typ – Payload type information

Return type

dict

tx.tx.RAWtoSTR(Bytes)

Byte-to-string decoder

Parameters: Bytes (list) – List of values
Return type: Decoded string characters

class tx.tx.SbdMessage(content, attach, imei)

Bases: object

SBD transmission message object

checkAttachment(attach): Check if attachment is present in email.message.Message object

checkAttachmentName(attach_file): Check if attachment is .sbd file

getKeyValue(content, seps, key, integer=True): Get attribute from email via keyword

getLocation(content, seps=' ', key='Unit Location'): Get latitude longitude unit location from email message

getPayloadFromEmail(attach, message_size): Get Sbd payload from email object

getPayloadFromFile(attach): Read Sbd payload from .sbd file

getStatus(content, seps1=': ', seps2=' ', key='Session Status'): Get session status from email message

tx.tx.addTail(in_file, out_dir, aws_name, header_names='', lines_limit=100)

Generate tails file from L0tx file

Parameters

in_file (str) – Input L0tx file
out_dir (str) – Output directory for tails file
aws_name (str) – AWS name
header_names (str, optional) – Header names. The default is ‘’.
lines_limit (int, optional) – Number of lines to append to tails file. The default is 100.

tx.tx.findDuplicates(lines)

Find duplicates lines in list of strings

Parameters: lines (list) – List of strings
Returns: unique_lines – List of unique strings
Return type: list

tx.tx.findLine(content, key)

Find keyword in line

Parameters

content (str) – String to find keyword in
key (str) – Keyword to find in string

Returns

line – Line that keyword appears on

Return type

str

tx.tx.getMail(mail_server, last_uid=1)

Retrieve new mail

Parameters

mail_server (imaplib.IMAP_SSL) – Mail server object
last_uid (int, optional) – Mail uid to start retrieval from. The default is 1.

Yields

str – Mail uid
str – Mail message

tx.tx.isModified(filename, time_threshold=1)

Return flag denoting if file is modified within a certain timeframe

Parameters

filename (str) – File path
time_threshold (int) – Time threshold (provided in hours)

Returns

Flag denoting if modified (True) or not (False)

Return type

bool

tx.tx.loadMsg(fname)

Load .msg email file into format compatible with EmailMessage and SbdMessage objects

Parameters: fname (str) – File path to .msg file
Returns: Email message object
Return type: email.message.Message

tx.tx.parseValue(line, seps)

Parse last value from line according to separating characters

Parameters

line (str) – String to split
sep (str) – Separator characters to split line by

Returns

value – Value extracted from line

Return type

str

tx.tx.readSBD(sbd_file)

Read encoded .sbd transmission file

Parameters: sbd_file (str) – Filepath to encoded .sbd file
Returns: data – Transmission message byte object
Return type: bytes

tx.tx.saveMsg(msg, fname)

Save email message object to .msg file

Parameters

msg (email.message.Message) – Email object to save to file
fname (str) – File path to outputted .msg file

tx.tx.sortLines(in_file, out_file, replace_unsorted=True)

Sort lines in text file

Parameters

in_file (str) – Input file path
out_file (str) – Output file path
replace_unsorted (bool, optional) – Flag to replace unsorted files with sorted files. The default is True.