process

process.aws

AWS data processing module

class process.aws.AWS(config_file, inpath, var_file=None, meta_file=None)

Bases: object

AWS object to load and process PROMICE AWS data

addAttributes(L3)

Add variable and attribute metadata

Parameters: L3 (xr.Dataset) – Level-3 data object
Returns: L3 – Level-3 data object with attributes
Return type: xr.Dataset

getL1(): Perform L0 to L1 data processing

getL2(): Perform L1 to L2 data processing

getL3(): Perform L2 to L3 data processing, including resampling and metadata and attribute population

loadConfig(config_file, inpath)

Load configuration from .toml file

Parameters

config_file (str) – TOML file path
inpath (str) – Input folder directory where L0 files can be found

Returns

conf – Configuration parameters

Return type

dict

loadL0()

Load level 0 (L0) data from associated TOML-formatted config file and L0 data file

Try readL0file() using the config with msg_lat & msg_lon appended. The specific ParserError except will occur when the number of columns in the tx file does not match the expected columns. In this case, remove msg_lat & msg_lon from the config and call readL0file() again. These station files either have no data after Nov 2022 (when msg_lat & msg_lon were added to processing), or for whatever reason these fields did not exist in the modem message and were not added.

Returns: ds_list – List of L0 xr.Dataset objects
Return type: list

process(): Perform L0 to L3 data processing

readL0file(conf)

Read L0 .txt file to Dataset object using config dictionary and populate with initial metadata

Parameters: conf (dict) – Configuration parameters
Returns: ds – L0 data
Return type: xr.Dataset

write(outpath): Write L3 data to .csv and .nc file

writeArr(outpath)

Write L3 data to .nc and .csv hourly and daily files

Parameters

outpath (str) – Output directory
L3 (AWS.L3) – Level-3 data object

class process.aws.TestProcess(methodName='runTest')

Bases: TestCase

testAddAll(): Test variable and metadata attributes added to Dataset

testCLIgetl3(): Test get_l3 CLI

testCLIjoinl3(): Test join_l3 CLI

testL0toL3(): Test L0 to L3 processing

testgetMeta(): Test AWS names retrieval

testgetVars(): Test variable table lookup retrieval

process.aws.addBasicMeta(ds, vars_df)

Use a variable lookup table DataFrame to add the basic metadata to the xarray dataset. This is later amended to finalise L3

Parameters

ds (xr.Dataset) – Dataset to add metadata to
vars_df (pd.DataFrame) – Metadata dataframe

Returns

ds – Dataset with added metadata

Return type

xr.Dataset

process.aws.addMeta(ds, meta)

Add metadata attributes from file to dataset

Parameters

ds (xarray.Dataset) – Dataset to add metadata attributes to
meta (dict) – Metadata file

Returns

ds – Dataset with metadata

Return type

xarray.Dataset

process.aws.addVars(ds, variables)

Add variable attributes from file to dataset

Parameters

ds (xarray.Dataset) – Dataset to add variable attributes to
variables (pandas.DataFrame) – Variables lookup table file

Returns

ds – Dataset with metadata

Return type

xarray.Dataset

process.aws.calculateSaturationVaporPressure(t, T_0=273.15, T_100=373.15, es_0=6.1071, es_100=1013.246, eps=0.622)

Calculate specific humidity

Parameters

T_0 (float) – Steam point temperature. Default is 273.15.
T_100 (float) – Steam point temperature in Kelvin
t (xarray.DataArray) – Air temperature
es_0 (float) – Saturation vapour pressure at the melting point (hPa)
es_100 (float) – Saturation vapour pressure at steam point temperature (hPa)

Returns

xarray.DataArray – Saturation vapour pressure with regard to water above 0 C (hPa)
xarray.DataArray – Saturation vapour pressure where subfreezing timestamps are with regards to ice (hPa)

process.aws.getColNames(vars_df, booms=None, data_type=None, bedrock=False)

Get all variable names for a given data type, based on a variables look-up table

Parameters

vars_df (pd.DataFrame) – Variables look-up table
booms (int, optional) – Number of booms. If this parameter is empty then all variables regardless of boom type will be passed. The default is None.
data_type (str, optional) – Data type, “tx”, “STM” or “raw”. If this parameter is empty then all variables regardless of data type will be passed. The default is None.

Returns

Variable names

Return type

list

process.aws.getConfig(config_file, inpath, default_columns: Sequence[str] = ('msg_lat', 'msg_lon'))

Load configuration from .toml file. PROMICE .toml files support defining features at the top level which apply to all nested properties, but do not overwrite nested properties if they are defined

Parameters

config_file (str) – TOML file path
inpath (str) – Input folder directory where L0 files can be found

Returns

conf – Configuration dictionary

Return type

dict

process.aws.getL0(infile, nodata, cols, skiprows, file_version, delimiter=',', comment='#', time_offset: Optional[float] = None) → Dataset

Read L0 data file into pandas DataFrame object

Parameters

infile (str) – L0 file path
nodata (list) – List containing value for nan values and reassigned value
cols (list) – List of columns in file
skiprows (int) – Skip rows value
file_version (int) – Version of L0 file
delimiter (str) – String delimiter for L0 file
comment (str) – Notifier of commented sections in L0 file
time_offset (Optional[float]) – Time offset in hours for correcting for non utc time data.

Returns

ds – L0 Dataset

Return type

xarray.Dataset

process.aws.getMeta(m_file=None, delimiter=',')

Load metadata table

Parameters

m_file (str) – Metadata file path
delimiter (str) – Metadata character delimiter. The default is “,”

Returns

meta – Metadata dictionary

Return type

dict

process.aws.getVars(v_file=None)

Load variables.csv file

Parameters: v_file (str) – Variable lookup table file path
Returns: Variables dataframe
Return type: pandas.DataFrame

process.aws.popCols(ds, names)

Populate dataset with all given variable names

Parammeters

dsxr.Dataset: Dataset
nameslist: List of variable names to populate

process.aws.populateMeta(ds, conf, skip)

Populate L0 Dataset with metadata dictionary

Parameters

ds (xarray.Dataset) – L0 dataset
conf (dict) – Metadata dictionary
skip (list) – List of column names to skip parsing to metadata

Returns

ds – L0 dataset with metadata populated as Dataset attributes

Return type

xarray.Dataset

process.aws.resampleL3(ds_h, t)

Resample L3 AWS data, e.g. hourly to daily average. This uses pandas DataFrame resampling at the moment as a work-around to the xarray Dataset resampling. As stated, xarray resampling is a lengthy process that takes ~2-3 minutes per operation: ds_d = ds_h.resample({‘time’:”1D”}).mean() This has now been fixed, so needs implementing: https://github.com/pydata/xarray/issues/4498#event-6610799698

Parameters

ds_h (xarray.Dataset) – L3 AWS dataset either at 10 min (for raw data) or hourly (for tx data)
t (str) – Resample factor, same variable definition as in pandas.DataFrame.resample()

Returns

ds_d – L3 AWS dataset resampled to the frequency defined by t

Return type

xarray.Dataset

process.aws.roundValues(ds, df, col='max_decimals')

Round all variable values in data array based on pre-defined rounding value in variables look-up table DataFrame

Parameters

ds (xr.Dataset) – Dataset to round values in
df (pd.Dataframe) – Variable look-up table with rounding values
col (str) – Column in variable look-up table that contains rounding values. The default is “max_decimals”

process.aws.writeAll(outpath, station_id, l3_h, l3_d, l3_m, csv_order=None)

Write L3 hourly, daily and monthly datasets to .nc and .csv files

outpathstr: Output file path
station_idstr: Station name
l3_hxr.Dataset: L3 hourly data
l3_dxr.Dataset: L3 daily data
l3_mxr.Dataset: L3 monthly data
csv_orderlist, optional: List order of variables

process.aws.writeCSV(outfile, Lx, csv_order)

Write data product to CSV file

Parameters

outfile (str) – Output file path
Lx (xr.Dataset) – Dataset to write to file
csv_order (list) – List order of variables

process.aws.writeNC(outfile, Lx, col_names=None)

Write data product to NetCDF file

Parameters

outfile (str) – Output file path
Lx (xr.Dataset) – Dataset to write to file

process.L0toL1

AWS Level 0 (L0) to Level 1 (L1) data processing

process.L0toL1.addTimeShift(ds, vars_df)

Shift times based on file format and logger type (shifting only hourly averaged values, and not instantaneous variables). For raw (10 min), all values are sampled instantaneously so do not shift. For STM (1 hour), values are averaged and assigned to end-of-hour by the logger, so shift by -1 hr. For TX (time frequency depends on v2 or v3) then time is shifted depending on logger type. We use the ‘instantaneous_hourly’ boolean from variables.csv to determine if a variable is considered instantaneous at hourly samples.

This approach creates two separate sub-dataframes, one for hourly-averaged variables and another for instantaneous variables. The instantaneous dataframe should never be shifted. We apply shifting only to the hourly average dataframe, then concat the two dataframes back together.

It is possible to use pandas merge or join instead of concat, there are equivalent methods in each. In this case, we use concat throughout.

Fausto et al. 2021 specifies the convention of assigning hourly averages to start-of-hour, so we need to retain this unless clearly communicated to users.

Parameters

ds (xarray.Dataset) – Dataset to apply time shift to
vars_df (pd.DataFrame) – Metadata dataframe

Returns

ds_out – Dataset with shifted times

Return type

xarray.Dataset

process.L0toL1.decodeGPS(ds, gps_names)

Decode GPS information based on names of GPS attributes. This should be applied if gps information does not consist of float values

Parameters

ds (xr.Dataset) – Data set
gps_names (list) – Variable names for GPS information, such as “gps_lat”, “gps_lon” and “gps_alt”

Returns

ds – Data set with decoded GPS information

Return type

xr.Dataset

process.L0toL1.getPressDepth(z_pt, p, pt_antifreeze, pt_z_factor, pt_z_coef, pt_z_p_coef)

Adjust pressure depth and calculate pressure transducer depth based on pressure transducer fluid density

Parameters

z_pt (xr.Dataarray) – Pressure transducer height (corrected for offset)
p (xr.Dataarray) – Air pressure
pt_antifreeze (float) – Pressure transducer anti-freeze percentage for fluid density correction
pt_z_factor (float) – Pressure transducer factor
pt_z_coef (float) – Pressure transducer coefficient
pt_z_p_coef (float) – Pressure transducer coefficient

Returns

z_pt_cor (xr.Dataarray) – Pressure transducer height corrected
z_pt (xr.Dataarray) – Pressure transducer depth

process.L0toL1.getTiltDegrees(tilt, threshold)

Filter tilt with given threshold, and convert from voltage to degrees. Voltage-to-degrees converseion is based on the equation in 3.2.9 at https://essd.copernicus.org/articles/13/3819/2021/#section3

Parameters

tilt (xarray.DataArray) – Array (either ‘tilt_x’ or ‘tilt_y’), tilt values (voltage)
threshold (int) – Values below this threshold (-100) will not be retained.

Returns

dst.interpolate_na() – Array (either ‘tilt_x’ or ‘tilt_y’), tilt values (degrees)

Return type

xarray.DataArray

process.L0toL1.interpTemp(temp, var_configurations, max_interp=Timedelta('0 days 12:00:00'))

Clip and interpolate temperature dataset for use in corrections

Parameters

temp (xarray.DataArray) – Array of temperature data
vars_df (pandas.DataFrame) – Dataframe to retrieve attribute hi-lo values from for temperature clipping
max_interp (pandas.Timedelta) – Maximum time steps to interpolate across. The default is 12 hours.

Returns

temp_interp – Array of interpolatedtemperature data

Return type

xarray.DataArray

process.L0toL1.reformatGPS(pos_arr, attrs)

Correct latitude and longitude from native format to decimal degrees.

v2 stations should send “NH6429.01544”,”WH04932.86061” (NUK_L 2022) v3 stations should send coordinates as “6628.93936”,”04617.59187” (DY2) or 6430,4916 (NUK_Uv3) decodeGPS should have decoded these strings to floats in ddmm.mmmm format v1 stations however only saved decimal minutes (mm.mmmmm) as float<=60. ‘ In this case, we use the integer part of the latitude given in the config file and append the gps value after it.

Parameters

pos_arr (xr.Dataarray) – Array of latitude or longitude measured by the GPS
attrs (dict) – The global attribute ‘latitude’ or ‘longitude’ associated with the file being processed. It is the standard latitude/longitude given in the config file for that station.

Returns

pos_arr – Formatted GPS position array in decimal degree

Return type

xr.Dataarray

process.L0toL1.smoothTilt(tilt, win_size)

Smooth tilt values using a rolling window. This is translated from the previous IDL/GDL smoothing algorithm: tiltX = smooth(tiltX,7,/EDGE_MIRROR,MISSING=-999) & tiltY = smooth(tiltY,7,/EDGE_MIRROR, MISSING=-999) endif In Python, this should be dstxy = dstxy.rolling(time=7, win_type=’boxcar’, center=True).mean() But the EDGE_MIRROR makes it a bit more complicated

Parameters

tilt (xarray.DataArray) – Array (either ‘tilt_x’ or ‘tilt_y’), tilt values (can be in degrees or voltage)
win_size (int) – Window size to use in pandas ‘rolling’ method. e.g. a value of 7 spans 70 minutes using 10 minute data.

Returns

tdf_rolling – The numpy array is the tilt values, smoothed with a rolling mean

Return type

tuple, as: (str, numpy.ndarray)

process.L0toL1.toL1(L0, vars_df, T_0=273.15, tilt_threshold=-100)

Process one Level 0 (L0) product to Level 1

Parameters

L0 (xarray.Dataset) – Level 0 dataset
vars_df (pd.DataFrame) – Metadata dataframe
T_0 (int) – Air temperature for sonic ranger adjustment
tilt_threshold (int) – Tilt-o-meter threshold for valid measurements

Returns

ds – Level 1 dataset

Return type

xarray.Dataset

process.L1toL2

AWS Level 1 (L1) to Level 2 (L2) data processing

process.L1toL2.toL2(L1: Dataset, vars_df: DataFrame, T_0=273.15, ews=1013.246, ei0=6.1071, eps_overcast=1.0, eps_clear=9.36508e-06, emissivity=0.97) → Dataset

Process one Level 1 (L1) product to Level 2

Parameters

L1 (xarray.Dataset) – Level 1 dataset
vars_df (pd.DataFrame) – Metadata dataframe
T_0 (float) – Ice point temperature in K. The default is 273.15.
ews (float) – Saturation pressure (normal atmosphere) at steam point temperature. The default is 1013.246.
ei0 (float) – Saturation pressure (normal atmosphere) at ice-point temperature. The default is 6.1071.
eps_overcast (int) – Cloud overcast. The default is 1..
eps_clear (float) – Cloud clear. The default is 9.36508e-6.
emissivity (float) – Emissivity. The default is 0.97.

Returns

ds – Level 2 dataset

Return type

xarray.Dataset

process.L2toL3

AWS Level 2 (L2) to Level 3 (L3) data processing

process.L2toL3.calcDirWindSpeeds(wspd, wdir, deg2rad=0.017453292519943295)

Calculate directional wind speed from wind speed and direction

Parameters

wspd (xr.Dataarray) – Wind speed data array
wdir (xr.Dataarray) – Wind direction data array
deg2rad (float) – Degree to radians coefficient. The default is np.pi/180

Returns

wspd_x (xr.Dataarray) – Wind speed in X direction
wspd_y (xr.Datarray) – Wind speed in Y direction

process.L2toL3.calcHeatFlux(T_0, T_h, Tsurf_h, rho_atm, WS_h, z_WS, z_T, nu, q_h, p_h, kappa=0.4, WS_lim=1.0, z_0=0.001, g=9.82, es_0=6.1071, eps=0.622, gamma=16.0, L_sub=2830000.0, L_dif_max=0.01, c_pd=1005.0, aa=0.7, bb=0.75, cc=5.0, dd=0.35)

Calculate latent and sensible heat flux using the bulk calculation method

Parameters

T_0 (int) – Steam point temperature
T_h (xarray.DataArray) – Air temperature
Tsurf_h (xarray.DataArray) – Surface temperature
rho_atm (float) – Atmopsheric density
WS_h (xarray.DataArray) – Wind speed
z_WS (float) – Height of anemometer
z_T (float) – Height of thermometer
nu (float) – Kinematic viscosity of air
q_h (xarray.DataArray) – Specific humidity
p_h (xarray.DataArray) – Air pressure
kappa (int) – Von Karman constant (0.35-0.42). Default is 0.4.
WS_lim (int) – Default is 1.
z_0 (int) – Aerodynamic surface roughness length for momention, assumed constant for all ice/snow surfaces. Default is 0.001.
g (int) – Gravitational acceleration (m/s2). Default is 9.82.
es_0 (int) – Saturation vapour pressure at the melting point (hPa). Default is 6.1071.
eps (int) – Ratio of molar masses of vapor and dry air (0.622).
gamma (int) – Flux profile correction (Paulson & Dyer). Default is 16..
L_sub (int) – Latent heat of sublimation (J/kg). Default is 2.83e6.
L_dif_max (int) – Default is 0.01.
c_pd (int) – Specific heat of dry air (J/kg/K). Default is 1005..
aa (int) – Flux profile correction constants (Holtslag & De Bruin ‘88). Default is 0.7.
bb (int) – Flux profile correction constants (Holtslag & De Bruin ‘88). Default is 0.75.
cc (int) – Flux profile correction constants (Holtslag & De Bruin ‘88). Default is 5.
dd (int) – Flux profile correction constants (Holtslag & De Bruin ‘88). Default is 0.35.

Returns

SHF_h (xarray.DataArray) – Sensible heat flux
LHF_h (xarray.DataArray) – Latent heat flux

process.L2toL3.calcHumid(T_0, T_100, T_h, es_0, es_100, eps, p_h, RH_cor_h)

Calculate specific humidity

Parameters

T_0 (float) – Steam point temperature. Default is 273.15.
T_100 (float) – Steam point temperature in Kelvin
T_h (xarray.DataArray) – Air temperature
eps (int) – ratio of molar masses of vapor and dry air (0.622)
es_0 (float) – Saturation vapour pressure at the melting point (hPa)
es_100 (float) – Saturation vapour pressure at steam point temperature (hPa)
p_h (xarray.DataArray) – Air pressure
RH_cor_h (xarray.DataArray) – Relative humidity corrected

Returns

Specific humidity data array

Return type

xarray.DataArray

process.L2toL3.calcVisc(T_h, T_0, rho_atm)

Calculate kinematic viscosity of air

Parameters

T_h (xarray.DataArray) – Air temperature
T_0 (float) – Steam point temperature
rho_atm (xarray.DataArray) – Surface temperature

Returns

Kinematic viscosity

Return type

xarray.DataArray

process.L2toL3.cleanHeatFlux(SHF, LHF, T, Tsurf, p, WS, RH_cor, z_boom)

Find invalid heat flux data values and replace with NaNs, based on air temperature, surface temperature, air pressure, wind speed, corrected relative humidity, and boom height

Parameters

SHF (xarray.DataArray) – Sensible heat flux
LHF (xarray.DataArray) – Latent heat flux
T (xarray.DataArray) – Air temperature
Tsurf (xarray.DataArray) – Surface temperature
p (xarray.DataArray) – Air pressure
WS (xarray.DataArray) – Wind speed
RH_cor (xarray.DataArray) – Relative humidity corrected
z_boom (xarray.DataArray) – Boom height

Returns

SHF (xarray.DataArray) – Sensible heat flux corrected
LHF (xarray.DataArray) – Latent heat flux corrected

process.L2toL3.cleanSpHumid(q_h, T, Tsurf, p, RH_cor)

Find invalid specific humidity data values and replace with NaNs, based on air temperature, surface temperature, air pressure, and corrected relative humidity

Parameters

q_h (xarray.DataArray) – Specific humidity
T (xarray.DataArray) – Air temperature
Tsurf (xarray.DataArray) – Surface temperature
p (xarray.DataArray) – Air pressure
RH_cor (xarray.DataArray) – Relative humidity corrected

Returns

q_h – Specific humidity corrected

Return type

xarray.DataArray

process.L2toL3.toL3(L2, T_0=273.15, z_0=0.001, R_d=287.05, eps=0.622, es_0=6.1071, es_100=1013.246)

Process one Level 2 (L2) product to Level 3 (L3)

Parameters

L2 (xarray:Dataset) – L2 AWS data
T_0 (int) – Steam point temperature. Default is 273.15.
z_0 (int) – Aerodynamic surface roughness length for momention, assumed constant for all ice/snow surfaces. Default is 0.001.
R_d (int) – Gas constant of dry air. Default is 287.05.
eps (int) – Default is 0.622.
es_0 (int) – Saturation vapour pressure at the melting point (hPa). Default is 6.1071.
es_100 (int) – Saturation vapour pressure at steam point temperature (hPa). Default is 1013.246.

postprocess

postprocess.csv2bufr

qc

qc.persistence

qc.persistence.count_consecutive_persistent_values(data: Series, max_diff: float) → Series

qc.persistence.find_persistent_regions(data: Series, min_repeats: int, max_diff: float) → Series: Algorithm that ensures values can stay the same within the outliers_mask

qc.persistence.persistence_qc(ds: Dataset, variable_thresholds: Optional[Mapping] = None) → Dataset

Detect and filter data points that seems to be persistent within a certain period.

TODO: It could be nice to have a reference to the logger or description of the behaviour here. The AWS logger program is know to return the last successfully read value if it fails reading from the sensor.

Parameters

ds (xr.Dataset) – Level 1 datset
variable_thresholds (Mapping) – Define threshold dict to hold limit values, and the difference values. Limit values indicate how much a variable has to change to the previous value period is how many hours a value can stay the same without being set to NaN * are used to calculate and define all limits, which are then applied to *_u, *_l and *_i

Returns

ds_out – Level 1 dataset with difference outliers set to NaN

Return type

xr.Dataset

get

get.get

AWS data retrieval module

class get.get.TestGet(methodName='runTest')

Bases: TestCase

testGetCLI(): Test get_promice_data

testURL(): Test URL retrieval

get.get.aws_data(aws_name)

Return PROMICE and GC-Net AWS L3 v3 hourly observations

Returns: df – AWS observations dataframe
Return type: pandas.DataFrame

get.get.aws_names(): Return PROMICE and GC-Net AWS names that can be used in get.aws_data() fetching

get.get.lookup_table(base_dois, server='https://dataverse.geus.dk')

Fetch dictionary of data files and download URLs from a DOI entry in the GEUS Dataverse

Parameters

base_dois (list) – List of DOIs to search
server (str, optional) – DOI server. The default is “https://dataverse.geus.dk”

get.get.watson_discharge(t='hour')

Return PROMICE hourly Watson river discharge

Parameters: t (str) – Temporal resolution of the data - “hour”, “day” or “year”
Returns: df – Watson river discharge dataframe
Return type: pandas.DataFrame

tx

tx.tx

AWS Level 0 (L0) data transmission fetching module

class tx.tx.EmailMessage(email_msg, sender_name)

Bases: SbdMessage

Email message object

checkEmail(email_msg): Check if email is Message object

checkSender(sender_name): Check email message from field matches sender name or names

getEmailBody(): Get email message body

getEmailInfo(): Parse message in email object

getIMEI(): Get modem identifier from email subject string

tx.tx.GFP2toDEC(Bytes)

Two-bit decoder

Parameters: Bytes (list) – List of two values
Returns: Decoded value
Return type: float

tx.tx.GLI4toDEC(Bytes)

Four-bit decoder

Parameters: Bytes (list) – List of four values
Returns: Decoded value
Return type: float

class tx.tx.L0tx(email_msg, format_file=None, type_file=None, sender_name=['sbdservice', 'ice@geus.dk', 'emailrelay@konectgds.com'], UnixEpochOffset=0, CRbasicEpochOffset=631152000)

Bases: EmailMessage, PayloadFormat

L0 tranmission data object

check2BitNAN(msg, type_letter, letter_flag=['g', 'n', 'e'], nan_value=8191): Check if byte is a 2-bit NAN. This occurs when the GPS data is not available and the logger sends a 2-bytes NAN instead of a 4-bytes value

checkByte(b): Check byte format against payload formatter object

checkLength()

checkPayload(): Check message payload

getByteValue(ValueBytesCount, BinaryMessage, idx): Get values from byte range in binary message

getDataLine()

Get data line from transmission message

Returns: Dataline string if found
Return type: str or None

getFirstByte(): Get first byte in payload

getFormat()

Get binary format type from first byte in payload

Returns

bval (int or None) – Format value
bfor (str or None) – Format string characters
bname (str or None) – Format name
blength (int or None) – Expected format length
bidx (int) – Format index
bool – Valid format flag

isDiagnostics(DataLine): Flag if message is diagnostics

isObservations(DataLine): Flag if message is observations

isSummer(DataLine): Flag if message is summer message

isWatsonObservation(DataLine): Flag if message is Watson River measurement

isWithInstance(DataLine): Flag if message is with instance

updateByteCounter(value): Update byte counter for decoding message

writeEntry(entry, i): Write out comma-formatted data entry from message

class tx.tx.PayloadFormat(format_file=None, type_file=None)

Bases: object

Payload formatter object

readFile(in_file)

Read lines from file

Parameters: in_file (str) – Input file path
Returns: lines – List of file line contents
Return type: list

readFormatter(in_file, delimiter=',')

Read payload formatter from file. Outputted dictionary set as key[number]: [expected_length, format_characters, description]. Flag column (info[4]) used to signify if entry should be written to output

Parameters

in_file (str) – Input file path
delimiter (str, optional) – File delimiter. The default is “,”

Returns

payload_fmt – Payload format information

Return type

dict

readPkgFile(fname)

Read lines from internal package file

Parameters: fname (str) – Package file name
Returns: lines – List of file line contents
Return type: list

readType(in_file, delimiter=',')

Read payload type setter from file. Outputted dictionary set as key[type_letter]: number_of_bytes

Parameters

in_file (str) – Input file path
delimiter (str, optional) – File delimiter. The default is “,”

Returns

payload_typ – Payload type information

Return type

dict

tx.tx.RAWtoSTR(Bytes)

Byte-to-string decoder

Parameters: Bytes (list) – List of values
Return type: Decoded string characters

class tx.tx.SbdMessage(content, attach, imei)

Bases: object

SBD transmission message object

checkAttachment(attach): Check if attachment is present in email.message.Message object

checkAttachmentName(attach_file): Check if attachment is .sbd file

getKeyValue(content, seps, key, integer=True): Get attribute from email via keyword

getLocation(content, seps=' ', key='Unit Location'): Get latitude longitude unit location from email message

getPayloadFromEmail(attach, message_size): Get Sbd payload from email object

getPayloadFromFile(attach): Read Sbd payload from .sbd file

getStatus(content, seps1=': ', seps2=' ', key='Session Status'): Get session status from email message

class tx.tx.TestTX(methodName='runTest')

Bases: TestCase

testCLIl0tx(): Test get_l0tx CLI

testCLImsg(): Test get_msg CLI

testCLIwatson(): Test get_watsontx CLI

testEmailMessage(): Test EmailMessage object initialisation from .msg file

testL0tx(): Test L0tx object initialisation

testPayloadFormat(): Test PayloadFormat object initialisation

tx.tx.addTail(in_file, out_dir, aws_name, header_names='', lines_limit=100)

Generate tails file from L0tx file

Parameters

in_file (str) – Input L0tx file
out_dir (str) – Output directory for tails file
aws_name (str) – AWS name
header_names (str, optional) – Header names. The default is ‘’.
lines_limit (int, optional) – Number of lines to append to tails file. The default is 100.

tx.tx.findDuplicates(lines)

Find duplicates lines in list of strings

Parameters: lines (list) – List of strings
Returns: unique_lines – List of unique strings
Return type: list

tx.tx.findLine(content, key)

Find keyword in line

Parameters

content (str) – String to find keyword in
key (str) – Keyword to find in string

Returns

line – Line that keyword appears on

Return type

str

tx.tx.getMail(mail_server, last_uid=1)

Retrieve new mail

Parameters

mail_server (imaplib.IMAP_SSL) – Mail server object
last_uid (int, optional) – Mail uid to start retrieval from. The default is 1.

Yields

str – Mail uid
str – Mail message

tx.tx.loadMsg(fname)

Load .msg email file into format compatible with EmailMessage and SbdMessage objects

Parameters: fname (str) – File path to .msg file
Returns: Email message object
Return type: email.message.Message

tx.tx.parseValue(line, seps)

Parse last value from line according to separating characters

Parameters

line (str) – String to split
sep (str) – Separator characters to split line by

Returns

value – Value extracted from line

Return type

str

tx.tx.readSBD(sbd_file)

Read encoded .sbd transmission file

Parameters: sbd_file (str) – Filepath to encoded .sbd file
Returns: data – Transmission message byte object
Return type: bytes

tx.tx.saveMsg(msg, fname)

Save email message object to .msg file

Parameters

msg (email.message.Message) – Email object to save to file
fname (str) – File path to outputted .msg file

tx.tx.sortLines(in_file, out_file, replace_unsorted=True)

Sort lines in text file

Parameters

in_file (str) – Input file path
out_file (str) – Output file path
replace_unsorted (bool, optional) – Flag to replace unsorted files with sorted files. The default is True.