Skip to content

Commit 77008ee

Browse files
author
Klarer Sandro
committed
Speed up parsing the NMEEA messages. Using the pandas concat() function is very time consuming. Changed it according to:
https://stackoverflow.com/questions/57000903/what-is-the-fastest-and-most-efficient-way-to-append-rows-to-a-dataframehttps://stackoverflow.com/questions/57000903/what-is-the-fastest-and-most-efficient-way-to-append-rows-to-a-dataframe Parsing 800MB of MNEA messages takes 3500s, now it takes 180s
1 parent 880bbf7 commit 77008ee

File tree

2 files changed

+7
-5
lines changed

2 files changed

+7
-5
lines changed

CONTRIBUTORS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@ Shivam Soni
77
Sriramya Bhamidipati
88
Dalton Vega
99
Daniel Neamati
10+
Sandro Klarer

gnss_lib_py/parsers/nmea.py

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ def __init__(self, input_path, msg_types=None,
6767
if msg_types is None:
6868
# Use default message types
6969
msg_types = ['GGA', 'RMC']
70-
pd_df = pd.DataFrame()
70+
temporary_dictionary_list_df = []
7171
field_dict = {}
7272
prev_timestamp = None
7373

@@ -101,8 +101,7 @@ def __init__(self, input_path, msg_types=None,
101101
delta_t = datetime.datetime.combine(datestamp(date), timestamp(time))
102102
field_dict['gps_millis'] = datetime_to_gps_millis(delta_t)
103103

104-
new_row = pd.DataFrame([field_dict])
105-
pd_df = pd.concat([pd_df, new_row])
104+
temporary_dictionary_list_df.append(field_dict)
106105
field_dict = {}
107106
prev_timestamp = msg.timestamp
108107
if "sentence_type" in msg.__dir__() and msg.sentence_type in msg_types:
@@ -134,8 +133,10 @@ def __init__(self, input_path, msg_types=None,
134133
date = field_dict.pop('datestamp')
135134
delta_t = datetime.datetime.combine(datestamp(date), timestamp(time))
136135
field_dict['gps_millis'] = datetime_to_gps_millis(delta_t)
137-
new_row = pd.DataFrame([field_dict])
138-
pd_df = pd.concat([pd_df, new_row])
136+
137+
temporary_dictionary_list_df.append(field_dict)
138+
139+
pd_df = pd.DataFrame.from_dict(temporary_dictionary_list_df)
139140
# As per `gnss_lib_py` standards, convert the heading from degrees
140141
# to radians
141142
pd_df['true_course_rad'] = (np.pi/180.)*pd_df['true_course']\

0 commit comments

Comments
 (0)