Sparse (NaN) dataset

Community
1

I’ve been collecting telemetry form my PHEV car. The data comes with a always filled time column (no NaNs in it), but the rest of the columns are not ‘polled’ on every poll time (It’s hard to describe, sorry), so one row has only some of the data points and the next will have other data points and the one after that have another set. I think it takes between 4 or 5 polls to go back to the original data, but I have so much data (35k rows) I can’t check it all by hand.

Of course, I’ve been trying to plot all this, but the removal of NaN support back in '08 or before (!!!) means I can only plot with marker but not only lines. I can try and mask the data, but since the time column is always not-NaN, I get the wrong graphs. I can call panda’s fillna, but then I can’t recognize a sequence of equal values from a sequence of NaNs being skipped.

Is there an alternative to turn on ‘skip NaNs’ while drawing lines? Or at least a link to what’s referred as r4833 in NaN support - #4 by Michael_Droettboom (it’s clearly a SVN or HG revision number)?

2

I’m not a pandas user, but can’t you use dropna to get a version of the timeseries without the NaN?

3

I can try and mask the data, but since the time column is always
not-NaN, I get the wrong graphs. I can call panda’s fillna, but then
I can’t recognize a sequence of equal values from a sequence of NaNs
being skipped.

Can you elaborate on this? I’m assuming you mean something like the
'ffill' mode, which propagates the last non-NaN value forward. Could
you post some example rows with how you’d like to fill them? If you can
characterise this you can write some pure Python to implement your own
fill method. Can you explain this stuff about “recognize a sequence of
equal values from a sequence of NaNs being skipped”. I think you mean
the 'ffill' would replace NaNs with equal prior values, thus obscuring
“real” distinct measurements which were genuinely equal.

What do you want to see in your plot where these NaN gaps exist?

Is there an alternative to turn on ‘skip NaNs’ while drawing lines? Or at least a link to what’s referred as r4833 in NaN support - #4 by Michael_Droettboom (it’s clearly a SVN or HG revision number)?

Could you plot individual lines in multiple calls onto the same axes?
You might call out distinct single column dataframes per line, and drop
the NaNs from those subsidiary dataframes? Then plot.

Cheers,
Cameron Simpson cs@cskk.id.au

···

On 11Sep2022 12:39, Marcos Dione via Matplotlib nobody@discourse.matplotlib.org wrote:

4

I don’t want to fill them, just skip them.

time | speed | batt level | temp
--------------------------------
0000 |     0 |        100 | NaN
0001 |   NaN |        100 | 18
0002 |   NaN |        100 | NaN
0003 |   NaN |        NaN | 19
0004 |    10 |         99 | NaN

The line connects the last point before the NaNs to the first point after them. so, with the values above, a line from 0 to 10 for speed, a line that goes 100, 100, 100, 99 for batt level, and one from 18 to 19 for temp, all aligned with the time series as the X axis. Think of it as a scatter plot. It seems it was the behavior 14y ago.

5

Again, just make three plots where you use dropna on each column of the original data frame.

6

The revision number is indeed an svn revision which is now Handle NaNs in the data (on-the-fly as it's drawn). · matplotlib/matplotlib@6229098 · GitHub

The functionality referred to did get merged, but it does not do what you want it to do. When there are nan (or masked data) in the data, we do not know where to draw that point so we can not draw a line from the last valid point to the missing data. Thus, we “skip” the invalid point and move (in the Path drawing sense) to the next valid point which effectively breaks the line. Because you need two sequential non-invalid points to draw a line, your data is going to show up as either very very short segments 2 or 3 point segments (which if the time base is very small compared to the total time maybe effectively invisible with the default anti-aliasing as they are shorter than a pixel long) or 1-ended segments (which do not get drawn because you need two sequential points to draw a line).

I think your option here are either to do 3 plots against the time index where you dropna before passing to Matplotlib (as described above) or to pick some interpolation scheme you like to resample all of the data sets to the same time-basis.

7

And the obvious question is: why is it so? Why not draw a line between a non-invalid point and the next non-invalid point? Can’t it be f.i. optional? Remember, this is for a scatter-like plot, so ‘skipping’ points should be natural.

8

How to deal with “invalid” data is a complex subject that depends a lot of the context. A value being nan because (as in your case) there really is no data there and a value being nan because the reading is there but invalid are very different things.

If we were to effectively do dropna inside of Line2D then it would make your case much easier, but it would make the second case much harder (because to get the broken lines you would have to do the breaks and then do a variable number of lines on each segment which would both be annoying to write and carry a (maybe steep) performance cost.

Having Line2D say “I will draw your data by connecting the point pairwise with linear lines, if a point is invalid I won’t draw the segment on either side of it because” is a simple and straight forward way to implement this. If you are in case 2 it “just works” and in “case 1” the user will have to drop the invalid points (because only the user can tell which of the nan are of which type).

I think the core of the problem here is that the data you have is not well modeled by a table. It is really N 1D time series of uneven length which cover the same time range but are not synchronously sampled not a sequence of records (all alike).

I see why the car software does it this way (1 file is simpler than N, probably just dumping a bunch of registers on a clock (and clearing them on write) so they do not having to deal with interrupts / push based logging…), but on reading the data in to work on, the first thing I would do would be to split the the table by column and then work with a dictionary of (t, v) pairs (or a pd.Series with t as the index) each of which only has nan for truly invalid data, not artifacts of the data acquisition system.

9

Just for reference, this is real data:

time,Airbag_H_wire_Duty,Auxiliary_Battery_Voltage_V,Available_Charge_Power_kW,Available_Discharge_Power_kW,Battery_Cell_Voltage_Deviation_V,Battery_Current_A,Battery_DC_Voltage_V,Battery_Fan_Feedback_Hz,Battery_Fan_Status,Battery_Heater_1_Temperature_℃,Battery_Heater_2_Temperature_℃,Battery_Inlet_Temperature_℃,Battery_Max_Temperature_℃,Battery_Min_Temperature_℃,Battery_Module_01_Temperature_℃,Battery_Module_02_Temperature_℃,Battery_Module_03_Temperature_℃,Battery_Module_04_Temperature_℃,Battery_Module_05_Temperature_℃,Battery_Module_06_Temperature_℃,Battery_Module_07_Temperature_℃,Battery_Module_08_Temperature_℃,Battery_Module_09_Temperature_℃,Battery_Module_10_Temperature_℃,Battery_Module_11_Temperature_℃,Battery_Module_12_Temperature_℃,Battery_Power_kW,BMS_Ignition,BMS_Main_Relay,Cumulative_Charge_Current_Ah,Cumulative_Discharge_Current_Ah,Cumulative_Energy_Charged_kWh,Cumulative_Energy_Discharged_kWh,Drive_Motor_Speed_1_rpm,Drive_Motor_Speed_2_rpm,HV_Charging,Inverter_Capacitor_Voltage_V,Isolation_Resistance_kOhm,Maximum_Cell_Voltage_V,Maximum_Cell_Voltage_No.,Maximum_Deterioration_Cell_No.,Minimum_Cell_Voltage_V,Minimum_Cell_Voltage_No.,Minimum_Deterioration_percent,Minimum_Deterioration_Cell_No.,Normal_Charge_Port,Operating_Time_Hours,Rapid_Charge_Port,State_of_Charge_BMS_percent,State_of_Charge_Display_percent,State_of_Health_percent,Ambient_air_temperature_℃,Average_fuel_consumption_L_100km,Average_fuel_consumption_total_L_100km,Average_fuel_consumption_10_sec_L_100km,Average_speed_km_h,Calculated_instant_fuel_consumption_L_100km,Calculated_instant_fuel_rate_L_h,Distance_travelled_km,Distance_travelled_total_km,Engine_fuel_rate_L_h,Engine_RPM_rpm,EV_Instant_Energy_Consumption_km_kWh,EV_Instant_Energy_Consumption_[kWh_100km]_kWh_100km,Fuel_used_L,Fuel_used_total_L,Fuel_used_price_€,Fuel_used_price_total_€,Hybrid_EV_battery_pack_remaining_charge_percent,Hybrid_EV_Battery_Power_kW,Hybrid_EV_Battery_System_Current_A,Hybrid_EV_Battery_System_Voltage_V,Instant_engine_power_based_on_fuel_consumption_kW,Throttle_position_percent,Vehicle_acceleration_g,Vehicle_speed_km_h,
17:46:00.191,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,29,0,0.542250077137425,,3.38414388601658,0,0,0.00441198208333333,3.45478583838888,0,0,,,0,0.0187335788735965,0,0.09,,,,,0,16.8627450980392,0,3,
17:46:00.284,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,7.3,336.65625,,,,,
17:46:00.369,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,20,,,,,,,,
17:46:00.558,,14.4,47,58.8,,10.3,336.4,255,0,,,28,29,26,29,29,28,27,26,,,,,,,,3.46492,1,1,494.1,496.7,169.6,165.3,631,,0,335,1000,3.5,5,,3.5,9,,,0,90.3747222222222,0,20,,,,,,,,,,,,,,,,,,,,,,,,,,,,
17:46:00.715,0,,,,0,,,,,0,0,,,,,,,,,0,27,18,92,22,-8,0,,,,,,,,,,,,,,,0,,,0,0,,,,,20.5,0,,,,,,,,,,,,,,,,,,,,,,,,,,

That 5 line segment seems to repeat indefinitely once I start driving.