HERE new data pulling plan

[chmnata]

We are gonna get some new columns from here, set up a plan on how to implement pulling new columns.

New Columns:

• harmonic_mean: Harmonic Means
• vdt : Vehicle Distance Travelled
• probe_count: Probe Count

We should keep pulling all the other columns we are pulling for now, even if they are similar to these new columns:
e.g.

• mean, arithmetic mean I guess
• sample_size, depending on if you are pulling path only, or probe mix, the count differs
  We can re-evaluate whether we want to keep pulling them after getting the actual data.

Step 1:
Pull a sample of new data and observe the order of columns.

Step 2:
Add new columns to here.ta and here.ta_path

Step 3:
Edit here.ta_view and here.ta_path_view to include new columns

Step 4:
Edit insert trigger to include new columns in the correct order.

Step 5:
Edit here pulling api, to request for these extra columns.

[chmnata]

reportFields= [LOCATION_REF, DATE, EPOCH, LENGTH, FREEFLOW, SPDLIMIT, COUNT, MEAN, HARMONIC_MEAN, VDT, PROBE_COUNT, STDDEV, MIN, MAX, CONFIDENCE, PERCENTILES]

[chmnata]

The new columns only applied for path data... and not probe_mix. The selection for those columns does not show up for probe_mix requests.

[chmnata]

The oldest date we can backfill is 2018-01-01, the oldest data we have currently in the database is 2017 Sept. So if we were to backfill we will only loss a few months of data.