.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/data/_02_working_with_ref_data.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_data__02_working_with_ref_data.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_data__02_working_with_ref_data.py:


Working with reference data
===========================

Often you want to test an algorithmic step in isolation or validate the output of an algorithm.
In both cases, you need reference data - either as input or as a comparison.

As explained in the `data example <data_loading_example>`_, reference data that is stored in .mat files can be easily
loaded using the existing tooling.

In this example, we will go into more detail about common patterns of using this reference data.

Ref Data as input on a GS level
-------------------------------
Most algorithms (after the GS detection) expect the data of only a single GS.
If you want to test such an algorithm, you need to use the GS/WB information of the reference data to cut the data
accordingly.
Further, you might also want to get the reference information belonging to the GS/WB.

This can be achieved using the :func:`~mobgap.pipeline.GsIterator`
(or the :func:`~mobgap.pipeline.iter_gs` function).

But first, we need to load some example data.

.. GENERATED FROM PYTHON SOURCE LINES 25-35

.. code-block:: default


    from mobgap.data import LabExampleDataset

    dataset = LabExampleDataset(reference_system="INDIP")
    datapoint = dataset.get_subset(
        cohort="HA", participant_id="001", test="Test11", trial="Trial1"
    )
    data = datapoint.data_ss
    data






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>acc_x</th>
          <th>acc_y</th>
          <th>acc_z</th>
          <th>gyr_x</th>
          <th>gyr_y</th>
          <th>gyr_z</th>
        </tr>
        <tr>
          <th>time</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>2020-12-10 13:09:03.815000057+00:00</th>
          <td>9.690482</td>
          <td>-0.498529</td>
          <td>-0.135459</td>
          <td>-2.326209</td>
          <td>2.967921</td>
          <td>-0.916732</td>
        </tr>
        <tr>
          <th>2020-12-10 13:09:03.825000048+00:00</th>
          <td>9.656917</td>
          <td>-0.489721</td>
          <td>-0.132339</td>
          <td>-1.993893</td>
          <td>2.647065</td>
          <td>-0.813600</td>
        </tr>
        <tr>
          <th>2020-12-10 13:09:03.835000038+00:00</th>
          <td>9.611984</td>
          <td>-0.476001</td>
          <td>-0.097731</td>
          <td>-1.535527</td>
          <td>2.188699</td>
          <td>-0.561499</td>
        </tr>
        <tr>
          <th>2020-12-10 13:09:03.845000029+00:00</th>
          <td>9.624743</td>
          <td>-0.467484</td>
          <td>-0.144403</td>
          <td>-0.882355</td>
          <td>1.879302</td>
          <td>-0.355234</td>
        </tr>
        <tr>
          <th>2020-12-10 13:09:03.855000019+00:00</th>
          <td>9.636681</td>
          <td>-0.488557</td>
          <td>-0.197406</td>
          <td>-0.320856</td>
          <td>1.478231</td>
          <td>-0.595876</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>2020-12-10 13:11:21.355000019+00:00</th>
          <td>9.605895</td>
          <td>-0.272558</td>
          <td>-0.233749</td>
          <td>-2.245995</td>
          <td>1.294885</td>
          <td>-2.417882</td>
        </tr>
        <tr>
          <th>2020-12-10 13:11:21.365000010+00:00</th>
          <td>9.606560</td>
          <td>-0.333937</td>
          <td>-0.245102</td>
          <td>-2.727279</td>
          <td>0.641713</td>
          <td>-2.910626</td>
        </tr>
        <tr>
          <th>2020-12-10 13:11:21.375000+00:00</th>
          <td>9.577996</td>
          <td>-0.333360</td>
          <td>-0.271078</td>
          <td>-4.549285</td>
          <td>-0.160428</td>
          <td>-3.701307</td>
        </tr>
        <tr>
          <th>2020-12-10 13:11:21.384999990+00:00</th>
          <td>9.592582</td>
          <td>-0.389587</td>
          <td>-0.202253</td>
          <td>-5.890006</td>
          <td>-1.226130</td>
          <td>-4.365938</td>
        </tr>
        <tr>
          <th>2020-12-10 13:11:21.394999981+00:00</th>
          <td>9.640033</td>
          <td>-0.392071</td>
          <td>-0.276513</td>
          <td>-7.013003</td>
          <td>-2.234535</td>
          <td>-4.595122</td>
        </tr>
      </tbody>
    </table>
    <p>13759 rows × 6 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 36-42

Then we load the reference data.
There are two versions available:
One version where all values are provided relative to the start of the recording and one version where the values are
provided relative to the start of the respective GS/WB.
Below we can see the first version, here both the walking bouts and the initial contacts (and other parameters) are
provided relative to the start of the recording.

.. GENERATED FROM PYTHON SOURCE LINES 42-45

.. code-block:: default

    ref_data = datapoint.reference_parameters_
    ref_data.wb_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>n_strides</th>
          <th>duration_s</th>
          <th>length_m</th>
          <th>avg_walking_speed_mps</th>
          <th>avg_cadence_spm</th>
          <th>avg_stride_length_m</th>
          <th>termination_reason</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>632</td>
          <td>988</td>
          <td>5</td>
          <td>3.55</td>
          <td>3.428989</td>
          <td>0.975373</td>
          <td>104.069084</td>
          <td>1.124391</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2864</td>
          <td>3325</td>
          <td>4</td>
          <td>4.60</td>
          <td>1.452572</td>
          <td>0.411857</td>
          <td>81.296475</td>
          <td>0.581029</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3853</td>
          <td>5085</td>
          <td>16</td>
          <td>12.31</td>
          <td>7.044042</td>
          <td>0.617801</td>
          <td>89.246331</td>
          <td>0.838960</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7641</td>
          <td>8621</td>
          <td>12</td>
          <td>9.79</td>
          <td>4.396574</td>
          <td>0.510108</td>
          <td>94.370318</td>
          <td>0.645176</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9451</td>
          <td>9932</td>
          <td>6</td>
          <td>4.80</td>
          <td>3.545277</td>
          <td>0.755728</td>
          <td>88.778698</td>
          <td>1.021695</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>5</th>
          <td>11989</td>
          <td>12517</td>
          <td>6</td>
          <td>5.27</td>
          <td>3.514735</td>
          <td>0.880632</td>
          <td>95.832693</td>
          <td>1.021576</td>
          <td>Pause</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 46-48

.. code-block:: default

    ref_data.ic_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>632</td>
          <td>left</td>
        </tr>
        <tr>
          <th>1</th>
          <td>709</td>
          <td>right</td>
        </tr>
        <tr>
          <th>2</th>
          <td>763</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>824</td>
          <td>right</td>
        </tr>
        <tr>
          <th>4</th>
          <td>876</td>
          <td>left</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>3</th>
          <td>12162</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12220</td>
          <td>right</td>
        </tr>
        <tr>
          <th>5</th>
          <td>12277</td>
          <td>left</td>
        </tr>
        <tr>
          <th>6</th>
          <td>12335</td>
          <td>right</td>
        </tr>
        <tr>
          <th>7</th>
          <td>12516</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    <p>63 rows × 2 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 49-53

However, as we want to use the reference data as input to an algorithm on a GS level, we use the version that provides
values relative to the start of the GS/WB.

The start and end values of reference WB are of course still relative to the start of the recording.

.. GENERATED FROM PYTHON SOURCE LINES 53-57

.. code-block:: default

    ref_data_rel = datapoint.reference_parameters_relative_to_wb_
    ref_walking_bouts = ref_data_rel.wb_list
    ref_walking_bouts






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>n_strides</th>
          <th>duration_s</th>
          <th>length_m</th>
          <th>avg_walking_speed_mps</th>
          <th>avg_cadence_spm</th>
          <th>avg_stride_length_m</th>
          <th>termination_reason</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>632</td>
          <td>988</td>
          <td>5</td>
          <td>3.55</td>
          <td>3.428989</td>
          <td>0.975373</td>
          <td>104.069084</td>
          <td>1.124391</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2864</td>
          <td>3325</td>
          <td>4</td>
          <td>4.60</td>
          <td>1.452572</td>
          <td>0.411857</td>
          <td>81.296475</td>
          <td>0.581029</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3853</td>
          <td>5085</td>
          <td>16</td>
          <td>12.31</td>
          <td>7.044042</td>
          <td>0.617801</td>
          <td>89.246331</td>
          <td>0.838960</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7641</td>
          <td>8621</td>
          <td>12</td>
          <td>9.79</td>
          <td>4.396574</td>
          <td>0.510108</td>
          <td>94.370318</td>
          <td>0.645176</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9451</td>
          <td>9932</td>
          <td>6</td>
          <td>4.80</td>
          <td>3.545277</td>
          <td>0.755728</td>
          <td>88.778698</td>
          <td>1.021695</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>5</th>
          <td>11989</td>
          <td>12517</td>
          <td>6</td>
          <td>5.27</td>
          <td>3.514735</td>
          <td>0.880632</td>
          <td>95.832693</td>
          <td>1.021576</td>
          <td>Pause</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 58-59

But the ICs time-samples are now relative to the start of the respective GS/WB.

.. GENERATED FROM PYTHON SOURCE LINES 59-62

.. code-block:: default

    ref_ics_rel = ref_data_rel.ic_list
    ref_ics_rel.loc[0]  # First WB






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>left</td>
        </tr>
        <tr>
          <th>1</th>
          <td>77</td>
          <td>right</td>
        </tr>
        <tr>
          <th>2</th>
          <td>131</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>192</td>
          <td>right</td>
        </tr>
        <tr>
          <th>4</th>
          <td>244</td>
          <td>left</td>
        </tr>
        <tr>
          <th>5</th>
          <td>301</td>
          <td>right</td>
        </tr>
        <tr>
          <th>6</th>
          <td>355</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 63-65

.. code-block:: default

    ref_ics_rel.loc[1]  # Second WB






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>right</td>
        </tr>
        <tr>
          <th>1</th>
          <td>71</td>
          <td>left</td>
        </tr>
        <tr>
          <th>2</th>
          <td>132</td>
          <td>right</td>
        </tr>
        <tr>
          <th>3</th>
          <td>194</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>263</td>
          <td>right</td>
        </tr>
        <tr>
          <th>5</th>
          <td>460</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 66-68

Now we can use the :func:`~mobgap.pipeline.GsIterator` to iterate over the data.
Check out the `gs_iterator example <gs_iterator_example>`_ for more information.

.. GENERATED FROM PYTHON SOURCE LINES 68-76

.. code-block:: default

    from mobgap.pipeline import GsIterator

    gs_iterator = GsIterator()

    # For most use-cases, the default configuration of the :class:`~mobgap.pipeline.GsIterator` should be sufficient.
    # This allows you to specify the following results:
    gs_iterator.data_type








.. GENERATED FROM PYTHON SOURCE LINES 77-81

If you want to change the default behaviour, you can create a custom dataclass (check the example linked above)

The iterator provides us the cut data and an object representing all information of the respective GS/WB.
The latter can be used to index other aspects of the reference data.

.. GENERATED FROM PYTHON SOURCE LINES 81-95

.. code-block:: default

    for (wb, data_per_wb), result in gs_iterator.iterate(data, ref_walking_bouts):
        print("GS/WB id: ", wb.id)
        print(
            "Expected N-samples in wb: ",
            ref_walking_bouts.loc[wb.id].end - ref_walking_bouts.loc[wb.id].start,
        )
        print("N-samples in wb: ", len(data_per_wb))

        # We can use the wb.id to get the reference initial contacts that belong to this GS/WB
        ics_per_wb = ref_ics_rel.loc[wb.id]
        # These could be used in some algorithm.
        # Here we will just store them in the results.
        result.ic_list = ics_per_wb





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    GS/WB id:  0
    Expected N-samples in wb:  356
    N-samples in wb:  356
    GS/WB id:  1
    Expected N-samples in wb:  461
    N-samples in wb:  461
    GS/WB id:  2
    Expected N-samples in wb:  1232
    N-samples in wb:  1232
    GS/WB id:  3
    Expected N-samples in wb:  980
    N-samples in wb:  980
    GS/WB id:  4
    Expected N-samples in wb:  481
    N-samples in wb:  481
    GS/WB id:  5
    Expected N-samples in wb:  528
    N-samples in wb:  528




.. GENERATED FROM PYTHON SOURCE LINES 96-99

The iterator will also conveniently aggregate the results for us.
You can see that the initial contacts are now stored in a single dataframe and the values are transformed back to be
relative to the start of the recording and not the GS anymore.

.. GENERATED FROM PYTHON SOURCE LINES 99-100

.. code-block:: default

    gs_iterator.results_.ic_list





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>632</td>
          <td>left</td>
        </tr>
        <tr>
          <th>1</th>
          <td>709</td>
          <td>right</td>
        </tr>
        <tr>
          <th>2</th>
          <td>763</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>824</td>
          <td>right</td>
        </tr>
        <tr>
          <th>4</th>
          <td>876</td>
          <td>left</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>3</th>
          <td>12162</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12220</td>
          <td>right</td>
        </tr>
        <tr>
          <th>5</th>
          <td>12277</td>
          <td>left</td>
        </tr>
        <tr>
          <th>6</th>
          <td>12335</td>
          <td>right</td>
        </tr>
        <tr>
          <th>7</th>
          <td>12516</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    <p>63 rows × 2 columns</p>
    </div>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 1.711 seconds)

**Estimated memory usage:**  9 MB


.. _sphx_glr_download_auto_examples_data__02_working_with_ref_data.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _02_working_with_ref_data.py <_02_working_with_ref_data.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _02_working_with_ref_data.ipynb <_02_working_with_ref_data.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_