.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/pipeline/_03_dmo_evaluation_on_wb_level.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_pipeline__03_dmo_evaluation_on_wb_level.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_pipeline__03_dmo_evaluation_on_wb_level.py:


.. _gsd_evaluation_parameter:

Evaluation of final walking bout level DMOs
================================================

This example shows how to evaluate the performance of parameters on a walking bout (WB) level by comparing against a reference.
On this level, we usually need to deal with the issue that the WB identified by the algorithm pipeline might not match
the reference WBs.
This makes comparing the parameters within them difficult.
In general, two approaches can be taken here [1]_:

1. First aggregate the WB-level parameters of both systems to a common level (e.g. per trial, per day, per hour, ...)
   and then compare the aggregated values.
2. Identify the subset of WBs that match between the two systems and compare the parameters only within these WBs.

In the following example we will show both approaches.

But first some general setup.

.. [1] Kirk, C., Küderle, A., Micó-Amigo, M.E. et al. Mobilise-D insights to estimate real-world walking speed in
       multiple conditions with a wearable device. Sci Rep 14, 1754 (2024). https://doi.org/10.1038/s41598-024-51766-5

.. GENERATED FROM PYTHON SOURCE LINES 26-34

Loading some example data
-------------------------
We simply load some example DMO data and their reference that we provide with the package.
Usually, the "detected" data would be the output of your algorithm pipeline and the "reference" data would be the
ground truth.

.. note :: This data is randomly generated and not physiologically meaningful. However, it has the same structure as
   any other typical input data for this evaluation.

.. GENERATED FROM PYTHON SOURCE LINES 34-50

.. code-block:: default

    from pprint import pprint

    import numpy as np
    import pandas as pd
    from mobgap import PACKAGE_ROOT

    DATA_PATH = PACKAGE_ROOT.parent / "example_data/dmo_data/dummy_dmo_data"

    detected_dmo = pd.read_csv(DATA_PATH / "detected_dmo_data.csv").set_index(
        ["visit_type", "participant_id", "measurement_date", "wb_id"]
    )

    reference_dmo = pd.read_csv(DATA_PATH / "reference_dmo_data.csv").set_index(
        ["visit_type", "participant_id", "measurement_date", "wb_id"]
    )








.. GENERATED FROM PYTHON SOURCE LINES 51-55

In both dataframes each row represents one WB with all of its parameters.
The index contains multiple levels, including the visit type, participant_id, measurement day, and WB id,
The start and end index of each WB in samples relative to the start of the respective recording is
contained in the columns `start` and `end`.

.. GENERATED FROM PYTHON SOURCE LINES 55-57

.. code-block:: default

    detected_dmo






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>duration_s</th>
          <th>n_steps</th>
          <th>cadence_spm</th>
          <th>walking_speed_mps</th>
          <th>stride_length_m</th>
          <th>stride_duration_s</th>
          <th>n_turns</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="8" valign="top">T1</th>
          <th rowspan="8" valign="top">12345</th>
          <th rowspan="8" valign="top">2023-01-01</th>
          <th>0</th>
          <td>0</td>
          <td>5</td>
          <td>5.130315</td>
          <td>8</td>
          <td>100.232432</td>
          <td>2.061659</td>
          <td>2.907343</td>
          <td>2.384807</td>
          <td>1</td>
        </tr>
        <tr>
          <th>1</th>
          <td>10</td>
          <td>15</td>
          <td>5.436672</td>
          <td>7</td>
          <td>101.677896</td>
          <td>2.722036</td>
          <td>2.469691</td>
          <td>2.439419</td>
          <td>1</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20</td>
          <td>25</td>
          <td>9.140576</td>
          <td>12</td>
          <td>87.484329</td>
          <td>2.255931</td>
          <td>2.572195</td>
          <td>3.472869</td>
          <td>3</td>
        </tr>
        <tr>
          <th>3</th>
          <td>30</td>
          <td>35</td>
          <td>17.204985</td>
          <td>28</td>
          <td>92.096962</td>
          <td>1.141349</td>
          <td>1.595533</td>
          <td>3.507587</td>
          <td>1</td>
        </tr>
        <tr>
          <th>4</th>
          <td>40</td>
          <td>45</td>
          <td>6.217228</td>
          <td>9</td>
          <td>93.988941</td>
          <td>2.548155</td>
          <td>2.338223</td>
          <td>3.313120</td>
          <td>0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>50</td>
          <td>55</td>
          <td>3.521295</td>
          <td>8</td>
          <td>99.425820</td>
          <td>1.820821</td>
          <td>2.882743</td>
          <td>2.388743</td>
          <td>0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>60</td>
          <td>65</td>
          <td>14.034649</td>
          <td>17</td>
          <td>87.428880</td>
          <td>2.783120</td>
          <td>3.309131</td>
          <td>2.627496</td>
          <td>1</td>
        </tr>
        <tr>
          <th>7</th>
          <td>70</td>
          <td>75</td>
          <td>8.296356</td>
          <td>12</td>
          <td>86.372700</td>
          <td>2.240491</td>
          <td>2.721844</td>
          <td>1.653604</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 58-60

.. code-block:: default

    reference_dmo






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>duration_s</th>
          <th>n_steps</th>
          <th>cadence_spm</th>
          <th>walking_speed_mps</th>
          <th>stride_length_m</th>
          <th>stride_duration_s</th>
          <th>n_turns</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="10" valign="top">T1</th>
          <th rowspan="10" valign="top">12345</th>
          <th rowspan="10" valign="top">2023-01-01</th>
          <th>0</th>
          <td>0</td>
          <td>4</td>
          <td>4.74702</td>
          <td>8</td>
          <td>99.82188</td>
          <td>1.16079</td>
          <td>2.51885</td>
          <td>1.58675</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>15</td>
          <td>19</td>
          <td>5.13150</td>
          <td>7</td>
          <td>101.16429</td>
          <td>2.57881</td>
          <td>1.57243</td>
          <td>1.46537</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20</td>
          <td>24</td>
          <td>8.52727</td>
          <td>12</td>
          <td>86.53527</td>
          <td>1.60044</td>
          <td>1.66305</td>
          <td>2.56092</td>
          <td>2</td>
        </tr>
        <tr>
          <th>3</th>
          <td>35</td>
          <td>39</td>
          <td>16.24554</td>
          <td>27</td>
          <td>91.49977</td>
          <td>0.95558</td>
          <td>0.88961</td>
          <td>3.14549</td>
          <td>1</td>
        </tr>
        <tr>
          <th>4</th>
          <td>40</td>
          <td>44</td>
          <td>6.09907</td>
          <td>8</td>
          <td>93.69895</td>
          <td>2.33230</td>
          <td>1.95969</td>
          <td>2.35295</td>
          <td>0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>55</td>
          <td>59</td>
          <td>3.25806</td>
          <td>7</td>
          <td>99.33525</td>
          <td>1.14732</td>
          <td>2.37307</td>
          <td>1.81262</td>
          <td>0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>60</td>
          <td>64</td>
          <td>13.52080</td>
          <td>17</td>
          <td>87.09312</td>
          <td>2.09538</td>
          <td>2.41782</td>
          <td>2.00582</td>
          <td>1</td>
        </tr>
        <tr>
          <th>7</th>
          <td>75</td>
          <td>79</td>
          <td>7.49830</td>
          <td>12</td>
          <td>85.96436</td>
          <td>2.23757</td>
          <td>1.89026</td>
          <td>1.55788</td>
          <td>0</td>
        </tr>
        <tr>
          <th>8</th>
          <td>80</td>
          <td>84</td>
          <td>8.21455</td>
          <td>10</td>
          <td>75.12352</td>
          <td>0.59915</td>
          <td>2.16121</td>
          <td>2.31160</td>
          <td>0</td>
        </tr>
        <tr>
          <th>9</th>
          <td>95</td>
          <td>99</td>
          <td>6.84377</td>
          <td>9</td>
          <td>76.61402</td>
          <td>2.22903</td>
          <td>1.03362</td>
          <td>3.17821</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 61-68

Approach 1: Aggregate then compare
----------------------------------
First, we combine the detected and reference data,
which can easily be done as both dataframes have the same index levels.
To sustain the information about the origin of the data, we add a column level assigning `"detected"`
and `"reference"` to the respective dmos.
Furthermore, we rearrange the columns to have the DMO metrics as the first level of the column index.

.. GENERATED FROM PYTHON SOURCE LINES 68-77

.. code-block:: default

    combined_dmos = (
        pd.concat(
            [detected_dmo, reference_dmo], keys=["detected", "reference"], axis=1
        )
        .reorder_levels((1, 0), axis=1)
        .sort_index(axis=1)
    )
    combined_dmos






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }

        .dataframe thead tr:last-of-type th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th colspan="2" halign="left">cadence_spm</th>
          <th colspan="2" halign="left">duration_s</th>
          <th colspan="2" halign="left">end</th>
          <th colspan="2" halign="left">n_steps</th>
          <th colspan="2" halign="left">n_turns</th>
          <th colspan="2" halign="left">start</th>
          <th colspan="2" halign="left">stride_duration_s</th>
          <th colspan="2" halign="left">stride_length_m</th>
          <th colspan="2" halign="left">walking_speed_mps</th>
        </tr>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="10" valign="top">T1</th>
          <th rowspan="10" valign="top">12345</th>
          <th rowspan="10" valign="top">2023-01-01</th>
          <th>0</th>
          <td>100.232432</td>
          <td>99.82188</td>
          <td>5.130315</td>
          <td>4.74702</td>
          <td>5.0</td>
          <td>4</td>
          <td>8.0</td>
          <td>8</td>
          <td>1.0</td>
          <td>0</td>
          <td>0.0</td>
          <td>0</td>
          <td>2.384807</td>
          <td>1.58675</td>
          <td>2.907343</td>
          <td>2.51885</td>
          <td>2.061659</td>
          <td>1.16079</td>
        </tr>
        <tr>
          <th>1</th>
          <td>101.677896</td>
          <td>101.16429</td>
          <td>5.436672</td>
          <td>5.13150</td>
          <td>15.0</td>
          <td>19</td>
          <td>7.0</td>
          <td>7</td>
          <td>1.0</td>
          <td>0</td>
          <td>10.0</td>
          <td>15</td>
          <td>2.439419</td>
          <td>1.46537</td>
          <td>2.469691</td>
          <td>1.57243</td>
          <td>2.722036</td>
          <td>2.57881</td>
        </tr>
        <tr>
          <th>2</th>
          <td>87.484329</td>
          <td>86.53527</td>
          <td>9.140576</td>
          <td>8.52727</td>
          <td>25.0</td>
          <td>24</td>
          <td>12.0</td>
          <td>12</td>
          <td>3.0</td>
          <td>2</td>
          <td>20.0</td>
          <td>20</td>
          <td>3.472869</td>
          <td>2.56092</td>
          <td>2.572195</td>
          <td>1.66305</td>
          <td>2.255931</td>
          <td>1.60044</td>
        </tr>
        <tr>
          <th>3</th>
          <td>92.096962</td>
          <td>91.49977</td>
          <td>17.204985</td>
          <td>16.24554</td>
          <td>35.0</td>
          <td>39</td>
          <td>28.0</td>
          <td>27</td>
          <td>1.0</td>
          <td>1</td>
          <td>30.0</td>
          <td>35</td>
          <td>3.507587</td>
          <td>3.14549</td>
          <td>1.595533</td>
          <td>0.88961</td>
          <td>1.141349</td>
          <td>0.95558</td>
        </tr>
        <tr>
          <th>4</th>
          <td>93.988941</td>
          <td>93.69895</td>
          <td>6.217228</td>
          <td>6.09907</td>
          <td>45.0</td>
          <td>44</td>
          <td>9.0</td>
          <td>8</td>
          <td>0.0</td>
          <td>0</td>
          <td>40.0</td>
          <td>40</td>
          <td>3.313120</td>
          <td>2.35295</td>
          <td>2.338223</td>
          <td>1.95969</td>
          <td>2.548155</td>
          <td>2.33230</td>
        </tr>
        <tr>
          <th>5</th>
          <td>99.425820</td>
          <td>99.33525</td>
          <td>3.521295</td>
          <td>3.25806</td>
          <td>55.0</td>
          <td>59</td>
          <td>8.0</td>
          <td>7</td>
          <td>0.0</td>
          <td>0</td>
          <td>50.0</td>
          <td>55</td>
          <td>2.388743</td>
          <td>1.81262</td>
          <td>2.882743</td>
          <td>2.37307</td>
          <td>1.820821</td>
          <td>1.14732</td>
        </tr>
        <tr>
          <th>6</th>
          <td>87.428880</td>
          <td>87.09312</td>
          <td>14.034649</td>
          <td>13.52080</td>
          <td>65.0</td>
          <td>64</td>
          <td>17.0</td>
          <td>17</td>
          <td>1.0</td>
          <td>1</td>
          <td>60.0</td>
          <td>60</td>
          <td>2.627496</td>
          <td>2.00582</td>
          <td>3.309131</td>
          <td>2.41782</td>
          <td>2.783120</td>
          <td>2.09538</td>
        </tr>
        <tr>
          <th>7</th>
          <td>86.372700</td>
          <td>85.96436</td>
          <td>8.296356</td>
          <td>7.49830</td>
          <td>75.0</td>
          <td>79</td>
          <td>12.0</td>
          <td>12</td>
          <td>1.0</td>
          <td>0</td>
          <td>70.0</td>
          <td>75</td>
          <td>1.653604</td>
          <td>1.55788</td>
          <td>2.721844</td>
          <td>1.89026</td>
          <td>2.240491</td>
          <td>2.23757</td>
        </tr>
        <tr>
          <th>8</th>
          <td>NaN</td>
          <td>75.12352</td>
          <td>NaN</td>
          <td>8.21455</td>
          <td>NaN</td>
          <td>84</td>
          <td>NaN</td>
          <td>10</td>
          <td>NaN</td>
          <td>0</td>
          <td>NaN</td>
          <td>80</td>
          <td>NaN</td>
          <td>2.31160</td>
          <td>NaN</td>
          <td>2.16121</td>
          <td>NaN</td>
          <td>0.59915</td>
        </tr>
        <tr>
          <th>9</th>
          <td>NaN</td>
          <td>76.61402</td>
          <td>NaN</td>
          <td>6.84377</td>
          <td>NaN</td>
          <td>99</td>
          <td>NaN</td>
          <td>9</td>
          <td>NaN</td>
          <td>0</td>
          <td>NaN</td>
          <td>95</td>
          <td>NaN</td>
          <td>3.17821</td>
          <td>NaN</td>
          <td>1.03362</td>
          <td>NaN</td>
          <td>2.22903</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 78-91

This provides us with a dataframe containing the detected and reference values for all detected and reference WBs.
Some entries are NaN, as the number of WBs in the detected and reference data might differ.
The single rows in this dataframe should not be compared directly,
as the same WB ids from a detected and a reference WB might not actually belong to the same WB.
Therefore, we need to aggregate the DMO data based on an index level of choice, e.g., per day, to retrieve meaningful
and interpretable results. This can for instance be done by grouping the data and averaging over the groups.
If required, apart from simple groupwise averaging, other aggregation functions
(e.g., moving averages or averaging over a span of several days) can be applied.
As long as the same aggregation method is applied to both the detected and reference data,
further processing can be done in the same way as shown below.

.. note:: In case of missing data, applying `dropna()` to the resulting dataframe might be helpful to remove all
          groups were either detected or reference data is missing.

.. GENERATED FROM PYTHON SOURCE LINES 91-100

.. code-block:: default

    daily_matches = (
        combined_dmos.groupby(
            level=["visit_type", "participant_id", "measurement_date"], axis=0
        )
        .mean()
        .dropna()
    )
    daily_matches.T





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/examples/pipeline/_03_dmo_evaluation_on_wb_level.py:92: FutureWarning: The 'axis' keyword in DataFrame.groupby is deprecated and will be removed in a future version.
      combined_dmos.groupby(


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th>T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th>12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th>2023-01-01</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="2" valign="top">cadence_spm</th>
          <th>detected</th>
          <td>93.588495</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>89.685043</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">duration_s</th>
          <th>detected</th>
          <td>8.622759</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>8.008588</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">end</th>
          <th>detected</th>
          <td>40.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>51.500000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_steps</th>
          <th>detected</th>
          <td>12.625000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>11.700000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_turns</th>
          <th>detected</th>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.400000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">start</th>
          <th>detected</th>
          <td>35.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>47.500000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_duration_s</th>
          <th>detected</th>
          <td>2.723456</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>2.197761</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_length_m</th>
          <th>detected</th>
          <td>2.599588</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.847961</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">walking_speed_mps</th>
          <th>detected</th>
          <td>2.196695</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.693637</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 101-105

The resulting dataframe contains the average detected and reference values for each DMO
per visit type, participant, and day.
This in conveniently provided as a multiindex column, so that selecting a single DMO, yields a DataFrame with the
detected and reference values.

.. GENERATED FROM PYTHON SOURCE LINES 105-107

.. code-block:: default

    daily_matches["cadence_spm"]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th>detected</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>T1</th>
          <th>12345</th>
          <th>2023-01-01</th>
          <td>93.588495</td>
          <td>89.685043</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 108-114

In our example data, we only have data from a single day, so the aggregated result only has one row.
Normally, you would have multiple rows, one for each group of WBs.
From here on, further processing to retrieve the aggregated error metrics is identical to the further processing when
following approach 2, and is shown below.

But let's first show how to calculate the error metrics on a WB-by-WB basis.

.. GENERATED FROM PYTHON SOURCE LINES 116-134

Approach 2: Match then compare
------------------------------
As the first step we need to indentify WBs that match between the detected and reference data.
As it is unlikely that the WBs are exactly the same, we need to define a threshold for the overlap between the WBs
to consider them as a match.
This matching can be done using the :func:`~mobgap.pipeline.evaluation.categorize_intervals` function.
It classifies every WB in the data either as true positive (TP), false positive (FP), or false negative
(TP).
In case our data has only WBs from a single recording, we could directly provide the detected and reference data
to the function.

However, in most cases data would contain WBs from multiple recordings, trials, and participants, ... .
In our case, we actually only have WBs from a single recording, but we will still show the approach assuming that
the data is more complex.

To avoid, that WBs from different recordings are matched (as the matching is just performed based on the start/end
index), we need to group the data by the relevant index levels first and apply the matching function to each group.
This can be done using the :func:`~mobgap.utils.array_handling.create_multi_groupby` helper function.

.. GENERATED FROM PYTHON SOURCE LINES 134-142

.. code-block:: default

    from mobgap.utils.df_operations import create_multi_groupby

    per_trial_participant_day_grouper = create_multi_groupby(
        detected_dmo,
        reference_dmo,
        groupby=["visit_type", "participant_id", "measurement_date"],
    )








.. GENERATED FROM PYTHON SOURCE LINES 143-152

This provides us with a groupby-object that is similar to the normal pandas groupby-object that can be created from a
single dataframe.
The ``MultiGroupBy`` object allows us to apply a function to each group across all dataframes.

Here we apply :func:`~mobgap.pipeline.evaluation.categorize_intervals` with a threshold of 0.8 to each group.
The `overlap_threshold` parameter defines the minimum overlap between the detected and reference WBs to be
considered a match.
It can be chosen according to your needs, whereby a value closer to 0.5 will yield more matches
than a value closer to 1.

.. GENERATED FROM PYTHON SOURCE LINES 152-165

.. code-block:: default


    from mobgap.pipeline.evaluation import categorize_intervals

    wb_tp_fp_fn = per_trial_participant_day_grouper.apply(
        lambda det, ref: categorize_intervals(
            gsd_list_detected=det,
            gsd_list_reference=ref,
            overlap_threshold=0.8,
            multiindex_warning=False,
        )
    )
    wb_tp_fp_fn






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>gs_id_detected</th>
          <th>gs_id_reference</th>
          <th>match_type</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th>match_id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="14" valign="top">T1</th>
          <th rowspan="14" valign="top">12345</th>
          <th rowspan="14" valign="top">2023-01-01</th>
          <th>0</th>
          <td>(T1, 12345, 2023-01-01, 0)</td>
          <td>(T1, 12345, 2023-01-01, 0)</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>1</th>
          <td>(T1, 12345, 2023-01-01, 1)</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>2</th>
          <td>(T1, 12345, 2023-01-01, 2)</td>
          <td>(T1, 12345, 2023-01-01, 2)</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>3</th>
          <td>(T1, 12345, 2023-01-01, 3)</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>4</th>
          <td>(T1, 12345, 2023-01-01, 4)</td>
          <td>(T1, 12345, 2023-01-01, 4)</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>5</th>
          <td>(T1, 12345, 2023-01-01, 5)</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>6</th>
          <td>(T1, 12345, 2023-01-01, 6)</td>
          <td>(T1, 12345, 2023-01-01, 6)</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>7</th>
          <td>(T1, 12345, 2023-01-01, 7)</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>8</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 1)</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>9</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 3)</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>10</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 5)</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>11</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 7)</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>12</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 8)</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>13</th>
          <td>NaN</td>
          <td>(T1, 12345, 2023-01-01, 9)</td>
          <td>fn</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 166-176

We can see that the function returns a dataframe with the same index as the input dataframes and each WB is classified
as TP, FP, or FN.
For the TP WBs, the corresponding reference WB is assigned.
For the comparison we want to perform here, only the matching WBs, i.e., the TPs, are of interest.
If you are interested in the FPs or FNs, have a look at the general :ref:`GSD evaluation example <gsd_evaluation>`.

Based on the positive matches, we can now extract the DMO data from detected and reference data that is to be
compared.
To make extracting all the TP WBs a little easier, we can use the
:func:`~mobgap.pipeline.evaluation.get_matching_intervals` function.

.. GENERATED FROM PYTHON SOURCE LINES 176-185

.. code-block:: default

    from mobgap.pipeline.evaluation import get_matching_intervals

    wb_matches = get_matching_intervals(
        metrics_detected=detected_dmo,
        metrics_reference=reference_dmo,
        matches=wb_tp_fp_fn,
    )
    wb_matches.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th colspan="4" halign="left">T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th colspan="4" halign="left">12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th colspan="4" halign="left">2023-01-01</th>
        </tr>
        <tr>
          <th></th>
          <th>match_id</th>
          <th>0</th>
          <th>2</th>
          <th>4</th>
          <th>6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="2" valign="top">cadence_spm</th>
          <th>detected</th>
          <td>100.232432</td>
          <td>87.484329</td>
          <td>93.988941</td>
          <td>87.428880</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>99.821880</td>
          <td>86.535270</td>
          <td>93.698950</td>
          <td>87.093120</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">duration_s</th>
          <th>detected</th>
          <td>5.130315</td>
          <td>9.140576</td>
          <td>6.217228</td>
          <td>14.034649</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>4.747020</td>
          <td>8.527270</td>
          <td>6.099070</td>
          <td>13.520800</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">end</th>
          <th>detected</th>
          <td>5.000000</td>
          <td>25.000000</td>
          <td>45.000000</td>
          <td>65.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>4.000000</td>
          <td>24.000000</td>
          <td>44.000000</td>
          <td>64.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_steps</th>
          <th>detected</th>
          <td>8.000000</td>
          <td>12.000000</td>
          <td>9.000000</td>
          <td>17.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>8.000000</td>
          <td>12.000000</td>
          <td>8.000000</td>
          <td>17.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_turns</th>
          <th>detected</th>
          <td>1.000000</td>
          <td>3.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">start</th>
          <th>detected</th>
          <td>0.000000</td>
          <td>20.000000</td>
          <td>40.000000</td>
          <td>60.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>20.000000</td>
          <td>40.000000</td>
          <td>60.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_duration_s</th>
          <th>detected</th>
          <td>2.384807</td>
          <td>3.472869</td>
          <td>3.313120</td>
          <td>2.627496</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.586750</td>
          <td>2.560920</td>
          <td>2.352950</td>
          <td>2.005820</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_length_m</th>
          <th>detected</th>
          <td>2.907343</td>
          <td>2.572195</td>
          <td>2.338223</td>
          <td>3.309131</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>2.518850</td>
          <td>1.663050</td>
          <td>1.959690</td>
          <td>2.417820</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">walking_speed_mps</th>
          <th>detected</th>
          <td>2.061659</td>
          <td>2.255931</td>
          <td>2.548155</td>
          <td>2.783120</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.160790</td>
          <td>1.600440</td>
          <td>2.332300</td>
          <td>2.095380</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">wb_id</th>
          <th>detected</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>4.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>4.000000</td>
          <td>6.000000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 186-189

The returned dataframe contains the detected and reference values for all DMOs of the matched WBs.
This in conveniently provided as a multiindex column, so that selecting a single DMO, yields a DataFrame with the
detected and reference values.

.. GENERATED FROM PYTHON SOURCE LINES 189-191

.. code-block:: default

    wb_matches["cadence_spm"]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>detected</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>visit_type</th>
          <th>participant_id</th>
          <th>measurement_date</th>
          <th>match_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="4" valign="top">T1</th>
          <th rowspan="4" valign="top">12345</th>
          <th rowspan="4" valign="top">2023-01-01</th>
          <th>0</th>
          <td>100.232432</td>
          <td>99.82188</td>
        </tr>
        <tr>
          <th>2</th>
          <td>87.484329</td>
          <td>86.53527</td>
        </tr>
        <tr>
          <th>4</th>
          <td>93.988941</td>
          <td>93.69895</td>
        </tr>
        <tr>
          <th>6</th>
          <td>87.428880</td>
          <td>87.09312</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 192-220

From here on, the aggregated DMOs (when following approach 1) or matched WBs (when following approach 2)
can be compared with the same methods to calculate error metrics. For the sake of simplicity, we will show the
calculation of error metrics for the matched WBs `wb_matches` (approach 2) here.
However, the input can also simply be replaced by the aggregated DMO dataframe ``

Estimate Errors in DMO data
---------------------------
The DMO data can now be compared day by day (approach 1) or WB by WB (approach 2).
We want to calculate general error metrics like the error, absolute error, relative error, and absolute relative error
for each day (WB) and DMO.
This can be done using the generic the :func:`~mobgap.utils.df_operations.apply_transformations` helper that allows
us to apply any list of transformation functions (transformation function -> WB in Series with same length out).
It further allows us to declaratively define which transformation/error should be applied
to which columns (i.e. which DMOs).

A simple definition of error metrics would look like this:
As input, it receives the matching DMO data and a list of transformations that should be applied to the data.
A transformation is characterized as a function that takes some subset of the input dataframe,
performs some operation on it, and returns a series with the same length as the input as output.
Calculating the differences between two sets of values, e.g., between detected and reference values,
is a common type of transformation that is applied to evaluate the performance of the DMO estimation.
For this purpose, the transformations are defined as aa list of tuples containing the DMO of interest
as the first element and the error functions applied to the detected and reference values as the second element.
This way, you can also define custom error functions and pass them as transformations.
Note that the columns of the detected and reference values are expected to be named `detected` and `reference`
per default.
For the standard error metrics (error, absolute error, relative error, absolute relative),
the :func:`~mobgap.pipeline.evaluation.get_default_error_transformations` returns the correct transformations.

.. GENERATED FROM PYTHON SOURCE LINES 220-228

.. code-block:: default

    from mobgap.pipeline.evaluation import ErrorTransformFuncs as E

    custom_errors = [
        ("cadence_spm", [E.abs_error, E.rel_error]),
        ("duration_s", [E.error]),
        ("n_turns", [E.rel_error]),
    ]








.. GENERATED FROM PYTHON SOURCE LINES 229-235

This definition should be relatively self-explanatory.

We can now apply these transformations to the DMO data using the
:func:`~mobgap.utils.df_operations.apply_transformations`.
Note, that there is no need to group the dataframe again, as all the transformations are applied row-wise to the
entire dataframe.

.. GENERATED FROM PYTHON SOURCE LINES 235-241

.. code-block:: default

    from mobgap.utils.df_operations import apply_transformations

    custom_wb_errors = apply_transformations(wb_matches, custom_errors)
    custom_wb_errors.T






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/pipeline/_error_metrics.py:84: UserWarning: Zero division occurred in rel_error because divisor contains zeroes. Affected error metrics are set to NaN.
      _handle_zero_division(ref, zero_division_hint, "rel_error")


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th colspan="4" halign="left">T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th colspan="4" halign="left">12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th colspan="4" halign="left">2023-01-01</th>
        </tr>
        <tr>
          <th></th>
          <th>match_id</th>
          <th>0</th>
          <th>2</th>
          <th>4</th>
          <th>6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="2" valign="top">cadence_spm</th>
          <th>abs_error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.004113</td>
          <td>0.010967</td>
          <td>0.003095</td>
          <td>0.003855</td>
        </tr>
        <tr>
          <th>duration_s</th>
          <th>error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th>n_turns</th>
          <th>rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 242-249

We can also modify the error metrics or provide custom error functions.
We will show three options here.

1. Use a usual error metric, but change some input parameters, and have the output under a new name.
   For this case, we just define a new function wrapping the old one.
   For example, we might want to suppress the warning that is raised when a zero division occurs in the relative error.
   As we saw above, this warning is raised for the `n_turns` parameter.

.. GENERATED FROM PYTHON SOURCE LINES 249-253

.. code-block:: default

    def rel_error_without_warning(x):
        return E.rel_error(x, zero_division_hint=np.nan)









.. GENERATED FROM PYTHON SOURCE LINES 254-258

2. When we want to keep the same name for the function, we could just overwrite the old function. But,
   to avoid accidentally messing up other code, that uses the function, we can also use a lambda function and manually
   set the name of the function.
   As a result, we supress the warning as above, but keep the function name for the aggregation.

.. GENERATED FROM PYTHON SOURCE LINES 258-262

.. code-block:: default

    rel_error_as_lambda = lambda x: E.rel_error(x, zero_division_hint=np.nan)
    rel_error_as_lambda.__name__ = "rel_error"









.. GENERATED FROM PYTHON SOURCE LINES 263-272

3. We can also define a completely new error function.
   The Dataframe we get as input here, contains the columns `detected` and `reference` with the detected and reference
   values for the DMO of interest.
   For this example here, we will create a nonsensical ``scaled_error`` function that scales the error by a factor
   of 2.

.. note::
   If you want to introduce custom, more complex transformation functions, you can also define them as
   :class:`~mobgap.utils.df_operations.CustomOperation` as shown for aggregations in the "Aggregation" section.

.. GENERATED FROM PYTHON SOURCE LINES 272-276

.. code-block:: default

    def scaled_error(x):
        return 2 * (x["detected"] - x["reference"])









.. GENERATED FROM PYTHON SOURCE LINES 277-281

Our custom functions can now be used in the transformations list and freely combined with other error metrics.

Also, keep in mind, that the definition is "just" Python, so we can use things like list comprehensions to generate
the list of transformations as shown below.

.. GENERATED FROM PYTHON SOURCE LINES 281-294

.. code-block:: default

    custom_errors = [
        ("cadence_spm", [E.error, scaled_error]),
        ("duration_s", [E.error]),
        ("n_turns", [rel_error_without_warning, rel_error_as_lambda]),
        *(
            (m, [E.abs_error, E.rel_error])
            for m in ["stride_duration_s", "stride_length_m"]
        ),
    ]

    custom_wb_errors = apply_transformations(wb_matches, custom_errors)
    custom_wb_errors.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th colspan="4" halign="left">T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th colspan="4" halign="left">12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th colspan="4" halign="left">2023-01-01</th>
        </tr>
        <tr>
          <th></th>
          <th>match_id</th>
          <th>0</th>
          <th>2</th>
          <th>4</th>
          <th>6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="2" valign="top">cadence_spm</th>
          <th>error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>scaled_error</th>
          <td>0.821103</td>
          <td>1.898118</td>
          <td>0.579983</td>
          <td>0.671520</td>
        </tr>
        <tr>
          <th>duration_s</th>
          <th>error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_turns</th>
          <th>rel_error_without_warning</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_duration_s</th>
          <th>abs_error</th>
          <td>0.798057</td>
          <td>0.911949</td>
          <td>0.960170</td>
          <td>0.621676</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.502951</td>
          <td>0.356102</td>
          <td>0.408071</td>
          <td>0.309936</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_length_m</th>
          <th>abs_error</th>
          <td>0.388493</td>
          <td>0.909145</td>
          <td>0.378533</td>
          <td>0.891311</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.154234</td>
          <td>0.546673</td>
          <td>0.193160</td>
          <td>0.368642</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 295-301

As expected, the resulting dataframe contains the error metrics for the specified DMOs and could now be further
processed, e.g., by aggregating the results.

As an alternative to defining a custom error definition, we provide a "default" error definition that can be used to
calculate the standard error metrics for the common DMOs.
In most cases, this is a good starting point for the evaluation of the DMOs.

.. GENERATED FROM PYTHON SOURCE LINES 301-307

.. code-block:: default

    from mobgap.pipeline.evaluation import get_default_error_transformations

    default_errors = get_default_error_transformations()

    pprint(default_errors)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [('cadence_spm',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('duration_s',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('n_steps',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('n_strides',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('n_turns',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('stride_duration_s',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('stride_length_m',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>]),
     ('walking_speed_mps',
      [<function error at 0x7fdc83ac5480>,
       <function rel_error at 0x7fdc82f9b5b0>,
       <function abs_error at 0x7fdc82f9b640>,
       <function abs_rel_error at 0x7fdc82f9b6d0>])]




.. GENERATED FROM PYTHON SOURCE LINES 308-312

While the visualization here is a little ugly, we can see that the default error transformation attempts to calculate
the error, the relative error, the absolute error, and the absolute relative error for all the core DMOs.

We can apply it as before.

.. GENERATED FROM PYTHON SOURCE LINES 312-316

.. code-block:: default

    wb_errors = apply_transformations(wb_matches, default_errors)
    wb_errors.T






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:365: UserWarning: One of the transformations requires the following columns: 'n_strides'. They are not found in the DataFrame.
      warnings.warn(str(e), stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/pipeline/_error_metrics.py:84: UserWarning: Zero division occurred in rel_error because divisor contains zeroes. Affected error metrics are set to NaN.
      _handle_zero_division(ref, zero_division_hint, "rel_error")
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/pipeline/_error_metrics.py:146: UserWarning: Zero division occurred in abs_rel_error because divisor contains zeroes. Affected error metrics are set to NaN.
      _handle_zero_division(ref, zero_division_hint, "abs_rel_error")


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th colspan="4" halign="left">T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th colspan="4" halign="left">12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th colspan="4" halign="left">2023-01-01</th>
        </tr>
        <tr>
          <th></th>
          <th>match_id</th>
          <th>0</th>
          <th>2</th>
          <th>4</th>
          <th>6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="4" valign="top">cadence_spm</th>
          <th>error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.004113</td>
          <td>0.010967</td>
          <td>0.003095</td>
          <td>0.003855</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.004113</td>
          <td>0.010967</td>
          <td>0.003095</td>
          <td>0.003855</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">duration_s</th>
          <th>error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.080744</td>
          <td>0.071923</td>
          <td>0.019373</td>
          <td>0.038004</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.080744</td>
          <td>0.071923</td>
          <td>0.019373</td>
          <td>0.038004</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">n_steps</th>
          <th>error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.125000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.125000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">n_turns</th>
          <th>error</th>
          <td>1.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>1.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">stride_duration_s</th>
          <th>error</th>
          <td>0.798057</td>
          <td>0.911949</td>
          <td>0.960170</td>
          <td>0.621676</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.502951</td>
          <td>0.356102</td>
          <td>0.408071</td>
          <td>0.309936</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.798057</td>
          <td>0.911949</td>
          <td>0.960170</td>
          <td>0.621676</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.502951</td>
          <td>0.356102</td>
          <td>0.408071</td>
          <td>0.309936</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">stride_length_m</th>
          <th>error</th>
          <td>0.388493</td>
          <td>0.909145</td>
          <td>0.378533</td>
          <td>0.891311</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.154234</td>
          <td>0.546673</td>
          <td>0.193160</td>
          <td>0.368642</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.388493</td>
          <td>0.909145</td>
          <td>0.378533</td>
          <td>0.891311</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.154234</td>
          <td>0.546673</td>
          <td>0.193160</td>
          <td>0.368642</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">walking_speed_mps</th>
          <th>error</th>
          <td>0.900869</td>
          <td>0.655491</td>
          <td>0.215855</td>
          <td>0.687740</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.776083</td>
          <td>0.409569</td>
          <td>0.092550</td>
          <td>0.328217</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.900869</td>
          <td>0.655491</td>
          <td>0.215855</td>
          <td>0.687740</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.776083</td>
          <td>0.409569</td>
          <td>0.092550</td>
          <td>0.328217</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 317-319

Before we now aggregate the results, we can also combine the error metrics with the reference and detected values
to have all the information in one dataframe.

.. GENERATED FROM PYTHON SOURCE LINES 319-322

.. code-block:: default

    wb_matches_with_errors = pd.concat([wb_matches, wb_errors], axis=1)
    wb_matches_with_errors.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th>visit_type</th>
          <th colspan="4" halign="left">T1</th>
        </tr>
        <tr>
          <th></th>
          <th>participant_id</th>
          <th colspan="4" halign="left">12345</th>
        </tr>
        <tr>
          <th></th>
          <th>measurement_date</th>
          <th colspan="4" halign="left">2023-01-01</th>
        </tr>
        <tr>
          <th></th>
          <th>match_id</th>
          <th>0</th>
          <th>2</th>
          <th>4</th>
          <th>6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="2" valign="top">cadence_spm</th>
          <th>detected</th>
          <td>100.232432</td>
          <td>87.484329</td>
          <td>93.988941</td>
          <td>87.428880</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>99.821880</td>
          <td>86.535270</td>
          <td>93.698950</td>
          <td>87.093120</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">duration_s</th>
          <th>detected</th>
          <td>5.130315</td>
          <td>9.140576</td>
          <td>6.217228</td>
          <td>14.034649</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>4.747020</td>
          <td>8.527270</td>
          <td>6.099070</td>
          <td>13.520800</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">end</th>
          <th>detected</th>
          <td>5.000000</td>
          <td>25.000000</td>
          <td>45.000000</td>
          <td>65.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>4.000000</td>
          <td>24.000000</td>
          <td>44.000000</td>
          <td>64.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_steps</th>
          <th>detected</th>
          <td>8.000000</td>
          <td>12.000000</td>
          <td>9.000000</td>
          <td>17.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>8.000000</td>
          <td>12.000000</td>
          <td>8.000000</td>
          <td>17.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">n_turns</th>
          <th>detected</th>
          <td>1.000000</td>
          <td>3.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">start</th>
          <th>detected</th>
          <td>0.000000</td>
          <td>20.000000</td>
          <td>40.000000</td>
          <td>60.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>20.000000</td>
          <td>40.000000</td>
          <td>60.000000</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_duration_s</th>
          <th>detected</th>
          <td>2.384807</td>
          <td>3.472869</td>
          <td>3.313120</td>
          <td>2.627496</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.586750</td>
          <td>2.560920</td>
          <td>2.352950</td>
          <td>2.005820</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">stride_length_m</th>
          <th>detected</th>
          <td>2.907343</td>
          <td>2.572195</td>
          <td>2.338223</td>
          <td>3.309131</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>2.518850</td>
          <td>1.663050</td>
          <td>1.959690</td>
          <td>2.417820</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">walking_speed_mps</th>
          <th>detected</th>
          <td>2.061659</td>
          <td>2.255931</td>
          <td>2.548155</td>
          <td>2.783120</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>1.160790</td>
          <td>1.600440</td>
          <td>2.332300</td>
          <td>2.095380</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">wb_id</th>
          <th>detected</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>4.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>0.000000</td>
          <td>2.000000</td>
          <td>4.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">cadence_spm</th>
          <th>error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.004113</td>
          <td>0.010967</td>
          <td>0.003095</td>
          <td>0.003855</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.410552</td>
          <td>0.949059</td>
          <td>0.289991</td>
          <td>0.335760</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.004113</td>
          <td>0.010967</td>
          <td>0.003095</td>
          <td>0.003855</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">duration_s</th>
          <th>error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.080744</td>
          <td>0.071923</td>
          <td>0.019373</td>
          <td>0.038004</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.383295</td>
          <td>0.613306</td>
          <td>0.118158</td>
          <td>0.513849</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.080744</td>
          <td>0.071923</td>
          <td>0.019373</td>
          <td>0.038004</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">n_steps</th>
          <th>error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.125000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.125000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">n_turns</th>
          <th>error</th>
          <td>1.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>1.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>NaN</td>
          <td>0.500000</td>
          <td>NaN</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">stride_duration_s</th>
          <th>error</th>
          <td>0.798057</td>
          <td>0.911949</td>
          <td>0.960170</td>
          <td>0.621676</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.502951</td>
          <td>0.356102</td>
          <td>0.408071</td>
          <td>0.309936</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.798057</td>
          <td>0.911949</td>
          <td>0.960170</td>
          <td>0.621676</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.502951</td>
          <td>0.356102</td>
          <td>0.408071</td>
          <td>0.309936</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">stride_length_m</th>
          <th>error</th>
          <td>0.388493</td>
          <td>0.909145</td>
          <td>0.378533</td>
          <td>0.891311</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.154234</td>
          <td>0.546673</td>
          <td>0.193160</td>
          <td>0.368642</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.388493</td>
          <td>0.909145</td>
          <td>0.378533</td>
          <td>0.891311</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.154234</td>
          <td>0.546673</td>
          <td>0.193160</td>
          <td>0.368642</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">walking_speed_mps</th>
          <th>error</th>
          <td>0.900869</td>
          <td>0.655491</td>
          <td>0.215855</td>
          <td>0.687740</td>
        </tr>
        <tr>
          <th>rel_error</th>
          <td>0.776083</td>
          <td>0.409569</td>
          <td>0.092550</td>
          <td>0.328217</td>
        </tr>
        <tr>
          <th>abs_error</th>
          <td>0.900869</td>
          <td>0.655491</td>
          <td>0.215855</td>
          <td>0.687740</td>
        </tr>
        <tr>
          <th>abs_rel_error</th>
          <td>0.776083</td>
          <td>0.409569</td>
          <td>0.092550</td>
          <td>0.328217</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 323-350

Aggregate Results
-----------------
Finally, the estimated DMO measures and their errors can be aggregated over all WBs (approach 2)
or all days (approach 1).
For this purpose, different aggregation functions can be applied to the error metrics, ranging from simple, built-in
aggregations like the mean or standard deviation to more complex functions like the limits of agreement or
5th and 95th percentiles.
This can be done using the :func:`~mobgap.utils.df_operations.apply_aggregations` function.
It operates similarly to the :func:`~mobgap.utils.df_operations.apply_transformations` function used above by taking
the error metrics dataframe and a list of aggregations as input.
In contrast to the transformations, an aggregation performed over a subset of dataframe columns
is expected to return a single value or a tuple of values stored in one cell of the resulting dataframe.
There are two ways to define aggregations:

1. As a tuple in the format ``(<identifier>, <aggregation>)``.
   In this case, the operation is performed based on exactly one column from the input df.
   Therefore, ``<identifier>`` can either be a string representing the name of the column to evaluate (for data with
   single-level columns), or a tuple of strings uniquely identifying the column to evaluate in case of multi-index
   columns.
   In our example, the identifier is a tuple ``(<metric>, <origin>)``, where ``<metric>`` is the metric column to
   evaluate, ``<origin>`` is the specific column from which data should be utilized (here, it would be either
   ``detected``, ``reference``, or one of the error columns).

   ``<aggregation>`` is the function or the list of functions to apply.
   The output dataframe will have a multilevel column with ``metric`` as the first level and ``origin`` as the second
   level.
   A valid aggregations list for all of our DMOs would consequently look like this:

.. GENERATED FROM PYTHON SOURCE LINES 350-366

.. code-block:: default

    metrics = [
        "cadence_spm",
        "duration_s",
        "n_steps",
        "n_turns",
        "stride_duration_s",
        "stride_length_m",
        "walking_speed_mps",
    ]
    aggregations_simple = [
        ((m, o), ["mean", "std"])
        for m in metrics
        for o in ["detected", "reference", "error"]
    ]
    pprint(aggregations_simple)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [(('cadence_spm', 'detected'), ['mean', 'std']),
     (('cadence_spm', 'reference'), ['mean', 'std']),
     (('cadence_spm', 'error'), ['mean', 'std']),
     (('duration_s', 'detected'), ['mean', 'std']),
     (('duration_s', 'reference'), ['mean', 'std']),
     (('duration_s', 'error'), ['mean', 'std']),
     (('n_steps', 'detected'), ['mean', 'std']),
     (('n_steps', 'reference'), ['mean', 'std']),
     (('n_steps', 'error'), ['mean', 'std']),
     (('n_turns', 'detected'), ['mean', 'std']),
     (('n_turns', 'reference'), ['mean', 'std']),
     (('n_turns', 'error'), ['mean', 'std']),
     (('stride_duration_s', 'detected'), ['mean', 'std']),
     (('stride_duration_s', 'reference'), ['mean', 'std']),
     (('stride_duration_s', 'error'), ['mean', 'std']),
     (('stride_length_m', 'detected'), ['mean', 'std']),
     (('stride_length_m', 'reference'), ['mean', 'std']),
     (('stride_length_m', 'error'), ['mean', 'std']),
     (('walking_speed_mps', 'detected'), ['mean', 'std']),
     (('walking_speed_mps', 'reference'), ['mean', 'std']),
     (('walking_speed_mps', 'error'), ['mean', 'std'])]




.. GENERATED FROM PYTHON SOURCE LINES 367-375

2. As a named tuple of Type `CustomOperation` taking three values: `identifier`, `function`, and `column_name`.
   `identifier` is a valid loc identifier selecting one or more columns from the dataframe, `function` is the (custom)
   aggregation function or list of functions to apply, and `column_name` is the name of the resulting column in the
   output dataframe (single-level column if `column_name` is a string, multi-level column if `column_name` is a
   tuple).
   This allows for more complex aggregations that require multiple columns as input, for example, the intraclass
   correlation coefficient (ICC) for the DMOs (see below).
   A valid aggregation list for calculating the ICC of all DMOs would look like this:

.. GENERATED FROM PYTHON SOURCE LINES 376-385

.. code-block:: default

    from mobgap.pipeline.evaluation import CustomErrorAggregations as A
    from mobgap.pipeline.evaluation import get_default_error_aggregations
    from mobgap.utils.df_operations import CustomOperation

    aggregations_custom = [
        CustomOperation(identifier=m, function=A.icc, column_name=(m, "all"))
        for m in metrics
    ]
    pprint(aggregations_custom)




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [CustomOperation(identifier='cadence_spm', function=<function icc at 0x7fdc82f9b7f0>, column_name=('cadence_spm', 'all')),
     CustomOperation(identifier='duration_s', function=<function icc at 0x7fdc82f9b7f0>, column_name=('duration_s', 'all')),
     CustomOperation(identifier='n_steps', function=<function icc at 0x7fdc82f9b7f0>, column_name=('n_steps', 'all')),
     CustomOperation(identifier='n_turns', function=<function icc at 0x7fdc82f9b7f0>, column_name=('n_turns', 'all')),
     CustomOperation(identifier='stride_duration_s', function=<function icc at 0x7fdc82f9b7f0>, column_name=('stride_duration_s', 'all')),
     CustomOperation(identifier='stride_length_m', function=<function icc at 0x7fdc82f9b7f0>, column_name=('stride_length_m', 'all')),
     CustomOperation(identifier='walking_speed_mps', function=<function icc at 0x7fdc82f9b7f0>, column_name=('walking_speed_mps', 'all'))]




.. GENERATED FROM PYTHON SOURCE LINES 386-391

In this case, the ICC function gets the entire "sub-dataframe" obtained by the selection
``wb_matches_with_errors.loc[:, m]`` as shown below for ``stride_duration_s`` as example, and could then perform
any required calculations.
The selection could theoretically be any valid loc selection.
So you could even select values across multiple DMOs.

.. GENERATED FROM PYTHON SOURCE LINES 391-393

.. code-block:: default

    sub_df = wb_matches_with_errors.loc[:, "stride_duration_s"]








.. GENERATED FROM PYTHON SOURCE LINES 394-395

The ICC function just takes the ``detected`` and ``reference`` columns and calculates the ICC.

.. GENERATED FROM PYTHON SOURCE LINES 395-397

.. code-block:: default

    A.icc(sub_df)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    (0.12564828430955782, array([-0.77,  0.9 ]))



.. GENERATED FROM PYTHON SOURCE LINES 398-403

Within one aggregation list, both types of aggregations can be combined
as long as the resulting output dataframes can be concatenated, i.e. have the same number of column levels.
Then, the :func:`~mobgap.utils.df_operations.apply_aggregations` function can be called.
This returns a pandas Series with the aggregated values for each metric and origin.
For better readability, we sort and format the resulting dataframe.

.. GENERATED FROM PYTHON SOURCE LINES 403-415

.. code-block:: default

    from mobgap.utils.df_operations import apply_aggregations

    aggregations = aggregations_simple + aggregations_custom
    agg_results = (
        apply_aggregations(wb_matches_with_errors, aggregations)
        .rename_axis(index=["aggregation", "metric", "origin"])
        .reorder_levels(["metric", "origin", "aggregation"])
        .sort_index(level=0)
        .to_frame("values")
    )
    agg_results






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th>values</th>
        </tr>
        <tr>
          <th>metric</th>
          <th>origin</th>
          <th>aggregation</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="7" valign="top">cadence_spm</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.9958867337397626, [0.96, 1.0])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>92.283646</td>
        </tr>
        <tr>
          <th>std</th>
          <td>6.128986</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.496341</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.305876</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>91.787305</td>
        </tr>
        <tr>
          <th>std</th>
          <td>6.267057</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">duration_s</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.9935066257807105, [0.94, 1.0])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>8.630692</td>
        </tr>
        <tr>
          <th>std</th>
          <td>3.980795</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.407152</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.214453</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>8.22354</td>
        </tr>
        <tr>
          <th>std</th>
          <td>3.86233</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">n_steps</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.9927710843373494, [0.93, 1.0])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>11.5</td>
        </tr>
        <tr>
          <th>std</th>
          <td>4.041452</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.25</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.5</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>11.25</td>
        </tr>
        <tr>
          <th>std</th>
          <td>4.272002</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">n_turns</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.8064516129032259, [-0.03, 0.99])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>1.25</td>
        </tr>
        <tr>
          <th>std</th>
          <td>1.258306</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.5</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.57735</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>0.75</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.957427</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">stride_duration_s</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.12564828430955782, [-0.77, 0.9])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>2.949573</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.525579</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.822963</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.150423</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>2.12661</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.426573</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">stride_length_m</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.1021600349369979, [-0.78, 0.9])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>2.781723</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.422111</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.64187</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.298442</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>2.139852</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.400293</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">walking_speed_mps</th>
          <th>all</th>
          <th>icc</th>
          <td>(0.20380942118620765, [-0.74, 0.92])</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">detected</th>
          <th>mean</th>
          <td>2.412216</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.317996</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">error</th>
          <th>mean</th>
          <td>0.614989</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.2875</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">reference</th>
          <th>mean</th>
          <td>1.797227</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.522486</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 416-418

If you simply want to apply a standard set of aggregations to the error metrics, you can use the
:func:`~mobgap.pipeline.evaluation.get_default_error_aggregations` function, resulting in the following list:

.. GENERATED FROM PYTHON SOURCE LINES 418-422

.. code-block:: default


    aggregations_default = get_default_error_aggregations()
    pprint(aggregations_default)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [(('cadence_spm', 'detected'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('cadence_spm', 'reference'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('cadence_spm', 'abs_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('cadence_spm', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('duration_s', 'detected'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('duration_s', 'reference'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('duration_s', 'abs_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('duration_s', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_steps', 'detected'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_steps', 'reference'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_steps', 'abs_error'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_steps', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_strides', 'detected'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_strides', 'reference'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_strides', 'abs_error'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_strides', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_turns', 'detected'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_turns', 'reference'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_turns', 'abs_error'), ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('n_turns', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_duration_s', 'detected'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_duration_s', 'reference'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_duration_s', 'abs_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_duration_s', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_length_m', 'detected'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_length_m', 'reference'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_length_m', 'abs_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('stride_length_m', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('walking_speed_mps', 'detected'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('walking_speed_mps', 'reference'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('walking_speed_mps', 'abs_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('walking_speed_mps', 'abs_rel_error'),
      ['mean', <function quantiles at 0x7fdc82f9b880>]),
     (('cadence_spm', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('cadence_spm', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('duration_s', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('duration_s', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_steps', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_steps', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_strides', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_strides', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_turns', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('n_turns', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('stride_duration_s', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('stride_duration_s', 'rel_error'),
      ['mean', <function loa at 0x7fdc82f9b910>]),
     (('stride_length_m', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('stride_length_m', 'rel_error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('walking_speed_mps', 'error'), ['mean', <function loa at 0x7fdc82f9b910>]),
     (('walking_speed_mps', 'rel_error'),
      ['mean', <function loa at 0x7fdc82f9b910>]),
     CustomOperation(identifier='cadence_spm', function=<function icc at 0x7fdc82f9b7f0>, column_name=('cadence_spm', 'all')),
     CustomOperation(identifier='duration_s', function=<function icc at 0x7fdc82f9b7f0>, column_name=('duration_s', 'all')),
     CustomOperation(identifier='n_steps', function=<function icc at 0x7fdc82f9b7f0>, column_name=('n_steps', 'all')),
     CustomOperation(identifier='n_strides', function=<function icc at 0x7fdc82f9b7f0>, column_name=('n_strides', 'all')),
     CustomOperation(identifier='n_turns', function=<function icc at 0x7fdc82f9b7f0>, column_name=('n_turns', 'all')),
     CustomOperation(identifier='stride_duration_s', function=<function icc at 0x7fdc82f9b7f0>, column_name=('stride_duration_s', 'all')),
     CustomOperation(identifier='stride_length_m', function=<function icc at 0x7fdc82f9b7f0>, column_name=('stride_length_m', 'all')),
     CustomOperation(identifier='walking_speed_mps', function=<function icc at 0x7fdc82f9b7f0>, column_name=('walking_speed_mps', 'all')),
     CustomOperation(identifier=None, function=<function n_datapoints at 0x7fdc82f9b9a0>, column_name=('all', 'all'))]




.. GENERATED FROM PYTHON SOURCE LINES 423-424

If you want to include further aggregations next to the default ones, you can also append them to this list.

.. GENERATED FROM PYTHON SOURCE LINES 424-428

.. code-block:: default

    aggregations_default_extended = aggregations_default + [
        *(((m, o), ["std"]) for m in metrics for o in ["detected", "reference"])
    ]








.. GENERATED FROM PYTHON SOURCE LINES 429-431

This list of standard aggregations can then also be passed to the
:func:`~mobgap.utils.df_operations.apply_aggregations` function.

.. GENERATED FROM PYTHON SOURCE LINES 431-440

.. code-block:: default

    default_agg_results = (
        apply_aggregations(wb_matches_with_errors, aggregations_default_extended)
        .rename_axis(index=["aggregation", "metric", "origin"])
        .reorder_levels(["metric", "origin", "aggregation"])
        .sort_index(level=0)
        .to_frame("values")
    )
    default_agg_results





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'detected')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'reference')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'abs_error')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'abs_rel_error')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'error')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:475: UserWarning: One of the transformations requires the following columns: '('n_strides', 'rel_error')'. They are not found in the DataFrame.
      warnings.warn(str(MissingDataColumnsError(key)), UserWarning, stacklevel=1)
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/utils/df_operations.py:566: UserWarning: One of the transformations requires the following columns: 'n_strides'. They are not found in the DataFrame.
      warnings.warn(str(e), UserWarning, stacklevel=1)


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th>values</th>
        </tr>
        <tr>
          <th>metric</th>
          <th>origin</th>
          <th>aggregation</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>all</th>
          <th>all</th>
          <th>n_datapoints</th>
          <td>4</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">cadence_spm</th>
          <th rowspan="2" valign="top">abs_error</th>
          <th>mean</th>
          <td>0.496341</td>
        </tr>
        <tr>
          <th>quantiles</th>
          <td>(0.29685672457991713, 0.8682829318918925)</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">abs_rel_error</th>
          <th>mean</th>
          <td>0.005508</td>
        </tr>
        <tr>
          <th>quantiles</th>
          <td>(0.003208965846203449, 0.00993913893132042)</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <th>...</th>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">walking_speed_mps</th>
          <th rowspan="3" valign="top">reference</th>
          <th>mean</th>
          <td>1.797227</td>
        </tr>
        <tr>
          <th>quantiles</th>
          <td>(1.2267375, 2.2967619999999997)</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.522486</td>
        </tr>
        <tr>
          <th rowspan="2" valign="top">rel_error</th>
          <th>loa</th>
          <td>(-0.15414805045818636, 0.9573580216047631)</td>
        </tr>
        <tr>
          <th>mean</th>
          <td>0.401605</td>
        </tr>
      </tbody>
    </table>
    <p>106 rows × 1 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 441-445

.. note::
  If you want to modify the default arguments of the aggregation functions, e.g. to change the calculated quantiles,
  you can either define custom aggregation functions or adapt the default functions as shown for the transformation
  functions above.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 4.901 seconds)

**Estimated memory usage:**  9 MB


.. _sphx_glr_download_auto_examples_pipeline__03_dmo_evaluation_on_wb_level.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _03_dmo_evaluation_on_wb_level.py <_03_dmo_evaluation_on_wb_level.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _03_dmo_evaluation_on_wb_level.ipynb <_03_dmo_evaluation_on_wb_level.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_