.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/gait_sequences/_03_gsd_evaluation.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_gait_sequences__03_gsd_evaluation.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_gait_sequences__03_gsd_evaluation.py:


.. _gsd_evaluation:

GSD Evaluation
==============

This example shows how to apply evaluation algorithms to GSD and thus how to rate the performance of a GSD algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 9-14

.. code-block:: Python


    import pandas as pd
    from mobgap.data import LabExampleDataset
    from mobgap.gait_sequences import GsdIluz








.. GENERATED FROM PYTHON SOURCE LINES 15-21

Loading some example data
-------------------------
First, we load example data and apply the GSD Iluz algorithm to it.
However, you can use any other GSD algorithm as well.
To have a reference to compare the results to, we also load the corresponding ground truth data.
These steps are explained in more detail in the :ref:`GSD Iluz example <gsd_iluz>`.

.. GENERATED FROM PYTHON SOURCE LINES 21-55

.. code-block:: Python

    from mobgap.utils.conversions import to_body_frame


    def load_data():
        lab_example_data = LabExampleDataset(reference_system="INDIP")
        single_test = lab_example_data.get_subset(
            cohort="MS", participant_id="001", test="Test11", trial="Trial1"
        )
        return single_test


    def calculate_gsd_iluz_output(single_test_data):
        """Calculate the GSD Iluz output for one sensor from the test data."""
        det_gsd = (
            GsdIluz()
            .detect(
                to_body_frame(single_test_data.data_ss),
                sampling_rate_hz=single_test_data.sampling_rate_hz,
            )
            .gs_list_
        )
        return det_gsd


    def load_reference(single_test_data):
        """Load the reference gait sequences from the test data."""
        ref_gsd = single_test_data.reference_parameters_.wb_list
        return ref_gsd


    test_data = load_data()
    detected_gsd_list = calculate_gsd_iluz_output(test_data)
    reference_gsd_list = load_reference(test_data)








.. GENERATED FROM PYTHON SOURCE LINES 56-57

that is characterized by its start and end index in samples.

.. GENERATED FROM PYTHON SOURCE LINES 57-59

.. code-block:: Python

    detected_gsd_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>900</td>
          <td>2101</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650</td>
          <td>6151</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9600</td>
          <td>10801</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11250</td>
          <td>12151</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12300</td>
          <td>14851</td>
        </tr>
        <tr>
          <th>5</th>
          <td>19950</td>
          <td>21151</td>
        </tr>
        <tr>
          <th>6</th>
          <td>21300</td>
          <td>22501</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 60-62

.. code-block:: Python

    reference_gsd_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>n_strides</th>
          <th>duration_s</th>
          <th>length_m</th>
          <th>avg_walking_speed_mps</th>
          <th>avg_cadence_spm</th>
          <th>avg_stride_length_m</th>
          <th>termination_reason</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1019</td>
          <td>1768</td>
          <td>9</td>
          <td>7.48</td>
          <td>4.468932</td>
          <td>0.847668</td>
          <td>107.795850</td>
          <td>0.942678</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4534</td>
          <td>5549</td>
          <td>11</td>
          <td>10.14</td>
          <td>2.900453</td>
          <td>0.365176</td>
          <td>93.396106</td>
          <td>0.483923</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9665</td>
          <td>10569</td>
          <td>9</td>
          <td>9.03</td>
          <td>2.140232</td>
          <td>0.294058</td>
          <td>75.981133</td>
          <td>0.506458</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12337</td>
          <td>14633</td>
          <td>28</td>
          <td>22.95</td>
          <td>11.201110</td>
          <td>0.634425</td>
          <td>92.337768</td>
          <td>0.803933</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>4</th>
          <td>20151</td>
          <td>20982</td>
          <td>11</td>
          <td>8.30</td>
          <td>2.390709</td>
          <td>0.371746</td>
          <td>87.915774</td>
          <td>0.507484</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21378</td>
          <td>22129</td>
          <td>9</td>
          <td>7.50</td>
          <td>2.517558</td>
          <td>0.492965</td>
          <td>95.365740</td>
          <td>0.599360</td>
          <td>Pause</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 63-94

Validation of algorithm output against a reference
--------------------------------------------------
Let's quantify how the algorithm output compares to the reference labels.
To gain a detailed insight into the performance of the algorithm, we can look into the individual matches between the
detected and reference gait sequences.
Note, that there are two different ways to approach this:

1. We can calculate the for each sample, whether it is correctly detected as gait or not.
2. We can check on the level of gait sequences, whether a detected gait sequence matches with a reference gait sequence (by a certain overlap threshold).

In mobgap, we provide functions to calculate both types of performance metrics. Let's start with the first one.

Sample-wise performance evaluation
----------------------------------
To do this, we use the :func:`~gaitlink.gait_sequences.evaluation.categorize_intervals_per_sample` function
to identify overlapping regions between the detected gait sequences and the reference gait sequences.
These overlapping regions can then be converted into sample-wise classifications of true positives, false positives, and false negatives.

As function arguments, besides the mandatory detected and reference gait sequences,
the total number of samples in the recording can be specified as optional parameter.
If provided, the intervals where no gait sequences are present in the reference and the detected list are also reported.
Later on, we can then use these categorized intervals to calculate a set of higher-level performance metrics.

As result, a DataFrame containing `start` and `end`  index of the resulting categorized intervals together with
a `match_type` column that contains the type of match for each interval, i.e. `tp` for true positive, `fp` for false
positive, and `fn` for false negative.
These intervals can not be interpreted as gait sequences, but are rather subsequences of the detected gait sequences
categorizing correctly detected samples (`tp`), falsely detected samples (`fp`), samples from the reference gsd
list that were not detected (`fn`), and (optionally) samples where no gait sequences are present in both the reference and detected gait sequences (`tn`).
Note that the tn intervals are not explicitly calculated, but are inferred from the total length of the recording
(if provided) and from the other intervals, as everything between them is considered as true negative.

.. GENERATED FROM PYTHON SOURCE LINES 94-104

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import categorize_intervals_per_sample

    categorized_intervals = categorize_intervals_per_sample(
        gsd_list_detected=detected_gsd_list,
        gsd_list_reference=reference_gsd_list,
        n_overall_samples=len(test_data.data_ss),
    )

    categorized_intervals






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>match_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>900</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>1</th>
          <td>900</td>
          <td>1019</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1019</td>
          <td>1768</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>3</th>
          <td>1768</td>
          <td>2101</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>4</th>
          <td>2101</td>
          <td>4534</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>5</th>
          <td>4534</td>
          <td>4650</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>6</th>
          <td>4650</td>
          <td>5549</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>7</th>
          <td>5549</td>
          <td>6151</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>8</th>
          <td>6151</td>
          <td>9600</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>9</th>
          <td>9600</td>
          <td>9665</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>10</th>
          <td>9665</td>
          <td>10569</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>11</th>
          <td>10569</td>
          <td>10801</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>12</th>
          <td>10801</td>
          <td>11250</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>13</th>
          <td>11250</td>
          <td>12151</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>14</th>
          <td>12151</td>
          <td>12300</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>15</th>
          <td>12300</td>
          <td>12337</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>16</th>
          <td>12337</td>
          <td>14633</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>17</th>
          <td>14633</td>
          <td>14851</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>18</th>
          <td>14851</td>
          <td>19950</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>19</th>
          <td>19950</td>
          <td>20151</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>20</th>
          <td>20151</td>
          <td>20982</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>21</th>
          <td>20982</td>
          <td>21151</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>22</th>
          <td>21151</td>
          <td>21300</td>
          <td>tn</td>
        </tr>
        <tr>
          <th>23</th>
          <td>21300</td>
          <td>21378</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>24</th>
          <td>21378</td>
          <td>22129</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>25</th>
          <td>22129</td>
          <td>22501</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>26</th>
          <td>22501</td>
          <td>22727</td>
          <td>tn</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 105-117

Based on the individually categorized tp, fp, fn, and tn intervals, common performance metrics,
e.g., F1 score, precision, or recall can be calculated.
For this purpose, the :func:`~gaitlink.gait_sequences.evaluation.calculate_matched_gsd_performance_metrics` function can be used.
It calculates the metrics based on the "matched" gsd intervals, i.e., the categorized interval list where every entry
has a match type (tp, fp, fn, tn) assigned.
Therefore, the function requires to call the :func:`~gaitlink.gait_sequences.evaluation.categorize_intervals_per_sample` function first.
The categorized intervals can then be passed as an argument
to :func:`~gaitlink.gait_sequences.evaluation.calculate_matched_gsd_performance_metrics`.
It returns a dictionary containing the metrics for the specified categorized intervals DataFrame.
Here, the total number of samples in every match type, precision, recall, F1 score, are always calculated.
Depending on whether true negatives are present in the categorized intervals,
specificity, negative predictive value, and accuracy will additionally be reported.

.. GENERATED FROM PYTHON SOURCE LINES 117-127

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import (
        calculate_matched_gsd_performance_metrics,
    )

    matched_metrics_dict = calculate_matched_gsd_performance_metrics(
        categorized_intervals
    )

    matched_metrics_dict





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    {'tp_samples': 6436, 'fp_samples': 3339, 'fn_samples': 117, 'precision': 0.6584143222506393, 'recall': 0.9821455821761026, 'f1_score': 0.7883390494855462, 'tn_samples': 12862, 'specificity': 0.7939016110116659, 'accuracy': 0.8481146172101609, 'npv': 0.9909854380152554}



.. GENERATED FROM PYTHON SOURCE LINES 128-135

Furthermore, there is a range of high-level performance metrics that are simply calculated based
on the overall amount of gait sequences/gait detected in reference and detected data.
Thus, they can be inferred from the reference and detected gait sequences directly without any intermediate steps
using the :func:`~gaitlink.gait_sequences.evaluation.calculate_unmatched_gsd_performance_metrics` function.
As some of the unmatched metrics are reported in seconds, the function requires the sampling frequency of the recorded
data as an additional argument.
It returns a dictionary containing all metrics for the specified detected and reference gait sequences.

.. GENERATED FROM PYTHON SOURCE LINES 135-146

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import (
        calculate_unmatched_gsd_performance_metrics,
    )

    unmatched_metrics_dict = calculate_unmatched_gsd_performance_metrics(
        gsd_list_detected=detected_gsd_list,
        gsd_list_reference=reference_gsd_list,
        sampling_rate_hz=test_data.sampling_rate_hz,
    )

    unmatched_metrics_dict




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    {'reference_gs_duration_s': 65.52, 'detected_gs_duration_s': 97.64, 'gs_duration_error_s': 32.120000000000005, 'gs_relative_duration_error': 0.4902319902319903, 'gs_absolute_duration_error_s': 32.120000000000005, 'gs_absolute_relative_duration_error': 0.4902319902319903, 'gs_absolute_relative_duration_error_log': 0.3989318059799454, 'detected_num_gs': 7, 'reference_num_gs': 6, 'num_gs_error': 1, 'num_gs_relative_error': 0.16666666666666666, 'num_gs_absolute_error': 1, 'num_gs_absolute_relative_error': 0.16666666666666666, 'num_gs_absolute_relative_error_log': 0.15415067982725836}



.. GENERATED FROM PYTHON SOURCE LINES 147-168

Direct Gait Sequence Matching
-----------------------------
Apart from the performance evaluation methods mentioned above, it might be useful in some cases to identify
how many and which detected gait sequences reliably match with the ground truth.

This is primarily useful, when further parameters are associated with each gaits sequence, e.g., the gait speed.
In this case, matching gait sequences that cover the same gait regions allows proper comparison of these parameters.
For more information on this, see the example on the overall parameter evaluation on Walking-Bout level (TODO).

For this purpose, the :func:`~gaitlink.gait_sequences.evaluation.categorize_intervals` can be used.
It returns all intervals of the detected gait sequences that overlap with the reference gait sequences by at least a
given amount.
The index of the result dataframe indicated the index of the detected gait sequence.
We can see that with an overlap threshold of 0.7 (70%), three of the six detected gait sequences are considered as
matches with the reference gait sequences for our example recording.
Note, that this threshold is enforced in both directions, i.e., the detected gait sequence must overlap with the
reference gait sequence by at least 70% and vice versa.
This means that only 1 to 1 matches are possible.
If multiple detected gait sequences overlap with the same reference gait sequence, only the one with the highest
overlap is considered as a match.
If one gait sequence is covered by multiple smaller once, possibly none of them is considered as a match.

.. GENERATED FROM PYTHON SOURCE LINES 168-178

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import categorize_intervals

    matches = categorize_intervals(
        gsd_list_detected=detected_gsd_list,
        gsd_list_reference=reference_gsd_list,
        overlap_threshold=0.7,
    )

    matches






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>gs_id_detected</th>
          <th>gs_id_reference</th>
          <th>match_type</th>
        </tr>
        <tr>
          <th>match_id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
          <td>2</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>3</th>
          <td>3</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>4</th>
          <td>4</td>
          <td>3</td>
          <td>tp</td>
        </tr>
        <tr>
          <th>5</th>
          <td>5</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>6</th>
          <td>6</td>
          <td>NaN</td>
          <td>fp</td>
        </tr>
        <tr>
          <th>7</th>
          <td>NaN</td>
          <td>0</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>8</th>
          <td>NaN</td>
          <td>1</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>9</th>
          <td>NaN</td>
          <td>4</td>
          <td>fn</td>
        </tr>
        <tr>
          <th>10</th>
          <td>NaN</td>
          <td>5</td>
          <td>fn</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 179-188

Running a full evaluation pipeline
----------------------------------
Instead of manually evaluating and investigating the performance of a GSD algorithm on a single piece of data, we
often want to run a full evaluation on an entire dataset.
This can be done using the :class:`~mobgap.gait_sequences.evaluation.GsdEvaluationPipeline` class and some ``tpcp``
functions.

But let's start with selecting some data.
We want to use all the simulated real-world walking data from the INDIP reference system (Test11).

.. GENERATED FROM PYTHON SOURCE LINES 188-193

.. code-block:: Python

    simulated_real_world_walking = LabExampleDataset(
        reference_system="INDIP"
    ).get_subset(test="Test11")

    simulated_real_world_walking





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <h4 style="margin-bottom: 0.1em;">LabExampleDataset [3 groups/rows]</h3>
    <div style="margin-top: 0em">
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table style="margin-left: 3em;">
      <thead>
        <tr style="text-align: right;">
          <th style="text-align: center;"></th>
          <th style="text-align: center;">cohort</th>
          <th style="text-align: center;">participant_id</th>
          <th style="text-align: center;">time_measure</th>
          <th style="text-align: center;">test</th>
          <th style="text-align: center;">trial</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th style="text-align: center;">0</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">HA</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">001</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
        <tr>
          <th style="text-align: center;">1</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">HA</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">002</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
        <tr>
          <th style="text-align: center;">2</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">MS</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">001</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 194-197

Now we can use the :class:`~mobgap.gait_sequences.evaluation.GsdEvaluationPipeline` class to directly run a Gsd
algorithm on a datapoint.
The pipeline takes care of extracting the required data.

.. GENERATED FROM PYTHON SOURCE LINES 197-203

.. code-block:: Python

    from mobgap.gait_sequences.pipeline import GsdEmulationPipeline

    pipeline = GsdEmulationPipeline(GsdIluz())

    pipeline.safe_run(simulated_real_world_walking[0]).gs_list_






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7800</td>
          <td>9001</td>
        </tr>
        <tr>
          <th>3</th>
          <td>9300</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>4</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>5</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 204-211

Note, that this did just "run" the pipeline on a single datapoint.
If we want to run it on all datapoints and evaluate the performance of the algorithm, we can use the
:func:`~tpcp.validate.validate` function.

For this we need to provide a score function that runs and evaluates the pipeline on a single datapoint.
We provide a default score function for GSD that calculates all the metrics shown above per datapoint and combined
accross all gait sequences (i.e. all gait sequences across all datapoints are pooled before metrics are calculated).

.. GENERATED FROM PYTHON SOURCE LINES 211-221

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import gsd_score
    from tpcp.validate import validate

    evaluation_results = pd.DataFrame(
        validate(pipeline, simulated_real_world_walking, scoring=gsd_score)
    )

    evaluation_results.drop(
        ["single__raw__reference", "single__raw__detected"], axis=1
    ).T




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Datapoints:   0%|          | 0/3 [00:00<?, ?it/s]
    Datapoints:  33%|███▎      | 1/3 [00:00<00:00,  5.15it/s]
    Datapoints:  67%|██████▋   | 2/3 [00:00<00:00,  5.19it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  5.05it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  5.08it/s]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>debug__score_time</th>
          <td>0.711474</td>
        </tr>
        <tr>
          <th>data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
        </tr>
        <tr>
          <th>single__reference_gs_duration_s</th>
          <td>[40.44, 40.82, 65.52]</td>
        </tr>
        <tr>
          <th>single__detected_gs_duration_s</th>
          <td>[48.12, 42.08, 97.64]</td>
        </tr>
        <tr>
          <th>single__gs_duration_error_s</th>
          <td>[7.68, 1.259999999999998, 32.120000000000005]</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
        </tr>
        <tr>
          <th>agg__combined__f1_score</th>
          <td>0.699236</td>
        </tr>
        <tr>
          <th>agg__combined__tn_samples</th>
          <td>30755</td>
        </tr>
        <tr>
          <th>agg__combined__specificity</th>
          <td>0.812507</td>
        </tr>
        <tr>
          <th>agg__combined__accuracy</th>
          <td>0.80828</td>
        </tr>
        <tr>
          <th>agg__combined__npv</th>
          <td>0.9118</td>
        </tr>
      </tbody>
    </table>
    <p>76 rows × 1 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 222-224

In addition to the metrics, the method also returns the raw reference and detected gait sequences.
These can be used for further custom analysis.

.. GENERATED FROM PYTHON SOURCE LINES 224-227

.. code-block:: Python


    evaluation_results["single__raw__reference"][0]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>wb_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="9" valign="top">HA</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>632</td>
          <td>988</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2864</td>
          <td>3325</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3853</td>
          <td>5085</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7641</td>
          <td>8621</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9451</td>
          <td>9932</td>
        </tr>
        <tr>
          <th>5</th>
          <td>11989</td>
          <td>12517</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">002</th>
          <th rowspan="3" valign="top">TimeMeasure1</th>
          <th rowspan="3" valign="top">Test11</th>
          <th rowspan="3" valign="top">Trial1</th>
          <th>0</th>
          <td>485</td>
          <td>1131</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1746</td>
          <td>3554</td>
        </tr>
        <tr>
          <th>2</th>
          <td>6083</td>
          <td>7708</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">MS</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>1019</td>
          <td>1768</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4534</td>
          <td>5549</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9665</td>
          <td>10569</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12337</td>
          <td>14633</td>
        </tr>
        <tr>
          <th>4</th>
          <td>20151</td>
          <td>20982</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21378</td>
          <td>22129</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 228-230

.. code-block:: Python

    evaluation_results["single__raw__detected"][0]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="10" valign="top">HA</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7800</td>
          <td>9001</td>
        </tr>
        <tr>
          <th>3</th>
          <td>9300</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>4</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>5</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">002</th>
          <th rowspan="4" valign="top">TimeMeasure1</th>
          <th rowspan="4" valign="top">Test11</th>
          <th rowspan="4" valign="top">Trial1</th>
          <th>0</th>
          <td>450</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700</td>
          <td>3301</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5700</td>
          <td>7951</td>
        </tr>
        <tr>
          <th>3</th>
          <td>15000</td>
          <td>15601</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">MS</th>
          <th rowspan="7" valign="top">001</th>
          <th rowspan="7" valign="top">TimeMeasure1</th>
          <th rowspan="7" valign="top">Test11</th>
          <th rowspan="7" valign="top">Trial1</th>
          <th>0</th>
          <td>900</td>
          <td>2101</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650</td>
          <td>6151</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9600</td>
          <td>10801</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11250</td>
          <td>12151</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12300</td>
          <td>14851</td>
        </tr>
        <tr>
          <th>5</th>
          <td>19950</td>
          <td>21151</td>
        </tr>
        <tr>
          <th>6</th>
          <td>21300</td>
          <td>22501</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 231-251

If you want to calculate additional metrics, you can either create a custom score function or subclass the pipeline
and overwrite the score function.

Parameter Optimization
----------------------
Simply applying an algorithm to the data for evaluation is often not enough.
In case, of machine learning algorithms or algorithms with tunable parameters, we might want to optimize these
parameters to get the best possible performance.
To avoid overfitting, we can use cross-validation to evaluate the performance of the algorithm on multiple splits of
the data.

Below we show that procedure by using a simple grid search to optimize the window length of the GSD Iluz algorithm
and evaluate this approach within a 3-fold cross-validation.
Per-fold we select the window length leading to the highest precision on the "train set" and evaluate the performance
on the "test set".

Note, that on a real world dataset, you would likely need to perform a group-wise stratified cross-validation to
avoid data leakage between multiple trials from the same participant and ensure equal distribution of patient cohorts
across the folds.
See the detailed ``tpcp`` examples on these topics.

.. GENERATED FROM PYTHON SOURCE LINES 251-282

.. code-block:: Python

    from sklearn.model_selection import ParameterGrid
    from tpcp.optimize import GridSearch
    from tpcp.validate import cross_validate

    para_grid = ParameterGrid({"algo__window_length_s": [2, 3, 4]})

    cross_validate_results = pd.DataFrame(
        cross_validate(
            GridSearch(
                GsdEmulationPipeline(GsdIluz()),
                para_grid,
                return_optimized="precision",
                scoring=gsd_score,
            ),
            simulated_real_world_walking,
            scoring=gsd_score,
            cv=3,
            return_train_score=True,
        )
    )

    cross_validate_results.drop(
        [
            "test__single__raw__reference",
            "test__single__raw__detected",
            "train__single__raw__reference",
            "train__single__raw__detected",
        ],
        axis=1,
    ).T





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]

    Parameter Combinations:   0%|          | 0/3 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  4.93it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  4.89it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  4.89it/s]


    Parameter Combinations:  33%|███▎      | 1/3 [00:00<00:01,  1.97it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.37it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.09it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.13it/s]


    Parameter Combinations:  67%|██████▋   | 2/3 [00:00<00:00,  2.02it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.38it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.12it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.15it/s]


    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.04it/s]
    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.03it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.19it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.18it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]

    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.39it/s]

    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.13it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.16it/s]

    CV Folds:  33%|███▎      | 1/3 [00:02<00:04,  2.25s/it]

    Parameter Combinations:   0%|          | 0/3 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.17it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.01it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.03it/s]


    Parameter Combinations:  33%|███▎      | 1/3 [00:00<00:00,  2.02it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.19it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.06it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.07it/s]


    Parameter Combinations:  67%|██████▋   | 2/3 [00:00<00:00,  2.03it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.18it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.05it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.06it/s]


    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.03it/s]
    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.03it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.44it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.43it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]

    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.17it/s]

    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.04it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.05it/s]

    CV Folds:  67%|██████▋   | 2/3 [00:04<00:02,  2.24s/it]

    Parameter Combinations:   0%|          | 0/3 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.22it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.33it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.31it/s]


    Parameter Combinations:  33%|███▎      | 1/3 [00:00<00:00,  2.11it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.16it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.28it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.26it/s]


    Parameter Combinations:  67%|██████▋   | 2/3 [00:00<00:00,  2.10it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]


    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.17it/s]


    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.30it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.28it/s]


    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.10it/s]
    Parameter Combinations: 100%|██████████| 3/3 [00:01<00:00,  2.10it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.89it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.88it/s]


    Datapoints:   0%|          | 0/2 [00:00<?, ?it/s]

    Datapoints:  50%|█████     | 1/2 [00:00<00:00,  5.16it/s]

    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.30it/s]
    Datapoints: 100%|██████████| 2/2 [00:00<00:00,  5.27it/s]

    CV Folds: 100%|██████████| 3/3 [00:06<00:00,  2.22s/it]
    CV Folds: 100%|██████████| 3/3 [00:06<00:00,  2.23s/it]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>debug__score_time</th>
          <td>0.267713</td>
          <td>0.257381</td>
          <td>0.279818</td>
        </tr>
        <tr>
          <th>debug__optimize_time</th>
          <td>1.486507</td>
          <td>1.483054</td>
          <td>1.433278</td>
        </tr>
        <tr>
          <th>train__data_labels</th>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
        </tr>
        <tr>
          <th>test__data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(MS, 001, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>test__single__reference_gs_duration_s</th>
          <td>[40.44]</td>
          <td>[40.82]</td>
          <td>[65.52]</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>train__agg__combined__f1_score</th>
          <td>0.756254</td>
          <td>0.74228</td>
          <td>0.680912</td>
        </tr>
        <tr>
          <th>train__agg__combined__tn_samples</th>
          <td>23439</td>
          <td>20410</td>
          <td>18330</td>
        </tr>
        <tr>
          <th>train__agg__combined__specificity</th>
          <td>0.833683</td>
          <td>0.786876</td>
          <td>0.846612</td>
        </tr>
        <tr>
          <th>train__agg__combined__accuracy</th>
          <td>0.845118</td>
          <td>0.819097</td>
          <td>0.813975</td>
        </tr>
        <tr>
          <th>train__agg__combined__npv</th>
          <td>0.946457</td>
          <td>0.949656</td>
          <td>0.892014</td>
        </tr>
      </tbody>
    </table>
    <p>152 rows × 3 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 283-286

In general, it is a good idea to use ``cross_validation`` also for algorithms that do not have tunable parameters.
This way you can ensure that the performance of the algorithm is stable across different splits of the data, and it
allows the direct comparison between tunable and non-tunable algorithms.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 10.678 seconds)

**Estimated memory usage:**  81 MB


.. _sphx_glr_download_auto_examples_gait_sequences__03_gsd_evaluation.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _03_gsd_evaluation.ipynb <_03_gsd_evaluation.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _03_gsd_evaluation.py <_03_gsd_evaluation.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: _03_gsd_evaluation.zip <_03_gsd_evaluation.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_