.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/gait_sequences/_04_gsd_evaluation_challenges.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py:


GSD Evaluation Challenges
-------------------------
The :ref:`gsd_evaluation` example demonstrates how to evaluate the performance of a GSD algorithm on a single datapoint
 and explains the individual performance metrics that are calculated.

With that you could set up a custom evaluation pipeline to run and then score the output of a GSD algorithm
multiple datapoints and then aggregate the results.
To make this process easier, we set up opinionated evaluation challenges that can be used to quickly perform the same
evaluation with multiple algorithms and datasets.

Below, we will show how to use them on the example dataset.

.. GENERATED FROM PYTHON SOURCE LINES 16-20

Dataset
-------
To use the challenges, we need to dataset with reference information in the expected format.
We will use the :class:`~mobgap.data.LabExampleDataset` for this purpose.

.. GENERATED FROM PYTHON SOURCE LINES 20-26

.. code-block:: default

    from mobgap.data import LabExampleDataset

    long_test = LabExampleDataset(reference_system="INDIP").get_subset(
        test="Test11"
    )








.. GENERATED FROM PYTHON SOURCE LINES 27-30

Algorithm
---------
Next we need to create an instance of a valid GSD algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 30-34

.. code-block:: default

    from mobgap.gait_sequences import GsdIluz

    algo = GsdIluz()








.. GENERATED FROM PYTHON SOURCE LINES 35-38

This algorithm needs to be wrapped in a :class:`~mobgap.gait_sequences.pipeline.GsdEmulationPipeline` to be used in
the challenges.
This pipeline takes care of extracting the correct data from the dataset and running the algorithm on it.

.. GENERATED FROM PYTHON SOURCE LINES 38-42

.. code-block:: default

    from mobgap.gait_sequences.pipeline import GsdEmulationPipeline

    pipe = GsdEmulationPipeline(algo)








.. GENERATED FROM PYTHON SOURCE LINES 43-44

Let's demonstrate that quickly on a single datapoint.

.. GENERATED FROM PYTHON SOURCE LINES 44-47

.. code-block:: default

    pipe_with_results = pipe.clone().run(long_test[0])
    pipe_with_results.gs_list_






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700</td>
          <td>4201</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7800</td>
          <td>8851</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9450</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>5</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>6</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 48-65

Evaluation Challenge
--------------------
This pipeline can now be used as part of an evaluation challenge.
An evaluation challenge takes care of two things:

- Running the pipeline on multiple datapoints
- Scoring the results per datapoint and then aggregating the results

We provide two challenges:

- :class:`~mobgap.gait_sequences.evaluation.GsdEvaluation`: This challenge simply runs the pipeline on all datapoints and then scores the results.
- :class:`~mobgap.gait_sequences.evaluation.GsdEvaluationCV`: This challenge runs a cross-validation on the dataset and then scores the results per fold.

Before we run the entire pipeline, let's look at the scoring.
We provide a default scoring function that calculates all the relevant performance metrics.

Let's look at the code of it first.

.. GENERATED FROM PYTHON SOURCE LINES 65-71

.. code-block:: default

    from inspect import getsource

    from mobgap.gait_sequences.evaluation import gsd_evaluation_scorer

    print(getsource(gsd_evaluation_scorer))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    def gsd_evaluation_scorer(pipeline: GsdEmulationPipeline, datapoint: BaseGaitDatasetWithReference) -> dict:
        """Evaluate the performance of a GSD algorithm on a single datapoint.

        This function is used to evaluate the performance of a GSD algorithm on a single datapoint.
        It calculates the performance metrics based on the detected gait sequences and the reference gait sequences.

        This is the default scoring functions for the GSD evaluation pipelines (``GsdEvaluation`` and ``GsdEvaluationCV``).

        Parameters
        ----------
        pipeline
            An instance of GSD emulation pipeline that wraps the algorithm that should be evaluated.
        datapoint
            The datapoint to be evaluated.

        Returns
        -------
        dict
            A dictionary containing the performance metrics.
            Note, that some results are wrapped in a ``NoAgg`` object or other aggregators.
            The results of this function are not expected to be parsed manually, but rather the function is expected to be
            used in the context of the :func:`~tpcp.validate.validate`/:func:`~tpcp.validate.cross_validate` functions or
            similar as scorer.
            This functions will aggregate the results and provide a summary of the performance metrics.

        """
        from mobgap.gait_sequences.evaluation import (
            calculate_matched_gsd_performance_metrics,
            calculate_unmatched_gsd_performance_metrics,
            categorize_intervals_per_sample,
        )

        # Run the algorithm on the datapoint
        detected_gs_list = pipeline.safe_run(datapoint).gs_list_
        reference_gs_list = datapoint.reference_parameters_.wb_list[["start", "end"]]
        n_overall_samples = len(datapoint.data_ss)
        sampling_rate_hz = datapoint.sampling_rate_hz

        matches = categorize_intervals_per_sample(
            gsd_list_detected=detected_gs_list, gsd_list_reference=reference_gs_list, n_overall_samples=n_overall_samples
        )

        # Calculate the performance metrics
        performance_metrics = {
            **calculate_unmatched_gsd_performance_metrics(
                gsd_list_detected=detected_gs_list,
                gsd_list_reference=reference_gs_list,
                sampling_rate_hz=sampling_rate_hz,
            ),
            **calculate_matched_gsd_performance_metrics(matches),
            "detected": NoAgg(detected_gs_list),
            "reference": NoAgg(reference_gs_list),
        }

        return performance_metrics





.. GENERATED FROM PYTHON SOURCE LINES 72-86

We can see that this method is relatively simple, using the gsd evaluation functions that we provide.
So if you want to run your own scoring function, it should be straightforward to do so.

Note, the :class:`~tpcp.validate.NoAgg` wrapping some of the return values.
This is a special aggregator that tells the challenge to not try to aggregate the respective values.
For all other values, the challenge will try average the values across all datapoints.

To learn more about these special aggregators, check out the `tpcp example
<https://tpcp.readthedocs.io/en/latest/auto_examples/validation/_03_custom_scorer.html>`_.

The scoring function takes care of running the pipeline.
So we can test the scorer, by just providing it with a pipeline and a datapoint.

Note, that we remove the :class:`~tpcp.validate.NoAgg` parameters from the results, as they don't visualize well.

.. GENERATED FROM PYTHON SOURCE LINES 86-93

.. code-block:: default

    from pprint import pprint

    single_dp_results = gsd_evaluation_scorer(pipe, long_test[0])
    single_dp_results.pop("detected")
    single_dp_results.pop("reference")
    pprint(single_dp_results)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {'accuracy': 0.7349292709466811,
     'detected_gs_duration_s': 60.14,
     'detected_num_gs': 7,
     'f1_score': 0.6371400198609732,
     'fn_samples': 839,
     'fp_samples': 2815,
     'gs_absolute_duration_error_s': 19.700000000000003,
     'gs_absolute_relative_duration_error': 0.487141444114738,
     'gs_absolute_relative_duration_error_log': 0.3968557834081124,
     'gs_duration_error_s': 19.700000000000003,
     'gs_relative_duration_error': 0.487141444114738,
     'npv': 0.8919093017263592,
     'num_gs_absolute_error': 1,
     'num_gs_absolute_relative_error': 0.16666666666666666,
     'num_gs_absolute_relative_error_log': 0.15415067982725836,
     'num_gs_error': 1,
     'num_gs_relative_error': 0.16666666666666666,
     'precision': 0.5326249377386685,
     'recall': 0.7926859402026192,
     'reference_gs_duration_s': 40.44,
     'reference_num_gs': 6,
     'specificity': 0.7109262682275621,
     'tn_samples': 6923,
     'tp_samples': 3208}




.. GENERATED FROM PYTHON SOURCE LINES 94-96

The challenge will call this scoring method for each datapoint in the dataset.
Let's test this with the `GsdEvaluation` challenge.

.. GENERATED FROM PYTHON SOURCE LINES 96-99

.. code-block:: default

    from mobgap.gait_sequences.evaluation import GsdEvaluation

    eval_challenge = GsdEvaluation(long_test, scoring=gsd_evaluation_scorer)







.. GENERATED FROM PYTHON SOURCE LINES 100-101

We can now run the challenge.

.. GENERATED FROM PYTHON SOURCE LINES 101-103

.. code-block:: default

    eval_challenge = eval_challenge.run(pipe)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Datapoints:   0%|          | 0/3 [00:00<?, ?it/s]
    Datapoints:  33%|███▎      | 1/3 [00:00<00:00,  3.79it/s]
    Datapoints:  67%|██████▋   | 2/3 [00:00<00:00,  3.64it/s]/home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/data/_mobilised_matlab_loader.py:1082: UserWarning: There were multiple ICs with the same index value, but different LR labels. This is likely an issue with the reference system you should further investigate. For now, we set the `lr_label` of the stride corresponding to this IC to Nan. However, both values still remain in the IC list.
      return parse_reference_parameters(

    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  3.48it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  3.53it/s]




.. GENERATED FROM PYTHON SOURCE LINES 104-106

The results are stored in the `results_` attribute and contain the aggregated and the raw results per datapoint.
To learn more about the results, check the :func:`~tpcp.validate.validate` documentation.

.. GENERATED FROM PYTHON SOURCE LINES 106-110

.. code-block:: default

    import pandas as pd

    validate_results = pd.DataFrame(eval_challenge.results_)








.. GENERATED FROM PYTHON SOURCE LINES 111-112

The aggregated results across all datapoints are available in all columns not starting with ``agg__``

.. GENERATED FROM PYTHON SOURCE LINES 112-115

.. code-block:: default

    agg_results = validate_results.filter(like="agg__")
    agg_results.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>agg__reference_gs_duration_s</th>
          <td>48.926667</td>
        </tr>
        <tr>
          <th>agg__detected_gs_duration_s</th>
          <td>58.620000</td>
        </tr>
        <tr>
          <th>agg__gs_duration_error_s</th>
          <td>9.693333</td>
        </tr>
        <tr>
          <th>agg__gs_relative_duration_error</th>
          <td>0.237191</td>
        </tr>
        <tr>
          <th>agg__gs_absolute_duration_error_s</th>
          <td>9.693333</td>
        </tr>
        <tr>
          <th>agg__gs_absolute_relative_duration_error</th>
          <td>0.237191</td>
        </tr>
        <tr>
          <th>agg__gs_absolute_relative_duration_error_log</th>
          <td>0.200297</td>
        </tr>
        <tr>
          <th>agg__detected_num_gs</th>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>agg__reference_num_gs</th>
          <td>5.000000</td>
        </tr>
        <tr>
          <th>agg__num_gs_error</th>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>agg__num_gs_relative_error</th>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>agg__num_gs_absolute_error</th>
          <td>1.666667</td>
        </tr>
        <tr>
          <th>agg__num_gs_absolute_relative_error</th>
          <td>0.444444</td>
        </tr>
        <tr>
          <th>agg__num_gs_absolute_relative_error_log</th>
          <td>0.333816</td>
        </tr>
        <tr>
          <th>agg__tp_samples</th>
          <td>3730.333333</td>
        </tr>
        <tr>
          <th>agg__fp_samples</th>
          <td>2138.666667</td>
        </tr>
        <tr>
          <th>agg__fn_samples</th>
          <td>1166.000000</td>
        </tr>
        <tr>
          <th>agg__precision</th>
          <td>0.632099</td>
        </tr>
        <tr>
          <th>agg__recall</th>
          <td>0.766355</td>
        </tr>
        <tr>
          <th>agg__f1_score</th>
          <td>0.688517</td>
        </tr>
        <tr>
          <th>agg__tn_samples</th>
          <td>10477.333333</td>
        </tr>
        <tr>
          <th>agg__specificity</th>
          <td>0.815958</td>
        </tr>
        <tr>
          <th>agg__accuracy</th>
          <td>0.802683</td>
        </tr>
        <tr>
          <th>agg__npv</th>
          <td>0.899915</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 116-117

The raw results are stored in the columns starting with ``single__``.

.. GENERATED FROM PYTHON SOURCE LINES 117-119

.. code-block:: default

    single_results = validate_results.filter(like="single__")








.. GENERATED FROM PYTHON SOURCE LINES 120-121

And it is often helpful to explode them to get a better overview.

.. GENERATED FROM PYTHON SOURCE LINES 121-135

.. code-block:: default

    exploded_results = (
        single_results.explode(single_results.columns.to_list())
        .rename_axis("fold")
        .set_index(
            pd.MultiIndex.from_tuples(
                (dl := validate_results["data_labels"].explode().to_list()),
                names=list(dl[0]._fields),
            ),
            append=True,
        )
    )
    exploded_results.columns = exploded_results.columns.str.removeprefix("single__")
    exploded_results.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th>fold</th>
          <th colspan="3" halign="left">0</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th colspan="2" halign="left">HA</th>
          <th>MS</th>
        </tr>
        <tr>
          <th>participant_id</th>
          <th>001</th>
          <th>002</th>
          <th>001</th>
        </tr>
        <tr>
          <th>time_measure</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
        </tr>
        <tr>
          <th>test</th>
          <th>Test11</th>
          <th>Test11</th>
          <th>Test11</th>
        </tr>
        <tr>
          <th>trial</th>
          <th>Trial1</th>
          <th>Trial1</th>
          <th>Trial1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>40.44</td>
          <td>40.82</td>
          <td>65.52</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>60.14</td>
          <td>49.62</td>
          <td>66.1</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>19.7</td>
          <td>8.8</td>
          <td>0.58</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>19.7</td>
          <td>8.8</td>
          <td>0.58</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.396856</td>
          <td>0.195222</td>
          <td>0.008813</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>7</td>
          <td>6</td>
          <td>5</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>6</td>
          <td>3</td>
          <td>6</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>1</td>
          <td>3</td>
          <td>-1</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.166667</td>
          <td>1.0</td>
          <td>-0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>1</td>
          <td>3</td>
          <td>1</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.166667</td>
          <td>1.0</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.154151</td>
          <td>0.693147</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>3208</td>
          <td>3132</td>
          <td>4851</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2815</td>
          <td>1835</td>
          <td>1766</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>839</td>
          <td>955</td>
          <td>1704</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.532625</td>
          <td>0.630562</td>
          <td>0.733112</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.792686</td>
          <td>0.766332</td>
          <td>0.740046</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.63714</td>
          <td>0.691849</td>
          <td>0.736562</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>6923</td>
          <td>10080</td>
          <td>14429</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.710926</td>
          <td>0.845992</td>
          <td>0.890954</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.734929</td>
          <td>0.825647</td>
          <td>0.847473</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.891909</td>
          <td>0.913457</td>
          <td>0.894378</td>
        </tr>
        <tr>
          <th>detected</th>
          <td>start    end
    gs_id              
    0     ...</td>
          <td>start    end
    gs_id              
    0     ...</td>
          <td>start    end
    gs_id              
    0     ...</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>start    end
    wb_id              
    0     ...</td>
          <td>start   end
    wb_id             
    0       ...</td>
          <td>start    end
    wb_id              
    0     ...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 136-138

The ``detected`` and ``reference`` columns in this dataframe contain the raw un-aggregated gait-sequences.
So if we want to perform further evaluation on them (e.g. visualize them), we can use them.

.. GENERATED FROM PYTHON SOURCE LINES 138-144

.. code-block:: default

    raw_gs_list = pd.concat(
        exploded_results.loc[:, ["detected", "reference"]].stack().to_dict(),
        names=[*exploded_results.index.names, "system"],
    ).unstack("system")
    raw_gs_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }

        .dataframe thead tr:last-of-type th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th colspan="2" halign="left">start</th>
          <th colspan="2" halign="left">end</th>
        </tr>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>system</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="19" valign="top">0</th>
          <th rowspan="13" valign="top">HA</th>
          <th rowspan="7" valign="top">001</th>
          <th rowspan="7" valign="top">TimeMeasure1</th>
          <th rowspan="7" valign="top">Test11</th>
          <th rowspan="7" valign="top">Trial1</th>
          <th>0</th>
          <td>600.0</td>
          <td>632.0</td>
          <td>1201.0</td>
          <td>988.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700.0</td>
          <td>2864.0</td>
          <td>4201.0</td>
          <td>3325.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4350.0</td>
          <td>3853.0</td>
          <td>5251.0</td>
          <td>5085.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7800.0</td>
          <td>7641.0</td>
          <td>8851.0</td>
          <td>8621.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9450.0</td>
          <td>9451.0</td>
          <td>10201.0</td>
          <td>9932.0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>10950.0</td>
          <td>11989.0</td>
          <td>11551.0</td>
          <td>12517.0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>13050.0</td>
          <td>NaN</td>
          <td>13651.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">002</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>450.0</td>
          <td>485.0</td>
          <td>1201.0</td>
          <td>1131.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2400.0</td>
          <td>1746.0</td>
          <td>3301.0</td>
          <td>3554.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3450.0</td>
          <td>6083.0</td>
          <td>4051.0</td>
          <td>7708.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>5700.0</td>
          <td>NaN</td>
          <td>7201.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>4</th>
          <td>7350.0</td>
          <td>NaN</td>
          <td>7951.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>5</th>
          <td>15000.0</td>
          <td>NaN</td>
          <td>15601.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">MS</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>750.0</td>
          <td>1019.0</td>
          <td>1651.0</td>
          <td>1768.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650.0</td>
          <td>4534.0</td>
          <td>6151.0</td>
          <td>5549.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>12900.0</td>
          <td>9665.0</td>
          <td>14851.0</td>
          <td>10569.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>20100.0</td>
          <td>12337.0</td>
          <td>21151.0</td>
          <td>14633.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>21300.0</td>
          <td>20151.0</td>
          <td>22501.0</td>
          <td>20982.0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>NaN</td>
          <td>21378.0</td>
          <td>NaN</td>
          <td>22129.0</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 145-146

Further there are some runtime information available (i.e. when the challenge was started, and how long it took).

.. GENERATED FROM PYTHON SOURCE LINES 146-148

.. code-block:: default

    eval_challenge.start_datetime_, eval_challenge.end_datetime_





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ('2024-10-21T12:43:10.783453+00:00', '2024-10-21T12:43:11.696589+00:00')



.. GENERATED FROM PYTHON SOURCE LINES 149-152

.. code-block:: default

    eval_challenge.runtime_s_






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0.9129904770015855



.. GENERATED FROM PYTHON SOURCE LINES 153-174

Using :class:`~mobgap.gait_sequences.evaluation.GsdEvaluation` is great, if you are only comparing (or planning to
compare) non-ML algorithms, or algorithms that don't require further optimization (e.g. through GridSearch).

Therefore, it is generally recommended to run a cross-validation with
:class:`~mobgap.gait_sequences.evaluation.GsdEvaluationCV`.
This allows you to evaluate the performance of the algorithm on multiple folds of the dataset and through the use
of :class:`~tpcp.optimize.DummyOptimize` you can also use algorithms without optimization in the same pipeline for
comparison.

Let's demonstrate the use of :class:`~mobgap.gait_sequences.evaluation.GsdEvaluationCV` on the example dataset using
the same algorithm once with and once without GridSearch.

For the CV-based challenge, we need to set up a cross-validation.
As we only have 3 datapoints here, we will use a 3-fold cross-validation without grouping or stratification.
In a real-world scenario, you would use a more sophisticated cross-validation strategy.
You can learn more about cross-validation in the `tpcp example
<https://tpcp.readthedocs.io/en/latest/auto_examples/validation/_04_advanced_cross_validation.html>`_.

Further, to speed things up, we are going to use multi-processing.
We can configure this using the ``n_jobs`` parameter that we pass to the internal
:func:`~tpcp.validate.cross_validate` function via the ``cv_params`` parameters

.. GENERATED FROM PYTHON SOURCE LINES 174-183

.. code-block:: default

    from mobgap.gait_sequences.evaluation import GsdEvaluationCV

    eval_challenge_cv = GsdEvaluationCV(
        long_test,
        cv_iterator=3,
        scoring=gsd_evaluation_scorer,
        cv_params={"n_jobs": 2, "return_optimizer": True},
    )








.. GENERATED FROM PYTHON SOURCE LINES 184-186

To use our pipeline from above, we need to wrap it in a :class:`~tpcp.optimize.DummyOptimize` instance.
This will basically skip any optimization on the train set and just apply the pipeline to the test set.

.. GENERATED FROM PYTHON SOURCE LINES 186-192

.. code-block:: default

    from tpcp.optimize import DummyOptimize

    eval_challenge_cv = eval_challenge_cv.run(
        DummyOptimize(pipe, ignore_potential_user_error_warning=True)
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]
    CV Folds:  33%|███▎      | 1/3 [00:05<00:10,  5.40s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.54s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.92s/it]




.. GENERATED FROM PYTHON SOURCE LINES 193-199

The results now are a little bit more complex, as they contain the results for each fold.
In addition, we have information for the train and the test set.
The test set results, are what we are usually looking for.
The train set results, are only calculated when providing the ``return_train_score`` parameter to the ``cv_params``.

As before our main results are the aggregated results.

.. GENERATED FROM PYTHON SOURCE LINES 199-203

.. code-block:: default

    cv_results = pd.DataFrame(eval_challenge_cv.cv_results_)
    agg_results_cv = cv_results.filter(like="test__agg__")
    agg_results_cv.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>test__agg__reference_gs_duration_s</th>
          <td>40.440000</td>
          <td>40.820000</td>
          <td>65.520000</td>
        </tr>
        <tr>
          <th>test__agg__detected_gs_duration_s</th>
          <td>60.140000</td>
          <td>49.620000</td>
          <td>66.100000</td>
        </tr>
        <tr>
          <th>test__agg__gs_duration_error_s</th>
          <td>19.700000</td>
          <td>8.800000</td>
          <td>0.580000</td>
        </tr>
        <tr>
          <th>test__agg__gs_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>test__agg__gs_absolute_duration_error_s</th>
          <td>19.700000</td>
          <td>8.800000</td>
          <td>0.580000</td>
        </tr>
        <tr>
          <th>test__agg__gs_absolute_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>test__agg__gs_absolute_relative_duration_error_log</th>
          <td>0.396856</td>
          <td>0.195222</td>
          <td>0.008813</td>
        </tr>
        <tr>
          <th>test__agg__detected_num_gs</th>
          <td>7.000000</td>
          <td>6.000000</td>
          <td>5.000000</td>
        </tr>
        <tr>
          <th>test__agg__reference_num_gs</th>
          <td>6.000000</td>
          <td>3.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>test__agg__num_gs_error</th>
          <td>1.000000</td>
          <td>3.000000</td>
          <td>-1.000000</td>
        </tr>
        <tr>
          <th>test__agg__num_gs_relative_error</th>
          <td>0.166667</td>
          <td>1.000000</td>
          <td>-0.166667</td>
        </tr>
        <tr>
          <th>test__agg__num_gs_absolute_error</th>
          <td>1.000000</td>
          <td>3.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>test__agg__num_gs_absolute_relative_error</th>
          <td>0.166667</td>
          <td>1.000000</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>test__agg__num_gs_absolute_relative_error_log</th>
          <td>0.154151</td>
          <td>0.693147</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>test__agg__tp_samples</th>
          <td>3208.000000</td>
          <td>3132.000000</td>
          <td>4851.000000</td>
        </tr>
        <tr>
          <th>test__agg__fp_samples</th>
          <td>2815.000000</td>
          <td>1835.000000</td>
          <td>1766.000000</td>
        </tr>
        <tr>
          <th>test__agg__fn_samples</th>
          <td>839.000000</td>
          <td>955.000000</td>
          <td>1704.000000</td>
        </tr>
        <tr>
          <th>test__agg__precision</th>
          <td>0.532625</td>
          <td>0.630562</td>
          <td>0.733112</td>
        </tr>
        <tr>
          <th>test__agg__recall</th>
          <td>0.792686</td>
          <td>0.766332</td>
          <td>0.740046</td>
        </tr>
        <tr>
          <th>test__agg__f1_score</th>
          <td>0.637140</td>
          <td>0.691849</td>
          <td>0.736562</td>
        </tr>
        <tr>
          <th>test__agg__tn_samples</th>
          <td>6923.000000</td>
          <td>10080.000000</td>
          <td>14429.000000</td>
        </tr>
        <tr>
          <th>test__agg__specificity</th>
          <td>0.710926</td>
          <td>0.845992</td>
          <td>0.890954</td>
        </tr>
        <tr>
          <th>test__agg__accuracy</th>
          <td>0.734929</td>
          <td>0.825647</td>
          <td>0.847473</td>
        </tr>
        <tr>
          <th>test__agg__npv</th>
          <td>0.891909</td>
          <td>0.913457</td>
          <td>0.894378</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 204-205

The raw results are stored in the columns starting with ``single__``.

.. GENERATED FROM PYTHON SOURCE LINES 205-222

.. code-block:: default

    single_results = cv_results.filter(like="test__single__")
    exploded_results_cv = (
        single_results.explode(single_results.columns.to_list())
        .rename_axis("fold")
        .set_index(
            pd.MultiIndex.from_tuples(
                (dl := cv_results["test__data_labels"].explode().to_list()),
                names=list(dl[0]._fields),
            ),
            append=True,
        )
    )
    exploded_results_cv.columns = exploded_results_cv.columns.str.removeprefix(
        "test__single__"
    )
    exploded_results_cv.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th>fold</th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th>HA</th>
          <th>HA</th>
          <th>MS</th>
        </tr>
        <tr>
          <th>participant_id</th>
          <th>001</th>
          <th>002</th>
          <th>001</th>
        </tr>
        <tr>
          <th>time_measure</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
        </tr>
        <tr>
          <th>test</th>
          <th>Test11</th>
          <th>Test11</th>
          <th>Test11</th>
        </tr>
        <tr>
          <th>trial</th>
          <th>Trial1</th>
          <th>Trial1</th>
          <th>Trial1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>40.44</td>
          <td>40.82</td>
          <td>65.52</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>60.14</td>
          <td>49.62</td>
          <td>66.1</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>19.7</td>
          <td>8.8</td>
          <td>0.58</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>19.7</td>
          <td>8.8</td>
          <td>0.58</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.487141</td>
          <td>0.215581</td>
          <td>0.008852</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.396856</td>
          <td>0.195222</td>
          <td>0.008813</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>7</td>
          <td>6</td>
          <td>5</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>6</td>
          <td>3</td>
          <td>6</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>1</td>
          <td>3</td>
          <td>-1</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.166667</td>
          <td>1.0</td>
          <td>-0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>1</td>
          <td>3</td>
          <td>1</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.166667</td>
          <td>1.0</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.154151</td>
          <td>0.693147</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>3208</td>
          <td>3132</td>
          <td>4851</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2815</td>
          <td>1835</td>
          <td>1766</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>839</td>
          <td>955</td>
          <td>1704</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.532625</td>
          <td>0.630562</td>
          <td>0.733112</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.792686</td>
          <td>0.766332</td>
          <td>0.740046</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.63714</td>
          <td>0.691849</td>
          <td>0.736562</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>6923</td>
          <td>10080</td>
          <td>14429</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.710926</td>
          <td>0.845992</td>
          <td>0.890954</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.734929</td>
          <td>0.825647</td>
          <td>0.847473</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.891909</td>
          <td>0.913457</td>
          <td>0.894378</td>
        </tr>
        <tr>
          <th>detected</th>
          <td>start    end
    gs_id              
    0     ...</td>
          <td>start    end
    gs_id              
    0     ...</td>
          <td>start    end
    gs_id              
    0     ...</td>
        </tr>
        <tr>
          <th>reference</th>
          <td>start    end
    wb_id              
    0     ...</td>
          <td>start   end
    wb_id             
    0       ...</td>
          <td>start    end
    wb_id              
    0     ...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 223-224

And the raw outputs:

.. GENERATED FROM PYTHON SOURCE LINES 224-230

.. code-block:: default

    raw_gs_list_cv = pd.concat(
        exploded_results_cv.loc[:, ["detected", "reference"]].stack().to_dict(),
        names=[*exploded_results_cv.index.names, "system"],
    ).unstack("system")
    raw_gs_list_cv






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }

        .dataframe thead tr:last-of-type th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th colspan="2" halign="left">start</th>
          <th colspan="2" halign="left">end</th>
        </tr>
        <tr>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>system</th>
          <th>detected</th>
          <th>reference</th>
          <th>detected</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="7" valign="top">0</th>
          <th rowspan="7" valign="top">HA</th>
          <th rowspan="7" valign="top">001</th>
          <th rowspan="7" valign="top">TimeMeasure1</th>
          <th rowspan="7" valign="top">Test11</th>
          <th rowspan="7" valign="top">Trial1</th>
          <th>0</th>
          <td>600.0</td>
          <td>632.0</td>
          <td>1201.0</td>
          <td>988.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700.0</td>
          <td>2864.0</td>
          <td>4201.0</td>
          <td>3325.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4350.0</td>
          <td>3853.0</td>
          <td>5251.0</td>
          <td>5085.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7800.0</td>
          <td>7641.0</td>
          <td>8851.0</td>
          <td>8621.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9450.0</td>
          <td>9451.0</td>
          <td>10201.0</td>
          <td>9932.0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>10950.0</td>
          <td>11989.0</td>
          <td>11551.0</td>
          <td>12517.0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>13050.0</td>
          <td>NaN</td>
          <td>13651.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">1</th>
          <th rowspan="6" valign="top">HA</th>
          <th rowspan="6" valign="top">002</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>450.0</td>
          <td>485.0</td>
          <td>1201.0</td>
          <td>1131.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2400.0</td>
          <td>1746.0</td>
          <td>3301.0</td>
          <td>3554.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3450.0</td>
          <td>6083.0</td>
          <td>4051.0</td>
          <td>7708.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>5700.0</td>
          <td>NaN</td>
          <td>7201.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>4</th>
          <td>7350.0</td>
          <td>NaN</td>
          <td>7951.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>5</th>
          <td>15000.0</td>
          <td>NaN</td>
          <td>15601.0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">2</th>
          <th rowspan="6" valign="top">MS</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>750.0</td>
          <td>1019.0</td>
          <td>1651.0</td>
          <td>1768.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650.0</td>
          <td>4534.0</td>
          <td>6151.0</td>
          <td>5549.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>12900.0</td>
          <td>9665.0</td>
          <td>14851.0</td>
          <td>10569.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>20100.0</td>
          <td>12337.0</td>
          <td>21151.0</td>
          <td>14633.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>21300.0</td>
          <td>20151.0</td>
          <td>22501.0</td>
          <td>20982.0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>NaN</td>
          <td>21378.0</td>
          <td>NaN</td>
          <td>22129.0</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 231-238

If we compare these results to the ones from the non-CV challenge, we can see that "single" results are identical,
just that they were called in multiple folds.
This is expected, as we used :class:`~tpcp.optimize.DummyOptimize` and thus didn't optimize the algorithm.

Let's try a :class:`~tpcp.optimize.GridSearch` on the algorithm to see how the results change.
For the gridsearch, we will re-use the same scoring function as before, but we need to specify, which scoring result
we want to optimize for.

.. GENERATED FROM PYTHON SOURCE LINES 238-244

.. code-block:: default

    from sklearn.model_selection import ParameterGrid
    from tpcp.optimize import GridSearch

    para_grid = ParameterGrid({"algo__window_length_s": [2, 3, 4]})
    optimizer = GridSearch(pipe, para_grid, return_optimized="precision")








.. GENERATED FROM PYTHON SOURCE LINES 245-248

The optimizer can now be used in the same CV challenge as before.
This way we can guarantee that the same folds are used for the optimization and the evaluation and ensure the best
possible comparison between the algorithms versions.

.. GENERATED FROM PYTHON SOURCE LINES 248-250

.. code-block:: default

    eval_challenge_cv = eval_challenge_cv.clone().run(optimizer)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]
    CV Folds:  33%|███▎      | 1/3 [00:03<00:07,  3.72s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.74s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.94s/it]




.. GENERATED FROM PYTHON SOURCE LINES 251-252

The results we are seeing now are generated by the internally optimized version of the algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 252-255

.. code-block:: default

    cv_results_gs = pd.DataFrame(eval_challenge_cv.cv_results_)
    agg_results_gs = cv_results.filter(like="test__agg__")








.. GENERATED FROM PYTHON SOURCE LINES 256-258

Because we used ``cv_params={"return_optimizer": True}`` we can also access the optimizer per fold directly.
This can be useful to get more insights into the optimization process and what the optimal parameters were.

.. GENERATED FROM PYTHON SOURCE LINES 258-261

.. code-block:: default

    opt_results = cv_results_gs["optimizer"]
    opt_results





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0    GridSearch(n_jobs=None, parameter_grid=<sklear...
    1    GridSearch(n_jobs=None, parameter_grid=<sklear...
    2    GridSearch(n_jobs=None, parameter_grid=<sklear...
    Name: optimizer, dtype: object



.. GENERATED FROM PYTHON SOURCE LINES 262-263

We can get the best parameters per fold by directly interacting with the optimizer instances.

.. GENERATED FROM PYTHON SOURCE LINES 263-266

.. code-block:: default

    best_params = opt_results.apply(lambda x: pd.Series(x.best_params_))
    best_params






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>algo__window_length_s</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>3</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 267-269

Or we can go much deeper, by getting all information about the optimization process.
Let's just look at the keys of the information that is available.

.. GENERATED FROM PYTHON SOURCE LINES 269-272

.. code-block:: default

    all_opti_results_fold0 = pd.DataFrame(opt_results.loc[0].gs_results_)
    all_opti_results_fold0.columns.to_list()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ['agg__reference_gs_duration_s', 'rank__agg__reference_gs_duration_s', 'agg__detected_gs_duration_s', 'rank__agg__detected_gs_duration_s', 'agg__gs_duration_error_s', 'rank__agg__gs_duration_error_s', 'agg__gs_relative_duration_error', 'rank__agg__gs_relative_duration_error', 'agg__gs_absolute_duration_error_s', 'rank__agg__gs_absolute_duration_error_s', 'agg__gs_absolute_relative_duration_error', 'rank__agg__gs_absolute_relative_duration_error', 'agg__gs_absolute_relative_duration_error_log', 'rank__agg__gs_absolute_relative_duration_error_log', 'agg__detected_num_gs', 'rank__agg__detected_num_gs', 'agg__reference_num_gs', 'rank__agg__reference_num_gs', 'agg__num_gs_error', 'rank__agg__num_gs_error', 'agg__num_gs_relative_error', 'rank__agg__num_gs_relative_error', 'agg__num_gs_absolute_error', 'rank__agg__num_gs_absolute_error', 'agg__num_gs_absolute_relative_error', 'rank__agg__num_gs_absolute_relative_error', 'agg__num_gs_absolute_relative_error_log', 'rank__agg__num_gs_absolute_relative_error_log', 'agg__tp_samples', 'rank__agg__tp_samples', 'agg__fp_samples', 'rank__agg__fp_samples', 'agg__fn_samples', 'rank__agg__fn_samples', 'agg__precision', 'rank__agg__precision', 'agg__recall', 'rank__agg__recall', 'agg__f1_score', 'rank__agg__f1_score', 'agg__tn_samples', 'rank__agg__tn_samples', 'agg__specificity', 'rank__agg__specificity', 'agg__accuracy', 'rank__agg__accuracy', 'agg__npv', 'rank__agg__npv', 'single__reference_gs_duration_s', 'single__detected_gs_duration_s', 'single__gs_duration_error_s', 'single__gs_relative_duration_error', 'single__gs_absolute_duration_error_s', 'single__gs_absolute_relative_duration_error', 'single__gs_absolute_relative_duration_error_log', 'single__detected_num_gs', 'single__reference_num_gs', 'single__num_gs_error', 'single__num_gs_relative_error', 'single__num_gs_absolute_error', 'single__num_gs_absolute_relative_error', 'single__num_gs_absolute_relative_error_log', 'single__tp_samples', 'single__fp_samples', 'single__fn_samples', 'single__precision', 'single__recall', 'single__f1_score', 'single__tn_samples', 'single__specificity', 'single__accuracy', 'single__npv', 'single__detected', 'single__reference', 'data_labels', 'debug__score_time', 'param__algo__window_length_s', 'params']



.. GENERATED FROM PYTHON SOURCE LINES 273-277

With that, we hope it becomes clear, how these challenges can be extremely valuable, when benchmarking algorithms
across datasets.
To see how we evaluate the performance of the algorithms available in mobgap, check out the other gsd evaluation
examples.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 16.891 seconds)

**Estimated memory usage:**  9 MB


.. _sphx_glr_download_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _04_gsd_evaluation_challenges.py <_04_gsd_evaluation_challenges.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _04_gsd_evaluation_challenges.ipynb <_04_gsd_evaluation_challenges.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_