.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/gait_sequences/_04_gsd_evaluation_challenges.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py:


GSD Evaluation Challenges
-------------------------
The :ref:`gsd_evaluation` example demonstrates how to evaluate the performance of a GSD algorithm on a single datapoint
 and explains the individual performance metrics that are calculated.

With that you could set up a custom evaluation pipeline to run and then score the output of a GSD algorithm
multiple datapoints and then aggregate the results.
To make this process easier, we set up opinionated evaluation challenges that can be used to quickly perform the same
evaluation with multiple algorithms and datasets.

Below, we will show how to use them on the example dataset.

.. GENERATED FROM PYTHON SOURCE LINES 14-16

.. code-block:: Python

    # TODO: Update based on new Scorer API








.. GENERATED FROM PYTHON SOURCE LINES 17-21

Dataset
-------
To use the challenges, we need to dataset with reference information in the expected format.
We will use the :class:`~mobgap.data.LabExampleDataset` for this purpose.

.. GENERATED FROM PYTHON SOURCE LINES 21-27

.. code-block:: Python

    from mobgap.data import LabExampleDataset

    long_test = LabExampleDataset(reference_system="INDIP").get_subset(
        test="Test11"
    )








.. GENERATED FROM PYTHON SOURCE LINES 28-31

Algorithm
---------
Next we need to create an instance of a valid GSD algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 31-35

.. code-block:: Python

    from mobgap.gait_sequences import GsdIluz

    algo = GsdIluz()








.. GENERATED FROM PYTHON SOURCE LINES 36-39

This algorithm needs to be wrapped in a :class:`~mobgap.gait_sequences.pipeline.GsdEmulationPipeline` to be used in
the challenges.
This pipeline takes care of extracting the correct data from the dataset and running the algorithm on it.

.. GENERATED FROM PYTHON SOURCE LINES 39-43

.. code-block:: Python

    from mobgap.gait_sequences.pipeline import GsdEmulationPipeline

    pipe = GsdEmulationPipeline(algo)








.. GENERATED FROM PYTHON SOURCE LINES 44-45

Let's demonstrate that quickly on a single datapoint.

.. GENERATED FROM PYTHON SOURCE LINES 45-48

.. code-block:: Python

    pipe_with_results = pipe.clone().run(long_test[0])
    pipe_with_results.gs_list_






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7800</td>
          <td>9001</td>
        </tr>
        <tr>
          <th>3</th>
          <td>9300</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>4</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>5</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 49-71

Evaluation Challenge
--------------------
This pipeline can now be used as part of an evaluation challenge.
An evaluation challenge takes care of two things:

- Running the pipeline on multiple datapoints
- Scoring the results per datapoint and then aggregating the results

We provide two challenges:

- :class:`~mobgap.gait_sequences.evaluation.GsdEvaluation`: This challenge simply runs the pipeline on all datapoints and then scores the results.
- :class:`~mobgap.gait_sequences.evaluation.GsdEvaluationCV`: This challenge runs a cross-validation on the dataset and then scores the results per fold.

Before we run the entire pipeline, let's look at the scoring.
Scoring is built based on tpcp's validation framework.
As we have relativly complex scoring, scoring is split across two functions:

- :func:`~mobgap.gait_sequences.evaluation.gsd_per_datapoint_score`: Run and score a single datapoint
- :func:`~mobgap.gait_sequences.evaluation.gsd_final_agg`: Perform final aggreagtion and scoring based on the results
                                                           per datapoint.

Let's look at the code of it first.

.. GENERATED FROM PYTHON SOURCE LINES 71-80

.. code-block:: Python

    from inspect import getsource

    from mobgap.gait_sequences.evaluation import (
        gsd_final_agg,
        gsd_per_datapoint_score,
    )

    print(getsource(gsd_per_datapoint_score))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    def gsd_per_datapoint_score(pipeline: GsdEmulationPipeline, datapoint: BaseGaitDatasetWithReference) -> dict:
        """Evaluate the performance of a GSD algorithm on a single datapoint.

        .. warning:: This function is not meant to be called directly, but as a scoring function in a
           :class:`tpcp.validate.Scorer`.
           If you are writing custom scoring functions, you can use this function as a template or wrap it in a new
           function.

        This function is used to evaluate the performance of a GSD algorithm on a single datapoint.
        It calculates the performance metrics based on the detected gait sequences and the reference gait sequences.

        The following performance metrics are calculated:

        - all outputs of :func:`~mobgap.gait_sequences.evaluation.calculate_unmatched_gsd_performance_metrics`
          (will be averaged over all datapoints)
        - all outputs of :func:`~mobgap.gait_sequences.evaluation.calculate_matched_gsd_performance_metrics`
          (will be averaged over all datapoints)
        - ``matches``: The matched gait sequences calculated by
          :func:`~mobgap.gait_sequences.evaluation.categorize_intervals_per_sample` (return as ``no_agg``)
        - ``detected``: The detected gait sequences (return as ``no_agg``)
        - ``reference``: The reference gait sequences (return as ``no_agg``)
        - ``sampling_rate_hz``: The sampling rate of the data (return as ``no_agg``)

        Parameters
        ----------
        pipeline
            An instance of GSD emulation pipeline that wraps the algorithm that should be evaluated.
        datapoint
            The datapoint to be evaluated.

        Returns
        -------
        dict
            A dictionary containing the performance metrics.
            Note, that some results are wrapped in a ``no_agg`` object or other aggregators.
            The results of this function are not expected to be parsed manually, but rather the function is expected to be
            used in the context of the :func:`~tpcp.validate.validate`/:func:`~tpcp.validate.cross_validate` functions or
            similar as scorer.
            This functions will aggregate the results and provide a summary of the performance metrics.

        """
        from mobgap.gait_sequences.evaluation import (
            calculate_matched_gsd_performance_metrics,
            calculate_unmatched_gsd_performance_metrics,
            categorize_intervals_per_sample,
        )

        with warnings.catch_warnings():
            # We know that these errors might happen, and they are usually not relevant for the evaluation
            warnings.filterwarnings("ignore", message="Zero division", category=UserWarning)
            warnings.filterwarnings("ignore", message="multiple ICs", category=UserWarning)

            # Run the algorithm on the datapoint
            pipeline.safe_run(datapoint)
            detected_gs_list = pipeline.gs_list_
            reference_gs_list = datapoint.reference_parameters_.wb_list[["start", "end"]]
            n_overall_samples = len(datapoint.data_ss)
            sampling_rate_hz = datapoint.sampling_rate_hz

            matches = categorize_intervals_per_sample(
                gsd_list_detected=detected_gs_list,
                gsd_list_reference=reference_gs_list,
                n_overall_samples=n_overall_samples,
            )

            # Calculate the performance metrics
            performance_metrics = {
                **calculate_unmatched_gsd_performance_metrics(
                    gsd_list_detected=detected_gs_list,
                    gsd_list_reference=reference_gs_list,
                    sampling_rate_hz=sampling_rate_hz,
                ),
                **calculate_matched_gsd_performance_metrics(matches),
                "matches": no_agg(matches),
                "detected": no_agg(detected_gs_list),
                "reference": no_agg(reference_gs_list),
                "sampling_rate_hz": no_agg(sampling_rate_hz),
                "runtime_s": getattr(pipeline.algo_, "perf_", {}).get("runtime_s", np.nan),
            }

        return performance_metrics





.. GENERATED FROM PYTHON SOURCE LINES 81-83

.. code-block:: Python

    print(getsource(gsd_final_agg))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    def gsd_final_agg(
        agg_results: dict[str, float],
        single_results: dict[str, list],
        pipeline: GsdEmulationPipeline,  # noqa: ARG001
        dataset: BaseGaitDatasetWithReference,
    ) -> tuple[dict[str, any], dict[str, list[any]]]:
        """Aggregate the performance metrics of a GSD algorithm over multiple datapoints.

        .. warning:: This function is not meant to be called directly, but as ``final_aggregator`` in a
           :class:`tpcp.validate.Scorer`.
           If you are writing custom scoring functions, you can use this function as a template or wrap it in a new
           function.

        This function aggregates the performance metrics as follows:

        - All raw outputs (``detected``, ``reference``, ``sampling_rate_hz``) are concatenated to a single
          dataframe, to make it easier to work with and are returned as part of the single results.
        - We recalculate all performance metrics from
          :func:`~mobgap.gait_sequences.evaluation.calculate_unmatched_gsd_performance_metrics`
          and :func:`~mobgap.gait_sequences.evaluation.calculate_matched_gsd_performance_metrics` on the combined data.
          The results are prefixed with ``combined__``.
          Compared to the per-datapoint results (which are calculated, as errors per recording -> average over all
          recordings), these metrics are calculated as combining all GSDs from all recordings and then calculating the
          performance metrics.
          Effectively, this means, that in the `per_datapoint` version, each recording is weighted equally, while in the
          `combined` version, each GS is weighted equally.

        Parameters
        ----------
        agg_results
            The aggregated results from all datapoints (see :class:`~tpcp.validate.Scorer`).
        single_results
            The per-datapoint results (see :class:`~tpcp.validate.Scorer`).
        pipeline
            The pipline that was passed to the scorer.
            This is ignored in this function, but might be useful in custom final aggregators.
        dataset
            The dataset that was passed to the scorer.

        Returns
        -------
        final_agg_results
            The final aggregated results.
        final_single_results
            The per-datapoint results, that are not aggregated.

        """
        from mobgap.gait_sequences.evaluation import (
            calculate_matched_gsd_performance_metrics,
            calculate_unmatched_gsd_performance_metrics,
        )

        data_labels = [d.group_label for d in dataset]
        data_label_names = data_labels[0]._fields
        # We combine each to a combined dataframe
        matches = single_results.pop("matches")
        matches = pd.concat(matches, keys=data_labels, names=[*data_label_names, *matches[0].index.names])
        detected = single_results.pop("detected")
        detected = pd.concat(detected, keys=data_labels, names=[*data_label_names, *detected[0].index.names])
        reference = single_results.pop("reference")
        reference = pd.concat(reference, keys=data_labels, names=[*data_label_names, *reference[0].index.names])

        aggregated_single_results = {
            "raw__detected": detected,
            "raw__reference": reference,
        }

        sampling_rate_hz = single_results.pop("sampling_rate_hz")
        if set(sampling_rate_hz) != {sampling_rate_hz[0]}:
            raise ValueError(
                "Sampling rate is not the same for all datapoints in the dataset. "
                "This not supported by this scorer. "
                "Provide a custom scorer that can handle this case."
            )

        combined_unmatched = {
            f"combined__{k}": v
            for k, v in calculate_unmatched_gsd_performance_metrics(
                gsd_list_detected=detected,
                gsd_list_reference=reference,
                sampling_rate_hz=sampling_rate_hz[0],
            ).items()
        }
        combined_matched = {f"combined__{k}": v for k, v in calculate_matched_gsd_performance_metrics(matches).items()}

        # Note, that we pass the "aggregated_single_results" out via the single results and not the aggregated results
        # The reason is that the aggregated results are expected to be a single value per metric, while the single results
        # can be anything.
        return {**agg_results, **combined_unmatched, **combined_matched}, {**single_results, **aggregated_single_results}





.. GENERATED FROM PYTHON SOURCE LINES 84-99

We can see that these method is relatively simple, using the lower level gsd evaluation functions that we provide.
`gsd_per_datapoint_score` calculates the raw results and all scores that can be calculated per datapoint.
`gsd_final_agg` handles the calculation of all scores, that require the raw results from all datapoints at once.
The remaining aggregation is handled by the :class:`~tpcp.validate.Scorer` class (see below).
So if you want to run your own scoring function, it should be straightforward to do so.

Note, the :func:`~tpcp.validate.no_agg` wrapping some of the return values.
This is a special aggregator that tells the challenge to not try to aggregate the respective values.
For all other values, the challenge will try average the values across all datapoints.

To learn more about these special aggregators, check out the `tpcp example
<https://tpcp.readthedocs.io/en/latest/auto_examples/validation/_03_custom_scorer.html>`_.

The scoring function takes care of running the pipeline.
So we can test the scorer, by just providing it with a pipeline and a datapoint.

.. GENERATED FROM PYTHON SOURCE LINES 99-106

.. code-block:: Python

    from pprint import pprint

    single_dp_results = gsd_per_datapoint_score(pipe, long_test[0])
    single_dp_results.pop("detected")
    single_dp_results.pop("reference")
    pprint(single_dp_results)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {'accuracy': 0.704708699122107,
     'detected_gs_duration_s': 48.12,
     'detected_num_gs': 6,
     'f1_score': 0.5408393501805054,
     'fn_samples': 1649,
     'fp_samples': 2421,
     'gs_absolute_duration_error_s': 7.68,
     'gs_absolute_relative_duration_error': 0.18991097922848665,
     'gs_absolute_relative_duration_error_log': 0.17387849695420737,
     'gs_duration_error_s': 7.68,
     'gs_relative_duration_error': 0.18991097922848665,
     'matches': _NoAgg(return_raw_scores=True)(    start    end match_type
    0       0    600         tn
    1     600    632         fp
    2     632    988         tp
    3     988   1201         fp
    4    1201   2864         tn
    5    2864   3325         fn
    6    3325   3853         tn
    7    3853   4350         fn
    8    4350   5085         tp
    9    5085   5251         fp
    10   5251   7641         tn
    11   7641   7800         fn
    12   7800   8621         tp
    13   8621   9001         fp
    14   9001   9300         tn
    15   9300   9451         fp
    16   9451   9932         tp
    17   9932  10201         fp
    18  10201  10950         tn
    19  10950  11551         fp
    20  11551  11989         tn
    21  11989  12517         fn
    22  12517  13050         tn
    23  13050  13651         fp
    24  13651  13758         tn),
     'npv': 0.8160624651422197,
     'num_gs_absolute_error': 0,
     'num_gs_absolute_relative_error': 0.0,
     'num_gs_absolute_relative_error_log': 0.0,
     'num_gs_error': 0,
     'num_gs_relative_error': 0.0,
     'precision': 0.4975093399750934,
     'recall': 0.592436974789916,
     'reference_gs_duration_s': 40.44,
     'reference_num_gs': 6,
     'runtime_s': 0.022248906999266183,
     'sampling_rate_hz': _NoAgg(return_raw_scores=True)(100.0),
     'specificity': 0.7513607887439663,
     'tn_samples': 7316,
     'tp_samples': 2397}




.. GENERATED FROM PYTHON SOURCE LINES 107-108

To use the two functions with a challenge, we need to wrap them into a :class:`~tpcp.validate.Scorer` instance.

.. GENERATED FROM PYTHON SOURCE LINES 108-114

.. code-block:: Python

    from tpcp.validate import Scorer

    gsd_evaluation_scorer = Scorer(
        gsd_per_datapoint_score, final_aggregator=gsd_final_agg
    )








.. GENERATED FROM PYTHON SOURCE LINES 115-123

The challenge will call this scorer for each group in the dataset.
The scorer itself will then call `gsd_per_datapoint_score` for each datapoint and then `gsd_final_agg` with the
combined results.

For these two default scoring functions, we also provide the scorer directly, so that you don't have to construct it
yourself.
However, in case you want to modify the scoring functions, you can do so by creating your own scorer.
We will continue to use the default scorer for the challenges.

.. GENERATED FROM PYTHON SOURCE LINES 123-127

.. code-block:: Python

    from mobgap.gait_sequences.evaluation import gsd_score

    gsd_evaluation_scorer = gsd_score








.. GENERATED FROM PYTHON SOURCE LINES 128-129

Let's put everything together and run the challenge.

.. GENERATED FROM PYTHON SOURCE LINES 129-133

.. code-block:: Python

    from mobgap.utils.evaluation import Evaluation

    eval_challenge = Evaluation(long_test, scoring=gsd_evaluation_scorer)








.. GENERATED FROM PYTHON SOURCE LINES 134-135

We can now run the challenge.

.. GENERATED FROM PYTHON SOURCE LINES 135-137

.. code-block:: Python

    eval_challenge = eval_challenge.run(pipe)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Datapoints:   0%|          | 0/3 [00:00<?, ?it/s]
    Datapoints:  33%|███▎      | 1/3 [00:00<00:00,  5.14it/s]
    Datapoints:  67%|██████▋   | 2/3 [00:00<00:00,  5.16it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  5.01it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  5.05it/s]




.. GENERATED FROM PYTHON SOURCE LINES 138-142

The results are stored in the `results_` attribute and contain the aggregated and the raw results per datapoint.
To learn more about the results, check the :func:`~tpcp.validate.validate` documentation.

Note, that we remove the :class:`~tpcp.validate.no_agg` parameters from the results, as they don't visualize well.

.. GENERATED FROM PYTHON SOURCE LINES 142-146

.. code-block:: Python

    import pandas as pd

    validate_results = pd.DataFrame(eval_challenge.results_)
    validate_results





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>debug__score_time</th>
          <th>data_labels</th>
          <th>single__reference_gs_duration_s</th>
          <th>single__detected_gs_duration_s</th>
          <th>single__gs_duration_error_s</th>
          <th>single__gs_relative_duration_error</th>
          <th>single__gs_absolute_duration_error_s</th>
          <th>single__gs_absolute_relative_duration_error</th>
          <th>single__gs_absolute_relative_duration_error_log</th>
          <th>single__detected_num_gs</th>
          <th>single__reference_num_gs</th>
          <th>single__num_gs_error</th>
          <th>single__num_gs_relative_error</th>
          <th>single__num_gs_absolute_error</th>
          <th>single__num_gs_absolute_relative_error</th>
          <th>single__num_gs_absolute_relative_error_log</th>
          <th>single__tp_samples</th>
          <th>single__fp_samples</th>
          <th>single__fn_samples</th>
          <th>single__precision</th>
          <th>single__recall</th>
          <th>single__f1_score</th>
          <th>single__tn_samples</th>
          <th>single__specificity</th>
          <th>single__accuracy</th>
          <th>single__npv</th>
          <th>single__runtime_s</th>
          <th>single__raw__detected</th>
          <th>single__raw__reference</th>
          <th>agg__reference_gs_duration_s</th>
          <th>agg__detected_gs_duration_s</th>
          <th>agg__gs_duration_error_s</th>
          <th>agg__gs_relative_duration_error</th>
          <th>agg__gs_absolute_duration_error_s</th>
          <th>agg__gs_absolute_relative_duration_error</th>
          <th>agg__gs_absolute_relative_duration_error_log</th>
          <th>agg__detected_num_gs</th>
          <th>agg__reference_num_gs</th>
          <th>agg__num_gs_error</th>
          <th>agg__num_gs_relative_error</th>
          <th>agg__num_gs_absolute_error</th>
          <th>agg__num_gs_absolute_relative_error</th>
          <th>agg__num_gs_absolute_relative_error_log</th>
          <th>agg__tp_samples</th>
          <th>agg__fp_samples</th>
          <th>agg__fn_samples</th>
          <th>agg__precision</th>
          <th>agg__recall</th>
          <th>agg__f1_score</th>
          <th>agg__tn_samples</th>
          <th>agg__specificity</th>
          <th>agg__accuracy</th>
          <th>agg__npv</th>
          <th>agg__runtime_s</th>
          <th>agg__combined__reference_gs_duration_s</th>
          <th>agg__combined__detected_gs_duration_s</th>
          <th>agg__combined__gs_duration_error_s</th>
          <th>agg__combined__gs_relative_duration_error</th>
          <th>agg__combined__gs_absolute_duration_error_s</th>
          <th>agg__combined__gs_absolute_relative_duration_error</th>
          <th>agg__combined__gs_absolute_relative_duration_error_log</th>
          <th>agg__combined__detected_num_gs</th>
          <th>agg__combined__reference_num_gs</th>
          <th>agg__combined__num_gs_error</th>
          <th>agg__combined__num_gs_relative_error</th>
          <th>agg__combined__num_gs_absolute_error</th>
          <th>agg__combined__num_gs_absolute_relative_error</th>
          <th>agg__combined__num_gs_absolute_relative_error_log</th>
          <th>agg__combined__tp_samples</th>
          <th>agg__combined__fp_samples</th>
          <th>agg__combined__fn_samples</th>
          <th>agg__combined__precision</th>
          <th>agg__combined__recall</th>
          <th>agg__combined__f1_score</th>
          <th>agg__combined__tn_samples</th>
          <th>agg__combined__specificity</th>
          <th>agg__combined__accuracy</th>
          <th>agg__combined__npv</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.712674</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
          <td>[40.44, 40.82, 65.52]</td>
          <td>[48.12, 42.08, 97.64]</td>
          <td>[7.68, 1.259999999999998, 32.120000000000005]</td>
          <td>[0.18991097922848665, 0.030867221950024448, 0....</td>
          <td>[7.68, 1.259999999999998, 32.120000000000005]</td>
          <td>[0.18991097922848665, 0.030867221950024448, 0....</td>
          <td>[0.17387849695420737, 0.03040041104775553, 0.3...</td>
          <td>[6, 4, 7]</td>
          <td>[6, 3, 6]</td>
          <td>[0, 1, 1]</td>
          <td>[0.0, 0.3333333333333333, 0.16666666666666666]</td>
          <td>[0, 1, 1]</td>
          <td>[0.0, 0.3333333333333333, 0.16666666666666666]</td>
          <td>[0.0, 0.28768207245178085, 0.15415067982725836]</td>
          <td>[2397, 2875, 6436]</td>
          <td>[2421, 1337, 3339]</td>
          <td>[1649, 1209, 117]</td>
          <td>[0.4975093399750934, 0.682573599240266, 0.6584...</td>
          <td>[0.592436974789916, 0.7039666993143977, 0.9821...</td>
          <td>[0.5408393501805054, 0.6931051108968177, 0.788...</td>
          <td>[7316, 10577, 12862]</td>
          <td>[0.7513607887439663, 0.8877790834312573, 0.793...</td>
          <td>[0.704708699122107, 0.8408551068883611, 0.8481...</td>
          <td>[0.8160624651422197, 0.8974206685898524, 0.990...</td>
          <td>[0.010609036999994714, 0.006414417000087269, 0...</td>
          <td>...</td>
          <td>...</td>
          <td>48.926667</td>
          <td>62.613333</td>
          <td>13.686667</td>
          <td>0.237003</td>
          <td>13.686667</td>
          <td>0.237003</td>
          <td>0.20107</td>
          <td>5.666667</td>
          <td>5.0</td>
          <td>0.666667</td>
          <td>0.166667</td>
          <td>0.666667</td>
          <td>0.166667</td>
          <td>0.147278</td>
          <td>3902.666667</td>
          <td>2365.666667</td>
          <td>991.666667</td>
          <td>0.612832</td>
          <td>0.759516</td>
          <td>0.674095</td>
          <td>10251.666667</td>
          <td>0.811014</td>
          <td>0.797893</td>
          <td>0.90149</td>
          <td>0.007851</td>
          <td>146.78</td>
          <td>187.84</td>
          <td>41.06</td>
          <td>0.279738</td>
          <td>41.06</td>
          <td>0.279738</td>
          <td>0.246656</td>
          <td>17</td>
          <td>15</td>
          <td>2</td>
          <td>0.133333</td>
          <td>2</td>
          <td>0.133333</td>
          <td>0.125163</td>
          <td>11708</td>
          <td>7097</td>
          <td>2975</td>
          <td>0.6226</td>
          <td>0.797385</td>
          <td>0.699236</td>
          <td>30755</td>
          <td>0.812507</td>
          <td>0.80828</td>
          <td>0.9118</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 147-150

As you can see, this is a very messy dataframe with a lot of information.
To make this easier to digest, the evaluation object has methods for extracting the different groups of information.
The first group is the aggregated results, which represent only a "single value" over the entire dataset.

.. GENERATED FROM PYTHON SOURCE LINES 150-153

.. code-block:: Python

    agg_results = eval_challenge.get_aggregated_results_as_df()
    agg_results.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th>fold</th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>48.926667</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>62.613333</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>13.686667</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.237003</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>13.686667</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.237003</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.201070</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>5.666667</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>5.000000</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>0.666667</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>0.666667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.147278</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>3902.666667</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2365.666667</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>991.666667</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.612832</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.759516</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.674095</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>10251.666667</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.811014</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.797893</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.901490</td>
        </tr>
        <tr>
          <th>runtime_s</th>
          <td>0.007851</td>
        </tr>
        <tr>
          <th>combined__reference_gs_duration_s</th>
          <td>146.780000</td>
        </tr>
        <tr>
          <th>combined__detected_gs_duration_s</th>
          <td>187.840000</td>
        </tr>
        <tr>
          <th>combined__gs_duration_error_s</th>
          <td>41.060000</td>
        </tr>
        <tr>
          <th>combined__gs_relative_duration_error</th>
          <td>0.279738</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_duration_error_s</th>
          <td>41.060000</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error</th>
          <td>0.279738</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error_log</th>
          <td>0.246656</td>
        </tr>
        <tr>
          <th>combined__detected_num_gs</th>
          <td>17.000000</td>
        </tr>
        <tr>
          <th>combined__reference_num_gs</th>
          <td>15.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_error</th>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_relative_error</th>
          <td>0.133333</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_error</th>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error</th>
          <td>0.133333</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error_log</th>
          <td>0.125163</td>
        </tr>
        <tr>
          <th>combined__tp_samples</th>
          <td>11708.000000</td>
        </tr>
        <tr>
          <th>combined__fp_samples</th>
          <td>7097.000000</td>
        </tr>
        <tr>
          <th>combined__fn_samples</th>
          <td>2975.000000</td>
        </tr>
        <tr>
          <th>combined__precision</th>
          <td>0.622600</td>
        </tr>
        <tr>
          <th>combined__recall</th>
          <td>0.797385</td>
        </tr>
        <tr>
          <th>combined__f1_score</th>
          <td>0.699236</td>
        </tr>
        <tr>
          <th>combined__tn_samples</th>
          <td>30755.000000</td>
        </tr>
        <tr>
          <th>combined__specificity</th>
          <td>0.812507</td>
        </tr>
        <tr>
          <th>combined__accuracy</th>
          <td>0.808280</td>
        </tr>
        <tr>
          <th>combined__npv</th>
          <td>0.911800</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 154-167

You might have seen, that many metrics appear twice, once with a `combined__` prefix and once without.
These represent two different things.
If you check in the source code of the scorer above, the metric without prefix is calculated per datapoint and then
averaged.
The metric with the prefix is calculated over the raw detected gait sequences of all datapoints combined.
Effectively, this is equivalent to different "weightings".
In the aggregated results without prefix, each recording has the same weight, independent of its length.
In the second case, each individual imu-sample has the same weight.
It does not matter, in which recording this sample was classified correctly or not, it has the same impact on the
combined metric.

Both approaches are valid, but you should be aware of the differences when comparing algorithms.
The way how you aggregate here, can have a big impact on the results.

.. GENERATED FROM PYTHON SOURCE LINES 167-178

.. code-block:: Python

    combined_metrics = agg_results.filter(like="combined__").rename(
        columns=lambda x: x.replace("combined__", "")
    )
    combined_vs_per_datapoint = pd.concat(
        {
            "combined": combined_metrics,
            "per_datapoint": agg_results[combined_metrics.columns],
        },
        axis=0,
    )
    combined_vs_per_datapoint.reset_index(level=-1, drop=True).T





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>combined</th>
          <th>per_datapoint</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>146.780000</td>
          <td>48.926667</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>187.840000</td>
          <td>62.613333</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>41.060000</td>
          <td>13.686667</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.279738</td>
          <td>0.237003</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>41.060000</td>
          <td>13.686667</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.279738</td>
          <td>0.237003</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.246656</td>
          <td>0.201070</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>17.000000</td>
          <td>5.666667</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>15.000000</td>
          <td>5.000000</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>2.000000</td>
          <td>0.666667</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.133333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>2.000000</td>
          <td>0.666667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.133333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.125163</td>
          <td>0.147278</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>11708.000000</td>
          <td>3902.666667</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>7097.000000</td>
          <td>2365.666667</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>2975.000000</td>
          <td>991.666667</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.622600</td>
          <td>0.612832</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.797385</td>
          <td>0.759516</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.699236</td>
          <td>0.674095</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>30755.000000</td>
          <td>10251.666667</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.812507</td>
          <td>0.811014</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.808280</td>
          <td>0.797893</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.911800</td>
          <td>0.901490</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 179-180

The "single" results represent the values per datapoint.

.. GENERATED FROM PYTHON SOURCE LINES 180-183

.. code-block:: Python

    single_results = eval_challenge.get_single_results_as_df()
    single_results.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead tr th {
            text-align: left;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr>
          <th>fold</th>
          <th colspan="3" halign="left">0</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th colspan="2" halign="left">HA</th>
          <th>MS</th>
        </tr>
        <tr>
          <th>participant_id</th>
          <th>001</th>
          <th>002</th>
          <th>001</th>
        </tr>
        <tr>
          <th>time_measure</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
          <th>TimeMeasure1</th>
        </tr>
        <tr>
          <th>test</th>
          <th>Test11</th>
          <th>Test11</th>
          <th>Test11</th>
        </tr>
        <tr>
          <th>trial</th>
          <th>Trial1</th>
          <th>Trial1</th>
          <th>Trial1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>40.44</td>
          <td>40.82</td>
          <td>65.52</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>48.12</td>
          <td>42.08</td>
          <td>97.64</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>7.68</td>
          <td>1.26</td>
          <td>32.12</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>7.68</td>
          <td>1.26</td>
          <td>32.12</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.173878</td>
          <td>0.0304</td>
          <td>0.398932</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>6</td>
          <td>4</td>
          <td>7</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>6</td>
          <td>3</td>
          <td>6</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>0</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.0</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>0</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.0</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.0</td>
          <td>0.287682</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>2397</td>
          <td>2875</td>
          <td>6436</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2421</td>
          <td>1337</td>
          <td>3339</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>1649</td>
          <td>1209</td>
          <td>117</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.497509</td>
          <td>0.682574</td>
          <td>0.658414</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.592437</td>
          <td>0.703967</td>
          <td>0.982146</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.540839</td>
          <td>0.693105</td>
          <td>0.788339</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>7316</td>
          <td>10577</td>
          <td>12862</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.751361</td>
          <td>0.887779</td>
          <td>0.793902</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.704709</td>
          <td>0.840855</td>
          <td>0.848115</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.816062</td>
          <td>0.897421</td>
          <td>0.990985</td>
        </tr>
        <tr>
          <th>runtime_s</th>
          <td>0.010609</td>
          <td>0.006414</td>
          <td>0.00653</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 184-187

And finally, we had a couple "raw" results in the scoring, that we passed through without calculating any error
metrics.
These are available as a dictionary of raw results.

.. GENERATED FROM PYTHON SOURCE LINES 187-189

.. code-block:: Python

    raw_results = eval_challenge.get_raw_results()
    list(raw_results.keys())




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ['detected', 'reference']



.. GENERATED FROM PYTHON SOURCE LINES 190-192

.. code-block:: Python

    raw_results["detected"]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="17" valign="top">0</th>
          <th rowspan="10" valign="top">HA</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7800</td>
          <td>9001</td>
        </tr>
        <tr>
          <th>3</th>
          <td>9300</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>4</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>5</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">002</th>
          <th rowspan="4" valign="top">TimeMeasure1</th>
          <th rowspan="4" valign="top">Test11</th>
          <th rowspan="4" valign="top">Trial1</th>
          <th>0</th>
          <td>450</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700</td>
          <td>3301</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5700</td>
          <td>7951</td>
        </tr>
        <tr>
          <th>3</th>
          <td>15000</td>
          <td>15601</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">MS</th>
          <th rowspan="7" valign="top">001</th>
          <th rowspan="7" valign="top">TimeMeasure1</th>
          <th rowspan="7" valign="top">Test11</th>
          <th rowspan="7" valign="top">Trial1</th>
          <th>0</th>
          <td>900</td>
          <td>2101</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650</td>
          <td>6151</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9600</td>
          <td>10801</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11250</td>
          <td>12151</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12300</td>
          <td>14851</td>
        </tr>
        <tr>
          <th>5</th>
          <td>19950</td>
          <td>21151</td>
        </tr>
        <tr>
          <th>6</th>
          <td>21300</td>
          <td>22501</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 193-195

.. code-block:: Python

    raw_results["reference"]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>wb_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="15" valign="top">0</th>
          <th rowspan="9" valign="top">HA</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>632</td>
          <td>988</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2864</td>
          <td>3325</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3853</td>
          <td>5085</td>
        </tr>
        <tr>
          <th>3</th>
          <td>7641</td>
          <td>8621</td>
        </tr>
        <tr>
          <th>4</th>
          <td>9451</td>
          <td>9932</td>
        </tr>
        <tr>
          <th>5</th>
          <td>11989</td>
          <td>12517</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">002</th>
          <th rowspan="3" valign="top">TimeMeasure1</th>
          <th rowspan="3" valign="top">Test11</th>
          <th rowspan="3" valign="top">Trial1</th>
          <th>0</th>
          <td>485</td>
          <td>1131</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1746</td>
          <td>3554</td>
        </tr>
        <tr>
          <th>2</th>
          <td>6083</td>
          <td>7708</td>
        </tr>
        <tr>
          <th rowspan="6" valign="top">MS</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>1019</td>
          <td>1768</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4534</td>
          <td>5549</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9665</td>
          <td>10569</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12337</td>
          <td>14633</td>
        </tr>
        <tr>
          <th>4</th>
          <td>20151</td>
          <td>20982</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21378</td>
          <td>22129</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 196-197

Further, there are some runtime information available (i.e. when the challenge was started, and how long it took).

.. GENERATED FROM PYTHON SOURCE LINES 197-199

.. code-block:: Python

    eval_challenge.perf_["start_datetime"], eval_challenge.perf_["end_datetime"]





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ('2025-07-11T14:37:37.242506+00:00', '2025-07-11T14:37:37.961028+00:00')



.. GENERATED FROM PYTHON SOURCE LINES 200-203

.. code-block:: Python

    eval_challenge.perf_["runtime_s"]






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0.7184769630002847



.. GENERATED FROM PYTHON SOURCE LINES 204-225

Using :class:`~mobgap.utils.evaluation.Evaluation` is great, if you are only comparing (or planning to
compare) non-ML algorithms, or algorithms that don't require further optimization (e.g. through GridSearch).

Therefore, it is generally recommended to run a cross-validation with
:class:`~mobgap.utils.evaluation.EvaluationCV`.
This allows you to evaluate the performance of the algorithm on multiple folds of the dataset and through the use
of :class:`~tpcp.optimize.DummyOptimize` you can also use algorithms without optimization in the same pipeline for
comparison.

Let's demonstrate the use of :class:`~mobgap.utils.evaluation.GsdEvaluationCV` on the example dataset using
the same algorithm once with and once without GridSearch.

For the CV-based challenge, we need to set up a cross-validation.
As we only have 3 datapoints here, we will use a 3-fold cross-validation without grouping or stratification.
In a real-world scenario, you would use a more sophisticated cross-validation strategy.
You can learn more about cross-validation in the `tpcp example
<https://tpcp.readthedocs.io/en/latest/auto_examples/validation/_04_advanced_cross_validation.html>`_.

Further, to speed things up, we are going to use multi-processing.
We can configure this using the ``n_jobs`` parameter that we pass to the internal
:func:`~tpcp.validate.cross_validate` function via the ``cv_params`` parameters

.. GENERATED FROM PYTHON SOURCE LINES 225-234

.. code-block:: Python

    from mobgap.utils.evaluation import EvaluationCV

    eval_challenge_cv = EvaluationCV(
        long_test,
        cv_iterator=3,
        scoring=gsd_evaluation_scorer,
        cv_params={"n_jobs": 2, "return_optimizer": True},
    )








.. GENERATED FROM PYTHON SOURCE LINES 235-237

To use our pipeline from above, we need to wrap it in a :class:`~tpcp.optimize.DummyOptimize` instance.
This will basically skip any optimization on the train set and just apply the pipeline to the test set.

.. GENERATED FROM PYTHON SOURCE LINES 237-243

.. code-block:: Python

    from tpcp.optimize import DummyOptimize

    eval_challenge_cv = eval_challenge_cv.run(
        DummyOptimize(pipe, ignore_potential_user_error_warning=True)
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]
    CV Folds:  33%|███▎      | 1/3 [00:08<00:16,  8.01s/it]
    CV Folds: 100%|██████████| 3/3 [00:08<00:00,  2.20s/it]
    CV Folds: 100%|██████████| 3/3 [00:08<00:00,  2.78s/it]




.. GENERATED FROM PYTHON SOURCE LINES 244-255

The results now are a little bit more complex, as they contain the results for each fold.
In addition, we have information for the train and the test set.
The test set results, are what we are usually looking for.
The train set results, are only calculated when providing the ``return_train_score`` parameter to the ``cv_params``.

As before all results are stored in the `results_` attribute, but it is usually recommended to use the helper methods
to access the data.

Note, that compared to the results above, we now have mutliple CV folds and the aggregated results present one value
per fold.
These parameters could be further aggregated, e.g. by calculating the mean of these values over all folds.

.. GENERATED FROM PYTHON SOURCE LINES 255-258

.. code-block:: Python

    agg_results_cv = eval_challenge_cv.get_aggregated_results_as_df()
    agg_results_cv.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th>fold</th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>40.440000</td>
          <td>40.820000</td>
          <td>65.520000</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>48.120000</td>
          <td>42.080000</td>
          <td>97.640000</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>7.680000</td>
          <td>1.260000</td>
          <td>32.120000</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>7.680000</td>
          <td>1.260000</td>
          <td>32.120000</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.173878</td>
          <td>0.030400</td>
          <td>0.398932</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>6.000000</td>
          <td>4.000000</td>
          <td>7.000000</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>6.000000</td>
          <td>3.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.000000</td>
          <td>0.287682</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>2397.000000</td>
          <td>2875.000000</td>
          <td>6436.000000</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2421.000000</td>
          <td>1337.000000</td>
          <td>3339.000000</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>1649.000000</td>
          <td>1209.000000</td>
          <td>117.000000</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.497509</td>
          <td>0.682574</td>
          <td>0.658414</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.592437</td>
          <td>0.703967</td>
          <td>0.982146</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.540839</td>
          <td>0.693105</td>
          <td>0.788339</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>7316.000000</td>
          <td>10577.000000</td>
          <td>12862.000000</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.751361</td>
          <td>0.887779</td>
          <td>0.793902</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.704709</td>
          <td>0.840855</td>
          <td>0.848115</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.816062</td>
          <td>0.897421</td>
          <td>0.990985</td>
        </tr>
        <tr>
          <th>runtime_s</th>
          <td>1.420231</td>
          <td>1.352697</td>
          <td>0.006638</td>
        </tr>
        <tr>
          <th>combined__reference_gs_duration_s</th>
          <td>40.440000</td>
          <td>40.820000</td>
          <td>65.520000</td>
        </tr>
        <tr>
          <th>combined__detected_gs_duration_s</th>
          <td>48.120000</td>
          <td>42.080000</td>
          <td>97.640000</td>
        </tr>
        <tr>
          <th>combined__gs_duration_error_s</th>
          <td>7.680000</td>
          <td>1.260000</td>
          <td>32.120000</td>
        </tr>
        <tr>
          <th>combined__gs_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_duration_error_s</th>
          <td>7.680000</td>
          <td>1.260000</td>
          <td>32.120000</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.030867</td>
          <td>0.490232</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error_log</th>
          <td>0.173878</td>
          <td>0.030400</td>
          <td>0.398932</td>
        </tr>
        <tr>
          <th>combined__detected_num_gs</th>
          <td>6.000000</td>
          <td>4.000000</td>
          <td>7.000000</td>
        </tr>
        <tr>
          <th>combined__reference_num_gs</th>
          <td>6.000000</td>
          <td>3.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error_log</th>
          <td>0.000000</td>
          <td>0.287682</td>
          <td>0.154151</td>
        </tr>
        <tr>
          <th>combined__tp_samples</th>
          <td>2397.000000</td>
          <td>2875.000000</td>
          <td>6436.000000</td>
        </tr>
        <tr>
          <th>combined__fp_samples</th>
          <td>2421.000000</td>
          <td>1337.000000</td>
          <td>3339.000000</td>
        </tr>
        <tr>
          <th>combined__fn_samples</th>
          <td>1649.000000</td>
          <td>1209.000000</td>
          <td>117.000000</td>
        </tr>
        <tr>
          <th>combined__precision</th>
          <td>0.497509</td>
          <td>0.682574</td>
          <td>0.658414</td>
        </tr>
        <tr>
          <th>combined__recall</th>
          <td>0.592437</td>
          <td>0.703967</td>
          <td>0.982146</td>
        </tr>
        <tr>
          <th>combined__f1_score</th>
          <td>0.540839</td>
          <td>0.693105</td>
          <td>0.788339</td>
        </tr>
        <tr>
          <th>combined__tn_samples</th>
          <td>7316.000000</td>
          <td>10577.000000</td>
          <td>12862.000000</td>
        </tr>
        <tr>
          <th>combined__specificity</th>
          <td>0.751361</td>
          <td>0.887779</td>
          <td>0.793902</td>
        </tr>
        <tr>
          <th>combined__accuracy</th>
          <td>0.704709</td>
          <td>0.840855</td>
          <td>0.848115</td>
        </tr>
        <tr>
          <th>combined__npv</th>
          <td>0.816062</td>
          <td>0.897421</td>
          <td>0.990985</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 259-262

The single results contain the CV fold as an additional index.
Otherwise, the output is identical to before.
Note, that if you use anything else then a KFold, splitter, you might have some datapoints duplicated across folds.

.. GENERATED FROM PYTHON SOURCE LINES 262-265

.. code-block:: Python

    single_results_cv = eval_challenge_cv.get_single_results_as_df()
    single_results_cv






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>reference_gs_duration_s</th>
          <th>detected_gs_duration_s</th>
          <th>gs_duration_error_s</th>
          <th>gs_relative_duration_error</th>
          <th>gs_absolute_duration_error_s</th>
          <th>gs_absolute_relative_duration_error</th>
          <th>gs_absolute_relative_duration_error_log</th>
          <th>detected_num_gs</th>
          <th>reference_num_gs</th>
          <th>num_gs_error</th>
          <th>num_gs_relative_error</th>
          <th>num_gs_absolute_error</th>
          <th>num_gs_absolute_relative_error</th>
          <th>num_gs_absolute_relative_error_log</th>
          <th>tp_samples</th>
          <th>fp_samples</th>
          <th>fn_samples</th>
          <th>precision</th>
          <th>recall</th>
          <th>f1_score</th>
          <th>tn_samples</th>
          <th>specificity</th>
          <th>accuracy</th>
          <th>npv</th>
          <th>runtime_s</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <th>HA</th>
          <th>001</th>
          <th>TimeMeasure1</th>
          <th>Test11</th>
          <th>Trial1</th>
          <td>40.44</td>
          <td>48.12</td>
          <td>7.68</td>
          <td>0.189911</td>
          <td>7.68</td>
          <td>0.189911</td>
          <td>0.173878</td>
          <td>6</td>
          <td>6</td>
          <td>0</td>
          <td>0.0</td>
          <td>0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>2397</td>
          <td>2421</td>
          <td>1649</td>
          <td>0.497509</td>
          <td>0.592437</td>
          <td>0.540839</td>
          <td>7316</td>
          <td>0.751361</td>
          <td>0.704709</td>
          <td>0.816062</td>
          <td>1.420231</td>
        </tr>
        <tr>
          <th>1</th>
          <th>HA</th>
          <th>002</th>
          <th>TimeMeasure1</th>
          <th>Test11</th>
          <th>Trial1</th>
          <td>40.82</td>
          <td>42.08</td>
          <td>1.26</td>
          <td>0.030867</td>
          <td>1.26</td>
          <td>0.030867</td>
          <td>0.0304</td>
          <td>4</td>
          <td>3</td>
          <td>1</td>
          <td>0.333333</td>
          <td>1</td>
          <td>0.333333</td>
          <td>0.287682</td>
          <td>2875</td>
          <td>1337</td>
          <td>1209</td>
          <td>0.682574</td>
          <td>0.703967</td>
          <td>0.693105</td>
          <td>10577</td>
          <td>0.887779</td>
          <td>0.840855</td>
          <td>0.897421</td>
          <td>1.352697</td>
        </tr>
        <tr>
          <th>2</th>
          <th>MS</th>
          <th>001</th>
          <th>TimeMeasure1</th>
          <th>Test11</th>
          <th>Trial1</th>
          <td>65.52</td>
          <td>97.64</td>
          <td>32.12</td>
          <td>0.490232</td>
          <td>32.12</td>
          <td>0.490232</td>
          <td>0.398932</td>
          <td>7</td>
          <td>6</td>
          <td>1</td>
          <td>0.166667</td>
          <td>1</td>
          <td>0.166667</td>
          <td>0.154151</td>
          <td>6436</td>
          <td>3339</td>
          <td>117</td>
          <td>0.658414</td>
          <td>0.982146</td>
          <td>0.788339</td>
          <td>12862</td>
          <td>0.793902</td>
          <td>0.848115</td>
          <td>0.990985</td>
          <td>0.006638</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 266-267

And the raw outputs:

.. GENERATED FROM PYTHON SOURCE LINES 267-270

.. code-block:: Python

    raw_results_cv = eval_challenge_cv.get_raw_results()
    raw_results_cv["detected"]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>fold</th>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="6" valign="top">0</th>
          <th rowspan="6" valign="top">HA</th>
          <th rowspan="6" valign="top">001</th>
          <th rowspan="6" valign="top">TimeMeasure1</th>
          <th rowspan="6" valign="top">Test11</th>
          <th rowspan="6" valign="top">Trial1</th>
          <th>0</th>
          <td>600</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4350</td>
          <td>5251</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7800</td>
          <td>9001</td>
        </tr>
        <tr>
          <th>3</th>
          <td>9300</td>
          <td>10201</td>
        </tr>
        <tr>
          <th>4</th>
          <td>10950</td>
          <td>11551</td>
        </tr>
        <tr>
          <th>5</th>
          <td>13050</td>
          <td>13651</td>
        </tr>
        <tr>
          <th rowspan="4" valign="top">1</th>
          <th rowspan="4" valign="top">HA</th>
          <th rowspan="4" valign="top">002</th>
          <th rowspan="4" valign="top">TimeMeasure1</th>
          <th rowspan="4" valign="top">Test11</th>
          <th rowspan="4" valign="top">Trial1</th>
          <th>0</th>
          <td>450</td>
          <td>1201</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2700</td>
          <td>3301</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5700</td>
          <td>7951</td>
        </tr>
        <tr>
          <th>3</th>
          <td>15000</td>
          <td>15601</td>
        </tr>
        <tr>
          <th rowspan="7" valign="top">2</th>
          <th rowspan="7" valign="top">MS</th>
          <th rowspan="7" valign="top">001</th>
          <th rowspan="7" valign="top">TimeMeasure1</th>
          <th rowspan="7" valign="top">Test11</th>
          <th rowspan="7" valign="top">Trial1</th>
          <th>0</th>
          <td>900</td>
          <td>2101</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4650</td>
          <td>6151</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9600</td>
          <td>10801</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11250</td>
          <td>12151</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12300</td>
          <td>14851</td>
        </tr>
        <tr>
          <th>5</th>
          <td>19950</td>
          <td>21151</td>
        </tr>
        <tr>
          <th>6</th>
          <td>21300</td>
          <td>22501</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 271-278

If we compare these results to the ones from the non-CV challenge, we can see that "single" results are identical,
just that they were called in multiple folds.
This is expected, as we used :class:`~tpcp.optimize.DummyOptimize` and thus didn't optimize the algorithm.

Let's try a :class:`~tpcp.optimize.GridSearch` on the algorithm to see how the results change.
For the gridsearch, we will re-use the same scoring function as before, but we need to specify, which scoring result
we want to optimize for.

.. GENERATED FROM PYTHON SOURCE LINES 278-286

.. code-block:: Python

    from sklearn.model_selection import ParameterGrid
    from tpcp.optimize import GridSearch

    para_grid = ParameterGrid({"algo__window_length_s": [2, 3, 4]})
    optimizer = GridSearch(
        pipe, para_grid, scoring=gsd_evaluation_scorer, return_optimized="precision"
    )








.. GENERATED FROM PYTHON SOURCE LINES 287-290

The optimizer can now be used in the same CV challenge as before.
This way we can guarantee that the same folds are used for the optimization and the evaluation and ensure the best
possible comparison between the algorithms versions.

.. GENERATED FROM PYTHON SOURCE LINES 290-292

.. code-block:: Python

    eval_challenge_gs = eval_challenge_cv.clone().run(optimizer)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]
    CV Folds:  33%|███▎      | 1/3 [00:03<00:07,  3.85s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.56s/it]
    CV Folds: 100%|██████████| 3/3 [00:05<00:00,  1.79s/it]




.. GENERATED FROM PYTHON SOURCE LINES 293-294

The results we are seeing now are generated by the internally optimized version of the algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 294-297

.. code-block:: Python

    agg_results_cv = eval_challenge_gs.get_aggregated_results_as_df()
    agg_results_cv.T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th>fold</th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>reference_gs_duration_s</th>
          <td>40.440000</td>
          <td>40.820000</td>
          <td>65.520000</td>
        </tr>
        <tr>
          <th>detected_gs_duration_s</th>
          <td>48.120000</td>
          <td>39.080000</td>
          <td>97.160000</td>
        </tr>
        <tr>
          <th>gs_duration_error_s</th>
          <td>7.680000</td>
          <td>-1.740000</td>
          <td>31.640000</td>
        </tr>
        <tr>
          <th>gs_relative_duration_error</th>
          <td>0.189911</td>
          <td>-0.042626</td>
          <td>0.482906</td>
        </tr>
        <tr>
          <th>gs_absolute_duration_error_s</th>
          <td>7.680000</td>
          <td>1.740000</td>
          <td>31.640000</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.042626</td>
          <td>0.482906</td>
        </tr>
        <tr>
          <th>gs_absolute_relative_duration_error_log</th>
          <td>0.173878</td>
          <td>0.041743</td>
          <td>0.394004</td>
        </tr>
        <tr>
          <th>detected_num_gs</th>
          <td>6.000000</td>
          <td>4.000000</td>
          <td>8.000000</td>
        </tr>
        <tr>
          <th>reference_num_gs</th>
          <td>6.000000</td>
          <td>3.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>num_gs_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>num_gs_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>num_gs_absolute_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>num_gs_absolute_relative_error_log</th>
          <td>0.000000</td>
          <td>0.287682</td>
          <td>0.287682</td>
        </tr>
        <tr>
          <th>tp_samples</th>
          <td>2397.000000</td>
          <td>2545.000000</td>
          <td>6153.000000</td>
        </tr>
        <tr>
          <th>fp_samples</th>
          <td>2421.000000</td>
          <td>1366.000000</td>
          <td>3573.000000</td>
        </tr>
        <tr>
          <th>fn_samples</th>
          <td>1649.000000</td>
          <td>1540.000000</td>
          <td>403.000000</td>
        </tr>
        <tr>
          <th>precision</th>
          <td>0.497509</td>
          <td>0.650729</td>
          <td>0.632634</td>
        </tr>
        <tr>
          <th>recall</th>
          <td>0.592437</td>
          <td>0.623011</td>
          <td>0.938530</td>
        </tr>
        <tr>
          <th>f1_score</th>
          <td>0.540839</td>
          <td>0.636568</td>
          <td>0.755804</td>
        </tr>
        <tr>
          <th>tn_samples</th>
          <td>7316.000000</td>
          <td>10547.000000</td>
          <td>12627.000000</td>
        </tr>
        <tr>
          <th>specificity</th>
          <td>0.751361</td>
          <td>0.885335</td>
          <td>0.779444</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.704709</td>
          <td>0.818352</td>
          <td>0.825277</td>
        </tr>
        <tr>
          <th>npv</th>
          <td>0.816062</td>
          <td>0.872590</td>
          <td>0.969071</td>
        </tr>
        <tr>
          <th>runtime_s</th>
          <td>0.014230</td>
          <td>0.006469</td>
          <td>0.006676</td>
        </tr>
        <tr>
          <th>combined__reference_gs_duration_s</th>
          <td>40.440000</td>
          <td>40.820000</td>
          <td>65.520000</td>
        </tr>
        <tr>
          <th>combined__detected_gs_duration_s</th>
          <td>48.120000</td>
          <td>39.080000</td>
          <td>97.160000</td>
        </tr>
        <tr>
          <th>combined__gs_duration_error_s</th>
          <td>7.680000</td>
          <td>-1.740000</td>
          <td>31.640000</td>
        </tr>
        <tr>
          <th>combined__gs_relative_duration_error</th>
          <td>0.189911</td>
          <td>-0.042626</td>
          <td>0.482906</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_duration_error_s</th>
          <td>7.680000</td>
          <td>1.740000</td>
          <td>31.640000</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error</th>
          <td>0.189911</td>
          <td>0.042626</td>
          <td>0.482906</td>
        </tr>
        <tr>
          <th>combined__gs_absolute_relative_duration_error_log</th>
          <td>0.173878</td>
          <td>0.041743</td>
          <td>0.394004</td>
        </tr>
        <tr>
          <th>combined__detected_num_gs</th>
          <td>6.000000</td>
          <td>4.000000</td>
          <td>8.000000</td>
        </tr>
        <tr>
          <th>combined__reference_num_gs</th>
          <td>6.000000</td>
          <td>3.000000</td>
          <td>6.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_error</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>2.000000</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error</th>
          <td>0.000000</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>combined__num_gs_absolute_relative_error_log</th>
          <td>0.000000</td>
          <td>0.287682</td>
          <td>0.287682</td>
        </tr>
        <tr>
          <th>combined__tp_samples</th>
          <td>2397.000000</td>
          <td>2545.000000</td>
          <td>6153.000000</td>
        </tr>
        <tr>
          <th>combined__fp_samples</th>
          <td>2421.000000</td>
          <td>1366.000000</td>
          <td>3573.000000</td>
        </tr>
        <tr>
          <th>combined__fn_samples</th>
          <td>1649.000000</td>
          <td>1540.000000</td>
          <td>403.000000</td>
        </tr>
        <tr>
          <th>combined__precision</th>
          <td>0.497509</td>
          <td>0.650729</td>
          <td>0.632634</td>
        </tr>
        <tr>
          <th>combined__recall</th>
          <td>0.592437</td>
          <td>0.623011</td>
          <td>0.938530</td>
        </tr>
        <tr>
          <th>combined__f1_score</th>
          <td>0.540839</td>
          <td>0.636568</td>
          <td>0.755804</td>
        </tr>
        <tr>
          <th>combined__tn_samples</th>
          <td>7316.000000</td>
          <td>10547.000000</td>
          <td>12627.000000</td>
        </tr>
        <tr>
          <th>combined__specificity</th>
          <td>0.751361</td>
          <td>0.885335</td>
          <td>0.779444</td>
        </tr>
        <tr>
          <th>combined__accuracy</th>
          <td>0.704709</td>
          <td>0.818352</td>
          <td>0.825277</td>
        </tr>
        <tr>
          <th>combined__npv</th>
          <td>0.816062</td>
          <td>0.872590</td>
          <td>0.969071</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 298-301

Because we used ``cv_params={"return_optimizer": True}`` we can also access the optimizer per fold directly from
the ``results_`` attribute.`
This can be useful to get more insights into the optimization process and what the optimal parameters were.

.. GENERATED FROM PYTHON SOURCE LINES 301-304

.. code-block:: Python

    opt_results = pd.Series(eval_challenge_gs.results_["optimizer"])
    opt_results





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0    GridSearch(n_jobs=None, parameter_grid=<sklear...
    1    GridSearch(n_jobs=None, parameter_grid=<sklear...
    2    GridSearch(n_jobs=None, parameter_grid=<sklear...
    dtype: object



.. GENERATED FROM PYTHON SOURCE LINES 305-306

We can get the best parameters per fold by directly interacting with the optimizer instances.

.. GENERATED FROM PYTHON SOURCE LINES 306-309

.. code-block:: Python

    best_params = opt_results.apply(lambda x: pd.Series(x.best_params_))
    best_params






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>algo__window_length_s</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>3</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 310-312

Or we can go much deeper, by getting all information about the optimization process.
Let's just look at the keys of the information that is available.

.. GENERATED FROM PYTHON SOURCE LINES 312-315

.. code-block:: Python

    all_opti_results_fold0 = pd.DataFrame(opt_results.loc[0].gs_results_)
    all_opti_results_fold0.columns.to_list()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ['agg__reference_gs_duration_s', 'rank__agg__reference_gs_duration_s', 'agg__detected_gs_duration_s', 'rank__agg__detected_gs_duration_s', 'agg__gs_duration_error_s', 'rank__agg__gs_duration_error_s', 'agg__gs_relative_duration_error', 'rank__agg__gs_relative_duration_error', 'agg__gs_absolute_duration_error_s', 'rank__agg__gs_absolute_duration_error_s', 'agg__gs_absolute_relative_duration_error', 'rank__agg__gs_absolute_relative_duration_error', 'agg__gs_absolute_relative_duration_error_log', 'rank__agg__gs_absolute_relative_duration_error_log', 'agg__detected_num_gs', 'rank__agg__detected_num_gs', 'agg__reference_num_gs', 'rank__agg__reference_num_gs', 'agg__num_gs_error', 'rank__agg__num_gs_error', 'agg__num_gs_relative_error', 'rank__agg__num_gs_relative_error', 'agg__num_gs_absolute_error', 'rank__agg__num_gs_absolute_error', 'agg__num_gs_absolute_relative_error', 'rank__agg__num_gs_absolute_relative_error', 'agg__num_gs_absolute_relative_error_log', 'rank__agg__num_gs_absolute_relative_error_log', 'agg__tp_samples', 'rank__agg__tp_samples', 'agg__fp_samples', 'rank__agg__fp_samples', 'agg__fn_samples', 'rank__agg__fn_samples', 'agg__precision', 'rank__agg__precision', 'agg__recall', 'rank__agg__recall', 'agg__f1_score', 'rank__agg__f1_score', 'agg__tn_samples', 'rank__agg__tn_samples', 'agg__specificity', 'rank__agg__specificity', 'agg__accuracy', 'rank__agg__accuracy', 'agg__npv', 'rank__agg__npv', 'agg__runtime_s', 'rank__agg__runtime_s', 'agg__combined__reference_gs_duration_s', 'rank__agg__combined__reference_gs_duration_s', 'agg__combined__detected_gs_duration_s', 'rank__agg__combined__detected_gs_duration_s', 'agg__combined__gs_duration_error_s', 'rank__agg__combined__gs_duration_error_s', 'agg__combined__gs_relative_duration_error', 'rank__agg__combined__gs_relative_duration_error', 'agg__combined__gs_absolute_duration_error_s', 'rank__agg__combined__gs_absolute_duration_error_s', 'agg__combined__gs_absolute_relative_duration_error', 'rank__agg__combined__gs_absolute_relative_duration_error', 'agg__combined__gs_absolute_relative_duration_error_log', 'rank__agg__combined__gs_absolute_relative_duration_error_log', 'agg__combined__detected_num_gs', 'rank__agg__combined__detected_num_gs', 'agg__combined__reference_num_gs', 'rank__agg__combined__reference_num_gs', 'agg__combined__num_gs_error', 'rank__agg__combined__num_gs_error', 'agg__combined__num_gs_relative_error', 'rank__agg__combined__num_gs_relative_error', 'agg__combined__num_gs_absolute_error', 'rank__agg__combined__num_gs_absolute_error', 'agg__combined__num_gs_absolute_relative_error', 'rank__agg__combined__num_gs_absolute_relative_error', 'agg__combined__num_gs_absolute_relative_error_log', 'rank__agg__combined__num_gs_absolute_relative_error_log', 'agg__combined__tp_samples', 'rank__agg__combined__tp_samples', 'agg__combined__fp_samples', 'rank__agg__combined__fp_samples', 'agg__combined__fn_samples', 'rank__agg__combined__fn_samples', 'agg__combined__precision', 'rank__agg__combined__precision', 'agg__combined__recall', 'rank__agg__combined__recall', 'agg__combined__f1_score', 'rank__agg__combined__f1_score', 'agg__combined__tn_samples', 'rank__agg__combined__tn_samples', 'agg__combined__specificity', 'rank__agg__combined__specificity', 'agg__combined__accuracy', 'rank__agg__combined__accuracy', 'agg__combined__npv', 'rank__agg__combined__npv', 'single__reference_gs_duration_s', 'single__detected_gs_duration_s', 'single__gs_duration_error_s', 'single__gs_relative_duration_error', 'single__gs_absolute_duration_error_s', 'single__gs_absolute_relative_duration_error', 'single__gs_absolute_relative_duration_error_log', 'single__detected_num_gs', 'single__reference_num_gs', 'single__num_gs_error', 'single__num_gs_relative_error', 'single__num_gs_absolute_error', 'single__num_gs_absolute_relative_error', 'single__num_gs_absolute_relative_error_log', 'single__tp_samples', 'single__fp_samples', 'single__fn_samples', 'single__precision', 'single__recall', 'single__f1_score', 'single__tn_samples', 'single__specificity', 'single__accuracy', 'single__npv', 'single__runtime_s', 'single__raw__detected', 'single__raw__reference', 'data_labels', 'debug__score_time', 'param__algo__window_length_s', 'params']



.. GENERATED FROM PYTHON SOURCE LINES 316-320

With that, we hope it becomes clear, how these challenges can be extremely valuable, when benchmarking algorithms
across datasets.
To see how we evaluate the performance of the algorithms available in mobgap, check out the other gsd evaluation
examples.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 21.736 seconds)

**Estimated memory usage:**  80 MB


.. _sphx_glr_download_auto_examples_gait_sequences__04_gsd_evaluation_challenges.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _04_gsd_evaluation_challenges.ipynb <_04_gsd_evaluation_challenges.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _04_gsd_evaluation_challenges.py <_04_gsd_evaluation_challenges.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: _04_gsd_evaluation_challenges.zip <_04_gsd_evaluation_challenges.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_