.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/laterality/_99_lrc_evaluation.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_laterality__99_lrc_evaluation.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_laterality__99_lrc_evaluation.py:


.. _lrc_evaluation:

LRC Evaluation
==============

This example demonstrates how to evaluate an LRC algorithm.
As left-right classification, is a balanced binary classification problem, we can apply simple metrics like accuracy to
evaluate the performance of the algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 12-19

.. code-block:: Python


    import pandas as pd
    from mobgap.data import LabExampleDataset
    from mobgap.laterality import LrcUllrich
    from mobgap.pipeline import GsIterator
    from mobgap.utils.conversions import to_body_frame








.. GENERATED FROM PYTHON SOURCE LINES 20-26

Loading some example data
-------------------------
First, we load some example data and apply the LrcUllrich algorithm with its default pre-trained model to it.
We use the reference initial contacts as input for the algorithm so that we can focus on the evaluation of the
L/R classification independently of the detection of the initial contacts.
However, you can use any other algorithm as well.

.. GENERATED FROM PYTHON SOURCE LINES 26-69

.. code-block:: Python



    def load_data():
        lab_example_data = LabExampleDataset(reference_system="INDIP")
        single_test = lab_example_data.get_subset(
            cohort="MS", participant_id="001", test="Test11", trial="Trial1"
        )
        return single_test


    def calculate_output(single_test_data):
        """Calculate the GSD Iluz output per WB."""
        iterator = GsIterator()
        ref_paras = single_test_data.reference_parameters_relative_to_wb_

        for (gs, data), r in iterator.iterate(
            to_body_frame(single_test_data.data_ss), ref_paras.wb_list
        ):
            ref_ics = ref_paras.ic_list.loc[gs.id]
            r.ic_list = (
                LrcUllrich()
                .predict(
                    data,
                    ref_ics,
                    sampling_rate_hz=single_test_data.sampling_rate_hz,
                )
                .ic_lr_list_
            )

        return iterator.results_.ic_list


    def load_reference(single_test_data):
        """Load the reference gait sequences from the test data."""
        ref_gsd = single_test_data.reference_parameters_.ic_list
        return ref_gsd


    test_data = load_data()
    calculated_ic_lr_list = calculate_output(test_data)
    reference_ic_lr_list = load_reference(test_data)









.. GENERATED FROM PYTHON SOURCE LINES 70-72

We can see that the calculated and the reference ic_list have the same structure with the ``lr_label`` column
providing the detected label per initial contact.

.. GENERATED FROM PYTHON SOURCE LINES 72-74

.. code-block:: Python

    calculated_ic_lr_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>1019</td>
          <td>right</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1065</td>
          <td>left</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1129</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>1172</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>1236</td>
          <td>right</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>7</th>
          <td>21896</td>
          <td>left</td>
        </tr>
        <tr>
          <th>8</th>
          <td>21951</td>
          <td>right</td>
        </tr>
        <tr>
          <th>9</th>
          <td>22042</td>
          <td>left</td>
        </tr>
        <tr>
          <th>10</th>
          <td>22089</td>
          <td>left</td>
        </tr>
        <tr>
          <th>11</th>
          <td>22128</td>
          <td>right</td>
        </tr>
      </tbody>
    </table>
    <p>91 rows × 2 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 75-77

.. code-block:: Python

    reference_ic_lr_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>1019</td>
          <td>right</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1065</td>
          <td>left</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1129</td>
          <td>right</td>
        </tr>
        <tr>
          <th>3</th>
          <td>1172</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>1236</td>
          <td>right</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>7</th>
          <td>21896</td>
          <td>right</td>
        </tr>
        <tr>
          <th>8</th>
          <td>21951</td>
          <td>left</td>
        </tr>
        <tr>
          <th>9</th>
          <td>22042</td>
          <td>right</td>
        </tr>
        <tr>
          <th>10</th>
          <td>22089</td>
          <td>left</td>
        </tr>
        <tr>
          <th>11</th>
          <td>22128</td>
          <td>right</td>
        </tr>
      </tbody>
    </table>
    <p>91 rows × 2 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 78-81

Visual comparison of the detected and reference labels
------------------------------------------------------
One easy way to compare the results is to visualize them as colorful bars.

.. GENERATED FROM PYTHON SOURCE LINES 81-110

.. code-block:: Python


    import matplotlib.pyplot as plt


    def plot_lr(ref, detected):
        fig, ax = plt.subplots(figsize=(15, 5))
        # We plot one box either (red or blue depending on the laterality) for each detected IC ignoring the actual time
        for (_, row), (_, ref_row) in zip(detected.iterrows(), ref.iterrows()):
            ax.plot(
                [row["ic"], row["ic"]],
                [0, 0.98],
                color="r" if row["lr_label"] == "left" else "b",
                linewidth=5,
            )
            ax.plot(
                [ref_row["ic"], ref_row["ic"]],
                [1.02, 2],
                color="r" if ref_row["lr_label"] == "left" else "b",
                linewidth=5,
            )

        ax.set_yticks([0.5, 1.5])
        ax.set_yticklabels(["Detected", "Reference"])
        return fig, ax


    fig, _ = plot_lr(reference_ic_lr_list, calculated_ic_lr_list)
    fig.show()




.. image-sg:: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_001.png
   :alt:  99 lrc evaluation
   :srcset: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 111-113

If we zoom in on a longer WB, we can see that for some ICs the L/R label does not match.
But, in particular for regular gait in the center of the WB, the labels match quite well.

.. GENERATED FROM PYTHON SOURCE LINES 113-118

.. code-block:: Python


    fig, ax = plot_lr(reference_ic_lr_list, calculated_ic_lr_list)
    ax.set_xlim(12000, 15000)
    fig.show()




.. image-sg:: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_002.png
   :alt:  99 lrc evaluation
   :srcset: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 119-122

Calculating evaluation metrics
------------------------------
We can also quantify the agreement between the detected and the reference labels using typical classification metrics.

.. GENERATED FROM PYTHON SOURCE LINES 122-140

.. code-block:: Python

    from sklearn.metrics import classification_report

    if not reference_ic_lr_list.empty:
        lr_classification_report = pd.DataFrame(
            classification_report(
                reference_ic_lr_list["lr_label"],
                calculated_ic_lr_list["lr_label"],
                target_names=["left", "right"],
                output_dict=True,
            )
        ).T
    else:
        lr_classification_report = pd.DataFrame(
            columns=["precision", "recall", "f1-score", "support"]
        )

    lr_classification_report






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>precision</th>
          <th>recall</th>
          <th>f1-score</th>
          <th>support</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>left</th>
          <td>0.782609</td>
          <td>0.720000</td>
          <td>0.750000</td>
          <td>50.000000</td>
        </tr>
        <tr>
          <th>right</th>
          <td>0.688889</td>
          <td>0.756098</td>
          <td>0.720930</td>
          <td>41.000000</td>
        </tr>
        <tr>
          <th>accuracy</th>
          <td>0.736264</td>
          <td>0.736264</td>
          <td>0.736264</td>
          <td>0.736264</td>
        </tr>
        <tr>
          <th>macro avg</th>
          <td>0.735749</td>
          <td>0.738049</td>
          <td>0.735465</td>
          <td>91.000000</td>
        </tr>
        <tr>
          <th>weighted avg</th>
          <td>0.740383</td>
          <td>0.736264</td>
          <td>0.736903</td>
          <td>91.000000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 141-143

In general we focus on the accuracy, as it is a balanced binary classification problem.
If you only want to calculate this you can just calculate the ``accuracy_score``

.. GENERATED FROM PYTHON SOURCE LINES 143-154

.. code-block:: Python

    from sklearn.metrics import accuracy_score

    lr_accuracy = (
        accuracy_score(
            reference_ic_lr_list["lr_label"], calculated_ic_lr_list["lr_label"]
        )
        if not reference_ic_lr_list.empty
        else float("nan")
    )
    lr_accuracy





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0.7362637362637363



.. GENERATED FROM PYTHON SOURCE LINES 155-156

Similarly, we could create a confusion matrix to get more insights into the performance of the algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 156-164

.. code-block:: Python

    from sklearn.metrics import ConfusionMatrixDisplay

    if not reference_ic_lr_list.empty:
        disp = ConfusionMatrixDisplay.from_predictions(
            reference_ic_lr_list["lr_label"], calculated_ic_lr_list["lr_label"]
        )
        disp.figure_.show()




.. image-sg:: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_003.png
   :alt:  99 lrc evaluation
   :srcset: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 165-173

Running a full evaluation pipeline
----------------------------------
Instead of manually evaluating and investigating the performance of an algorithm on a single piece of data, we
often want to run a full evaluation on an entire dataset.
This can be done using the :class:`~mobgap.laterality.base.LrdPipeline` class and some ``tpcp`` functions.

But let's start with selecting some data.
We want to use all the simulated real-world walking data from the INDIP reference system (Test11).

.. GENERATED FROM PYTHON SOURCE LINES 173-179

.. code-block:: Python

    simulated_real_world_walking = LabExampleDataset(
        reference_system="INDIP"
    ).get_subset(test="Test11")

    simulated_real_world_walking






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <h4 style="margin-bottom: 0.1em;">LabExampleDataset [3 groups/rows]</h3>
    <div style="margin-top: 0em">
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table style="margin-left: 3em;">
      <thead>
        <tr style="text-align: right;">
          <th style="text-align: center;"></th>
          <th style="text-align: center;">cohort</th>
          <th style="text-align: center;">participant_id</th>
          <th style="text-align: center;">time_measure</th>
          <th style="text-align: center;">test</th>
          <th style="text-align: center;">trial</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th style="text-align: center;">0</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">HA</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">001</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
        <tr>
          <th style="text-align: center;">1</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">HA</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">002</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
        <tr>
          <th style="text-align: center;">2</th>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">MS</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">001</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">TimeMeasure1</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Test11</td>
          <td style="text-align: center; padding-left: 2em; padding-right: 2em;">Trial1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 180-181

Now we can create a pipeline instance and directly run it on of the datapoints of the dataset.

.. GENERATED FROM PYTHON SOURCE LINES 181-187

.. code-block:: Python

    from mobgap.laterality.pipeline import LrcEmulationPipeline

    pipeline = LrcEmulationPipeline(LrcUllrich())

    pipeline.safe_run(simulated_real_world_walking[0]).ic_lr_list_






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
          <th>lr_label</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>632</td>
          <td>right</td>
        </tr>
        <tr>
          <th>1</th>
          <td>709</td>
          <td>left</td>
        </tr>
        <tr>
          <th>2</th>
          <td>763</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>824</td>
          <td>right</td>
        </tr>
        <tr>
          <th>4</th>
          <td>876</td>
          <td>left</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>3</th>
          <td>12162</td>
          <td>left</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12220</td>
          <td>right</td>
        </tr>
        <tr>
          <th>5</th>
          <td>12277</td>
          <td>left</td>
        </tr>
        <tr>
          <th>6</th>
          <td>12335</td>
          <td>right</td>
        </tr>
        <tr>
          <th>7</th>
          <td>12516</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    <p>63 rows × 2 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 188-194

This is exactly what we did before, just on a pipeline level, without manually extracting the data from the dataset.
To now actually run a validation, we need to iterate over all datapoints and calculate the accuracy for each of them.
This can be done using the :func:`~tpcp.validate.validate` function.

Note, that the ``LrdPipeline`` class already has a ``score`` method that returns the accuracy.
This is used by default, but you could supply your own scoring method as well.

.. GENERATED FROM PYTHON SOURCE LINES 194-206

.. code-block:: Python

    from mobgap.laterality.evaluation import lrc_score
    from tpcp.validate import validate

    evaluation_results_with_opti = pd.DataFrame(
        validate(
            pipeline,
            simulated_real_world_walking,
            scoring=lrc_score,
        )
    )
    evaluation_results_with_opti.drop(["single__raw__predictions"], axis=1).T





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Datapoints:   0%|          | 0/3 [00:00<?, ?it/s]
    Datapoints:  33%|███▎      | 1/3 [00:00<00:00,  4.44it/s]
    Datapoints:  67%|██████▋   | 2/3 [00:00<00:00,  4.77it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  4.43it/s]
    Datapoints: 100%|██████████| 3/3 [00:00<00:00,  4.48it/s]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>debug__score_time</th>
          <td>0.729573</td>
        </tr>
        <tr>
          <th>data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
        </tr>
        <tr>
          <th>single__accuracy</th>
          <td>[0.7619047619047619, 0.7391304347826086, 0.736...</td>
        </tr>
        <tr>
          <th>single__accuracy_pairwise</th>
          <td>[0.6935483870967742, 0.5333333333333333, 0.666...</td>
        </tr>
        <tr>
          <th>agg__accuracy</th>
          <td>0.745766</td>
        </tr>
        <tr>
          <th>agg__accuracy_pairwise</th>
          <td>0.631183</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 207-213

The accuracy provided is the mean accuracy over all datapoints.
The accuracy per datapoint can be found in the ``single_accuracy`` column.

In addition to the metrics, we also provide the raw results for each datapoint in the ``single_raw_results`` column.
This could be used for further analysis.
For example to calculate the confusion matrix over all ICs of all datapoints.

.. GENERATED FROM PYTHON SOURCE LINES 213-217

.. code-block:: Python

    raw_results = evaluation_results_with_opti["single__raw__predictions"][0]

    raw_results.head()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th>predicted</th>
          <th>reference</th>
        </tr>
        <tr>
          <th>cohort</th>
          <th>participant_id</th>
          <th>time_measure</th>
          <th>test</th>
          <th>trial</th>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">HA</th>
          <th rowspan="5" valign="top">001</th>
          <th rowspan="5" valign="top">TimeMeasure1</th>
          <th rowspan="5" valign="top">Test11</th>
          <th rowspan="5" valign="top">Trial1</th>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>right</td>
          <td>left</td>
        </tr>
        <tr>
          <th>1</th>
          <td>left</td>
          <td>right</td>
        </tr>
        <tr>
          <th>2</th>
          <td>left</td>
          <td>left</td>
        </tr>
        <tr>
          <th>3</th>
          <td>right</td>
          <td>right</td>
        </tr>
        <tr>
          <th>4</th>
          <td>left</td>
          <td>left</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 218-219

The confusion matrix can be calculated using the same functions as before.

.. GENERATED FROM PYTHON SOURCE LINES 219-225

.. code-block:: Python

    if not raw_results.empty:
        disp = ConfusionMatrixDisplay.from_predictions(
            raw_results["reference"], raw_results["predicted"]
        )
        disp.figure_.show()




.. image-sg:: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_004.png
   :alt:  99 lrc evaluation
   :srcset: /auto_examples/laterality/images/sphx_glr__99_lrc_evaluation_004.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 226-245

If you want to calculate additional metrics, you can either create a custom score function.

Parameter Optimization and Model Training
-----------------------------------------
Simply applying an algorithm for evaluation is one thing, but often we want to optimize the parameters of the
algorithm, train internal models, or both and evalute the performance of this optimization approach and not just a
fixed algorithm/model.

In this case, we need to create a train test split on the dataset and to ensure we have independent data for the
optimization.
In general, we would recommend using a cross-validation approach.
This can be done using the :func:`~tpcp.validate.cross_validate` function.

In the example below, we show the "most complicated" case, where we retrain the internal model of the ``LrcUllrich``
algorithm and optimize one of the Hyperparmeters of the internal SVM.
As we retrain the model and optimize hyperparameters, we need to use a :class:`~tpcp.optimize.GridSearchCV` nested
within the cross-validation loop.

Let's set this up first.

.. GENERATED FROM PYTHON SOURCE LINES 245-251

.. code-block:: Python

    from sklearn.model_selection import ParameterGrid
    from sklearn.pipeline import Pipeline
    from sklearn.preprocessing import MinMaxScaler
    from sklearn.svm import SVC
    from tpcp.optimize import GridSearchCV








.. GENERATED FROM PYTHON SOURCE LINES 252-253

We initialize the pipeline with an untrained model and an untrained scaler as a new pipeline.

.. GENERATED FROM PYTHON SOURCE LINES 253-258

.. code-block:: Python

    clf_pipeline = Pipeline(
        [("scaler", MinMaxScaler()), ("clf", SVC(kernel="linear"))]
    )
    pipeline = LrcEmulationPipeline(LrcUllrich(clf_pipe=clf_pipeline))








.. GENERATED FROM PYTHON SOURCE LINES 259-261

Then we can create a parameter Grid for the gridsearch.
Note, that we use ``__`` to set nested parameters.

.. GENERATED FROM PYTHON SOURCE LINES 261-263

.. code-block:: Python

    para_grid = ParameterGrid({"algo__clf_pipe__clf__C": [0.1, 1.0, 10.0]})








.. GENERATED FROM PYTHON SOURCE LINES 264-267

Then we path the pipeline to the optimizer.
We only select a 2-fold cross-validation for this example, as we will only have 2 datapoints per train set and we want
to minimize run time for this example.

.. GENERATED FROM PYTHON SOURCE LINES 267-275

.. code-block:: Python

    optimizer = GridSearchCV(
        pipeline,
        para_grid,
        return_optimized="accuracy",
        cv=2,
        scoring=lrc_score,
    )








.. GENERATED FROM PYTHON SOURCE LINES 276-277

Let's test the optimizer first on a manual train set.

.. GENERATED FROM PYTHON SOURCE LINES 277-279

.. code-block:: Python

    optimizer.optimize(simulated_real_world_walking[:2])





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Split-Para Combos:   0%|          | 0/6 [00:00<?, ?it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.63it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.63it/s]

    Split-Para Combos:  17%|█▋        | 1/6 [00:00<00:02,  2.23it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.56it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.55it/s]

    Split-Para Combos:  33%|███▎      | 2/6 [00:00<00:01,  2.41it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.70it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.69it/s]

    Split-Para Combos:  50%|█████     | 3/6 [00:01<00:01,  2.43it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.58it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.57it/s]

    Split-Para Combos:  67%|██████▋   | 4/6 [00:01<00:00,  2.48it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.72it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.71it/s]

    Split-Para Combos:  83%|████████▎ | 5/6 [00:02<00:00,  2.47it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.57it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.56it/s]

    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.50it/s]
    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.46it/s]

    GridSearchCV(cv=2, n_jobs=None, optimize_with_info=True, parameter_grid=<sklearn.model_selection._search.ParameterGrid object at 0x7dd58e96c3a0>, pipeline=LrcEmulationPipeline(algo=LrcUllrich(clf_pipe=Pipeline(steps=[('scaler', MinMaxScaler()), ('clf', SVC(kernel='linear'))]), smoothing_filter=ButterworthFilter(cutoff_freq_hz=(0.5, 2), filter_type='bandpass', order=4, zero_phase=True))), pre_dispatch='n_jobs', progress_bar=True, pure_parameters=False, return_optimized='accuracy', return_train_score=False, safe_optimize=True, scoring=Scorer(default_aggregator=<function mean_agg at 0x7dd58fc10c10>, final_aggregator=<function lrc_final_agg at 0x7dd59220f400>, n_jobs=None, pre_dispatch='2*n_jobs', progress_bar=True, score_func=<function lrc_per_datapoint_score at 0x7dd591dda0e0>, single_score_callback=None, verbose=0), verbose=0)



.. GENERATED FROM PYTHON SOURCE LINES 280-281

We can inspect the results:

.. GENERATED FROM PYTHON SOURCE LINES 281-284

.. code-block:: Python

    results = pd.DataFrame(optimizer.cv_results_)
    results.loc[:, ~results.columns.str.endswith("raw_results")].T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>mean__debug__optimize_time</th>
          <td>0.188295</td>
          <td>0.170197</td>
          <td>0.169777</td>
        </tr>
        <tr>
          <th>std__debug__optimize_time</th>
          <td>0.008776</td>
          <td>0.008983</td>
          <td>0.009409</td>
        </tr>
        <tr>
          <th>mean__debug__score_time</th>
          <td>0.221495</td>
          <td>0.219404</td>
          <td>0.219253</td>
        </tr>
        <tr>
          <th>std__debug__score_time</th>
          <td>0.017926</td>
          <td>0.01672</td>
          <td>0.016159</td>
        </tr>
        <tr>
          <th>split0__test__data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>split1__test__data_labels</th>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>split0__train__data_labels</th>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>split1__train__data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>param__algo__clf_pipe__clf__C</th>
          <td>0.1</td>
          <td>1.0</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>params</th>
          <td>{'algo__clf_pipe__clf__C': 0.1}</td>
          <td>{'algo__clf_pipe__clf__C': 1.0}</td>
          <td>{'algo__clf_pipe__clf__C': 10.0}</td>
        </tr>
        <tr>
          <th>split0__test__agg__accuracy</th>
          <td>0.507937</td>
          <td>0.746032</td>
          <td>0.793651</td>
        </tr>
        <tr>
          <th>split1__test__agg__accuracy</th>
          <td>0.652174</td>
          <td>0.695652</td>
          <td>0.782609</td>
        </tr>
        <tr>
          <th>mean__test__agg__accuracy</th>
          <td>0.580055</td>
          <td>0.720842</td>
          <td>0.78813</td>
        </tr>
        <tr>
          <th>std__test__agg__accuracy</th>
          <td>0.072119</td>
          <td>0.02519</td>
          <td>0.005521</td>
        </tr>
        <tr>
          <th>rank__test__agg__accuracy</th>
          <td>3</td>
          <td>2</td>
          <td>1</td>
        </tr>
        <tr>
          <th>split0__test__agg__accuracy_pairwise</th>
          <td>0.064516</td>
          <td>0.548387</td>
          <td>0.709677</td>
        </tr>
        <tr>
          <th>split1__test__agg__accuracy_pairwise</th>
          <td>0.688889</td>
          <td>0.644444</td>
          <td>0.644444</td>
        </tr>
        <tr>
          <th>mean__test__agg__accuracy_pairwise</th>
          <td>0.376703</td>
          <td>0.596416</td>
          <td>0.677061</td>
        </tr>
        <tr>
          <th>std__test__agg__accuracy_pairwise</th>
          <td>0.312186</td>
          <td>0.048029</td>
          <td>0.032616</td>
        </tr>
        <tr>
          <th>rank__test__agg__accuracy_pairwise</th>
          <td>3</td>
          <td>2</td>
          <td>1</td>
        </tr>
        <tr>
          <th>split0__test__single__accuracy</th>
          <td>[0.5079365079365079]</td>
          <td>[0.746031746031746]</td>
          <td>[0.7936507936507936]</td>
        </tr>
        <tr>
          <th>split1__test__single__accuracy</th>
          <td>[0.6521739130434783]</td>
          <td>[0.6956521739130435]</td>
          <td>[0.782608695652174]</td>
        </tr>
        <tr>
          <th>split0__test__single__accuracy_pairwise</th>
          <td>[0.06451612903225806]</td>
          <td>[0.5483870967741935]</td>
          <td>[0.7096774193548387]</td>
        </tr>
        <tr>
          <th>split1__test__single__accuracy_pairwise</th>
          <td>[0.6888888888888889]</td>
          <td>[0.6444444444444445]</td>
          <td>[0.6444444444444445]</td>
        </tr>
        <tr>
          <th>split0__test__single__raw__predictions</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>split1__test__single__raw__predictions</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 285-286

And apply/score the best performing and retrained model directly on the test set.

.. GENERATED FROM PYTHON SOURCE LINES 286-291

.. code-block:: Python


    lrc_score(optimizer.optimized_pipeline_, simulated_real_world_walking[2])[0][
        "accuracy"
    ]





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.17it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.17it/s]

    0.8681318681318682



.. GENERATED FROM PYTHON SOURCE LINES 292-293

Let's run everything combined with the external cross-validate to actually validate our optimization approach.

.. GENERATED FROM PYTHON SOURCE LINES 293-307

.. code-block:: Python

    from tpcp.validate import cross_validate

    evaluation_results_with_opti = pd.DataFrame(
        cross_validate(
            optimizer,
            simulated_real_world_walking,
            cv=3,
            scoring=lrc_score,
        )
    )
    evaluation_results_with_opti.loc[
        :, ~evaluation_results_with_opti.columns.str.endswith("raw__predictions")
    ].T





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]

    Split-Para Combos:   0%|          | 0/6 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.43it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.42it/s]


    Split-Para Combos:  17%|█▋        | 1/6 [00:00<00:02,  2.43it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.54it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.53it/s]


    Split-Para Combos:  33%|███▎      | 2/6 [00:00<00:01,  2.43it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.62it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.61it/s]


    Split-Para Combos:  50%|█████     | 3/6 [00:01<00:01,  2.49it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.59it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.58it/s]


    Split-Para Combos:  67%|██████▋   | 4/6 [00:01<00:00,  2.47it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.62it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.61it/s]


    Split-Para Combos:  83%|████████▎ | 5/6 [00:02<00:00,  2.50it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.57it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.57it/s]


    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.48it/s]
    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.47it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]

    CV Folds:  33%|███▎      | 1/3 [00:03<00:06,  3.14s/it]

    Split-Para Combos:   0%|          | 0/6 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]


    Split-Para Combos:  17%|█▋        | 1/6 [00:00<00:02,  2.22it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.59it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.58it/s]


    Split-Para Combos:  33%|███▎      | 2/6 [00:00<00:01,  2.29it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.76it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s]


    Split-Para Combos:  50%|█████     | 3/6 [00:01<00:01,  2.32it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.58it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.57it/s]


    Split-Para Combos:  67%|██████▋   | 4/6 [00:01<00:00,  2.33it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.73it/s]


    Split-Para Combos:  83%|████████▎ | 5/6 [00:02<00:00,  2.34it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.57it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.57it/s]


    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.34it/s]
    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.32it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.63it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.62it/s]

    CV Folds:  67%|██████▋   | 2/3 [00:06<00:03,  3.22s/it]

    Split-Para Combos:   0%|          | 0/6 [00:00<?, ?it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]


    Split-Para Combos:  17%|█▋        | 1/6 [00:00<00:02,  2.48it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.62it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.61it/s]


    Split-Para Combos:  33%|███▎      | 2/6 [00:00<00:01,  2.53it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.76it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s]


    Split-Para Combos:  50%|█████     | 3/6 [00:01<00:01,  2.51it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.64it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.63it/s]


    Split-Para Combos:  67%|██████▋   | 4/6 [00:01<00:00,  2.54it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s]


    Split-Para Combos:  83%|████████▎ | 5/6 [00:01<00:00,  2.52it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]


    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.63it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.62it/s]


    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.54it/s]
    Split-Para Combos: 100%|██████████| 6/6 [00:02<00:00,  2.53it/s]


    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.56it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.55it/s]

    CV Folds: 100%|██████████| 3/3 [00:09<00:00,  3.16s/it]
    CV Folds: 100%|██████████| 3/3 [00:09<00:00,  3.17s/it]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>debug__score_time</th>
          <td>0.233761</td>
          <td>0.200758</td>
          <td>0.242409</td>
        </tr>
        <tr>
          <th>debug__optimize_time</th>
          <td>2.892581</td>
          <td>3.070624</td>
          <td>2.827143</td>
        </tr>
        <tr>
          <th>train__data_labels</th>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
        </tr>
        <tr>
          <th>test__data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(MS, 001, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>test__single__accuracy</th>
          <td>[0.9047619047619048]</td>
          <td>[0.5869565217391305]</td>
          <td>[0.8681318681318682]</td>
        </tr>
        <tr>
          <th>test__single__accuracy_pairwise</th>
          <td>[0.8870967741935484]</td>
          <td>[0.4444444444444444]</td>
          <td>[0.7777777777777778]</td>
        </tr>
        <tr>
          <th>test__agg__accuracy</th>
          <td>0.904762</td>
          <td>0.586957</td>
          <td>0.868132</td>
        </tr>
        <tr>
          <th>test__agg__accuracy_pairwise</th>
          <td>0.887097</td>
          <td>0.444444</td>
          <td>0.777778</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 308-311

We can compare these results with the performance of the pre-trained model that was not optimized for the given
dataset, by using :class:`~tpcp.optimize.DummyOptimize`, to run a cross-validation, but without any optimization.
We simply evaluate the pre-trained model on exactly the same test sets as the optimized model.

.. GENERATED FROM PYTHON SOURCE LINES 311-329

.. code-block:: Python

    from tpcp.optimize import DummyOptimize

    optimizer = DummyOptimize(
        LrcEmulationPipeline(LrcUllrich()), ignore_potential_user_error_warning=True
    )

    evaluation_results_pre_trained = pd.DataFrame(
        cross_validate(
            optimizer,
            simulated_real_world_walking,
            cv=3,
            scoring=lrc_score,
        )
    )
    evaluation_results_pre_trained.loc[
        :, ~evaluation_results_pre_trained.columns.str.endswith("raw__predictions")
    ].T





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    CV Folds:   0%|          | 0/3 [00:00<?, ?it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.09it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.08it/s]

    CV Folds:  33%|███▎      | 1/3 [00:00<00:00,  3.54it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.17it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  5.16it/s]

    CV Folds:  67%|██████▋   | 2/3 [00:00<00:00,  4.00it/s]

    Datapoints:   0%|          | 0/1 [00:00<?, ?it/s]

    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.17it/s]
    Datapoints: 100%|██████████| 1/1 [00:00<00:00,  4.16it/s]

    CV Folds: 100%|██████████| 3/3 [00:00<00:00,  3.83it/s]
    CV Folds: 100%|██████████| 3/3 [00:00<00:00,  3.83it/s]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>debug__score_time</th>
          <td>0.268173</td>
          <td>0.216302</td>
          <td>0.262861</td>
        </tr>
        <tr>
          <th>debug__optimize_time</th>
          <td>0.002526</td>
          <td>0.001352</td>
          <td>0.001348</td>
        </tr>
        <tr>
          <th>train__data_labels</th>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (MS,...</td>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1), (HA,...</td>
        </tr>
        <tr>
          <th>test__data_labels</th>
          <td>[(HA, 001, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(HA, 002, TimeMeasure1, Test11, Trial1)]</td>
          <td>[(MS, 001, TimeMeasure1, Test11, Trial1)]</td>
        </tr>
        <tr>
          <th>test__single__accuracy</th>
          <td>[0.7619047619047619]</td>
          <td>[0.7391304347826086]</td>
          <td>[0.7362637362637363]</td>
        </tr>
        <tr>
          <th>test__single__accuracy_pairwise</th>
          <td>[0.6935483870967742]</td>
          <td>[0.5333333333333333]</td>
          <td>[0.6666666666666666]</td>
        </tr>
        <tr>
          <th>test__agg__accuracy</th>
          <td>0.761905</td>
          <td>0.73913</td>
          <td>0.736264</td>
        </tr>
        <tr>
          <th>test__agg__accuracy_pairwise</th>
          <td>0.693548</td>
          <td>0.533333</td>
          <td>0.666667</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 330-333

Note that using only so little data is not a good idea in practice.
There are many parameters, that you should tweak to make this a robust validation.
However, this example should provide a good starting point for your own experiments.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 19.137 seconds)

**Estimated memory usage:**  89 MB


.. _sphx_glr_download_auto_examples_laterality__99_lrc_evaluation.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _99_lrc_evaluation.ipynb <_99_lrc_evaluation.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _99_lrc_evaluation.py <_99_lrc_evaluation.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: _99_lrc_evaluation.zip <_99_lrc_evaluation.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_