.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_revalidation/laterality/_01_lrc_analysis.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_revalidation_laterality__01_lrc_analysis.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_revalidation_laterality__01_lrc_analysis.py:


.. _lrc_val_results:

Performance of the laterality classification algorithms on the TVS dataset
==========================================================================

The following provides an analysis and comparison of the stride length algorithms on the TVS dataset
(lab and free-living).
We look into the actual performance of the algorithms compared to the reference data.

Compared to the other revalidation scripts, this one does not load the old "matlab" results, as there are no old results.
The laterality algorithm by Ulrich et al. was validated independently and was already written in Python.
The implemented version follows the old version very closely.
The goal of this revalidation, is to validate the re-trained model (with the updated training code) on the TVS dataset.
We compare it against the old model and the McCamley algorithm.

.. note:: If you are interested in how these results are calculated, head over to the
    :ref:`processing page <lrc_val_gen>`.

.. GENERATED FROM PYTHON SOURCE LINES 23-29

Below are the list of algorithms that we will compare.
Note, that we use the prefix "MobGap" to refer to the newly trained model and "Original Implementation" refers to the
models trained as part of previous work.
We compare all the available models.
For context, the "MS_ALL" models are used by default in the pipelines.
For the McCamley algorithm, only a single version exists.

.. GENERATED FROM PYTHON SOURCE LINES 29-37

.. code-block:: Python


    algorithms = {
        "McCamley": ("McCamley", "-"),
        "UllrichOld__ms_all": ("Ullrich - MS-ALL", "Original Implementation"),
        "UllrichOld__ms_ms": ("Ullrich - MS-MS", "Original Implementation"),
        "UllrichNew__ms_all": ("Ullrich - MS-ALL", "MobGap"),
    }








.. GENERATED FROM PYTHON SOURCE LINES 38-45

The code below loads the data and prepares it for the analysis.
By default, the data will be downloaded from an online repository (and cached locally).
If you want to use a local copy of the data, you can set the `MOBGAP_VALIDATION_DATA_PATH` environment variable.
and the MOBGAP_VALIDATION_USE_LOCA_DATA to `1`.

The file download will print a couple log information, which can usually be ignored.
You can also change the `version` parameter to load a different version of the data.

.. GENERATED FROM PYTHON SOURCE LINES 45-116

.. code-block:: Python

    from pathlib import Path

    import pandas as pd
    from mobgap.data.validation_results import ValidationResultLoader
    from mobgap.utils.misc import get_env_var


    def format_loaded_results(
        values: dict[tuple[str, str], pd.DataFrame],
        index_cols: list[str],
    ) -> pd.DataFrame:
        formatted = (
            pd.concat(values, names=["algo", "version", *index_cols])
            .reset_index()
            .assign(
                algo_with_version=lambda df: (
                    df["algo"] + " (" + df["version"] + ")"
                ),
                _combined="combined",
            )
        )
        return formatted


    local_data_path = (
        Path(get_env_var("MOBGAP_VALIDATION_DATA_PATH")) / "results"
        if int(get_env_var("MOBGAP_VALIDATION_USE_LOCAL_DATA", 0))
        else None
    )
    __RESULT_VERSION = "v1.0.0"
    loader = ValidationResultLoader(
        "lrc", result_path=local_data_path, version=__RESULT_VERSION
    )


    free_living_index_cols = [
        "cohort",
        "participant_id",
        "time_measure",
        "recording",
        "recording_name",
        "recording_name_pretty",
    ]

    free_living_results = format_loaded_results(
        {
            v: loader.load_single_results(k, "free_living")
            for k, v in algorithms.items()
        },
        free_living_index_cols,
    )

    lab_index_cols = [
        "cohort",
        "participant_id",
        "time_measure",
        "test",
        "trial",
        "test_name",
        "test_name_pretty",
    ]

    lab_results = format_loaded_results(
        {
            v: loader.load_single_results(k, "laboratory")
            for k, v in algorithms.items()
        },
        lab_index_cols,
    )

    cohort_order = ["HA", "CHF", "COPD", "MS", "PD", "PFF"]




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|                                              | 0.00/2.45k [00:00<?, ?B/s]
      0%|                                              | 0.00/2.45k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 2.45k/2.45k [00:00<00:00, 13.3MB/s]

      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 2.43k/2.43k [00:00<00:00, 18.1MB/s]

      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 2.43k/2.43k [00:00<00:00, 15.6MB/s]

      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
      0%|                                              | 0.00/2.43k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 2.43k/2.43k [00:00<00:00, 18.0MB/s]

      0%|                                              | 0.00/9.57k [00:00<?, ?B/s]
      0%|                                              | 0.00/9.57k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 9.57k/9.57k [00:00<00:00, 65.9MB/s]

      0%|                                              | 0.00/9.67k [00:00<?, ?B/s]
      0%|                                              | 0.00/9.67k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 9.67k/9.67k [00:00<00:00, 69.4MB/s]

      0%|                                              | 0.00/9.66k [00:00<?, ?B/s]
      0%|                                              | 0.00/9.66k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 9.66k/9.66k [00:00<00:00, 62.3MB/s]

      0%|                                              | 0.00/9.06k [00:00<?, ?B/s]
      0%|                                              | 0.00/9.06k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 9.06k/9.06k [00:00<00:00, 65.7MB/s]




.. GENERATED FROM PYTHON SOURCE LINES 117-126

Performance metrics
-------------------
Below you can find the setup for all performance metrics that we will calculate.
For laterality, this is really simple, as we just calculate the accuracy of the binary classification and the
"pariwise accuracy" that checks if consecutive ICs have been assigned either the same or different laterality.
High "pairwise accuracy" provides an better indicator if steps and strides would be correctly defined based on the
laterality information.
This metrics explicitly ignores the actual label of the laterality, as would not impact the main gait metrics, if
the laterality is swapped consistently.

.. GENERATED FROM PYTHON SOURCE LINES 126-193

.. code-block:: Python

    from functools import partial

    from mobgap.pipeline.evaluation import CustomErrorAggregations as A
    from mobgap.utils.df_operations import (
        CustomOperation,
        apply_aggregations,
        apply_transformations,
    )
    from mobgap.utils.tables import FormatTransformer as F
    from mobgap.utils.tables import RevalidationInfo, revalidation_table_styles

    custom_aggs = [
        CustomOperation(
            identifier=None,
            function=A.n_datapoints,
            column_name=[("n_datapoints", "all")],
        ),
        ("accuracy", ["mean", A.conf_intervals]),
        ("accuracy_pairwise", ["mean", A.conf_intervals]),
    ]

    format_transforms = [
        CustomOperation(
            identifier=None,
            function=lambda df_: df_[("n_datapoints", "all")].astype(int),
            column_name="n_datapoints",
        ),
        CustomOperation(
            identifier=None,
            function=partial(
                F.value_with_metadata,
                value_col=("mean", "accuracy"),
                other_columns={"range": ("conf_intervals", "accuracy")},
            ),
            column_name="accuracy",
        ),
        CustomOperation(
            identifier=None,
            function=partial(
                F.value_with_metadata,
                value_col=("mean", "accuracy_pairwise"),
                other_columns={"range": ("conf_intervals", "accuracy_pairwise")},
            ),
            column_name="accuracy_pairwise",
        ),
    ]


    final_names = {
        "n_datapoints": "# participants",
        "accuracy": "Accuracy",
        "accuracy_pairwise": "Accuracy IC-pairs",
    }

    validation_thresholds = {
        "Accuracy": RevalidationInfo(threshold=0.7, higher_is_better=True),
    }


    def format_tables(df: pd.DataFrame) -> pd.DataFrame:
        return (
            df.pipe(apply_transformations, format_transforms)
            .rename(columns=final_names)
            .loc[:, list(final_names.values())]
        )









.. GENERATED FROM PYTHON SOURCE LINES 194-202

Free-Living Comparison
----------------------
We focus on the free-living data for the comparison as this is the expected use case for the algorithms.

All results across all cohorts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The results below represent the average performance across all participants independent of the
cohort.

.. GENERATED FROM PYTHON SOURCE LINES 202-233

.. code-block:: Python

    import matplotlib.pyplot as plt
    import seaborn as sns

    fig, ax = plt.subplots()
    sns.boxplot(
        data=free_living_results, x="algo_with_version", y="accuracy", ax=ax
    )
    plt.xticks(rotation=45, ha="right")
    fig.tight_layout()
    fig.show()

    fig, ax = plt.subplots()
    sns.boxplot(
        data=free_living_results,
        x="algo_with_version",
        y="accuracy_pairwise",
        ax=ax,
    )
    plt.xticks(rotation=45, ha="right")
    fig.tight_layout()
    fig.show()

    perf_metrics_all = (
        free_living_results.groupby(["algo", "version"])
        .apply(apply_aggregations, custom_aggs, include_groups=False)
        .pipe(format_tables)
    )
    perf_metrics_all.style.pipe(
        revalidation_table_styles, validation_thresholds, ["algo"]
    )




.. rst-class:: sphx-glr-horizontal


    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_001.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_001.png
         :class: sphx-glr-single-img

    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_002.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_002.png
         :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_25d21_row0_col0, #T_25d21_row0_col2, #T_25d21_row2_col0, #T_25d21_row2_col2, #T_25d21_row3_col0, #T_25d21_row3_col2 {
      border-bottom: 2px solid black;
    }
    #T_25d21_row0_col1, #T_25d21_row3_col1 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_25d21_row1_col1 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_25d21_row2_col1 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_25d21" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_25d21_level0_col0" class="col_heading level0 col0" ># participants</th>
          <th id="T_25d21_level0_col1" class="col_heading level0 col1" >Accuracy</th>
          <th id="T_25d21_level0_col2" class="col_heading level0 col2" >Accuracy IC-pairs</th>
        </tr>
        <tr>
          <th class="index_name level0" >algo</th>
          <th class="index_name level1" >version</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_25d21_level0_row0" class="row_heading level0 row0" >McCamley</th>
          <th id="T_25d21_level1_row0" class="row_heading level1 row0" >-</th>
          <td id="T_25d21_row0_col0" class="data row0 col0" >101</td>
          <td id="T_25d21_row0_col1" class="data row0 col1" ><span>0.78 [0.75, 0.82]</span></td>
          <td id="T_25d21_row0_col2" class="data row0 col2" ><span>0.77 [0.75, 0.79]</span></td>
        </tr>
        <tr>
          <th id="T_25d21_level0_row1" class="row_heading level0 row1" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_25d21_level1_row1" class="row_heading level1 row1" >MobGap</th>
          <td id="T_25d21_row1_col0" class="data row1 col0" >101</td>
          <td id="T_25d21_row1_col1" class="data row1 col1" ><span>0.80 [0.77, 0.84]</span></td>
          <td id="T_25d21_row1_col2" class="data row1 col2" ><span>0.81 [0.79, 0.83]</span></td>
        </tr>
        <tr>
          <th id="T_25d21_level1_row2" class="row_heading level1 row2" >Original Implementation</th>
          <td id="T_25d21_row2_col0" class="data row2 col0" >101</td>
          <td id="T_25d21_row2_col1" class="data row2 col1" ><span>0.77 [0.74, 0.81]</span></td>
          <td id="T_25d21_row2_col2" class="data row2 col2" ><span>0.74 [0.71, 0.77]</span></td>
        </tr>
        <tr>
          <th id="T_25d21_level0_row3" class="row_heading level0 row3" >Ullrich - MS-MS</th>
          <th id="T_25d21_level1_row3" class="row_heading level1 row3" >Original Implementation</th>
          <td id="T_25d21_row3_col0" class="data row3 col0" >101</td>
          <td id="T_25d21_row3_col1" class="data row3 col1" ><span>0.76 [0.72, 0.80]</span></td>
          <td id="T_25d21_row3_col2" class="data row3 col2" ><span>0.76 [0.73, 0.78]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 234-237

Per Cohort
~~~~~~~~~~
The results below represent the average performance across all participants within a cohort.

.. GENERATED FROM PYTHON SOURCE LINES 237-269

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=free_living_results,
        x="cohort",
        y="accuracy",
        hue="algo_with_version",
        order=cohort_order,
        ax=ax,
    )
    ax.set_title("Accuracy")
    fig.show()
    fig, ax = plt.subplots()
    sns.boxplot(
        data=free_living_results,
        x="cohort",
        y="accuracy_pairwise",
        hue="algo_with_version",
        order=cohort_order,
        ax=ax,
    )
    ax.set_title("Accuracy IC-pairs")
    fig.show()
    perf_metrics_cohort = (
        free_living_results.groupby(["cohort", "algo", "version"])
        .apply(apply_aggregations, custom_aggs, include_groups=False)
        .pipe(format_tables)
        .loc[cohort_order]
    )
    perf_metrics_cohort.style.pipe(
        revalidation_table_styles, validation_thresholds, ["cohort", "algo"]
    )




.. rst-class:: sphx-glr-horizontal


    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_003.png
         :alt: Accuracy
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_003.png
         :class: sphx-glr-single-img

    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_004.png
         :alt: Accuracy IC-pairs
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_004.png
         :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_8dae7_row0_col0, #T_8dae7_row0_col2, #T_8dae7_row2_col0, #T_8dae7_row2_col2, #T_8dae7_row3_col0, #T_8dae7_row3_col2, #T_8dae7_row4_col0, #T_8dae7_row4_col2, #T_8dae7_row6_col0, #T_8dae7_row6_col2, #T_8dae7_row7_col0, #T_8dae7_row7_col2, #T_8dae7_row8_col0, #T_8dae7_row8_col2, #T_8dae7_row10_col0, #T_8dae7_row10_col2, #T_8dae7_row11_col0, #T_8dae7_row11_col2, #T_8dae7_row12_col0, #T_8dae7_row12_col2, #T_8dae7_row14_col0, #T_8dae7_row14_col2, #T_8dae7_row15_col0, #T_8dae7_row15_col2, #T_8dae7_row16_col0, #T_8dae7_row16_col2, #T_8dae7_row18_col0, #T_8dae7_row18_col2, #T_8dae7_row19_col0, #T_8dae7_row19_col2, #T_8dae7_row20_col0, #T_8dae7_row20_col2, #T_8dae7_row22_col0, #T_8dae7_row22_col2, #T_8dae7_row23_col0, #T_8dae7_row23_col2 {
      border-bottom: 2px solid black;
    }
    #T_8dae7_row0_col1, #T_8dae7_row3_col1, #T_8dae7_row4_col1, #T_8dae7_row7_col1, #T_8dae7_row8_col1, #T_8dae7_row11_col1, #T_8dae7_row12_col1, #T_8dae7_row15_col1, #T_8dae7_row16_col1, #T_8dae7_row19_col1, #T_8dae7_row20_col1, #T_8dae7_row23_col1 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_8dae7_row1_col1, #T_8dae7_row5_col1, #T_8dae7_row9_col1, #T_8dae7_row13_col1, #T_8dae7_row17_col1, #T_8dae7_row21_col1 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_8dae7_row2_col1, #T_8dae7_row6_col1, #T_8dae7_row10_col1, #T_8dae7_row14_col1, #T_8dae7_row18_col1, #T_8dae7_row22_col1 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_8dae7" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_8dae7_level0_col0" class="col_heading level0 col0" ># participants</th>
          <th id="T_8dae7_level0_col1" class="col_heading level0 col1" >Accuracy</th>
          <th id="T_8dae7_level0_col2" class="col_heading level0 col2" >Accuracy IC-pairs</th>
        </tr>
        <tr>
          <th class="index_name level0" >cohort</th>
          <th class="index_name level1" >algo</th>
          <th class="index_name level2" >version</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_8dae7_level0_row0" class="row_heading level0 row0" rowspan="4">HA</th>
          <th id="T_8dae7_level1_row0" class="row_heading level1 row0" >McCamley</th>
          <th id="T_8dae7_level2_row0" class="row_heading level2 row0" >-</th>
          <td id="T_8dae7_row0_col0" class="data row0 col0" >20</td>
          <td id="T_8dae7_row0_col1" class="data row0 col1" ><span>0.85 [0.80, 0.90]</span></td>
          <td id="T_8dae7_row0_col2" class="data row0 col2" ><span>0.81 [0.76, 0.86]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row1" class="row_heading level1 row1" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row1" class="row_heading level2 row1" >MobGap</th>
          <td id="T_8dae7_row1_col0" class="data row1 col0" >20</td>
          <td id="T_8dae7_row1_col1" class="data row1 col1" ><span>0.86 [0.81, 0.91]</span></td>
          <td id="T_8dae7_row1_col2" class="data row1 col2" ><span>0.84 [0.80, 0.88]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row2" class="row_heading level2 row2" >Original Implementation</th>
          <td id="T_8dae7_row2_col0" class="data row2 col0" >20</td>
          <td id="T_8dae7_row2_col1" class="data row2 col1" ><span>0.84 [0.79, 0.88]</span></td>
          <td id="T_8dae7_row2_col2" class="data row2 col2" ><span>0.78 [0.73, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row3" class="row_heading level1 row3" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row3" class="row_heading level2 row3" >Original Implementation</th>
          <td id="T_8dae7_row3_col0" class="data row3 col0" >20</td>
          <td id="T_8dae7_row3_col1" class="data row3 col1" ><span>0.82 [0.76, 0.87]</span></td>
          <td id="T_8dae7_row3_col2" class="data row3 col2" ><span>0.78 [0.73, 0.83]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level0_row4" class="row_heading level0 row4" rowspan="4">CHF</th>
          <th id="T_8dae7_level1_row4" class="row_heading level1 row4" >McCamley</th>
          <th id="T_8dae7_level2_row4" class="row_heading level2 row4" >-</th>
          <td id="T_8dae7_row4_col0" class="data row4 col0" >10</td>
          <td id="T_8dae7_row4_col1" class="data row4 col1" ><span>0.82 [0.73, 0.91]</span></td>
          <td id="T_8dae7_row4_col2" class="data row4 col2" ><span>0.80 [0.72, 0.88]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row5" class="row_heading level1 row5" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row5" class="row_heading level2 row5" >MobGap</th>
          <td id="T_8dae7_row5_col0" class="data row5 col0" >10</td>
          <td id="T_8dae7_row5_col1" class="data row5 col1" ><span>0.83 [0.73, 0.93]</span></td>
          <td id="T_8dae7_row5_col2" class="data row5 col2" ><span>0.85 [0.80, 0.90]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row6" class="row_heading level2 row6" >Original Implementation</th>
          <td id="T_8dae7_row6_col0" class="data row6 col0" >10</td>
          <td id="T_8dae7_row6_col1" class="data row6 col1" ><span>0.80 [0.71, 0.90]</span></td>
          <td id="T_8dae7_row6_col2" class="data row6 col2" ><span>0.78 [0.72, 0.85]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row7" class="row_heading level1 row7" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row7" class="row_heading level2 row7" >Original Implementation</th>
          <td id="T_8dae7_row7_col0" class="data row7 col0" >10</td>
          <td id="T_8dae7_row7_col1" class="data row7 col1" ><span>0.72 [0.61, 0.84]</span></td>
          <td id="T_8dae7_row7_col2" class="data row7 col2" ><span>0.72 [0.63, 0.82]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level0_row8" class="row_heading level0 row8" rowspan="4">COPD</th>
          <th id="T_8dae7_level1_row8" class="row_heading level1 row8" >McCamley</th>
          <th id="T_8dae7_level2_row8" class="row_heading level2 row8" >-</th>
          <td id="T_8dae7_row8_col0" class="data row8 col0" >17</td>
          <td id="T_8dae7_row8_col1" class="data row8 col1" ><span>0.70 [0.62, 0.78]</span></td>
          <td id="T_8dae7_row8_col2" class="data row8 col2" ><span>0.71 [0.66, 0.76]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row9" class="row_heading level1 row9" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row9" class="row_heading level2 row9" >MobGap</th>
          <td id="T_8dae7_row9_col0" class="data row9 col0" >17</td>
          <td id="T_8dae7_row9_col1" class="data row9 col1" ><span>0.76 [0.68, 0.84]</span></td>
          <td id="T_8dae7_row9_col2" class="data row9 col2" ><span>0.78 [0.74, 0.82]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row10" class="row_heading level2 row10" >Original Implementation</th>
          <td id="T_8dae7_row10_col0" class="data row10 col0" >17</td>
          <td id="T_8dae7_row10_col1" class="data row10 col1" ><span>0.70 [0.62, 0.78]</span></td>
          <td id="T_8dae7_row10_col2" class="data row10 col2" ><span>0.69 [0.64, 0.73]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row11" class="row_heading level1 row11" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row11" class="row_heading level2 row11" >Original Implementation</th>
          <td id="T_8dae7_row11_col0" class="data row11 col0" >17</td>
          <td id="T_8dae7_row11_col1" class="data row11 col1" ><span>0.76 [0.70, 0.83]</span></td>
          <td id="T_8dae7_row11_col2" class="data row11 col2" ><span>0.74 [0.70, 0.78]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level0_row12" class="row_heading level0 row12" rowspan="4">MS</th>
          <th id="T_8dae7_level1_row12" class="row_heading level1 row12" >McCamley</th>
          <th id="T_8dae7_level2_row12" class="row_heading level2 row12" >-</th>
          <td id="T_8dae7_row12_col0" class="data row12 col0" >18</td>
          <td id="T_8dae7_row12_col1" class="data row12 col1" ><span>0.78 [0.68, 0.87]</span></td>
          <td id="T_8dae7_row12_col2" class="data row12 col2" ><span>0.79 [0.74, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row13" class="row_heading level1 row13" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row13" class="row_heading level2 row13" >MobGap</th>
          <td id="T_8dae7_row13_col0" class="data row13 col0" >18</td>
          <td id="T_8dae7_row13_col1" class="data row13 col1" ><span>0.79 [0.69, 0.88]</span></td>
          <td id="T_8dae7_row13_col2" class="data row13 col2" ><span>0.82 [0.77, 0.86]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row14" class="row_heading level2 row14" >Original Implementation</th>
          <td id="T_8dae7_row14_col0" class="data row14 col0" >18</td>
          <td id="T_8dae7_row14_col1" class="data row14 col1" ><span>0.75 [0.66, 0.84]</span></td>
          <td id="T_8dae7_row14_col2" class="data row14 col2" ><span>0.73 [0.67, 0.79]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row15" class="row_heading level1 row15" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row15" class="row_heading level2 row15" >Original Implementation</th>
          <td id="T_8dae7_row15_col0" class="data row15 col0" >18</td>
          <td id="T_8dae7_row15_col1" class="data row15 col1" ><span>0.75 [0.65, 0.86]</span></td>
          <td id="T_8dae7_row15_col2" class="data row15 col2" ><span>0.80 [0.75, 0.85]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level0_row16" class="row_heading level0 row16" rowspan="4">PD</th>
          <th id="T_8dae7_level1_row16" class="row_heading level1 row16" >McCamley</th>
          <th id="T_8dae7_level2_row16" class="row_heading level2 row16" >-</th>
          <td id="T_8dae7_row16_col0" class="data row16 col0" >19</td>
          <td id="T_8dae7_row16_col1" class="data row16 col1" ><span>0.79 [0.71, 0.87]</span></td>
          <td id="T_8dae7_row16_col2" class="data row16 col2" ><span>0.78 [0.72, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row17" class="row_heading level1 row17" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row17" class="row_heading level2 row17" >MobGap</th>
          <td id="T_8dae7_row17_col0" class="data row17 col0" >19</td>
          <td id="T_8dae7_row17_col1" class="data row17 col1" ><span>0.81 [0.71, 0.91]</span></td>
          <td id="T_8dae7_row17_col2" class="data row17 col2" ><span>0.83 [0.76, 0.90]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row18" class="row_heading level2 row18" >Original Implementation</th>
          <td id="T_8dae7_row18_col0" class="data row18 col0" >19</td>
          <td id="T_8dae7_row18_col1" class="data row18 col1" ><span>0.79 [0.70, 0.88]</span></td>
          <td id="T_8dae7_row18_col2" class="data row18 col2" ><span>0.78 [0.70, 0.85]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row19" class="row_heading level1 row19" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row19" class="row_heading level2 row19" >Original Implementation</th>
          <td id="T_8dae7_row19_col0" class="data row19 col0" >19</td>
          <td id="T_8dae7_row19_col1" class="data row19 col1" ><span>0.78 [0.68, 0.87]</span></td>
          <td id="T_8dae7_row19_col2" class="data row19 col2" ><span>0.78 [0.71, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level0_row20" class="row_heading level0 row20" rowspan="4">PFF</th>
          <th id="T_8dae7_level1_row20" class="row_heading level1 row20" >McCamley</th>
          <th id="T_8dae7_level2_row20" class="row_heading level2 row20" >-</th>
          <td id="T_8dae7_row20_col0" class="data row20 col0" >17</td>
          <td id="T_8dae7_row20_col1" class="data row20 col1" ><span>0.75 [0.67, 0.84]</span></td>
          <td id="T_8dae7_row20_col2" class="data row20 col2" ><span>0.73 [0.68, 0.78]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row21" class="row_heading level1 row21" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_8dae7_level2_row21" class="row_heading level2 row21" >MobGap</th>
          <td id="T_8dae7_row21_col0" class="data row21 col0" >17</td>
          <td id="T_8dae7_row21_col1" class="data row21 col1" ><span>0.76 [0.65, 0.86]</span></td>
          <td id="T_8dae7_row21_col2" class="data row21 col2" ><span>0.76 [0.69, 0.83]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level2_row22" class="row_heading level2 row22" >Original Implementation</th>
          <td id="T_8dae7_row22_col0" class="data row22 col0" >17</td>
          <td id="T_8dae7_row22_col1" class="data row22 col1" ><span>0.75 [0.65, 0.84]</span></td>
          <td id="T_8dae7_row22_col2" class="data row22 col2" ><span>0.70 [0.61, 0.79]</span></td>
        </tr>
        <tr>
          <th id="T_8dae7_level1_row23" class="row_heading level1 row23" >Ullrich - MS-MS</th>
          <th id="T_8dae7_level2_row23" class="row_heading level2 row23" >Original Implementation</th>
          <td id="T_8dae7_row23_col0" class="data row23 col0" >17</td>
          <td id="T_8dae7_row23_col1" class="data row23 col1" ><span>0.70 [0.60, 0.81]</span></td>
          <td id="T_8dae7_row23_col2" class="data row23 col2" ><span>0.70 [0.62, 0.77]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 270-274

Deep Dive Analysis of Main Algorithms
-------------------------------------
Below, we show the direct correlation between the results from the old and the new implementation.
Each datapoint represents one participant.

.. GENERATED FROM PYTHON SOURCE LINES 274-321

.. code-block:: Python

    from mobgap.plotting import (
        calc_min_max_with_margin,
        make_square,
        move_legend_outside,
        plot_regline,
    )


    def compare_scatter_plot(data, name):
        fig, ax = plt.subplots(figsize=(8, 8), constrained_layout=True)
        reformated_data = (
            data.pivot_table(
                values="accuracy",
                index=("cohort", "participant_id"),
                columns="version",
            )
            .reset_index()
            .dropna(how="any")
        )

        min_max = calc_min_max_with_margin(
            reformated_data["Original Implementation"], reformated_data["MobGap"]
        )
        sns.scatterplot(
            reformated_data,
            x="Original Implementation",
            y="MobGap",
            hue="cohort",
            ax=ax,
        )
        plot_regline(
            reformated_data["Original Implementation"],
            reformated_data["MobGap"],
            ax=ax,
        )
        make_square(ax, min_max, draw_diagonal=True)
        move_legend_outside(fig, ax)
        ax.set_title(name)
        ax.set_xlabel("Original Implementation")
        ax.set_ylabel("MobGap")
        plt.show()


    free_living_results.query("algo == 'Ullrich - MS-ALL'").pipe(
        compare_scatter_plot, "Ullrich - MS-ALL"
    )




.. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_005.png
   :alt: Ullrich - MS-ALL
   :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_005.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 322-329

Conclusion Free-Living
~~~~~~~~~~~~~~~~~~~~~~
It is good to see that the new version of the algorithm performs slightly better than the old version.
However, it is unclear, why the new model is different, as we used almost the same pipeline and the same data.
The non-ML algo (McCamly) performs suprisingly well, and much better than in the tests we did as part of Mobilise-D.
Overall, the performance is not as good as we would like it to be.
In particular for a couple of participants, where the performance is as low as 0.1.

.. GENERATED FROM PYTHON SOURCE LINES 331-339

Laboratory Comparison
---------------------
Every datapoint below is one trial of a test.
Note, that each datapoint is weighted equally in the calculation of the performance metrics.
This is a limitation of this simple approach, as the number of strides per trial and the complexity of the context
can vary significantly.
For a full picture, different groups of tests should be analyzed separately.
The approach below should still provide a good overview to compare the algorithms.

.. GENERATED FROM PYTHON SOURCE LINES 339-362

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(data=lab_results, x="algo_with_version", y="accuracy", ax=ax)
    plt.xticks(rotation=45, ha="right")
    fig.tight_layout()
    fig.show()

    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results, x="algo_with_version", y="accuracy_pairwise", ax=ax
    )
    plt.xticks(rotation=45, ha="right")
    fig.tight_layout()
    fig.show()

    perf_metrics_all = (
        lab_results.groupby(["algo", "version"])
        .apply(apply_aggregations, custom_aggs, include_groups=False)
        .pipe(format_tables)
    )
    perf_metrics_all.style.pipe(
        revalidation_table_styles, validation_thresholds, ["algo"]
    )




.. rst-class:: sphx-glr-horizontal


    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_006.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_006.png
         :class: sphx-glr-single-img

    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_007.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_007.png
         :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_f714b_row0_col0, #T_f714b_row0_col2, #T_f714b_row2_col0, #T_f714b_row2_col2, #T_f714b_row3_col0, #T_f714b_row3_col2 {
      border-bottom: 2px solid black;
    }
    #T_f714b_row0_col1, #T_f714b_row3_col1 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_f714b_row1_col1 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_f714b_row2_col1 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_f714b" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_f714b_level0_col0" class="col_heading level0 col0" ># participants</th>
          <th id="T_f714b_level0_col1" class="col_heading level0 col1" >Accuracy</th>
          <th id="T_f714b_level0_col2" class="col_heading level0 col2" >Accuracy IC-pairs</th>
        </tr>
        <tr>
          <th class="index_name level0" >algo</th>
          <th class="index_name level1" >version</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_f714b_level0_row0" class="row_heading level0 row0" >McCamley</th>
          <th id="T_f714b_level1_row0" class="row_heading level1 row0" >-</th>
          <td id="T_f714b_row0_col0" class="data row0 col0" >1168</td>
          <td id="T_f714b_row0_col1" class="data row0 col1" ><span>0.80 [0.79, 0.81]</span></td>
          <td id="T_f714b_row0_col2" class="data row0 col2" ><span>0.77 [0.76, 0.78]</span></td>
        </tr>
        <tr>
          <th id="T_f714b_level0_row1" class="row_heading level0 row1" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_f714b_level1_row1" class="row_heading level1 row1" >MobGap</th>
          <td id="T_f714b_row1_col0" class="data row1 col0" >1168</td>
          <td id="T_f714b_row1_col1" class="data row1 col1" ><span>0.85 [0.84, 0.86]</span></td>
          <td id="T_f714b_row1_col2" class="data row1 col2" ><span>0.83 [0.82, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_f714b_level1_row2" class="row_heading level1 row2" >Original Implementation</th>
          <td id="T_f714b_row2_col0" class="data row2 col0" >1168</td>
          <td id="T_f714b_row2_col1" class="data row2 col1" ><span>0.79 [0.78, 0.80]</span></td>
          <td id="T_f714b_row2_col2" class="data row2 col2" ><span>0.74 [0.73, 0.75]</span></td>
        </tr>
        <tr>
          <th id="T_f714b_level0_row3" class="row_heading level0 row3" >Ullrich - MS-MS</th>
          <th id="T_f714b_level1_row3" class="row_heading level1 row3" >Original Implementation</th>
          <td id="T_f714b_row3_col0" class="data row3 col0" >1168</td>
          <td id="T_f714b_row3_col1" class="data row3 col1" ><span>0.78 [0.77, 0.79]</span></td>
          <td id="T_f714b_row3_col2" class="data row3 col2" ><span>0.75 [0.74, 0.76]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 363-366

Per Cohort
~~~~~~~~~~
The results below represent the average performance across all trails of all participants within a cohort.

.. GENERATED FROM PYTHON SOURCE LINES 366-397

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results,
        x="cohort",
        y="accuracy",
        hue="algo_with_version",
        order=cohort_order,
        ax=ax,
    )
    fig.show()
    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results,
        x="cohort",
        y="accuracy_pairwise",
        hue="algo_with_version",
        order=cohort_order,
        ax=ax,
    )
    fig.show()
    perf_metrics_cohort = (
        lab_results.groupby(["cohort", "algo", "version"])
        .apply(apply_aggregations, custom_aggs, include_groups=False)
        .pipe(format_tables)
        .loc[cohort_order]
    )
    perf_metrics_cohort.style.pipe(
        revalidation_table_styles, validation_thresholds, ["cohort", "algo"]
    )





.. rst-class:: sphx-glr-horizontal


    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_008.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_008.png
         :class: sphx-glr-single-img

    *

      .. image-sg:: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_009.png
         :alt:  01 lrc analysis
         :srcset: /auto_revalidation/laterality/images/sphx_glr__01_lrc_analysis_009.png
         :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_9d772_row0_col0, #T_9d772_row0_col2, #T_9d772_row2_col0, #T_9d772_row2_col2, #T_9d772_row3_col0, #T_9d772_row3_col2, #T_9d772_row4_col0, #T_9d772_row4_col2, #T_9d772_row6_col0, #T_9d772_row6_col2, #T_9d772_row7_col0, #T_9d772_row7_col2, #T_9d772_row8_col0, #T_9d772_row8_col2, #T_9d772_row10_col0, #T_9d772_row10_col2, #T_9d772_row11_col0, #T_9d772_row11_col2, #T_9d772_row12_col0, #T_9d772_row12_col2, #T_9d772_row14_col0, #T_9d772_row14_col2, #T_9d772_row15_col0, #T_9d772_row15_col2, #T_9d772_row16_col0, #T_9d772_row16_col2, #T_9d772_row18_col0, #T_9d772_row18_col2, #T_9d772_row19_col0, #T_9d772_row19_col2, #T_9d772_row20_col0, #T_9d772_row20_col2, #T_9d772_row22_col0, #T_9d772_row22_col2, #T_9d772_row23_col0, #T_9d772_row23_col2 {
      border-bottom: 2px solid black;
    }
    #T_9d772_row0_col1, #T_9d772_row3_col1, #T_9d772_row4_col1, #T_9d772_row7_col1, #T_9d772_row8_col1, #T_9d772_row11_col1, #T_9d772_row12_col1, #T_9d772_row15_col1, #T_9d772_row16_col1, #T_9d772_row19_col1, #T_9d772_row20_col1, #T_9d772_row23_col1 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_9d772_row1_col1, #T_9d772_row5_col1, #T_9d772_row9_col1, #T_9d772_row13_col1, #T_9d772_row17_col1, #T_9d772_row21_col1 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_9d772_row2_col1, #T_9d772_row6_col1, #T_9d772_row10_col1, #T_9d772_row14_col1, #T_9d772_row18_col1, #T_9d772_row22_col1 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_9d772" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_9d772_level0_col0" class="col_heading level0 col0" ># participants</th>
          <th id="T_9d772_level0_col1" class="col_heading level0 col1" >Accuracy</th>
          <th id="T_9d772_level0_col2" class="col_heading level0 col2" >Accuracy IC-pairs</th>
        </tr>
        <tr>
          <th class="index_name level0" >cohort</th>
          <th class="index_name level1" >algo</th>
          <th class="index_name level2" >version</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_9d772_level0_row0" class="row_heading level0 row0" rowspan="4">HA</th>
          <th id="T_9d772_level1_row0" class="row_heading level1 row0" >McCamley</th>
          <th id="T_9d772_level2_row0" class="row_heading level2 row0" >-</th>
          <td id="T_9d772_row0_col0" class="data row0 col0" >227</td>
          <td id="T_9d772_row0_col1" class="data row0 col1" ><span>0.80 [0.77, 0.83]</span></td>
          <td id="T_9d772_row0_col2" class="data row0 col2" ><span>0.78 [0.75, 0.80]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row1" class="row_heading level1 row1" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row1" class="row_heading level2 row1" >MobGap</th>
          <td id="T_9d772_row1_col0" class="data row1 col0" >227</td>
          <td id="T_9d772_row1_col1" class="data row1 col1" ><span>0.82 [0.79, 0.84]</span></td>
          <td id="T_9d772_row1_col2" class="data row1 col2" ><span>0.82 [0.80, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row2" class="row_heading level2 row2" >Original Implementation</th>
          <td id="T_9d772_row2_col0" class="data row2 col0" >227</td>
          <td id="T_9d772_row2_col1" class="data row2 col1" ><span>0.76 [0.73, 0.79]</span></td>
          <td id="T_9d772_row2_col2" class="data row2 col2" ><span>0.74 [0.72, 0.77]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row3" class="row_heading level1 row3" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row3" class="row_heading level2 row3" >Original Implementation</th>
          <td id="T_9d772_row3_col0" class="data row3 col0" >227</td>
          <td id="T_9d772_row3_col1" class="data row3 col1" ><span>0.81 [0.79, 0.83]</span></td>
          <td id="T_9d772_row3_col2" class="data row3 col2" ><span>0.78 [0.75, 0.80]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level0_row4" class="row_heading level0 row4" rowspan="4">CHF</th>
          <th id="T_9d772_level1_row4" class="row_heading level1 row4" >McCamley</th>
          <th id="T_9d772_level2_row4" class="row_heading level2 row4" >-</th>
          <td id="T_9d772_row4_col0" class="data row4 col0" >106</td>
          <td id="T_9d772_row4_col1" class="data row4 col1" ><span>0.85 [0.82, 0.87]</span></td>
          <td id="T_9d772_row4_col2" class="data row4 col2" ><span>0.80 [0.77, 0.83]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row5" class="row_heading level1 row5" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row5" class="row_heading level2 row5" >MobGap</th>
          <td id="T_9d772_row5_col0" class="data row5 col0" >106</td>
          <td id="T_9d772_row5_col1" class="data row5 col1" ><span>0.88 [0.86, 0.90]</span></td>
          <td id="T_9d772_row5_col2" class="data row5 col2" ><span>0.86 [0.83, 0.88]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row6" class="row_heading level2 row6" >Original Implementation</th>
          <td id="T_9d772_row6_col0" class="data row6 col0" >106</td>
          <td id="T_9d772_row6_col1" class="data row6 col1" ><span>0.81 [0.79, 0.84]</span></td>
          <td id="T_9d772_row6_col2" class="data row6 col2" ><span>0.75 [0.71, 0.79]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row7" class="row_heading level1 row7" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row7" class="row_heading level2 row7" >Original Implementation</th>
          <td id="T_9d772_row7_col0" class="data row7 col0" >106</td>
          <td id="T_9d772_row7_col1" class="data row7 col1" ><span>0.73 [0.69, 0.77]</span></td>
          <td id="T_9d772_row7_col2" class="data row7 col2" ><span>0.69 [0.65, 0.73]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level0_row8" class="row_heading level0 row8" rowspan="4">COPD</th>
          <th id="T_9d772_level1_row8" class="row_heading level1 row8" >McCamley</th>
          <th id="T_9d772_level2_row8" class="row_heading level2 row8" >-</th>
          <td id="T_9d772_row8_col0" class="data row8 col0" >214</td>
          <td id="T_9d772_row8_col1" class="data row8 col1" ><span>0.80 [0.77, 0.83]</span></td>
          <td id="T_9d772_row8_col2" class="data row8 col2" ><span>0.77 [0.74, 0.80]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row9" class="row_heading level1 row9" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row9" class="row_heading level2 row9" >MobGap</th>
          <td id="T_9d772_row9_col0" class="data row9 col0" >214</td>
          <td id="T_9d772_row9_col1" class="data row9 col1" ><span>0.87 [0.85, 0.90]</span></td>
          <td id="T_9d772_row9_col2" class="data row9 col2" ><span>0.85 [0.83, 0.88]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row10" class="row_heading level2 row10" >Original Implementation</th>
          <td id="T_9d772_row10_col0" class="data row10 col0" >214</td>
          <td id="T_9d772_row10_col1" class="data row10 col1" ><span>0.79 [0.76, 0.82]</span></td>
          <td id="T_9d772_row10_col2" class="data row10 col2" ><span>0.76 [0.73, 0.78]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row11" class="row_heading level1 row11" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row11" class="row_heading level2 row11" >Original Implementation</th>
          <td id="T_9d772_row11_col0" class="data row11 col0" >214</td>
          <td id="T_9d772_row11_col1" class="data row11 col1" ><span>0.83 [0.81, 0.85]</span></td>
          <td id="T_9d772_row11_col2" class="data row11 col2" ><span>0.78 [0.75, 0.80]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level0_row12" class="row_heading level0 row12" rowspan="4">MS</th>
          <th id="T_9d772_level1_row12" class="row_heading level1 row12" >McCamley</th>
          <th id="T_9d772_level2_row12" class="row_heading level2 row12" >-</th>
          <td id="T_9d772_row12_col0" class="data row12 col0" >228</td>
          <td id="T_9d772_row12_col1" class="data row12 col1" ><span>0.79 [0.76, 0.82]</span></td>
          <td id="T_9d772_row12_col2" class="data row12 col2" ><span>0.77 [0.74, 0.79]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row13" class="row_heading level1 row13" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row13" class="row_heading level2 row13" >MobGap</th>
          <td id="T_9d772_row13_col0" class="data row13 col0" >228</td>
          <td id="T_9d772_row13_col1" class="data row13 col1" ><span>0.86 [0.84, 0.88]</span></td>
          <td id="T_9d772_row13_col2" class="data row13 col2" ><span>0.83 [0.80, 0.85]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row14" class="row_heading level2 row14" >Original Implementation</th>
          <td id="T_9d772_row14_col0" class="data row14 col0" >228</td>
          <td id="T_9d772_row14_col1" class="data row14 col1" ><span>0.81 [0.79, 0.83]</span></td>
          <td id="T_9d772_row14_col2" class="data row14 col2" ><span>0.74 [0.71, 0.77]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row15" class="row_heading level1 row15" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row15" class="row_heading level2 row15" >Original Implementation</th>
          <td id="T_9d772_row15_col0" class="data row15 col0" >228</td>
          <td id="T_9d772_row15_col1" class="data row15 col1" ><span>0.76 [0.73, 0.79]</span></td>
          <td id="T_9d772_row15_col2" class="data row15 col2" ><span>0.74 [0.71, 0.77]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level0_row16" class="row_heading level0 row16" rowspan="4">PD</th>
          <th id="T_9d772_level1_row16" class="row_heading level1 row16" >McCamley</th>
          <th id="T_9d772_level2_row16" class="row_heading level2 row16" >-</th>
          <td id="T_9d772_row16_col0" class="data row16 col0" >224</td>
          <td id="T_9d772_row16_col1" class="data row16 col1" ><span>0.77 [0.74, 0.79]</span></td>
          <td id="T_9d772_row16_col2" class="data row16 col2" ><span>0.75 [0.73, 0.78]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row17" class="row_heading level1 row17" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row17" class="row_heading level2 row17" >MobGap</th>
          <td id="T_9d772_row17_col0" class="data row17 col0" >224</td>
          <td id="T_9d772_row17_col1" class="data row17 col1" ><span>0.81 [0.79, 0.84]</span></td>
          <td id="T_9d772_row17_col2" class="data row17 col2" ><span>0.82 [0.79, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row18" class="row_heading level2 row18" >Original Implementation</th>
          <td id="T_9d772_row18_col0" class="data row18 col0" >224</td>
          <td id="T_9d772_row18_col1" class="data row18 col1" ><span>0.77 [0.74, 0.79]</span></td>
          <td id="T_9d772_row18_col2" class="data row18 col2" ><span>0.72 [0.69, 0.75]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row19" class="row_heading level1 row19" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row19" class="row_heading level2 row19" >Original Implementation</th>
          <td id="T_9d772_row19_col0" class="data row19 col0" >224</td>
          <td id="T_9d772_row19_col1" class="data row19 col1" ><span>0.75 [0.73, 0.78]</span></td>
          <td id="T_9d772_row19_col2" class="data row19 col2" ><span>0.73 [0.70, 0.75]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level0_row20" class="row_heading level0 row20" rowspan="4">PFF</th>
          <th id="T_9d772_level1_row20" class="row_heading level1 row20" >McCamley</th>
          <th id="T_9d772_level2_row20" class="row_heading level2 row20" >-</th>
          <td id="T_9d772_row20_col0" class="data row20 col0" >169</td>
          <td id="T_9d772_row20_col1" class="data row20 col1" ><span>0.83 [0.81, 0.85]</span></td>
          <td id="T_9d772_row20_col2" class="data row20 col2" ><span>0.77 [0.74, 0.80]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row21" class="row_heading level1 row21" rowspan="2">Ullrich - MS-ALL</th>
          <th id="T_9d772_level2_row21" class="row_heading level2 row21" >MobGap</th>
          <td id="T_9d772_row21_col0" class="data row21 col0" >169</td>
          <td id="T_9d772_row21_col1" class="data row21 col1" ><span>0.86 [0.84, 0.88]</span></td>
          <td id="T_9d772_row21_col2" class="data row21 col2" ><span>0.81 [0.79, 0.84]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level2_row22" class="row_heading level2 row22" >Original Implementation</th>
          <td id="T_9d772_row22_col0" class="data row22 col0" >169</td>
          <td id="T_9d772_row22_col1" class="data row22 col1" ><span>0.81 [0.79, 0.83]</span></td>
          <td id="T_9d772_row22_col2" class="data row22 col2" ><span>0.74 [0.72, 0.76]</span></td>
        </tr>
        <tr>
          <th id="T_9d772_level1_row23" class="row_heading level1 row23" >Ullrich - MS-MS</th>
          <th id="T_9d772_level2_row23" class="row_heading level2 row23" >Original Implementation</th>
          <td id="T_9d772_row23_col0" class="data row23 col0" >169</td>
          <td id="T_9d772_row23_col1" class="data row23 col1" ><span>0.81 [0.79, 0.84]</span></td>
          <td id="T_9d772_row23_col2" class="data row23 col2" ><span>0.76 [0.73, 0.79]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 3.584 seconds)

**Estimated memory usage:**  80 MB


.. _sphx_glr_download_auto_revalidation_laterality__01_lrc_analysis.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _01_lrc_analysis.ipynb <_01_lrc_analysis.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _01_lrc_analysis.py <_01_lrc_analysis.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: _01_lrc_analysis.zip <_01_lrc_analysis.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_