.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_revalidation/initial_contacts/_01_icd_analysis.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_revalidation_initial_contacts__01_icd_analysis.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_revalidation_initial_contacts__01_icd_analysis.py:


.. _icd_val_results:

Performance of the initial contact algorithms on the TVS dataset
================================================================

.. warning:: On this page you will find preliminary results for a standardized revalidation of the pipeline and all
  of its algorithm.
  The current state, **TECHNICAL EXPERIMENTATION**.
  Don't use these results or make any assumptions based on them.
  We will update this page incrementally and provide further information, as soon as the state of any of the validation
  steps changes.

The following provides an analysis and comparison of the icd performance on the TVS dataset (lab and free-living).
We look into the actual performance of the algorithms compared to the reference data and compare these results with
the performance of the original matlab algorithm.

.. note:: If you are interested in how these results are calculated, head over to the
   :ref:`processing page <icd_val_gen>`.

We focus on the `single_results` (aka the performance per trail) and will aggregate it over multiple levels.

.. GENERATED FROM PYTHON SOURCE LINES 26-29

Below are the list of algorithms that we will compare.
Note, that we use the prefix "MobGap" to refer to the reimplemented python algorithms and "Original Implementation" to
refer to the original matlab algorithms.

.. GENERATED FROM PYTHON SOURCE LINES 29-47

.. code-block:: Python


    # Note also that the IcdIonescu algorithm is the reimplementation of the Ani_McCamley algorithm in the original
    # matlab algorithms.
    # The  other two algorithms (IcdShinImproved and IcdHKLeeImproved) are actually cadence algorithms.
    # As they can also be used to detect initial contacts, we present their results as well.
    # However, you should check the dedicated cadence analysis for a more detailed comparison of these algorithms.
    algorithms = {
        "IcdIonescu": ("IcdIonescu", "MobGap"),
        "IcdShinImproved": ("IcdShinImproved", "MobGap"),
        "IcdHKLeeImproved": ("IcdHKLeeImproved", "MobGap"),
    }
    # We only load the matlab algorithms that we reimplemented
    algorithms.update(
        {
            "matlab_Ani_McCamley": ("IcdIonescu", "Original Implementation"),
        }
    )








.. GENERATED FROM PYTHON SOURCE LINES 48-55

The code below loads the data and prepares it for the analysis.
By default, the data will be downloaded from an online repository (and cached locally).
If you want to use a local copy of the data, you can set the `MOBGAP_VALIDATION_DATA_PATH` environment variable.
and the MOBGAP_VALIDATION_USE_LOCA_DATA to `1`.

The file download will print a couple log information, which can usually be ignored.
You can also change the `version` parameter to load a different version of the data.

.. GENERATED FROM PYTHON SOURCE LINES 55-111

.. code-block:: Python

    from pathlib import Path

    import pandas as pd
    from mobgap.data.validation_results import ValidationResultLoader
    from mobgap.utils.misc import get_env_var

    local_data_path = (
        Path(get_env_var("MOBGAP_VALIDATION_DATA_PATH")) / "results"
        if int(get_env_var("MOBGAP_VALIDATION_USE_LOCAL_DATA", 0))
        else None
    )
    __RESULT_VERSION = "v0.11.0"
    loader = ValidationResultLoader(
        "icd", result_path=local_data_path, version=__RESULT_VERSION
    )

    free_living_index_cols = [
        "cohort",
        "participant_id",
        "time_measure",
        "recording",
        "recording_name",
        "recording_name_pretty",
    ]

    results = {
        v: loader.load_single_results(k, "free_living")
        for k, v in algorithms.items()
    }
    results = pd.concat(results, names=["algo", "version", *free_living_index_cols])
    results_long = results.reset_index().assign(
        algo_with_version=lambda df: df["algo"] + " (" + df["version"] + ")",
        _combined="combined",
    )

    lab_index_cols = [
        "cohort",
        "participant_id",
        "time_measure",
        "test",
        "trial",
        "test_name",
        "test_name_pretty",
    ]

    lab_results = {
        v: loader.load_single_results(k, "laboratory")
        for k, v in algorithms.items()
    }
    lab_results = pd.concat(lab_results, names=["algo", "version", *lab_index_cols])
    lab_results_long = lab_results.reset_index().assign(
        algo_with_version=lambda df: df["algo"] + " (" + df["version"] + ")",
        _combined="combined",
    )

    cohort_order = ["HA", "CHF", "COPD", "MS", "PD", "PFF"]




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|                                              | 0.00/5.92k [00:00<?, ?B/s]
      0%|                                              | 0.00/5.92k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 5.92k/5.92k [00:00<00:00, 34.4MB/s]

      0%|                                              | 0.00/5.95k [00:00<?, ?B/s]
      0%|                                              | 0.00/5.95k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 5.95k/5.95k [00:00<00:00, 38.7MB/s]

      0%|                                              | 0.00/5.97k [00:00<?, ?B/s]
      0%|                                              | 0.00/5.97k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 5.97k/5.97k [00:00<00:00, 43.1MB/s]

      0%|                                              | 0.00/5.88k [00:00<?, ?B/s]
      0%|                                              | 0.00/5.88k [00:00<?, ?B/s]
    100%|█████████████████████████████████████| 5.88k/5.88k [00:00<00:00, 43.3MB/s]

      0%|                                              | 0.00/25.8k [00:00<?, ?B/s]
      0%|                                              | 0.00/25.8k [00:00<?, ?B/s]
    100%|██████████████████████████████████████| 25.8k/25.8k [00:00<00:00, 153MB/s]

      0%|                                              | 0.00/25.8k [00:00<?, ?B/s]
      0%|                                              | 0.00/25.8k [00:00<?, ?B/s]
    100%|██████████████████████████████████████| 25.8k/25.8k [00:00<00:00, 171MB/s]

      0%|                                              | 0.00/25.9k [00:00<?, ?B/s]
      0%|                                              | 0.00/25.9k [00:00<?, ?B/s]
    100%|██████████████████████████████████████| 25.9k/25.9k [00:00<00:00, 174MB/s]

      0%|                                              | 0.00/25.2k [00:00<?, ?B/s]
      0%|                                              | 0.00/25.2k [00:00<?, ?B/s]
    100%|██████████████████████████████████████| 25.2k/25.2k [00:00<00:00, 168MB/s]




.. GENERATED FROM PYTHON SOURCE LINES 112-124

Performance metrics
-------------------
For each participant, performance metrics were calculated by classifying the detected initial contacts as TP, FP or
FN matches. Based on these values, recall (sensitivity), precision (positive predictive value), F1 score were
calculated.
On top of that, absolute error for each true positive initial contact was calculated as the temporal difference
between detected and reference values. Relative error was calculated by dividing all absolute errors, within a walking
bout, by the average step duration estimated from the reference system.
From these, we calculate the mean and confidence interval for both systems, the bias and limits of agreement (LoA)
between the algorithm output and the reference data, and the ICC.

Below the functions that calculate these metrics are defined.

.. GENERATED FROM PYTHON SOURCE LINES 124-237

.. code-block:: Python

    from functools import partial

    from mobgap.pipeline.evaluation import CustomErrorAggregations as A
    from mobgap.utils.df_operations import (
        CustomOperation,
        apply_aggregations,
        apply_transformations,
        multilevel_groupby_apply_merge,
    )
    from mobgap.utils.tables import FormatTransformer as F
    from mobgap.utils.tables import RevalidationInfo, revalidation_table_styles
    from mobgap.utils.tables import StatsFunctions as S

    custom_aggs = [
        CustomOperation(
            identifier=None,
            function=A.n_datapoints,
            column_name=[("n_datapoints", "all")],
        ),
        ("recall", ["mean", A.conf_intervals]),
        ("precision", ["mean", A.conf_intervals]),
        ("f1_score", ["mean", A.conf_intervals]),
        ("tp_absolute_timing_error_s", ["mean", A.loa]),
        ("tp_relative_timing_error", ["mean", A.loa]),
    ]

    stats_transform = [
        CustomOperation(
            identifier=None,
            function=partial(
                S.pairwise_tests,
                value_col=c,
                between="version",
                reference_group_key="Original Implementation",
            ),
            column_name=[("stats_metadata", c)],
        )
        for c in [
            "recall",
            "precision",
            "f1_score",
        ]
    ]

    format_transforms = [
        CustomOperation(
            identifier=None,
            function=lambda df_: df_[("n_datapoints", "all")].astype(int),
            column_name=("General", "n_datapoints"),
        ),
        *(
            CustomOperation(
                identifier=None,
                function=partial(
                    F.value_with_metadata,
                    value_col=("mean", c),
                    other_columns={
                        "range": ("conf_intervals", c),
                        "stats_metadata": ("stats_metadata", c),
                    },
                ),
                column_name=("ICD", c),
            )
            for c in [
                "recall",
                "precision",
                "f1_score",
            ]
        ),
        *(
            CustomOperation(
                identifier=None,
                function=partial(
                    F.value_with_metadata,
                    value_col=("mean", c),
                    other_columns={"range": ("loa", c)},
                ),
                column_name=("IC Timing", c),
            )
            for c in [
                "tp_absolute_timing_error_s",
                "tp_relative_timing_error",
            ]
        ),
    ]

    final_names = {
        "n_datapoints": "# recordings",
        "recall": "Recall",
        "precision": "Precision",
        "f1_score": "F1 Score",
        "tp_absolute_timing_error_s": "Abs. Error [s]",
        "tp_relative_timing_error": "Bias and LoA",
    }


    validation_thresholds = {
        ("ICD", "Recall"): RevalidationInfo(threshold=0.7, higher_is_better=True),
        ("ICD", "Precision"): RevalidationInfo(
            threshold=0.7, higher_is_better=True
        ),
        ("ICD", "F1 Score"): RevalidationInfo(threshold=0.7, higher_is_better=True),
    }


    def format_results(df: pd.DataFrame) -> pd.DataFrame:
        return (
            df.pipe(apply_transformations, format_transforms)
            .rename(columns=final_names)
            .loc[:, pd.IndexSlice[:, list(final_names.values())]]
        )









.. GENERATED FROM PYTHON SOURCE LINES 238-246

Free-Living Comparison
----------------------
We focus the comparison on the free-living data, as this is the most relevant considering our final use-case.
In the free-living data, there is one 2.5 hour recording per participant.
This means, each datapoint in the plots below and in the summary statistics represents one participant.

All results across all cohorts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. GENERATED FROM PYTHON SOURCE LINES 246-281

.. code-block:: Python

    import matplotlib.pyplot as plt
    import seaborn as sns

    hue_order = ["Original Implementation", "MobGap"]

    fig, ax = plt.subplots()
    sns.boxplot(
        data=results_long,
        x="algo",
        y="f1_score",
        hue="version",
        hue_order=hue_order,
        ax=ax,
    )
    fig.show()

    perf_metrics_all = results_long.pipe(
        multilevel_groupby_apply_merge,
        [
            (
                ["algo", "version"],
                partial(apply_aggregations, aggregations=custom_aggs),
            ),
            (
                ["algo"],
                partial(apply_transformations, transformations=stats_transform),
            ),
        ],
    ).pipe(format_results)
    perf_metrics_all.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["algo"],
    )




.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_001.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]

    <pandas.io.formats.style.Styler object at 0x7c10efad2140>



.. GENERATED FROM PYTHON SOURCE LINES 282-285

Per Cohort
~~~~~~~~~~
While this provides a good overview, it does not fully reflect how these algorithms perform on the different cohorts.

.. GENERATED FROM PYTHON SOURCE LINES 285-315

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=results_long, x="cohort", y="f1_score", hue="algo_with_version", ax=ax
    )
    fig.show()

    perf_metrics_per_cohort = (
        results_long.pipe(
            multilevel_groupby_apply_merge,
            [
                (
                    ["cohort", "algo", "version"],
                    partial(apply_aggregations, aggregations=custom_aggs),
                ),
                (
                    ["cohort", "algo"],
                    partial(apply_transformations, transformations=stats_transform),
                ),
            ],
        )
        .pipe(format_results)
        .loc[cohort_order]
    )
    perf_metrics_per_cohort.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["cohort", "algo"],
    )





.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_002.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_002.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]

    <pandas.io.formats.style.Styler object at 0x7c10f19d5b10>



.. GENERATED FROM PYTHON SOURCE LINES 316-321

Only relevant algorithms
~~~~~~~~~~~~~~~~~~~~~~~~
Finally, we present comparison of the old and new implementations of IcdIonescu. IcdShinImproved and
IcdHKLeeImproved are excluded because they are cadence algorithms and we don't calculate ICs with these algos in the
old Matlab implementation.

.. GENERATED FROM PYTHON SOURCE LINES 321-343

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=results_long.query("algo == 'IcdIonescu'"),
        x="cohort",
        y="f1_score",
        hue="algo_with_version",
        ax=ax,
    )
    fig.show()

    final_perf_metrics = (
        perf_metrics_per_cohort.copy()
        .query("algo == 'IcdIonescu'")
        .reset_index(level="algo", drop=True)
    )

    final_perf_metrics.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["cohort"],
    )




.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_003.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_003.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_c04c1_row0_col1, #T_c04c1_row0_col2, #T_c04c1_row0_col3, #T_c04c1_row4_col1, #T_c04c1_row4_col2, #T_c04c1_row4_col3, #T_c04c1_row6_col2, #T_c04c1_row8_col1, #T_c04c1_row8_col3, #T_c04c1_row10_col1 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_c04c1_row1_col0, #T_c04c1_row1_col4, #T_c04c1_row1_col5, #T_c04c1_row3_col0, #T_c04c1_row3_col4, #T_c04c1_row3_col5, #T_c04c1_row5_col0, #T_c04c1_row5_col4, #T_c04c1_row5_col5, #T_c04c1_row7_col0, #T_c04c1_row7_col4, #T_c04c1_row7_col5, #T_c04c1_row9_col0, #T_c04c1_row9_col4, #T_c04c1_row9_col5, #T_c04c1_row11_col0, #T_c04c1_row11_col4, #T_c04c1_row11_col5 {
      border-bottom: 2px solid black;
    }
    #T_c04c1_row1_col1, #T_c04c1_row1_col2, #T_c04c1_row1_col3, #T_c04c1_row5_col1, #T_c04c1_row5_col2, #T_c04c1_row5_col3, #T_c04c1_row7_col2, #T_c04c1_row9_col1, #T_c04c1_row9_col3, #T_c04c1_row11_col1 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_c04c1_row2_col1, #T_c04c1_row2_col2, #T_c04c1_row2_col3, #T_c04c1_row6_col1, #T_c04c1_row6_col3, #T_c04c1_row8_col2, #T_c04c1_row10_col2, #T_c04c1_row10_col3 {
      background-color: lightgreen;
    }
    #T_c04c1_row3_col1, #T_c04c1_row3_col2, #T_c04c1_row3_col3, #T_c04c1_row7_col1, #T_c04c1_row7_col3, #T_c04c1_row9_col2, #T_c04c1_row11_col2, #T_c04c1_row11_col3 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_c04c1" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_c04c1_level0_col0" class="col_heading level0 col0" >General</th>
          <th id="T_c04c1_level0_col1" class="col_heading level0 col1" colspan="3">ICD</th>
          <th id="T_c04c1_level0_col4" class="col_heading level0 col4" colspan="2">IC Timing</th>
        </tr>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level1" >&nbsp;</th>
          <th id="T_c04c1_level1_col0" class="col_heading level1 col0" ># recordings</th>
          <th id="T_c04c1_level1_col1" class="col_heading level1 col1" >Recall</th>
          <th id="T_c04c1_level1_col2" class="col_heading level1 col2" >Precision</th>
          <th id="T_c04c1_level1_col3" class="col_heading level1 col3" >F1 Score</th>
          <th id="T_c04c1_level1_col4" class="col_heading level1 col4" >Abs. Error [s]</th>
          <th id="T_c04c1_level1_col5" class="col_heading level1 col5" >Bias and LoA</th>
        </tr>
        <tr>
          <th class="index_name level0" >cohort</th>
          <th class="index_name level1" >&nbsp;</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
          <th class="blank col3" >&nbsp;</th>
          <th class="blank col4" >&nbsp;</th>
          <th class="blank col5" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_c04c1_level0_row0" class="row_heading level0 row0" rowspan="2">HA</th>
          <th id="T_c04c1_level1_row0" class="row_heading level1 row0" >MobGap</th>
          <td id="T_c04c1_row0_col0" class="data row0 col0" >20</td>
          <td id="T_c04c1_row0_col1" class="data row0 col1" ><span>0.92 [0.91, 0.94]</span></td>
          <td id="T_c04c1_row0_col2" class="data row0 col2" ><span>0.94 [0.93, 0.96]</span></td>
          <td id="T_c04c1_row0_col3" class="data row0 col3" ><span>0.93 [0.92, 0.94]</span></td>
          <td id="T_c04c1_row0_col4" class="data row0 col4" ><span>0.05 [0.03, 0.08]</span></td>
          <td id="T_c04c1_row0_col5" class="data row0 col5" ><span>0.08 [0.05, 0.11]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row1" class="row_heading level1 row1" >Original Implementation</th>
          <td id="T_c04c1_row1_col0" class="data row1 col0" >20</td>
          <td id="T_c04c1_row1_col1" class="data row1 col1" ><span>0.92 [0.90, 0.94]</span></td>
          <td id="T_c04c1_row1_col2" class="data row1 col2" ><span>0.94 [0.93, 0.96]</span></td>
          <td id="T_c04c1_row1_col3" class="data row1 col3" ><span>0.93 [0.92, 0.94]</span></td>
          <td id="T_c04c1_row1_col4" class="data row1 col4" ><span>0.05 [0.02, 0.08]</span></td>
          <td id="T_c04c1_row1_col5" class="data row1 col5" ><span>0.08 [0.04, 0.11]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level0_row2" class="row_heading level0 row2" rowspan="2">CHF</th>
          <th id="T_c04c1_level1_row2" class="row_heading level1 row2" >MobGap</th>
          <td id="T_c04c1_row2_col0" class="data row2 col0" >10</td>
          <td id="T_c04c1_row2_col1" class="data row2 col1" ><span>0.92 [0.89, 0.95]</span></td>
          <td id="T_c04c1_row2_col2" class="data row2 col2" ><span>0.95 [0.93, 0.97]</span></td>
          <td id="T_c04c1_row2_col3" class="data row2 col3" ><span>0.94 [0.91, 0.96]</span></td>
          <td id="T_c04c1_row2_col4" class="data row2 col4" ><span>0.06 [0.01, 0.10]</span></td>
          <td id="T_c04c1_row2_col5" class="data row2 col5" ><span>0.08 [0.02, 0.14]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row3" class="row_heading level1 row3" >Original Implementation</th>
          <td id="T_c04c1_row3_col0" class="data row3 col0" >10</td>
          <td id="T_c04c1_row3_col1" class="data row3 col1" ><span>0.93 [0.90, 0.95]</span></td>
          <td id="T_c04c1_row3_col2" class="data row3 col2" ><span>0.95 [0.94, 0.97]</span></td>
          <td id="T_c04c1_row3_col3" class="data row3 col3" ><span>0.94 [0.92, 0.96]</span></td>
          <td id="T_c04c1_row3_col4" class="data row3 col4" ><span>0.05 [0.00, 0.10]</span></td>
          <td id="T_c04c1_row3_col5" class="data row3 col5" ><span>0.08 [0.01, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level0_row4" class="row_heading level0 row4" rowspan="2">COPD</th>
          <th id="T_c04c1_level1_row4" class="row_heading level1 row4" >MobGap</th>
          <td id="T_c04c1_row4_col0" class="data row4 col0" >17</td>
          <td id="T_c04c1_row4_col1" class="data row4 col1" ><span>0.89 [0.87, 0.90]</span></td>
          <td id="T_c04c1_row4_col2" class="data row4 col2" ><span>0.92 [0.91, 0.94]</span></td>
          <td id="T_c04c1_row4_col3" class="data row4 col3" ><span>0.91 [0.89, 0.92]</span></td>
          <td id="T_c04c1_row4_col4" class="data row4 col4" ><span>0.07 [0.02, 0.12]</span></td>
          <td id="T_c04c1_row4_col5" class="data row4 col5" ><span>0.09 [0.04, 0.14]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row5" class="row_heading level1 row5" >Original Implementation</th>
          <td id="T_c04c1_row5_col0" class="data row5 col0" >17</td>
          <td id="T_c04c1_row5_col1" class="data row5 col1" ><span>0.89 [0.87, 0.90]</span></td>
          <td id="T_c04c1_row5_col2" class="data row5 col2" ><span>0.92 [0.90, 0.94]</span></td>
          <td id="T_c04c1_row5_col3" class="data row5 col3" ><span>0.90 [0.89, 0.92]</span></td>
          <td id="T_c04c1_row5_col4" class="data row5 col4" ><span>0.07 [0.03, 0.12]</span></td>
          <td id="T_c04c1_row5_col5" class="data row5 col5" ><span>0.09 [0.04, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level0_row6" class="row_heading level0 row6" rowspan="2">MS</th>
          <th id="T_c04c1_level1_row6" class="row_heading level1 row6" >MobGap</th>
          <td id="T_c04c1_row6_col0" class="data row6 col0" >18</td>
          <td id="T_c04c1_row6_col1" class="data row6 col1" ><span>0.93 [0.91, 0.95]</span></td>
          <td id="T_c04c1_row6_col2" class="data row6 col2" ><span>0.94 [0.92, 0.96]</span></td>
          <td id="T_c04c1_row6_col3" class="data row6 col3" ><span>0.93 [0.91, 0.95]</span></td>
          <td id="T_c04c1_row6_col4" class="data row6 col4" ><span>0.06 [-0.00, 0.13]</span></td>
          <td id="T_c04c1_row6_col5" class="data row6 col5" ><span>0.09 [0.02, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row7" class="row_heading level1 row7" >Original Implementation</th>
          <td id="T_c04c1_row7_col0" class="data row7 col0" >18</td>
          <td id="T_c04c1_row7_col1" class="data row7 col1" ><span>0.93 [0.91, 0.95]</span></td>
          <td id="T_c04c1_row7_col2" class="data row7 col2" ><span>0.94 [0.92, 0.96]</span></td>
          <td id="T_c04c1_row7_col3" class="data row7 col3" ><span>0.93 [0.91, 0.96]</span></td>
          <td id="T_c04c1_row7_col4" class="data row7 col4" ><span>0.06 [-0.00, 0.13]</span></td>
          <td id="T_c04c1_row7_col5" class="data row7 col5" ><span>0.09 [0.03, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level0_row8" class="row_heading level0 row8" rowspan="2">PD</th>
          <th id="T_c04c1_level1_row8" class="row_heading level1 row8" >MobGap</th>
          <td id="T_c04c1_row8_col0" class="data row8 col0" >19</td>
          <td id="T_c04c1_row8_col1" class="data row8 col1" ><span>0.92 [0.89, 0.95]</span></td>
          <td id="T_c04c1_row8_col2" class="data row8 col2" ><span>0.94 [0.91, 0.97]</span></td>
          <td id="T_c04c1_row8_col3" class="data row8 col3" ><span>0.93 [0.90, 0.96]</span></td>
          <td id="T_c04c1_row8_col4" class="data row8 col4" ><span>0.06 [0.01, 0.11]</span></td>
          <td id="T_c04c1_row8_col5" class="data row8 col5" ><span>0.09 [0.04, 0.14]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row9" class="row_heading level1 row9" >Original Implementation</th>
          <td id="T_c04c1_row9_col0" class="data row9 col0" >19</td>
          <td id="T_c04c1_row9_col1" class="data row9 col1" ><span>0.92 [0.89, 0.95]</span></td>
          <td id="T_c04c1_row9_col2" class="data row9 col2" ><span>0.94 [0.91, 0.97]</span></td>
          <td id="T_c04c1_row9_col3" class="data row9 col3" ><span>0.93 [0.90, 0.96]</span></td>
          <td id="T_c04c1_row9_col4" class="data row9 col4" ><span>0.06 [0.01, 0.10]</span></td>
          <td id="T_c04c1_row9_col5" class="data row9 col5" ><span>0.08 [0.03, 0.14]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level0_row10" class="row_heading level0 row10" rowspan="2">PFF</th>
          <th id="T_c04c1_level1_row10" class="row_heading level1 row10" >MobGap</th>
          <td id="T_c04c1_row10_col0" class="data row10 col0" >17</td>
          <td id="T_c04c1_row10_col1" class="data row10 col1" ><span>0.84 [0.73, 0.95]</span></td>
          <td id="T_c04c1_row10_col2" class="data row10 col2" ><span>0.84 [0.73, 0.95]</span></td>
          <td id="T_c04c1_row10_col3" class="data row10 col3" ><span>0.84 [0.73, 0.94]</span></td>
          <td id="T_c04c1_row10_col4" class="data row10 col4" ><span>0.07 [0.02, 0.11]</span></td>
          <td id="T_c04c1_row10_col5" class="data row10 col5" ><span>0.08 [0.03, 0.13]</span></td>
        </tr>
        <tr>
          <th id="T_c04c1_level1_row11" class="row_heading level1 row11" >Original Implementation</th>
          <td id="T_c04c1_row11_col0" class="data row11 col0" >17</td>
          <td id="T_c04c1_row11_col1" class="data row11 col1" ><span>0.84 [0.73, 0.95]</span></td>
          <td id="T_c04c1_row11_col2" class="data row11 col2" ><span>0.84 [0.73, 0.95]</span></td>
          <td id="T_c04c1_row11_col3" class="data row11 col3" ><span>0.84 [0.73, 0.95]</span></td>
          <td id="T_c04c1_row11_col4" class="data row11 col4" ><span>0.06 [0.02, 0.11]</span></td>
          <td id="T_c04c1_row11_col5" class="data row11 col5" ><span>0.08 [0.03, 0.13]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 344-352

Laboratory Comparison
---------------------
Every datapoint below is one trial of a test.
Note, that each datapoint is weighted equally in the calculation of the performance metrics.
This is a limitation of this simple approach, as the number of strides per trial and the complexity of the context
can vary significantly.
For a full picture, different groups of tests should be analyzed separately.
The approach below should still provide a good overview to compare the algorithms.

.. GENERATED FROM PYTHON SOURCE LINES 352-384

.. code-block:: Python

    hue_order = ["Original Implementation", "MobGap"]

    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results_long,
        x="algo",
        y="f1_score",
        hue="version",
        hue_order=hue_order,
        ax=ax,
    )
    fig.show()

    perf_metrics_all = lab_results_long.pipe(
        multilevel_groupby_apply_merge,
        [
            (
                ["algo", "version"],
                partial(apply_aggregations, aggregations=custom_aggs),
            ),
            (
                ["algo"],
                partial(apply_transformations, transformations=stats_transform),
            ),
        ],
    ).pipe(format_results)
    perf_metrics_all.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["algo"],
    )




.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_004.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_004.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]

    <pandas.io.formats.style.Styler object at 0x7c10d776f370>



.. GENERATED FROM PYTHON SOURCE LINES 385-388

Per Cohort
~~~~~~~~~~
While this provides a good overview, it does not fully reflect how these algorithms perform on the different cohorts.

.. GENERATED FROM PYTHON SOURCE LINES 388-422

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results_long,
        x="cohort",
        y="f1_score",
        hue="algo_with_version",
        ax=ax,
    )
    fig.show()

    perf_metrics_per_cohort = (
        lab_results_long.pipe(
            multilevel_groupby_apply_merge,
            [
                (
                    ["cohort", "algo", "version"],
                    partial(apply_aggregations, aggregations=custom_aggs),
                ),
                (
                    ["cohort", "algo"],
                    partial(apply_transformations, transformations=stats_transform),
                ),
            ],
        )
        .pipe(format_results)
        .loc[cohort_order]
    )
    perf_metrics_per_cohort.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["cohort", "algo"],
    )





.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_005.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_005.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]
    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.11.0/src/mobgap/utils/df_operations.py:703: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
      results = [df.groupby(key).apply(func, **apply_kwargs) for key, func in groupbys]

    <pandas.io.formats.style.Styler object at 0x7c10ef8a0490>



.. GENERATED FROM PYTHON SOURCE LINES 423-428

Only relevant algorithms
~~~~~~~~~~~~~~~~~~~~~~~~
Finally, we present comparison of the old and new implementations of IcdIonescu. IcdShinImproved and
IcdHKLeeImproved are excluded because they are cadence algorithms and we don't calculate ICs with these algos in the
old Matlab implementation.

.. GENERATED FROM PYTHON SOURCE LINES 428-449

.. code-block:: Python

    fig, ax = plt.subplots()
    sns.boxplot(
        data=lab_results_long.query("algo == 'IcdIonescu'"),
        x="cohort",
        y="f1_score",
        hue="algo_with_version",
        ax=ax,
    )
    fig.show()

    final_perf_metrics = (
        perf_metrics_per_cohort.copy()
        .query("algo == 'IcdIonescu'")
        .reset_index(level="algo", drop=True)
    )

    final_perf_metrics.style.pipe(
        revalidation_table_styles,
        validation_thresholds,
        ["cohort"],
    )



.. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_006.png
   :alt:  01 icd analysis
   :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_006.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_e3480_row0_col1 {
      font-weight: bold;
      background-color: lightcoral;
    }
    #T_e3480_row0_col2, #T_e3480_row0_col3, #T_e3480_row2_col1, #T_e3480_row2_col2, #T_e3480_row2_col3, #T_e3480_row6_col1, #T_e3480_row6_col2, #T_e3480_row6_col3, #T_e3480_row8_col1, #T_e3480_row8_col2, #T_e3480_row8_col3, #T_e3480_row10_col1, #T_e3480_row10_col2, #T_e3480_row10_col3 {
      font-weight: bold;
      background-color: lightgreen;
    }
    #T_e3480_row1_col0, #T_e3480_row1_col4, #T_e3480_row1_col5, #T_e3480_row3_col0, #T_e3480_row3_col4, #T_e3480_row3_col5, #T_e3480_row5_col0, #T_e3480_row5_col4, #T_e3480_row5_col5, #T_e3480_row7_col0, #T_e3480_row7_col4, #T_e3480_row7_col5, #T_e3480_row9_col0, #T_e3480_row9_col4, #T_e3480_row9_col5, #T_e3480_row11_col0, #T_e3480_row11_col4, #T_e3480_row11_col5 {
      border-bottom: 2px solid black;
    }
    #T_e3480_row1_col1, #T_e3480_row9_col1 {
      background-color: lightcoral;
      border-bottom: 2px solid black;
    }
    #T_e3480_row1_col2, #T_e3480_row1_col3, #T_e3480_row3_col1, #T_e3480_row3_col2, #T_e3480_row3_col3, #T_e3480_row7_col1, #T_e3480_row7_col2, #T_e3480_row7_col3, #T_e3480_row9_col2, #T_e3480_row9_col3, #T_e3480_row11_col1, #T_e3480_row11_col2, #T_e3480_row11_col3 {
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    #T_e3480_row4_col1 {
      background-color: lightcoral;
    }
    #T_e3480_row4_col2, #T_e3480_row4_col3 {
      background-color: lightgreen;
    }
    #T_e3480_row5_col1 {
      font-weight: bold;
      background-color: lightcoral;
      border-bottom: 2px solid black;
    }
    #T_e3480_row5_col2, #T_e3480_row5_col3 {
      font-weight: bold;
      background-color: lightgreen;
      border-bottom: 2px solid black;
    }
    </style>
    <table id="T_e3480" class="dataframe">
      <thead>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_e3480_level0_col0" class="col_heading level0 col0" >General</th>
          <th id="T_e3480_level0_col1" class="col_heading level0 col1" colspan="3">ICD</th>
          <th id="T_e3480_level0_col4" class="col_heading level0 col4" colspan="2">IC Timing</th>
        </tr>
        <tr>
          <th class="blank" >&nbsp;</th>
          <th class="blank level1" >&nbsp;</th>
          <th id="T_e3480_level1_col0" class="col_heading level1 col0" ># recordings</th>
          <th id="T_e3480_level1_col1" class="col_heading level1 col1" >Recall</th>
          <th id="T_e3480_level1_col2" class="col_heading level1 col2" >Precision</th>
          <th id="T_e3480_level1_col3" class="col_heading level1 col3" >F1 Score</th>
          <th id="T_e3480_level1_col4" class="col_heading level1 col4" >Abs. Error [s]</th>
          <th id="T_e3480_level1_col5" class="col_heading level1 col5" >Bias and LoA</th>
        </tr>
        <tr>
          <th class="index_name level0" >cohort</th>
          <th class="index_name level1" >&nbsp;</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
          <th class="blank col3" >&nbsp;</th>
          <th class="blank col4" >&nbsp;</th>
          <th class="blank col5" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_e3480_level0_row0" class="row_heading level0 row0" rowspan="2">HA</th>
          <th id="T_e3480_level1_row0" class="row_heading level1 row0" >MobGap</th>
          <td id="T_e3480_row0_col0" class="data row0 col0" >227</td>
          <td id="T_e3480_row0_col1" class="data row0 col1" ><span>0.67 [0.63, 0.71]</span></td>
          <td id="T_e3480_row0_col2" class="data row0 col2" ><span>0.82 [0.78, 0.87]</span></td>
          <td id="T_e3480_row0_col3" class="data row0 col3" ><span>0.74 [0.69, 0.78]</span></td>
          <td id="T_e3480_row0_col4" class="data row0 col4" ><span>0.04 [-0.03, 0.11]</span></td>
          <td id="T_e3480_row0_col5" class="data row0 col5" ><span>0.06 [-0.04, 0.16]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row1" class="row_heading level1 row1" >Original Implementation</th>
          <td id="T_e3480_row1_col0" class="data row1 col0" >227</td>
          <td id="T_e3480_row1_col1" class="data row1 col1" ><span>0.67 [0.63, 0.71]</span></td>
          <td id="T_e3480_row1_col2" class="data row1 col2" ><span>0.82 [0.77, 0.87]</span></td>
          <td id="T_e3480_row1_col3" class="data row1 col3" ><span>0.73 [0.69, 0.78]</span></td>
          <td id="T_e3480_row1_col4" class="data row1 col4" ><span>0.04 [-0.03, 0.10]</span></td>
          <td id="T_e3480_row1_col5" class="data row1 col5" ><span>0.06 [-0.04, 0.16]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level0_row2" class="row_heading level0 row2" rowspan="2">CHF</th>
          <th id="T_e3480_level1_row2" class="row_heading level1 row2" >MobGap</th>
          <td id="T_e3480_row2_col0" class="data row2 col0" >106</td>
          <td id="T_e3480_row2_col1" class="data row2 col1" ><span>0.74 [0.70, 0.79]</span></td>
          <td id="T_e3480_row2_col2" class="data row2 col2" ><span>0.89 [0.84, 0.94]</span></td>
          <td id="T_e3480_row2_col3" class="data row2 col3" ><span>0.81 [0.76, 0.85]</span></td>
          <td id="T_e3480_row2_col4" class="data row2 col4" ><span>0.06 [-0.02, 0.13]</span></td>
          <td id="T_e3480_row2_col5" class="data row2 col5" ><span>0.08 [-0.02, 0.18]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row3" class="row_heading level1 row3" >Original Implementation</th>
          <td id="T_e3480_row3_col0" class="data row3 col0" >106</td>
          <td id="T_e3480_row3_col1" class="data row3 col1" ><span>0.74 [0.70, 0.79]</span></td>
          <td id="T_e3480_row3_col2" class="data row3 col2" ><span>0.89 [0.84, 0.94]</span></td>
          <td id="T_e3480_row3_col3" class="data row3 col3" ><span>0.80 [0.76, 0.85]</span></td>
          <td id="T_e3480_row3_col4" class="data row3 col4" ><span>0.05 [-0.02, 0.12]</span></td>
          <td id="T_e3480_row3_col5" class="data row3 col5" ><span>0.08 [-0.02, 0.17]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level0_row4" class="row_heading level0 row4" rowspan="2">COPD</th>
          <th id="T_e3480_level1_row4" class="row_heading level1 row4" >MobGap</th>
          <td id="T_e3480_row4_col0" class="data row4 col0" >214</td>
          <td id="T_e3480_row4_col1" class="data row4 col1" ><span>0.68 [0.64, 0.72]</span></td>
          <td id="T_e3480_row4_col2" class="data row4 col2" ><span>0.83 [0.78, 0.88]</span></td>
          <td id="T_e3480_row4_col3" class="data row4 col3" ><span>0.74 [0.70, 0.79]</span></td>
          <td id="T_e3480_row4_col4" class="data row4 col4" ><span>0.04 [-0.02, 0.10]</span></td>
          <td id="T_e3480_row4_col5" class="data row4 col5" ><span>0.06 [-0.03, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row5" class="row_heading level1 row5" >Original Implementation</th>
          <td id="T_e3480_row5_col0" class="data row5 col0" >214</td>
          <td id="T_e3480_row5_col1" class="data row5 col1" ><span>0.68 [0.64, 0.72]</span></td>
          <td id="T_e3480_row5_col2" class="data row5 col2" ><span>0.83 [0.78, 0.88]</span></td>
          <td id="T_e3480_row5_col3" class="data row5 col3" ><span>0.74 [0.70, 0.79]</span></td>
          <td id="T_e3480_row5_col4" class="data row5 col4" ><span>0.04 [-0.02, 0.10]</span></td>
          <td id="T_e3480_row5_col5" class="data row5 col5" ><span>0.06 [-0.03, 0.16]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level0_row6" class="row_heading level0 row6" rowspan="2">MS</th>
          <th id="T_e3480_level1_row6" class="row_heading level1 row6" >MobGap</th>
          <td id="T_e3480_row6_col0" class="data row6 col0" >228</td>
          <td id="T_e3480_row6_col1" class="data row6 col1" ><span>0.78 [0.76, 0.81]</span></td>
          <td id="T_e3480_row6_col2" class="data row6 col2" ><span>0.94 [0.91, 0.96]</span></td>
          <td id="T_e3480_row6_col3" class="data row6 col3" ><span>0.85 [0.83, 0.87]</span></td>
          <td id="T_e3480_row6_col4" class="data row6 col4" ><span>0.07 [-0.02, 0.16]</span></td>
          <td id="T_e3480_row6_col5" class="data row6 col5" ><span>0.10 [-0.00, 0.21]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row7" class="row_heading level1 row7" >Original Implementation</th>
          <td id="T_e3480_row7_col0" class="data row7 col0" >228</td>
          <td id="T_e3480_row7_col1" class="data row7 col1" ><span>0.75 [0.72, 0.78]</span></td>
          <td id="T_e3480_row7_col2" class="data row7 col2" ><span>0.90 [0.87, 0.94]</span></td>
          <td id="T_e3480_row7_col3" class="data row7 col3" ><span>0.82 [0.79, 0.85]</span></td>
          <td id="T_e3480_row7_col4" class="data row7 col4" ><span>0.06 [-0.02, 0.15]</span></td>
          <td id="T_e3480_row7_col5" class="data row7 col5" ><span>0.09 [-0.01, 0.20]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level0_row8" class="row_heading level0 row8" rowspan="2">PD</th>
          <th id="T_e3480_level1_row8" class="row_heading level1 row8" >MobGap</th>
          <td id="T_e3480_row8_col0" class="data row8 col0" >225</td>
          <td id="T_e3480_row8_col1" class="data row8 col1" ><span>0.71 [0.67, 0.75]</span></td>
          <td id="T_e3480_row8_col2" class="data row8 col2" ><span>0.85 [0.81, 0.89]</span></td>
          <td id="T_e3480_row8_col3" class="data row8 col3" ><span>0.77 [0.73, 0.81]</span></td>
          <td id="T_e3480_row8_col4" class="data row8 col4" ><span>0.05 [-0.02, 0.11]</span></td>
          <td id="T_e3480_row8_col5" class="data row8 col5" ><span>0.07 [-0.02, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row9" class="row_heading level1 row9" >Original Implementation</th>
          <td id="T_e3480_row9_col0" class="data row9 col0" >225</td>
          <td id="T_e3480_row9_col1" class="data row9 col1" ><span>0.69 [0.65, 0.73]</span></td>
          <td id="T_e3480_row9_col2" class="data row9 col2" ><span>0.83 [0.78, 0.88]</span></td>
          <td id="T_e3480_row9_col3" class="data row9 col3" ><span>0.75 [0.71, 0.79]</span></td>
          <td id="T_e3480_row9_col4" class="data row9 col4" ><span>0.04 [-0.02, 0.11]</span></td>
          <td id="T_e3480_row9_col5" class="data row9 col5" ><span>0.06 [-0.02, 0.15]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level0_row10" class="row_heading level0 row10" rowspan="2">PFF</th>
          <th id="T_e3480_level1_row10" class="row_heading level1 row10" >MobGap</th>
          <td id="T_e3480_row10_col0" class="data row10 col0" >169</td>
          <td id="T_e3480_row10_col1" class="data row10 col1" ><span>0.83 [0.81, 0.84]</span></td>
          <td id="T_e3480_row10_col2" class="data row10 col2" ><span>0.95 [0.93, 0.96]</span></td>
          <td id="T_e3480_row10_col3" class="data row10 col3" ><span>0.88 [0.87, 0.89]</span></td>
          <td id="T_e3480_row10_col4" class="data row10 col4" ><span>0.06 [-0.00, 0.13]</span></td>
          <td id="T_e3480_row10_col5" class="data row10 col5" ><span>0.09 [0.02, 0.16]</span></td>
        </tr>
        <tr>
          <th id="T_e3480_level1_row11" class="row_heading level1 row11" >Original Implementation</th>
          <td id="T_e3480_row11_col0" class="data row11 col0" >169</td>
          <td id="T_e3480_row11_col1" class="data row11 col1" ><span>0.81 [0.79, 0.83]</span></td>
          <td id="T_e3480_row11_col2" class="data row11 col2" ><span>0.93 [0.90, 0.95]</span></td>
          <td id="T_e3480_row11_col3" class="data row11 col3" ><span>0.86 [0.84, 0.88]</span></td>
          <td id="T_e3480_row11_col4" class="data row11 col4" ><span>0.06 [-0.01, 0.13]</span></td>
          <td id="T_e3480_row11_col5" class="data row11 col5" ><span>0.08 [0.01, 0.15]</span></td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 4.450 seconds)

**Estimated memory usage:**  80 MB


.. _sphx_glr_download_auto_revalidation_initial_contacts__01_icd_analysis.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _01_icd_analysis.ipynb <_01_icd_analysis.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _01_icd_analysis.py <_01_icd_analysis.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: _01_icd_analysis.zip <_01_icd_analysis.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_