.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_revalidation/initial_contacts/_01_icd_analysis.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_revalidation_initial_contacts__01_icd_analysis.py: .. _icd_val_results: Performance of the initial contact algorithms on the TVS dataset ================================================================ The following provides an analysis and comparison of the icd performance on the TVS dataset (lab and free-living). We look into the actual performance of the algorithms compared to the reference data and compare these results with the performance of the original matlab algorithm. .. note:: If you are interested in how these results are calculated, head over to the :ref:`processing page `. We focus on the `single_results` (aka the performance per trail) and will aggregate it over multiple levels. .. GENERATED FROM PYTHON SOURCE LINES 19-22 Below are the list of algorithms that we will compare. Note, that we use the prefix "MobGap" to refer to the reimplemented python algorithms and "Original Implementation" to refer to the original matlab algorithms. .. GENERATED FROM PYTHON SOURCE LINES 22-40 .. code-block:: Python # Note also that the IcdIonescu algorithm is the reimplementation of the Ani_McCamley algorithm in the original # matlab algorithms. # The other two algorithms (IcdShinImproved and IcdHKLeeImproved) are actually cadence algorithms. # As they can also be used to detect initial contacts, we present their results as well. # However, you should check the dedicated cadence analysis for a more detailed comparison of these algorithms. algorithms = { "IcdIonescu": ("IcdIonescu", "MobGap"), "IcdShinImproved": ("IcdShinImproved", "MobGap"), "IcdHKLeeImproved": ("IcdHKLeeImproved", "MobGap"), } # We only load the matlab algorithms that we reimplemented algorithms.update( { "matlab_Ani_McCamley": ("IcdIonescu", "Original Implementation"), } ) .. GENERATED FROM PYTHON SOURCE LINES 41-48 The code below loads the data and prepares it for the analysis. By default, the data will be downloaded from an online repository (and cached locally). If you want to use a local copy of the data, you can set the `MOBGAP_VALIDATION_DATA_PATH` environment variable. and the MOBGAP_VALIDATION_USE_LOCA_DATA to `1`. The file download will print a couple log information, which can usually be ignored. You can also change the `version` parameter to load a different version of the data. .. GENERATED FROM PYTHON SOURCE LINES 48-104 .. code-block:: Python from pathlib import Path import pandas as pd from mobgap.data.validation_results import ValidationResultLoader from mobgap.utils.misc import get_env_var local_data_path = ( Path(get_env_var("MOBGAP_VALIDATION_DATA_PATH")) / "results" if int(get_env_var("MOBGAP_VALIDATION_USE_LOCAL_DATA", 0)) else None ) __RESULT_VERSION = "v1.2.0" loader = ValidationResultLoader( "icd", result_path=local_data_path, version=__RESULT_VERSION ) free_living_index_cols = [ "cohort", "participant_id", "time_measure", "recording", "recording_name", "recording_name_pretty", ] results = { v: loader.load_single_results(k, "free_living") for k, v in algorithms.items() } results = pd.concat(results, names=["algo", "version", *free_living_index_cols]) results_long = results.reset_index().assign( algo_with_version=lambda df: df["algo"] + " (" + df["version"] + ")", _combined="combined", ) lab_index_cols = [ "cohort", "participant_id", "time_measure", "test", "trial", "test_name", "test_name_pretty", ] lab_results = { v: loader.load_single_results(k, "laboratory") for k, v in algorithms.items() } lab_results = pd.concat(lab_results, names=["algo", "version", *lab_index_cols]) lab_results_long = lab_results.reset_index().assign( algo_with_version=lambda df: df["algo"] + " (" + df["version"] + ")", _combined="combined", ) cohort_order = ["HA", "CHF", "COPD", "MS", "PD", "PFF"] .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/5.94k [00:00 pd.DataFrame: return ( df.pipe(apply_transformations, format_transforms) .rename(columns=final_names) .loc[:, pd.IndexSlice[:, list(final_names.values())]] ) .. GENERATED FROM PYTHON SOURCE LINES 231-239 Free-Living Comparison ---------------------- We focus the comparison on the free-living data, as this is the most relevant considering our final use-case. In the free-living data, there is one 2.5 hour recording per participant. This means, each datapoint in the plots below and in the summary statistics represents one participant. All results across all cohorts ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 239-274 .. code-block:: Python import matplotlib.pyplot as plt import seaborn as sns hue_order = ["Original Implementation", "MobGap"] fig, ax = plt.subplots() sns.boxplot( data=results_long, x="algo", y="f1_score", hue="version", hue_order=hue_order, ax=ax, ) fig.show() perf_metrics_all = results_long.pipe( multilevel_groupby_apply_merge, [ ( ["algo", "version"], partial(apply_aggregations, aggregations=custom_aggs), ), ( ["algo"], partial(apply_transformations, transformations=stats_transform), ), ], ).pipe(format_results) perf_metrics_all.style.pipe( revalidation_table_styles, validation_thresholds, ["algo"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_001.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_001.png :class: sphx-glr-single-img .. raw:: html
    General ICD IC Timing
    # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
algo              
IcdHKLeeImproved MobGap 101 0.93 [0.91, 0.95] 0.91 [0.89, 0.93] 0.92 [0.90, 0.94] 0.14 [0.08, 0.21] 0.21 [0.12, 0.29]
IcdIonescu MobGap 101 0.90 [0.88, 0.92] 0.92 [0.90, 0.94] 0.91 [0.89, 0.93] 0.06 [0.01, 0.11] 0.08 [0.03, 0.14]
Original Implementation 101 0.90 [0.88, 0.92] 0.92 [0.90, 0.94] 0.91 [0.89, 0.93] 0.06 [0.01, 0.11] 0.08 [0.03, 0.14]
IcdShinImproved MobGap 101 0.93 [0.91, 0.95] 0.92 [0.90, 0.94] 0.93 [0.91, 0.95] 0.06 [0.02, 0.11] 0.09 [0.04, 0.13]


.. GENERATED FROM PYTHON SOURCE LINES 275-278 Per Cohort ~~~~~~~~~~ While this provides a good overview, it does not fully reflect how these algorithms perform on the different cohorts. .. GENERATED FROM PYTHON SOURCE LINES 278-308 .. code-block:: Python fig, ax = plt.subplots() sns.boxplot( data=results_long, x="cohort", y="f1_score", hue="algo_with_version", ax=ax ) fig.show() perf_metrics_per_cohort = ( results_long.pipe( multilevel_groupby_apply_merge, [ ( ["cohort", "algo", "version"], partial(apply_aggregations, aggregations=custom_aggs), ), ( ["cohort", "algo"], partial(apply_transformations, transformations=stats_transform), ), ], ) .pipe(format_results) .loc[cohort_order] ) perf_metrics_per_cohort.style.pipe( revalidation_table_styles, validation_thresholds, ["cohort", "algo"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_002.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_002.png :class: sphx-glr-single-img .. raw:: html
      General ICD IC Timing
      # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
cohort algo              
HA IcdHKLeeImproved MobGap 20 0.94 [0.93, 0.96] 0.93 [0.91, 0.94] 0.93 [0.92, 0.95] 0.14 [0.09, 0.18] 0.20 [0.15, 0.26]
IcdIonescu MobGap 20 0.92 [0.91, 0.94] 0.94 [0.93, 0.96] 0.93 [0.92, 0.95] 0.05 [0.03, 0.08] 0.08 [0.05, 0.11]
Original Implementation 20 0.92 [0.90, 0.94] 0.94 [0.93, 0.96] 0.93 [0.92, 0.94] 0.05 [0.02, 0.08] 0.08 [0.04, 0.11]
IcdShinImproved MobGap 20 0.94 [0.93, 0.96] 0.94 [0.93, 0.95] 0.94 [0.93, 0.95] 0.06 [0.03, 0.08] 0.08 [0.05, 0.11]
CHF IcdHKLeeImproved MobGap 10 0.94 [0.92, 0.96] 0.93 [0.91, 0.96] 0.94 [0.92, 0.96] 0.14 [0.07, 0.20] 0.21 [0.11, 0.30]
IcdIonescu MobGap 10 0.92 [0.89, 0.95] 0.95 [0.93, 0.97] 0.94 [0.91, 0.96] 0.06 [0.01, 0.10] 0.08 [0.02, 0.14]
Original Implementation 10 0.93 [0.90, 0.95] 0.95 [0.94, 0.97] 0.94 [0.92, 0.96] 0.05 [0.00, 0.10] 0.08 [0.01, 0.15]
IcdShinImproved MobGap 10 0.95 [0.93, 0.96] 0.95 [0.94, 0.97] 0.95 [0.93, 0.96] 0.06 [0.02, 0.10] 0.09 [0.04, 0.14]
COPD IcdHKLeeImproved MobGap 17 0.92 [0.91, 0.93] 0.89 [0.87, 0.91] 0.91 [0.89, 0.92] 0.17 [0.10, 0.23] 0.22 [0.14, 0.31]
IcdIonescu MobGap 17 0.89 [0.88, 0.91] 0.92 [0.91, 0.94] 0.91 [0.89, 0.92] 0.07 [0.02, 0.12] 0.09 [0.04, 0.14]
Original Implementation 17 0.89 [0.87, 0.90] 0.92 [0.91, 0.94] 0.90 [0.89, 0.92] 0.07 [0.03, 0.12] 0.09 [0.04, 0.15]
IcdShinImproved MobGap 17 0.93 [0.92, 0.94] 0.92 [0.91, 0.94] 0.93 [0.91, 0.94] 0.07 [0.03, 0.11] 0.09 [0.04, 0.14]
MS IcdHKLeeImproved MobGap 18 0.95 [0.93, 0.97] 0.93 [0.90, 0.95] 0.94 [0.91, 0.96] 0.14 [0.08, 0.19] 0.20 [0.11, 0.29]
IcdIonescu MobGap 18 0.93 [0.91, 0.95] 0.94 [0.92, 0.96] 0.93 [0.91, 0.95] 0.06 [-0.00, 0.13] 0.09 [0.02, 0.15]
Original Implementation 18 0.93 [0.91, 0.95] 0.94 [0.92, 0.96] 0.93 [0.91, 0.96] 0.06 [-0.00, 0.13] 0.09 [0.03, 0.15]
IcdShinImproved MobGap 18 0.94 [0.92, 0.97] 0.94 [0.91, 0.97] 0.94 [0.92, 0.97] 0.06 [0.00, 0.12] 0.09 [0.03, 0.15]
PD IcdHKLeeImproved MobGap 19 0.93 [0.91, 0.96] 0.93 [0.90, 0.96] 0.93 [0.91, 0.95] 0.14 [0.09, 0.20] 0.21 [0.14, 0.28]
IcdIonescu MobGap 19 0.92 [0.89, 0.95] 0.94 [0.91, 0.97] 0.93 [0.90, 0.96] 0.06 [0.01, 0.11] 0.09 [0.04, 0.14]
Original Implementation 19 0.92 [0.89, 0.95] 0.94 [0.91, 0.97] 0.93 [0.90, 0.96] 0.06 [0.01, 0.10] 0.08 [0.03, 0.14]
IcdShinImproved MobGap 19 0.94 [0.91, 0.96] 0.95 [0.92, 0.97] 0.94 [0.92, 0.96] 0.06 [0.02, 0.10] 0.09 [0.04, 0.13]
PFF IcdHKLeeImproved MobGap 17 0.88 [0.77, 0.99] 0.84 [0.73, 0.95] 0.86 [0.75, 0.96] 0.15 [0.05, 0.25] 0.19 [0.07, 0.30]
IcdIonescu MobGap 17 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.84 [0.73, 0.94] 0.07 [0.02, 0.11] 0.08 [0.03, 0.13]
Original Implementation 17 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.06 [0.02, 0.11] 0.08 [0.03, 0.13]
IcdShinImproved MobGap 17 0.87 [0.76, 0.98] 0.85 [0.73, 0.96] 0.86 [0.75, 0.97] 0.06 [0.01, 0.11] 0.08 [0.02, 0.13]


.. GENERATED FROM PYTHON SOURCE LINES 309-314 Only relevant algorithms ~~~~~~~~~~~~~~~~~~~~~~~~ Finally, we present comparison of the old and new implementations of IcdIonescu. IcdShinImproved and IcdHKLeeImproved are excluded because they are cadence algorithms and we don't calculate ICs with these algos in the old Matlab implementation. .. GENERATED FROM PYTHON SOURCE LINES 314-336 .. code-block:: Python fig, ax = plt.subplots() sns.boxplot( data=results_long.query("algo == 'IcdIonescu'"), x="cohort", y="f1_score", hue="algo_with_version", ax=ax, ) fig.show() final_perf_metrics = ( perf_metrics_per_cohort.copy() .query("algo == 'IcdIonescu'") .reset_index(level="algo", drop=True) ) final_perf_metrics.style.pipe( revalidation_table_styles, validation_thresholds, ["cohort"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_003.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_003.png :class: sphx-glr-single-img .. raw:: html
    General ICD IC Timing
    # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
cohort              
HA MobGap 20 0.92 [0.91, 0.94] 0.94 [0.93, 0.96] 0.93 [0.92, 0.95] 0.05 [0.03, 0.08] 0.08 [0.05, 0.11]
Original Implementation 20 0.92 [0.90, 0.94] 0.94 [0.93, 0.96] 0.93 [0.92, 0.94] 0.05 [0.02, 0.08] 0.08 [0.04, 0.11]
CHF MobGap 10 0.92 [0.89, 0.95] 0.95 [0.93, 0.97] 0.94 [0.91, 0.96] 0.06 [0.01, 0.10] 0.08 [0.02, 0.14]
Original Implementation 10 0.93 [0.90, 0.95] 0.95 [0.94, 0.97] 0.94 [0.92, 0.96] 0.05 [0.00, 0.10] 0.08 [0.01, 0.15]
COPD MobGap 17 0.89 [0.88, 0.91] 0.92 [0.91, 0.94] 0.91 [0.89, 0.92] 0.07 [0.02, 0.12] 0.09 [0.04, 0.14]
Original Implementation 17 0.89 [0.87, 0.90] 0.92 [0.91, 0.94] 0.90 [0.89, 0.92] 0.07 [0.03, 0.12] 0.09 [0.04, 0.15]
MS MobGap 18 0.93 [0.91, 0.95] 0.94 [0.92, 0.96] 0.93 [0.91, 0.95] 0.06 [-0.00, 0.13] 0.09 [0.02, 0.15]
Original Implementation 18 0.93 [0.91, 0.95] 0.94 [0.92, 0.96] 0.93 [0.91, 0.96] 0.06 [-0.00, 0.13] 0.09 [0.03, 0.15]
PD MobGap 19 0.92 [0.89, 0.95] 0.94 [0.91, 0.97] 0.93 [0.90, 0.96] 0.06 [0.01, 0.11] 0.09 [0.04, 0.14]
Original Implementation 19 0.92 [0.89, 0.95] 0.94 [0.91, 0.97] 0.93 [0.90, 0.96] 0.06 [0.01, 0.10] 0.08 [0.03, 0.14]
PFF MobGap 17 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.84 [0.73, 0.94] 0.07 [0.02, 0.11] 0.08 [0.03, 0.13]
Original Implementation 17 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.84 [0.73, 0.95] 0.06 [0.02, 0.11] 0.08 [0.03, 0.13]


.. GENERATED FROM PYTHON SOURCE LINES 337-345 Laboratory Comparison --------------------- Every datapoint below is one trial of a test. Note, that each datapoint is weighted equally in the calculation of the performance metrics. This is a limitation of this simple approach, as the number of strides per trial and the complexity of the context can vary significantly. For a full picture, different groups of tests should be analyzed separately. The approach below should still provide a good overview to compare the algorithms. .. GENERATED FROM PYTHON SOURCE LINES 345-377 .. code-block:: Python hue_order = ["Original Implementation", "MobGap"] fig, ax = plt.subplots() sns.boxplot( data=lab_results_long, x="algo", y="f1_score", hue="version", hue_order=hue_order, ax=ax, ) fig.show() perf_metrics_all = lab_results_long.pipe( multilevel_groupby_apply_merge, [ ( ["algo", "version"], partial(apply_aggregations, aggregations=custom_aggs), ), ( ["algo"], partial(apply_transformations, transformations=stats_transform), ), ], ).pipe(format_results) perf_metrics_all.style.pipe( revalidation_table_styles, validation_thresholds, ["algo"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_004.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_004.png :class: sphx-glr-single-img .. raw:: html
    General ICD IC Timing
    # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
algo              
IcdHKLeeImproved MobGap 1168 0.80 [0.79, 0.82] 0.88 [0.86, 0.89] 0.83 [0.82, 0.85] 0.12 [0.02, 0.22] 0.18 [0.02, 0.34]
IcdIonescu MobGap 1168 0.73 [0.72, 0.75] 0.88 [0.86, 0.89] 0.79 [0.78, 0.81] 0.05 [-0.02, 0.13] 0.08 [-0.02, 0.18]
Original Implementation 1168 0.72 [0.70, 0.73] 0.86 [0.84, 0.88] 0.78 [0.76, 0.80] 0.05 [-0.02, 0.12] 0.07 [-0.02, 0.17]
IcdShinImproved MobGap 1168 0.79 [0.77, 0.80] 0.88 [0.86, 0.90] 0.83 [0.81, 0.84] 0.05 [-0.01, 0.12] 0.08 [-0.01, 0.17]


.. GENERATED FROM PYTHON SOURCE LINES 378-381 Per Cohort ~~~~~~~~~~ While this provides a good overview, it does not fully reflect how these algorithms perform on the different cohorts. .. GENERATED FROM PYTHON SOURCE LINES 381-415 .. code-block:: Python fig, ax = plt.subplots() sns.boxplot( data=lab_results_long, x="cohort", y="f1_score", hue="algo_with_version", ax=ax, ) fig.show() perf_metrics_per_cohort = ( lab_results_long.pipe( multilevel_groupby_apply_merge, [ ( ["cohort", "algo", "version"], partial(apply_aggregations, aggregations=custom_aggs), ), ( ["cohort", "algo"], partial(apply_transformations, transformations=stats_transform), ), ], ) .pipe(format_results) .loc[cohort_order] ) perf_metrics_per_cohort.style.pipe( revalidation_table_styles, validation_thresholds, ["cohort", "algo"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_005.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_005.png :class: sphx-glr-single-img .. raw:: html
      General ICD IC Timing
      # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
cohort algo              
HA IcdHKLeeImproved MobGap 227 0.74 [0.69, 0.78] 0.83 [0.78, 0.87] 0.78 [0.73, 0.82] 0.10 [0.01, 0.20] 0.17 [0.01, 0.33]
IcdIonescu MobGap 227 0.67 [0.63, 0.71] 0.82 [0.78, 0.87] 0.74 [0.69, 0.78] 0.04 [-0.03, 0.11] 0.06 [-0.04, 0.16]
Original Implementation 227 0.67 [0.63, 0.71] 0.82 [0.77, 0.87] 0.73 [0.69, 0.78] 0.04 [-0.03, 0.10] 0.06 [-0.04, 0.16]
IcdShinImproved MobGap 227 0.72 [0.67, 0.76] 0.82 [0.78, 0.87] 0.76 [0.72, 0.81] 0.05 [-0.02, 0.11] 0.08 [-0.02, 0.18]
CHF IcdHKLeeImproved MobGap 106 0.81 [0.76, 0.86] 0.89 [0.83, 0.94] 0.84 [0.79, 0.89] 0.12 [0.02, 0.21] 0.18 [0.02, 0.33]
IcdIonescu MobGap 106 0.74 [0.70, 0.79] 0.89 [0.84, 0.94] 0.81 [0.76, 0.85] 0.06 [-0.02, 0.13] 0.08 [-0.02, 0.18]
Original Implementation 106 0.74 [0.70, 0.79] 0.89 [0.84, 0.94] 0.80 [0.76, 0.85] 0.05 [-0.02, 0.12] 0.08 [-0.02, 0.17]
IcdShinImproved MobGap 106 0.80 [0.75, 0.85] 0.89 [0.84, 0.95] 0.84 [0.79, 0.89] 0.05 [-0.01, 0.12] 0.08 [-0.01, 0.17]
COPD IcdHKLeeImproved MobGap 214 0.74 [0.70, 0.79] 0.82 [0.78, 0.87] 0.78 [0.73, 0.83] 0.11 [-0.00, 0.23] 0.18 [-0.01, 0.38]
IcdIonescu MobGap 214 0.68 [0.64, 0.72] 0.83 [0.78, 0.88] 0.74 [0.70, 0.79] 0.04 [-0.02, 0.10] 0.06 [-0.03, 0.15]
Original Implementation 214 0.68 [0.64, 0.72] 0.83 [0.78, 0.88] 0.74 [0.70, 0.79] 0.04 [-0.02, 0.10] 0.06 [-0.03, 0.16]
IcdShinImproved MobGap 214 0.73 [0.69, 0.78] 0.83 [0.78, 0.88] 0.77 [0.73, 0.82] 0.04 [-0.02, 0.11] 0.07 [-0.03, 0.16]
MS IcdHKLeeImproved MobGap 228 0.87 [0.85, 0.89] 0.94 [0.92, 0.96] 0.90 [0.88, 0.92] 0.13 [0.03, 0.22] 0.19 [0.05, 0.33]
IcdIonescu MobGap 228 0.78 [0.76, 0.81] 0.94 [0.91, 0.96] 0.85 [0.83, 0.87] 0.07 [-0.02, 0.16] 0.10 [-0.00, 0.21]
Original Implementation 228 0.75 [0.72, 0.78] 0.90 [0.87, 0.94] 0.82 [0.79, 0.85] 0.06 [-0.02, 0.15] 0.09 [-0.01, 0.20]
IcdShinImproved MobGap 228 0.85 [0.82, 0.87] 0.94 [0.92, 0.96] 0.89 [0.87, 0.91] 0.07 [-0.01, 0.14] 0.10 [0.00, 0.19]
PD IcdHKLeeImproved MobGap 224 0.78 [0.74, 0.82] 0.84 [0.80, 0.89] 0.81 [0.77, 0.85] 0.12 [0.01, 0.23] 0.18 [0.02, 0.35]
IcdIonescu MobGap 224 0.71 [0.67, 0.74] 0.85 [0.81, 0.89] 0.77 [0.73, 0.81] 0.05 [-0.02, 0.11] 0.07 [-0.02, 0.15]
Original Implementation 224 0.69 [0.65, 0.73] 0.83 [0.78, 0.87] 0.75 [0.71, 0.79] 0.04 [-0.02, 0.11] 0.06 [-0.02, 0.15]
IcdShinImproved MobGap 224 0.77 [0.73, 0.81] 0.85 [0.81, 0.90] 0.81 [0.77, 0.85] 0.05 [-0.01, 0.11] 0.07 [-0.01, 0.16]
PFF IcdHKLeeImproved MobGap 169 0.90 [0.89, 0.92] 0.95 [0.94, 0.97] 0.92 [0.91, 0.93] 0.13 [0.05, 0.21] 0.19 [0.09, 0.29]
IcdIonescu MobGap 169 0.83 [0.81, 0.84] 0.95 [0.93, 0.96] 0.88 [0.87, 0.89] 0.06 [-0.00, 0.13] 0.09 [0.02, 0.16]
Original Implementation 169 0.81 [0.79, 0.83] 0.93 [0.90, 0.95] 0.86 [0.84, 0.88] 0.06 [-0.01, 0.13] 0.08 [0.01, 0.15]
IcdShinImproved MobGap 169 0.89 [0.87, 0.90] 0.95 [0.94, 0.97] 0.91 [0.90, 0.92] 0.06 [0.00, 0.12] 0.09 [0.01, 0.17]


.. GENERATED FROM PYTHON SOURCE LINES 416-421 Only relevant algorithms ~~~~~~~~~~~~~~~~~~~~~~~~ Finally, we present comparison of the old and new implementations of IcdIonescu. IcdShinImproved and IcdHKLeeImproved are excluded because they are cadence algorithms and we don't calculate ICs with these algos in the old Matlab implementation. .. GENERATED FROM PYTHON SOURCE LINES 421-440 .. code-block:: Python fig, ax = plt.subplots() sns.boxplot( data=lab_results_long.query("algo == 'IcdIonescu'"), x="cohort", y="f1_score", hue="algo_with_version", ax=ax, ) fig.show() final_perf_metrics = perf_metrics_per_cohort.query( "algo == 'IcdIonescu'" ).reset_index(level="algo", drop=True) final_perf_metrics.style.pipe( revalidation_table_styles, validation_thresholds, ["cohort"], ) .. image-sg:: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_006.png :alt: 01 icd analysis :srcset: /auto_revalidation/initial_contacts/images/sphx_glr__01_icd_analysis_006.png :class: sphx-glr-single-img .. raw:: html
    General ICD IC Timing
    # recordings Recall Precision F1 Score Abs. Error [s] Bias and LoA
cohort              
HA MobGap 227 0.67 [0.63, 0.71] 0.82 [0.78, 0.87] 0.74 [0.69, 0.78] 0.04 [-0.03, 0.11] 0.06 [-0.04, 0.16]
Original Implementation 227 0.67 [0.63, 0.71] 0.82 [0.77, 0.87] 0.73 [0.69, 0.78] 0.04 [-0.03, 0.10] 0.06 [-0.04, 0.16]
CHF MobGap 106 0.74 [0.70, 0.79] 0.89 [0.84, 0.94] 0.81 [0.76, 0.85] 0.06 [-0.02, 0.13] 0.08 [-0.02, 0.18]
Original Implementation 106 0.74 [0.70, 0.79] 0.89 [0.84, 0.94] 0.80 [0.76, 0.85] 0.05 [-0.02, 0.12] 0.08 [-0.02, 0.17]
COPD MobGap 214 0.68 [0.64, 0.72] 0.83 [0.78, 0.88] 0.74 [0.70, 0.79] 0.04 [-0.02, 0.10] 0.06 [-0.03, 0.15]
Original Implementation 214 0.68 [0.64, 0.72] 0.83 [0.78, 0.88] 0.74 [0.70, 0.79] 0.04 [-0.02, 0.10] 0.06 [-0.03, 0.16]
MS MobGap 228 0.78 [0.76, 0.81] 0.94 [0.91, 0.96] 0.85 [0.83, 0.87] 0.07 [-0.02, 0.16] 0.10 [-0.00, 0.21]
Original Implementation 228 0.75 [0.72, 0.78] 0.90 [0.87, 0.94] 0.82 [0.79, 0.85] 0.06 [-0.02, 0.15] 0.09 [-0.01, 0.20]
PD MobGap 224 0.71 [0.67, 0.74] 0.85 [0.81, 0.89] 0.77 [0.73, 0.81] 0.05 [-0.02, 0.11] 0.07 [-0.02, 0.15]
Original Implementation 224 0.69 [0.65, 0.73] 0.83 [0.78, 0.87] 0.75 [0.71, 0.79] 0.04 [-0.02, 0.11] 0.06 [-0.02, 0.15]
PFF MobGap 169 0.83 [0.81, 0.84] 0.95 [0.93, 0.96] 0.88 [0.87, 0.89] 0.06 [-0.00, 0.13] 0.09 [0.02, 0.16]
Original Implementation 169 0.81 [0.79, 0.83] 0.93 [0.90, 0.95] 0.86 [0.84, 0.88] 0.06 [-0.01, 0.13] 0.08 [0.01, 0.15]


.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 5.690 seconds) **Estimated memory usage:** 81 MB .. _sphx_glr_download_auto_revalidation_initial_contacts__01_icd_analysis.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: _01_icd_analysis.ipynb <_01_icd_analysis.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: _01_icd_analysis.py <_01_icd_analysis.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: _01_icd_analysis.zip <_01_icd_analysis.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_