GSD performance under simulated mounting errors

This analysis compares free-living gait sequence detection performance across simulated lower-back sensor mounting orientations. It uses the result files generated by Revalidation of gait sequence detection under simulated mounting errors.

This analysis is more of a confirmation which algorithms work even when the sensor is mounted incorrectly and which mounting orientations are more problematic for the algorithms.

The plot shows the per-recording F1 score for each algorithm and orientation. The table reports the overall number of detected gait sequences. Each simulated orientation is produced by converting the TVS recording to body frame and applying the rough mounting rotations used by the reorientation validation.

Note

The standard GSD validation remains the reference for performance under the default Mobilise-D mounting assumption. This page isolates orientation robustness and therefore only compares Python algorithms that can be rerun on the simulated recordings.

from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from mobgap.data.validation_results import ValidationResultLoader
from mobgap.plotting import move_legend_outside
from mobgap.re_orientation.pipeline import REORIENTATION_LABELS
from mobgap.utils.misc import get_env_var

Algorithms

algorithms = {
    "GsdIonescu": ("GsdIonescu", "MobGap"),
    "GsdIluz": ("GsdIluz", "MobGap"),
    "GsdIluz_orig_peak": ("GsdIluz", "MobGap (original peak)"),
}

Loading Results

The result files add orientation to the usual TVS index. The standard validation loader keeps unknown CSV columns as data columns, so we can still use it for local and remote loading.

local_data_path = (
    Path(get_env_var("MOBGAP_VALIDATION_DATA_PATH")) / "results"
    if int(get_env_var("MOBGAP_VALIDATION_USE_LOCAL_DATA", 0))
    else None
)
__RESULT_VERSION = "main"
loader = ValidationResultLoader(
    "gsd_misorientation",
    result_path=local_data_path,
    version=__RESULT_VERSION,
)

orientation_order = list(REORIENTATION_LABELS)
algorithm_order = [
    f"{algo} ({version})" for algo, version in algorithms.values()
]


def load_free_living_results() -> pd.DataFrame:
    results = []
    for folder_name, (algo, version) in algorithms.items():
        result = (
            loader.load_single_results(folder_name, "free_living")
            .reset_index()
            .assign(
                algo=algo,
                version=version,
                algo_with_version=f"{algo} ({version})",
            )
        )
        results.append(result)
    return pd.concat(results, ignore_index=True)


free_living_results = load_free_living_results()

  0%|                                              | 0.00/19.4k [00:00<?, ?B/s]
  0%|                                              | 0.00/19.4k [00:00<?, ?B/s]
100%|█████████████████████████████████████| 19.4k/19.4k [00:00<00:00, 58.9MB/s]

  0%|                                              | 0.00/36.7k [00:00<?, ?B/s]
  0%|                                              | 0.00/36.7k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 36.7k/36.7k [00:00<00:00, 191MB/s]

  0%|                                              | 0.00/35.7k [00:00<?, ?B/s]
  0%|                                              | 0.00/35.7k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 35.7k/35.7k [00:00<00:00, 183MB/s]

Plot Helpers

sns.set_context("talk")


def plot_f1_by_orientation(data: pd.DataFrame, *, title: str) -> None:
    fig, ax = plt.subplots(figsize=(18, 7), constrained_layout=True)
    sns.boxplot(
        data=data,
        x="algo_with_version",
        y="f1_score",
        hue="orientation",
        order=algorithm_order,
        hue_order=orientation_order,
        showmeans=True,
        ax=ax,
    )
    ax.set_title(title)
    ax.set_xlabel("Algorithm")
    ax.set_ylabel("F1 score")
    ax.set_ylim(-0.02, 1.02)
    ax.tick_params(axis="x", rotation=20)
    ax.grid(True, axis="y", alpha=0.3)
    move_legend_outside(fig, ax, ncol=4)
    plt.show()


def detected_gs_count_table(data: pd.DataFrame) -> pd.DataFrame:
    return (
        data.pivot_table(
            index="algo_with_version",
            columns="orientation",
            values="detected_num_gs",
            aggfunc="sum",
            sort=False,
        )
        .reindex(index=algorithm_order, columns=orientation_order)
        .rename_axis(index="Algorithm", columns="Orientation")
        .astype("Int64")
    )

Free-Living Comparison

Every datapoint below is one free-living recording under one simulated orientation.

plot_f1_by_orientation(
    free_living_results,
    title="Free-living GSD F1 score by simulated orientation",
)

⤢

Detected GS Counts

detected_gs_count_table(free_living_results)

⤢

Orientation	identity	pa_normal__rot_pa_pos90	pa_normal__rot_pa_180	pa_normal__rot_pa_neg90	pa_flipped__rot_pa_0	pa_flipped__rot_pa_pos90	pa_flipped__rot_pa_180	pa_flipped__rot_pa_neg90
Algorithm
GsdIonescu (MobGap)	5538	5538	5538	5538	5538	5538	5538	5538
GsdIluz (MobGap)	7260	11	1	1	7057	2	1	12
GsdIluz (MobGap (original peak))	4381	6	0	0	4662	0	0	7

Conclusion

As expected GsdIonescu works identically across all orientations, as it relies on the norm of the accelerometer signal. GsdIluz is heavily dependent on the correct orientation. If the gravity direction is on the wrong axis, the algorithm fails to find any gait sequence (with very few exceptions). Flipping the sensor front-back (pa_flipped__rot_pa_0) is less problematic and seems to result in only a small drop in performance for GsdIluz. But as expected, it makes no sense to use GsdIluz if the sensor is expected to be mounted in a different orientation than the default Mobilise-D one.

Estimated memory usage: 82 MB