.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/pipeline/_01_gs_iterator.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_pipeline__01_gs_iterator.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_pipeline__01_gs_iterator.py:


.. _gs_iterator_example:

Gait Sequence Iterator
======================

As part of most pipelines, we need to iterate over the gait sequences to apply all further algorithms to them
individually.
This can be a bit cumbersome, as we need to iterate over the data and aggregate the results at the same time.
Hence, we provide some helpers for that.

We provide two ways of iterating.
The first one, only handles the iteration and does not aggregate the results.
The second approach attempts to also support you in aggregating the results.

Getting Some Example Data
-------------------------

.. GENERATED FROM PYTHON SOURCE LINES 20-33

.. code-block:: default


    import numpy as np
    import pandas as pd
    from mobgap.data import LabExampleDataset

    lab_example_data = LabExampleDataset(reference_system="INDIP")
    long_trial = lab_example_data.get_subset(
        cohort="MS", participant_id="001", test="Test11", trial="Trial1"
    )
    long_trial_gs = long_trial.reference_parameters_.wb_list

    long_trial_gs





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mobgap/checkouts/v0.9.0/mobgap/data/_mobilised_matlab_loader.py:1082: UserWarning: There were multiple ICs with the same index value, but different LR labels. This is likely an issue with the reference system you should further investigate. For now, we set the `lr_label` of the stride corresponding to this IC to Nan. However, both values still remain in the IC list.
      return parse_reference_parameters(


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>n_strides</th>
          <th>duration_s</th>
          <th>length_m</th>
          <th>avg_walking_speed_mps</th>
          <th>avg_cadence_spm</th>
          <th>avg_stride_length_m</th>
          <th>termination_reason</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1019</td>
          <td>1768</td>
          <td>9</td>
          <td>7.48</td>
          <td>4.468932</td>
          <td>0.847668</td>
          <td>107.795850</td>
          <td>0.942678</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4534</td>
          <td>5549</td>
          <td>11</td>
          <td>10.14</td>
          <td>2.900453</td>
          <td>0.365176</td>
          <td>93.396106</td>
          <td>0.483923</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9665</td>
          <td>10569</td>
          <td>9</td>
          <td>9.03</td>
          <td>2.140232</td>
          <td>0.294058</td>
          <td>75.981133</td>
          <td>0.506458</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12337</td>
          <td>14633</td>
          <td>28</td>
          <td>22.95</td>
          <td>11.201110</td>
          <td>0.634425</td>
          <td>92.337768</td>
          <td>0.803933</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>4</th>
          <td>20151</td>
          <td>20982</td>
          <td>11</td>
          <td>8.30</td>
          <td>2.390709</td>
          <td>0.371746</td>
          <td>87.915774</td>
          <td>0.507484</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21378</td>
          <td>22129</td>
          <td>9</td>
          <td>7.50</td>
          <td>2.517558</td>
          <td>0.492965</td>
          <td>95.365740</td>
          <td>0.599360</td>
          <td>Pause</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 34-44

Simple Functional Interface
---------------------------
We provide the :func:`~mobgap.pipeline.iter_gs` function to iterate over the gait sequences.
It simply takes the data and the gait sequence list and cuts the data accordingly to iterate over it.
The function yields the gait sequence information as tuple (i.e. the "row" of the gs dataframe as namedtuple) and the
data for each iteration.
Note that the index of the data is not changed.
Hence, we recommend using `iloc` to access the data (`iloc[0]` will return the first sample of the gait sequence).

Using our example data and gs, we can iterate over the data as follows:

.. GENERATED FROM PYTHON SOURCE LINES 44-54

.. code-block:: default

    from mobgap.pipeline import iter_gs

    for gs, data in iter_gs(long_trial.data_ss, long_trial_gs):
        # Note that the key to access the id is called "wb_id" here, as we loaded the WB from the reference system.
        # If this is an "actual" gait sequences, as calculated by one of the GSD algorithms, the key would be "gs_id".
        print("Gait Sequence: ", gs)
        print("Expected N-samples in gs: ", gs.end - gs.start)
        print("N-samples in gs: ", len(data))
        print("First sample of gs:\n", data.iloc[0], end="\n\n")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Gait Sequence:  Region(id=0, start=1019, end=1768, id_origin='wb_id')
    Expected N-samples in gs:  749
    N-samples in gs:  749
    First sample of gs:
     acc_x    10.746760
    acc_y     0.390074
    acc_z    -2.088171
    gyr_x    13.226900
    gyr_y    -4.914900
    gyr_z    19.874400
    Name: 2020-10-30 12:53:33.211999893+00:00, dtype: float64

    Gait Sequence:  Region(id=1, start=4534, end=5549, id_origin='wb_id')
    Expected N-samples in gs:  1015
    N-samples in gs:  1015
    First sample of gs:
     acc_x     9.993886
    acc_y    -0.864273
    acc_z    -0.297190
    gyr_x     3.193700
    gyr_y    12.861200
    gyr_z    -5.371300
    Name: 2020-10-30 12:54:08.361999989+00:00, dtype: float64

    Gait Sequence:  Region(id=2, start=9665, end=10569, id_origin='wb_id')
    Expected N-samples in gs:  904
    N-samples in gs:  904
    First sample of gs:
     acc_x    8.889232
    acc_y   -0.136809
    acc_z   -3.005737
    gyr_x   -7.543100
    gyr_y   -5.688700
    gyr_z    0.470900
    Name: 2020-10-30 12:54:59.671999931+00:00, dtype: float64

    Gait Sequence:  Region(id=3, start=12337, end=14633, id_origin='wb_id')
    Expected N-samples in gs:  2296
    N-samples in gs:  2296
    First sample of gs:
     acc_x    10.232635
    acc_y    -1.686375
    acc_z    -0.285600
    gyr_x    84.994600
    gyr_y   -17.645800
    gyr_z   -24.538000
    Name: 2020-10-30 12:55:26.391999960+00:00, dtype: float64

    Gait Sequence:  Region(id=4, start=20151, end=20982, id_origin='wb_id')
    Expected N-samples in gs:  831
    N-samples in gs:  831
    First sample of gs:
     acc_x      9.132286
    acc_y     -1.796825
    acc_z      0.479311
    gyr_x   -137.510300
    gyr_y     16.480800
    gyr_z    -21.008900
    Name: 2020-10-30 12:56:44.532000064+00:00, dtype: float64

    Gait Sequence:  Region(id=5, start=21378, end=22129, id_origin='wb_id')
    Expected N-samples in gs:  751
    N-samples in gs:  751
    First sample of gs:
     acc_x     9.891845
    acc_y    -3.079999
    acc_z    -1.069887
    gyr_x    56.595700
    gyr_y    -5.242100
    gyr_z   -16.025800
    Name: 2020-10-30 12:56:56.802000046+00:00, dtype: float64





.. GENERATED FROM PYTHON SOURCE LINES 55-96

.. note:: The ``gs`` named-tuples returned by the iterator is of type ``Region``.
          It contains the fields ``id``, ``start``, and ``end`` in this order.
          When using the named access the ``id`` field corresponds to either the ``gs_id`` or ``wb_id`` of the input
          dataframe, depending on what type of list was provided.

You can see that this way it is pretty easy to iterate over the data.
However, if you are planning to run calculations on the data, you need to aggregate the results yourself.
If you are planning to collect multiple pieces of results, this can become cumbersome.
See the is `tpcp example <https://tpcp.readthedocs.io/en/latest/auto_examples/recipies/_04_typed_iterator.html>`__
for more information about this.
Therefore, we also provide an Iterator Class based on :class:`~tpcp.misc.TypedIterator`.

Class based Interface
---------------------
.. note:: Learn more about the general approach of using :class:`~tpcp.misc.TypedIterator` classes in this
          `tpcp example <https://tpcp.readthedocs.io/en/latest/auto_examples/recipies/_04_typed_iterator.html>`__.

Compared to the functional interface, the class interface attempts to also solve the problem of collecting the
and aggregating results that you produce per GS.
In a typical pipeline you might want to calculate the initial contacts, cadence, stride length, and gait speed for
each gait sequence.
With the class based interface, you can easily collect all of these results and then aggregate them into one
predefined data structure.

The class based interface can be used in two ways.
First in the "default" configuration, which is set up to work with the typical calculations and results that you would
expect from a typical processing pipeline.
And second, in a custom way, where you need to define expected "results" per iteration yourself.

The simple case
---------------
The simple case basically requires no more setup as the functional interface.
However, it assumes that your results are a subset of initial contacts, cadence, stride length, and gait speed, and
that all of them are stored in the expected mobgap datatypes (aka pandas dataframes).
The iterator will then automatically aggregate the results the dataframes per iteration into one combined dataframe,
handling the sample offsets of the gait sequences for you.

Below we will show how this works, by "simulating" the calculation of some initial contacts and cadence.

We start by setting up an iterator object.
We can leave everything at the default values, as we do not need any custom aggregation functions.

.. GENERATED FROM PYTHON SOURCE LINES 96-101

.. code-block:: default

    from mobgap.pipeline import GsIterator

    iterator = GsIterator()
    dt = iterator.data_type








.. GENERATED FROM PYTHON SOURCE LINES 102-103

The default result datatype per iteration is defined as follows:

.. GENERATED FROM PYTHON SOURCE LINES 103-109

.. code-block:: default

    import inspect

    from IPython.core.display_functions import display

    display(inspect.getsource(dt))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    @dataclass
    class FullPipelinePerGsResult:
        """Default expected result type for the gait-sequence iterator.

        When using the :class:`~mobgap.pipeline.GsIterator` with the default configuration, an instance of this dataclass
        will be created for each gait-sequence.

        Each value is expected to be a dataframe.

        Attributes
        ----------
        ic_list
            The initial contacts for each gait-sequence.
            This is a dataframe with a column called ``ic``.
            The values of this ic-column are expected to be samples relative to the start of the gait-sequence.
        turn_list
            The turn list for each gait-sequence.
            The dataframe has at least columns called ``start`` and ``end``.
            The values of these columns are expected to be samples relative to the start of the gait-sequence.
        cadence_per_sec
            The cadence values within each gait-sequence.
            This dataframe has no further requirements relevant for the iterator.
        stride_length_per_sec
            The stride length values within each gait-sequence.
            This dataframe has no further requirements relevant for the iterator.
        walking_speed_per_sec
            The gait speed values within each gait-sequence.
            This dataframe has no further requirements relevant for the iterator.

        """

        ic_list: pd.DataFrame
        turn_list: pd.DataFrame
        cadence_per_sec: pd.DataFrame
        stride_length_per_sec: pd.DataFrame
        walking_speed_per_sec: pd.DataFrame





.. GENERATED FROM PYTHON SOURCE LINES 110-116

This means you are only allowed to use the available attributes.
But, you don't need to specify all of them.
Below we will only "calculate" the initial contacts and cadence.

In each iteration the iterator will give us a tuple of the gait sequence information, the data for the iteration, and
a new empty result object.

.. GENERATED FROM PYTHON SOURCE LINES 116-133

.. code-block:: default

    from mobgap.utils.conversions import as_samples

    for (gs, data), result in iterator.iterate(long_trial.data_ss, long_trial_gs):
        # Now we can just "calculate" the initial contacts and set it on the result object.
        result.ic_list = pd.DataFrame(
            np.arange(0, len(data), 100, dtype="int64"), columns=["ic"]
        ).rename_axis(index="step_id")
        # For cadence, we just set a dummy value to the wb_id for each 1 second bout of the data.
        n_seconds = int(len(data) // long_trial.sampling_rate_hz)
        result.cadence_per_sec = pd.DataFrame(
            [gs.id] * n_seconds,
            columns=["cadence_spm"],
            index=as_samples(
                np.arange(0, n_seconds) + 0.5, long_trial.sampling_rate_hz
            ),
        ).rename_axis(index="sec_center_samples")








.. GENERATED FROM PYTHON SOURCE LINES 134-135

After the iteration, we can access the aggregated results using the `results_` property of the iterator

.. GENERATED FROM PYTHON SOURCE LINES 135-137

.. code-block:: default

    iterator.results_.ic_list






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>ic</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>step_id</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>0</th>
          <td>1019</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1119</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1219</td>
        </tr>
        <tr>
          <th>3</th>
          <td>1319</td>
        </tr>
        <tr>
          <th>4</th>
          <td>1419</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>3</th>
          <td>21678</td>
        </tr>
        <tr>
          <th>4</th>
          <td>21778</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21878</td>
        </tr>
        <tr>
          <th>6</th>
          <td>21978</td>
        </tr>
        <tr>
          <th>7</th>
          <td>22078</td>
        </tr>
      </tbody>
    </table>
    <p>69 rows × 1 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 138-145

We can see that we only get a single dataframe with all the results.
And all ICs are offset, so that they are relative to the start of the recording and not the start of the gait
sequence anymore.

For the cadence value, the index represents the sample of the center of the second the cadence value belongs to.
This value was originally relative to the start of the GS.
We can see that in the aggregated results this is transformed back to be relative to the start of the recording.

.. GENERATED FROM PYTHON SOURCE LINES 145-148

.. code-block:: default

    iterator.results_.cadence_per_sec







.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>cadence_spm</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>sec_center_samples</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="5" valign="top">0</th>
          <th>1069</th>
          <td>0</td>
        </tr>
        <tr>
          <th>1169</th>
          <td>0</td>
        </tr>
        <tr>
          <th>1269</th>
          <td>0</td>
        </tr>
        <tr>
          <th>1369</th>
          <td>0</td>
        </tr>
        <tr>
          <th>1469</th>
          <td>0</td>
        </tr>
        <tr>
          <th>...</th>
          <th>...</th>
          <td>...</td>
        </tr>
        <tr>
          <th rowspan="5" valign="top">5</th>
          <th>21628</th>
          <td>5</td>
        </tr>
        <tr>
          <th>21728</th>
          <td>5</td>
        </tr>
        <tr>
          <th>21828</th>
          <td>5</td>
        </tr>
        <tr>
          <th>21928</th>
          <td>5</td>
        </tr>
        <tr>
          <th>22028</th>
          <td>5</td>
        </tr>
      </tbody>
    </table>
    <p>63 rows × 1 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 149-164

But what to do, if you don't want to use the default result datatype?

Custom Results
--------------

This requires a little bit more setup.
First we need to decide what results we expect.
This is done by defining a dataclass that represents the results.

Here we create a new dataclass that only expect two dummy results, but you can add as many as you want.
You could also subclass the default dataclass and just add the additional results.

The first result here is ``n_samples`` which is just a dummy results indicating the number of samples the data has.
The second result is ``filtered_data`` (we will just add some dummy data here).
This is expected to be a pd.DataFrame to demonstrate that you can also return more complex results.

.. GENERATED FROM PYTHON SOURCE LINES 164-173

.. code-block:: default

    from dataclasses import dataclass


    @dataclass
    class ResultType:
        n_samples: int
        filtered_data: pd.DataFrame









.. GENERATED FROM PYTHON SOURCE LINES 174-184

For each iteration (i.e. for each gait sequence), we will create one instance of this dataclass.
The list of these instances will be available as the `raw_results_` attribute of the iterator.

We can also decide to aggregate the results.
We provide some default aggregations functions (see ``GsIterator.DEFAULT_AGGREGATORS``), that you could use.
However, here we will create our own aggregation function.

It might be nice to turn the ``n_samples`` into a pandas series with the gs identifier as index.
For this we define an aggregation function that expects the list of ``TypedIteratorResultTuple``.
These are named tuples of the following shape:

.. GENERATED FROM PYTHON SOURCE LINES 184-189

.. code-block:: default

    from tpcp.misc import TypedIteratorResultTuple

    display(inspect.getsource(TypedIteratorResultTuple))






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    class TypedIteratorResultTuple(NamedTuple, Generic[InputTypeT, ResultT]):
        iteration_name: str
        input: InputTypeT
        result: ResultT
        iteration_context: dict[str, Any]





.. GENERATED FROM PYTHON SOURCE LINES 190-207

The type of the ``input`` and the ``result`` depend on the dataclass you defined and the iterator you use.
For the gait sequence iterator the input-type will be ``tuple[Region, pd.DataFrame]`` and the result-type will
the dataclass you defined.
The other arguments provide additional context, that might be needed in advanced cases (see lower down in this
example).

To simplify typing of functions that use these types, we provide ``GsIterator.IteratorResult`` which already has the
input type bound and is generic with respect to the output type.
We can see in the function below how to use it.

As mentioned, an aggregation function will get a list of these named tuples.
Note, that the values get passed the entire result object and that parts of the result objects might be ``NOT_SET``.
To filter out the ``NOT_SET`` values and replace the ``result`` attribute with just one specific value, we provide
the ``GsIterator.filter_iterator_results`` function (see below).

With that, out aggregate function, takes the gs-id from the inputs and the n_samples from the results and creates a
pandas series with the gs-id as index and the n_samples as values.

.. GENERATED FROM PYTHON SOURCE LINES 207-217

.. code-block:: default

    def aggregate_n_samples(values: list[GsIterator.IteratorResult[ResultType]]):
        non_null_results: list[GsIterator.IteratorResult[int]] = (
            GsIterator.filter_iterator_results(values, "n_samples")
        )
        results = {r.input[0].id: r.result for r in non_null_results}
        return pd.Series(results, name="N-Samples")


    aggregations = [("n_samples", aggregate_n_samples)]








.. GENERATED FROM PYTHON SOURCE LINES 218-222

Now we can create an instance of the iterator.
Note, that if we want to correctly infer the result type, we need to use the somewhat weird square bracket-typing
syntax, when creating the iterator.
This will allow to autocomplete the attributes of the result type.

.. GENERATED FROM PYTHON SOURCE LINES 222-225

.. code-block:: default


    custom_iterator = GsIterator[ResultType](ResultType, aggregations=aggregations)








.. GENERATED FROM PYTHON SOURCE LINES 226-228

Iterating over the iterator now provides us the row from the gait sequence list (which we ignore here), the data for
each iteration, and the empty result object, we can fill up each iteration.

.. GENERATED FROM PYTHON SOURCE LINES 228-238

.. code-block:: default


    for (_, data), custom_result in custom_iterator.iterate(
        long_trial.data_ss, long_trial_gs
    ):
        # We just calculate the length, but you can image any other calculation here.
        # Then we just set the result.
        custom_result.n_samples = len(data)
        # For the "filtered" data we just subtract 1 form the input
        custom_result.filtered_data = data - 1








.. GENERATED FROM PYTHON SOURCE LINES 239-243

Then we can easily inspect the aggregated results.
Note, while the typing system can correctly infer the available attributes of the result object, the typing of the
attributes might be wrong as Python can not infer the types based on the aggregations.
We have to explicitly cast the value if we care about the type-correctness,

.. GENERATED FROM PYTHON SOURCE LINES 243-248

.. code-block:: default

    from typing import cast

    n_samples = cast(pd.Series, custom_iterator.results_.n_samples)
    n_samples





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    0     749
    1    1015
    2     904
    3    2296
    4     831
    5     751
    Name: N-Samples, dtype: int64



.. GENERATED FROM PYTHON SOURCE LINES 249-250

For the filtered data, we did not apply any aggregation and hence just get a list of all results.

.. GENERATED FROM PYTHON SOURCE LINES 250-254

.. code-block:: default

    filtered_data = cast(list[pd.DataFrame], custom_iterator.results_.filtered_data)
    filtered_data






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    [                                        acc_x     acc_y  ...    gyr_y    gyr_z
    time                                                     ...                  
    2020-10-30 12:53:33.211999893+00:00  9.746760 -0.609926  ...  -5.9149  18.8744
    2020-10-30 12:53:33.221999884+00:00  9.212541 -1.843216  ... -16.0051  24.6507
    2020-10-30 12:53:33.232000113+00:00  8.315173 -2.915390  ... -21.3116  26.9805
    2020-10-30 12:53:33.242000103+00:00  7.357514 -3.853508  ... -22.8495  25.4291
    2020-10-30 12:53:33.252000092+00:00  6.386339 -4.408384  ... -21.4698  21.1185
    ...                                       ...       ...  ...      ...      ...
    2020-10-30 12:53:40.651999950+00:00  7.841718 -3.162747  ...   3.7649 -10.4524
    2020-10-30 12:53:40.661999941+00:00  7.816116 -3.056435  ...   5.7176 -11.1201
    2020-10-30 12:53:40.671999931+00:00  7.897385 -2.964248  ...   6.3879 -11.5961
    2020-10-30 12:53:40.681999922+00:00  7.913504 -2.904070  ...   5.8304 -12.1774
    2020-10-30 12:53:40.691999912+00:00  8.062056 -2.800103  ...   6.3839 -12.1929

    [749 rows x 6 columns],                                         acc_x     acc_y  ...    gyr_y    gyr_z
    time                                                     ...                  
    2020-10-30 12:54:08.361999989+00:00  8.993886 -1.864273  ...  11.8612  -6.3713
    2020-10-30 12:54:08.371999979+00:00  9.006853 -1.821367  ...  11.6298  -4.6801
    2020-10-30 12:54:08.381999969+00:00  9.066191 -1.717037  ...  11.0973  -3.1574
    2020-10-30 12:54:08.391999960+00:00  9.389386 -1.495249  ...   9.9199  -0.4397
    2020-10-30 12:54:08.401999950+00:00  9.781203 -1.396804  ...   7.7379   3.6206
    ...                                       ...       ...  ...      ...      ...
    2020-10-30 12:54:18.461999893+00:00  7.044788  0.514322  ... -27.7005  35.8898
    2020-10-30 12:54:18.471999884+00:00  6.966694  0.439490  ... -26.9190  37.7245
    2020-10-30 12:54:18.482000113+00:00  6.716884  0.505251  ... -26.5421  38.8936
    2020-10-30 12:54:18.492000103+00:00  6.448211  0.742959  ... -26.5874  39.8177
    2020-10-30 12:54:18.502000093+00:00  6.191566  0.939965  ... -26.6665  40.8314

    [1015 rows x 6 columns],                                         acc_x     acc_y  ...    gyr_y   gyr_z
    time                                                     ...                 
    2020-10-30 12:54:59.671999931+00:00  7.889232 -1.136809  ...  -6.6887 -0.5291
    2020-10-30 12:54:59.681999922+00:00  7.987170 -1.144480  ...  -7.2629 -0.7600
    2020-10-30 12:54:59.691999912+00:00  8.074680 -1.182571  ...  -7.6096 -1.1997
    2020-10-30 12:54:59.701999903+00:00  8.150059 -1.142325  ...  -7.9950 -1.5970
    2020-10-30 12:54:59.711999893+00:00  8.128522 -1.144701  ...  -8.3241 -1.6867
    ...                                       ...       ...  ...      ...     ...
    2020-10-30 12:55:08.661999941+00:00  8.133023 -2.345900  ...  16.1862 -7.2686
    2020-10-30 12:55:08.671999931+00:00  8.149110 -2.267799  ...  17.1322 -8.0022
    2020-10-30 12:55:08.681999922+00:00  8.179423 -2.251566  ...  17.5279 -8.3387
    2020-10-30 12:55:08.691999912+00:00  8.302335 -2.481964  ...  18.2932 -8.8593
    2020-10-30 12:55:08.701999903+00:00  8.430995 -2.731003  ...  18.5796 -7.7286

    [904 rows x 6 columns],                                         acc_x     acc_y  ...    gyr_y    gyr_z
    time                                                     ...                  
    2020-10-30 12:55:26.391999960+00:00  9.232635 -2.686375  ... -18.6458 -25.5380
    2020-10-30 12:55:26.401999950+00:00  9.363166 -2.469547  ... -20.7625 -24.8423
    2020-10-30 12:55:26.411999941+00:00  9.262668 -2.263549  ... -18.0376 -24.7323
    2020-10-30 12:55:26.421999931+00:00  8.993393 -2.105027  ... -10.3876 -24.2681
    2020-10-30 12:55:26.431999922+00:00  8.917060 -1.906179  ...  -1.9655 -23.3928
    ...                                       ...       ...  ...      ...      ...
    2020-10-30 12:55:49.302000046+00:00  7.714016  0.326042  ...  15.9933   7.4403
    2020-10-30 12:55:49.312000036+00:00  7.725419  0.574019  ...  15.3581   5.5792
    2020-10-30 12:55:49.322000027+00:00  7.813756  0.665189  ...  15.8199   4.5902
    2020-10-30 12:55:49.332000017+00:00  8.071623  0.308971  ...  17.9715   3.6861
    2020-10-30 12:55:49.342000008+00:00  8.267141 -0.338464  ...  20.9984   3.2169

    [2296 rows x 6 columns],                                         acc_x     acc_y  ...    gyr_y    gyr_z
    time                                                     ...                  
    2020-10-30 12:56:44.532000064+00:00  8.132286 -2.796825  ...  15.4808 -22.0089
    2020-10-30 12:56:44.542000055+00:00  8.472326 -2.708309  ...  16.3716 -23.4909
    2020-10-30 12:56:44.552000046+00:00  8.850238 -2.523199  ...  15.7210 -24.0634
    2020-10-30 12:56:44.562000036+00:00  9.348547 -2.173151  ...  14.0635 -24.1058
    2020-10-30 12:56:44.572000027+00:00  9.730036 -1.798863  ...  12.3673 -23.7012
    ...                                       ...       ...  ...      ...      ...
    2020-10-30 12:56:52.792000055+00:00  8.033643 -2.490667  ...  13.5385   1.7647
    2020-10-30 12:56:52.802000046+00:00  8.542713 -2.340519  ...  11.4386   6.4245
    2020-10-30 12:56:52.812000036+00:00  8.818854 -2.348388  ...  12.2948  10.9618
    2020-10-30 12:56:52.822000027+00:00  9.123138 -2.335998  ...  15.8081  14.4861
    2020-10-30 12:56:52.832000017+00:00  9.461653 -2.308538  ...  17.4874  17.1938

    [831 rows x 6 columns],                                         acc_x     acc_y  ...    gyr_y    gyr_z
    time                                                     ...                  
    2020-10-30 12:56:56.802000046+00:00  8.891845 -4.079999  ...  -6.2421 -17.0258
    2020-10-30 12:56:56.812000036+00:00  8.841221 -4.293471  ... -12.8417 -14.2214
    2020-10-30 12:56:56.822000027+00:00  8.438049 -4.353188  ... -10.4114 -12.7980
    2020-10-30 12:56:56.832000017+00:00  7.941118 -4.371653  ...  -0.0473 -10.9684
    2020-10-30 12:56:56.842000008+00:00  7.879858 -4.434756  ...  11.1010  -7.4715
    ...                                       ...       ...  ...      ...      ...
    2020-10-30 12:57:04.262000084+00:00  7.598598  1.484613  ...  -2.9975  -8.6763
    2020-10-30 12:57:04.272000074+00:00  8.016810  1.889539  ...  -1.3660  -9.6691
    2020-10-30 12:57:04.282000065+00:00  8.434351  2.205847  ...  -0.3714  -8.8523
    2020-10-30 12:57:04.292000055+00:00  8.746053  2.303438  ...  -2.6950  -7.8553
    2020-10-30 12:57:04.302000046+00:00  9.100351  2.272933  ... -10.0859  -6.3404

    [751 rows x 6 columns]]



.. GENERATED FROM PYTHON SOURCE LINES 255-278

Sub-Iterations (Advanced)
-------------------------
Using the iterator to iterate GSs or other types of regions of interest works well, if all of them are defined at the
start of the processing.
However, sometimes you might want to iterate over sub-regions of the gait sequences where the regions are only
calculated during the iteration.
In this case, you would need to start creating multiple instances of the iterator.
However, this is cumbersome and redundant, as both iterator share a lot of information.
Hence, we support this special case with the ``iterate_subregions`` method.
It takes a gait-sequence list as input, that is defined relative to the gait sequence that is currently processed.
It then iterates over the sub-regions, provides new result objects for each sub-region, and then magically aggregates
everything after the main iteration ends.

.. note:: There is one usecase, we don't support at the moment, and that is accessing the results of the
          sub-iterations in the outer loop.
          The results are only available after the main iteration ends.
          However, for this you can create a new instance of your iterator within the outer loop instead of using
          ``iterate_subregions``.

Below we show an "artificial" example, where we split each outer gs dynamically into 3 subparts.
We then calculate the length of each subpart and detect some "fake" events.

As before, we start by defining a type for the results.

.. GENERATED FROM PYTHON SOURCE LINES 278-285

.. code-block:: default

    @dataclass
    class CustomNestedResults:
        n_samples: int
        outer_regions: pd.DataFrame
        events: pd.DataFrame









.. GENERATED FROM PYTHON SOURCE LINES 286-294

And 3 aggregators:

1. A df-aggregator that adjusts the index of the events to be relative to the start of the original data.
2. A df-aggregator that adjusts the start/end of the outer_regions to be relative to the start of the original data.
3. An aggregator that turns the n_samples into a pandas series with the gs identifier as index.

For the first two aggregator, we can just use the default aggregator for dataframes and tell is that we want to modify
the ``ev`` column based on the start of the respective GS.

.. GENERATED FROM PYTHON SOURCE LINES 294-300

.. code-block:: default

    events_agg = GsIterator.DefaultAggregators.create_aggregate_df("events", ["ev"])
    outer_regions_agg = GsIterator.DefaultAggregators.create_aggregate_df(
        "outer_regions", ["start", "end"]
    )









.. GENERATED FROM PYTHON SOURCE LINES 301-304

For the second, we will use a modified version of the aggregator we used before.
The only difference is that we will make use of the ``iteration_context``.
In case of a nested iteration, the context will contain the parent-GS.

.. GENERATED FROM PYTHON SOURCE LINES 304-321

.. code-block:: default

    def aggregate_n_samples(values: list[GsIterator.IteratorResult[ResultType]]):
        non_null_results: list[GsIterator.IteratorResult[int]] = (
            GsIterator.filter_iterator_results(values, "n_samples")
        )
        results = [r.result for r in non_null_results]
        ids = [
            (r.iteration_context["parent_region"].id, r.input[0].id)
            for r in non_null_results
        ]
        index_col_names = [
            non_null_results[0].iteration_context["parent_region"].id_origin,
            non_null_results[0].input.region.id_origin,
        ]
        index = pd.MultiIndex.from_tuples(ids, names=index_col_names)
        return pd.Series(results, index=index, name="N-Samples")









.. GENERATED FROM PYTHON SOURCE LINES 322-323

Now we can define the iterator.

.. GENERATED FROM PYTHON SOURCE LINES 323-332

.. code-block:: default

    nested_iterator = GsIterator[CustomNestedResults](
        CustomNestedResults,
        aggregations=[
            ("n_samples", aggregate_n_samples),
            ("events", events_agg),
            ("outer_regions", outer_regions_agg),
        ],
    )








.. GENERATED FROM PYTHON SOURCE LINES 333-336

When we loop the iterator, we will reuse the outer iteration as before, but then "simulate" an algorithm that
identifies sub-regions within the gait sequence.
Note, that we can write some results in the outer scope and some results in the inner scope.

.. GENERATED FROM PYTHON SOURCE LINES 336-356

.. code-block:: default

    for (_, data), r in nested_iterator.iterate(long_trial.data_ss, long_trial_gs):
        print(
            f"Length of outer data: {len(data)} samples. Divided by 3: {len(data) // 3} samples."
        )
        r.outer_regions = pd.DataFrame(
            {
                "start": [0, len(data) // 3, 2 * len(data) // 3],
                "end": [len(data) // 3, 2 * len(data) // 3, len(data)],
            }
        ).rename_axis("sub_roi_id")

        # Then we iterate over the sub-regions and calculate the length of each sub-region and identify fake events
        for (_, nested_data), nr in nested_iterator.iterate_subregions(
            r.outer_regions
        ):
            nr.n_samples = len(nested_data)
            nr.events = pd.DataFrame(
                {"ev": np.linspace(0, len(nested_data), 3, dtype="int64")}
            ).rename_axis("step_id")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Length of outer data: 749 samples. Divided by 3: 249 samples.
    Length of outer data: 1015 samples. Divided by 3: 338 samples.
    Length of outer data: 904 samples. Divided by 3: 301 samples.
    Length of outer data: 2296 samples. Divided by 3: 765 samples.
    Length of outer data: 831 samples. Divided by 3: 277 samples.
    Length of outer data: 751 samples. Divided by 3: 250 samples.




.. GENERATED FROM PYTHON SOURCE LINES 357-363

After the iteration, we can access the aggregated results.
Let's start with the unspectacular ``outer_regions``.

As we wrote them in the outer scope, iteration and aggreagtion worked just like before.
We can see that the start and end values are now relative to the start of the recording and match the orignal gait
sequences (see below).

.. GENERATED FROM PYTHON SOURCE LINES 363-365

.. code-block:: default

    nested_iterator.results_.outer_regions






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>sub_roi_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>1019</td>
          <td>1268</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1268</td>
          <td>1518</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1518</td>
          <td>1768</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>4534</td>
          <td>4872</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4872</td>
          <td>5210</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5210</td>
          <td>5549</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>9665</td>
          <td>9966</td>
        </tr>
        <tr>
          <th>1</th>
          <td>9966</td>
          <td>10267</td>
        </tr>
        <tr>
          <th>2</th>
          <td>10267</td>
          <td>10569</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">3</th>
          <th>0</th>
          <td>12337</td>
          <td>13102</td>
        </tr>
        <tr>
          <th>1</th>
          <td>13102</td>
          <td>13867</td>
        </tr>
        <tr>
          <th>2</th>
          <td>13867</td>
          <td>14633</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">4</th>
          <th>0</th>
          <td>20151</td>
          <td>20428</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20428</td>
          <td>20705</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20705</td>
          <td>20982</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">5</th>
          <th>0</th>
          <td>21378</td>
          <td>21628</td>
        </tr>
        <tr>
          <th>1</th>
          <td>21628</td>
          <td>21878</td>
        </tr>
        <tr>
          <th>2</th>
          <td>21878</td>
          <td>22129</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 366-367

For reference the outer GSs:

.. GENERATED FROM PYTHON SOURCE LINES 367-369

.. code-block:: default

    long_trial_gs






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>start</th>
          <th>end</th>
          <th>n_strides</th>
          <th>duration_s</th>
          <th>length_m</th>
          <th>avg_walking_speed_mps</th>
          <th>avg_cadence_spm</th>
          <th>avg_stride_length_m</th>
          <th>termination_reason</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1019</td>
          <td>1768</td>
          <td>9</td>
          <td>7.48</td>
          <td>4.468932</td>
          <td>0.847668</td>
          <td>107.795850</td>
          <td>0.942678</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4534</td>
          <td>5549</td>
          <td>11</td>
          <td>10.14</td>
          <td>2.900453</td>
          <td>0.365176</td>
          <td>93.396106</td>
          <td>0.483923</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9665</td>
          <td>10569</td>
          <td>9</td>
          <td>9.03</td>
          <td>2.140232</td>
          <td>0.294058</td>
          <td>75.981133</td>
          <td>0.506458</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12337</td>
          <td>14633</td>
          <td>28</td>
          <td>22.95</td>
          <td>11.201110</td>
          <td>0.634425</td>
          <td>92.337768</td>
          <td>0.803933</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>4</th>
          <td>20151</td>
          <td>20982</td>
          <td>11</td>
          <td>8.30</td>
          <td>2.390709</td>
          <td>0.371746</td>
          <td>87.915774</td>
          <td>0.507484</td>
          <td>Pause</td>
        </tr>
        <tr>
          <th>5</th>
          <td>21378</td>
          <td>22129</td>
          <td>9</td>
          <td>7.50</td>
          <td>2.517558</td>
          <td>0.492965</td>
          <td>95.365740</td>
          <td>0.599360</td>
          <td>Pause</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 370-372

We can see that our n_samples are now a multi-index series with both gs-levels as index.
The length roughly matches the length of the outer scope that we printed during iteration (see above).

.. GENERATED FROM PYTHON SOURCE LINES 372-374

.. code-block:: default

    nested_iterator.results_.n_samples





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    wb_id  sub_roi_id
    0      0             249
           1             250
           2             250
    1      0             338
           1             338
           2             339
    2      0             301
           1             301
           2             302
    3      0             765
           1             765
           2             766
    4      0             277
           1             277
           2             277
    5      0             250
           1             250
           2             251
    Name: N-Samples, dtype: int64



.. GENERATED FROM PYTHON SOURCE LINES 375-377

The events are also a multi-index dataframe containin both gs-levels.
All ev values are modified to be relative to the start of the recording.

.. GENERATED FROM PYTHON SOURCE LINES 377-379

.. code-block:: default

    nested_iterator.results_.events






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th>ev</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>sub_roi_id</th>
          <th>step_id</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="9" valign="top">0</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>1019</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1143</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1268</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>1268</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1393</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1518</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>1518</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1643</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1768</td>
        </tr>
        <tr>
          <th rowspan="9" valign="top">1</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>4534</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4703</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4872</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>4872</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5041</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5210</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>5210</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5379</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5549</td>
        </tr>
        <tr>
          <th rowspan="9" valign="top">2</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>9665</td>
        </tr>
        <tr>
          <th>1</th>
          <td>9815</td>
        </tr>
        <tr>
          <th>2</th>
          <td>9966</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>9966</td>
        </tr>
        <tr>
          <th>1</th>
          <td>10116</td>
        </tr>
        <tr>
          <th>2</th>
          <td>10267</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>10267</td>
        </tr>
        <tr>
          <th>1</th>
          <td>10418</td>
        </tr>
        <tr>
          <th>2</th>
          <td>10569</td>
        </tr>
        <tr>
          <th rowspan="9" valign="top">3</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>12337</td>
        </tr>
        <tr>
          <th>1</th>
          <td>12719</td>
        </tr>
        <tr>
          <th>2</th>
          <td>13102</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>13102</td>
        </tr>
        <tr>
          <th>1</th>
          <td>13484</td>
        </tr>
        <tr>
          <th>2</th>
          <td>13867</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>13867</td>
        </tr>
        <tr>
          <th>1</th>
          <td>14250</td>
        </tr>
        <tr>
          <th>2</th>
          <td>14633</td>
        </tr>
        <tr>
          <th rowspan="9" valign="top">4</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>20151</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20289</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20428</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>20428</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20566</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20705</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>20705</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20843</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20982</td>
        </tr>
        <tr>
          <th rowspan="9" valign="top">5</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>21378</td>
        </tr>
        <tr>
          <th>1</th>
          <td>21503</td>
        </tr>
        <tr>
          <th>2</th>
          <td>21628</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th>0</th>
          <td>21628</td>
        </tr>
        <tr>
          <th>1</th>
          <td>21753</td>
        </tr>
        <tr>
          <th>2</th>
          <td>21878</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th>0</th>
          <td>21878</td>
        </tr>
        <tr>
          <th>1</th>
          <td>22003</td>
        </tr>
        <tr>
          <th>2</th>
          <td>22129</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 380-392

Single nested regions/aka refined GS (advanced)
-----------------------------------------------
In some cases, you might want to iterate over a single sub-region of the gait sequence.
While you could use the ``iterate_subregions`` method, this is a bit cumbersome and makes the code harder to read.
For this we provide the ``with_subregion`` and the ``subregion`` method, where the latter is syntactic sugar for the
former.
Both methods simply return the same output that you would get per iteration, but simply once.
Below a short example on how this works.
We start with the ``subregion`` version, as this is actually the recommended way to use it, as we think it is easier
to read, even though it might be a bit surprising that Python allows this.

We are going to reuse most of the setup from the previous example.

.. GENERATED FROM PYTHON SOURCE LINES 392-401

.. code-block:: default

    flat_nested_iterator = GsIterator[CustomNestedResults](
        CustomNestedResults,
        aggregations=[
            ("n_samples", aggregate_n_samples),
            ("events", events_agg),
            ("outer_regions", outer_regions_agg),
        ],
    )








.. GENERATED FROM PYTHON SOURCE LINES 402-405

But then we will use the ``subregion`` to run some computations in the context of the refined GS.
The return value of ``subregion`` acts as contextmanager, that allows to visually encapsulate the code that is run in
the context of the refined GS.

.. GENERATED FROM PYTHON SOURCE LINES 405-425

.. code-block:: default

    for (_, data), r in flat_nested_iterator.iterate(
        long_trial.data_ss, long_trial_gs
    ):
        r.outer_regions = pd.DataFrame(
            {
                "start": [5],
                "end": [len(data) - 5],
            }
        ).rename_axis("refined_gs_id")

        with flat_nested_iterator.subregion(r.outer_regions.iloc[[0]]) as (
            (_, refined_data),
            refined_result,
        ):
            refined_result.n_samples = len(refined_data)
            refined_result.events = pd.DataFrame(
                {"ev": np.linspace(0, len(refined_data), 3, dtype="int64")}
            ).rename_axis("step_id")









.. GENERATED FROM PYTHON SOURCE LINES 426-427

This is equivalent to the following code, using ``with_subregion``:

.. GENERATED FROM PYTHON SOURCE LINES 427-445

.. code-block:: default

    for (_, data), r in flat_nested_iterator.iterate(
        long_trial.data_ss, long_trial_gs
    ):
        r.outer_regions = pd.DataFrame(
            {
                "start": [5],
                "end": [len(data) - 5],
            }
        ).rename_axis("refined_gs_id")

        (_, refined_data), refined_result = flat_nested_iterator.with_subregion(
            r.outer_regions.iloc[[0]]
        )
        refined_result.n_samples = len(refined_data)
        refined_result.events = pd.DataFrame(
            {"ev": np.linspace(0, len(refined_data), 3, dtype="int64")}
        ).rename_axis("step_id")








.. GENERATED FROM PYTHON SOURCE LINES 446-447

And in both cases everything is aggregated as expected.

.. GENERATED FROM PYTHON SOURCE LINES 447-449

.. code-block:: default

    flat_nested_iterator.results_.outer_regions






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>start</th>
          <th>end</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>refined_gs_id</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <th>0</th>
          <td>1024</td>
          <td>1763</td>
        </tr>
        <tr>
          <th>1</th>
          <th>0</th>
          <td>4539</td>
          <td>5544</td>
        </tr>
        <tr>
          <th>2</th>
          <th>0</th>
          <td>9670</td>
          <td>10564</td>
        </tr>
        <tr>
          <th>3</th>
          <th>0</th>
          <td>12342</td>
          <td>14628</td>
        </tr>
        <tr>
          <th>4</th>
          <th>0</th>
          <td>20156</td>
          <td>20977</td>
        </tr>
        <tr>
          <th>5</th>
          <th>0</th>
          <td>21383</td>
          <td>22124</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 450-452

.. code-block:: default

    flat_nested_iterator.results_.n_samples





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    wb_id  refined_gs_id
    0      0                 739
    1      0                1005
    2      0                 894
    3      0                2286
    4      0                 821
    5      0                 741
    Name: N-Samples, dtype: int64



.. GENERATED FROM PYTHON SOURCE LINES 453-455

.. code-block:: default

    flat_nested_iterator.results_.events






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th></th>
          <th>ev</th>
        </tr>
        <tr>
          <th>wb_id</th>
          <th>refined_gs_id</th>
          <th>step_id</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th rowspan="3" valign="top">0</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>1024</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1393</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1763</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">1</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>4539</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5041</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5544</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">2</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>9670</td>
        </tr>
        <tr>
          <th>1</th>
          <td>10117</td>
        </tr>
        <tr>
          <th>2</th>
          <td>10564</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">3</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>12342</td>
        </tr>
        <tr>
          <th>1</th>
          <td>13485</td>
        </tr>
        <tr>
          <th>2</th>
          <td>14628</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">4</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>20156</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20566</td>
        </tr>
        <tr>
          <th>2</th>
          <td>20977</td>
        </tr>
        <tr>
          <th rowspan="3" valign="top">5</th>
          <th rowspan="3" valign="top">0</th>
          <th>0</th>
          <td>21383</td>
        </tr>
        <tr>
          <th>1</th>
          <td>21753</td>
        </tr>
        <tr>
          <th>2</th>
          <td>22124</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 456-469

Nested Iterations - under the Hood
----------------------------------
These nested iterators a re a little bit black magic...
If you are working with them, it might be nice to have some understanding of what is going on.

When a new item is yielded during iteration (in the outer or the inner), the iterator will create a new instance of
result object and will internally store this object together with some metadata.
This metadata includes an indicator, if we are in the parent or sub-iteration scope and in case of the subscope it
contains the parent GS we are iterating.

We can see the stored information by inspecting ``raw_results_``.
We will do that for the nested iterator we used before.
We will format them a little to make things easier to read.

.. GENERATED FROM PYTHON SOURCE LINES 469-478

.. code-block:: default

    from pprint import pprint

    pprint(
        [
            v._replace(result="...", input=(v.input[0], "..."))
            for v in nested_iterator.raw_results_
        ]
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=0, start=1019, end=1768, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=249, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=249, end=499, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=499, end=749, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=1, start=4534, end=5549, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=338, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=338, end=676, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=676, end=1015, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=2, start=9665, end=10569, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=301, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=301, end=602, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=602, end=904, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=3, start=12337, end=14633, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=765, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=765, end=1530, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=1530, end=2296, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=4, start=20151, end=20982, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=277, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=277, end=554, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=554, end=831, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__main__', input=(Region(id=5, start=21378, end=22129, id_origin='wb_id'), '...'), result='...', iteration_context={}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=250, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=250, end=500, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=500, end=751, id_origin='sub_roi_id'), '...'), result='...', iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')})]




.. GENERATED FROM PYTHON SOURCE LINES 479-485

All iterations that are marked as ``__sub_iter__`` are the sub-iterations and we can see that they have the parent GS
in the context.

If we look at the result values, we can see that the ``n_samples`` are only on result objects that come from the
inner scope.
For the result objects from the outer scope, the ``n_samples`` are set to ``NOT_SET``.

.. GENERATED FROM PYTHON SOURCE LINES 485-492

.. code-block:: default

    pprint(
        [
            (v.iteration_name, v.result.n_samples)
            for v in nested_iterator.raw_results_
        ]
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [('__main__', NOT_SET),
     ('__sub_iter__', 249),
     ('__sub_iter__', 250),
     ('__sub_iter__', 250),
     ('__main__', NOT_SET),
     ('__sub_iter__', 338),
     ('__sub_iter__', 338),
     ('__sub_iter__', 339),
     ('__main__', NOT_SET),
     ('__sub_iter__', 301),
     ('__sub_iter__', 301),
     ('__sub_iter__', 302),
     ('__main__', NOT_SET),
     ('__sub_iter__', 765),
     ('__sub_iter__', 765),
     ('__sub_iter__', 766),
     ('__main__', NOT_SET),
     ('__sub_iter__', 277),
     ('__sub_iter__', 277),
     ('__sub_iter__', 277),
     ('__main__', NOT_SET),
     ('__sub_iter__', 250),
     ('__sub_iter__', 250),
     ('__sub_iter__', 251)]




.. GENERATED FROM PYTHON SOURCE LINES 493-496

The second piece of "magic" happens in the aggregation functions.
There we use the ``filter_iterator_results`` function to filter out the ``NOT_SET`` values, so that we can operate on
the actual values and use their context to make adjustments/aggregate them.

.. GENERATED FROM PYTHON SOURCE LINES 496-505

.. code-block:: default

    pprint(
        [
            v._replace(input=(v.input[0], "..."))
            for v in GsIterator.filter_iterator_results(
                nested_iterator.raw_results_, "n_samples"
            )
        ]
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=249, id_origin='sub_roi_id'), '...'), result=249, iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=249, end=499, id_origin='sub_roi_id'), '...'), result=250, iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=499, end=749, id_origin='sub_roi_id'), '...'), result=250, iteration_context={'parent_region': Region(id=0, start=1019, end=1768, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=338, id_origin='sub_roi_id'), '...'), result=338, iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=338, end=676, id_origin='sub_roi_id'), '...'), result=338, iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=676, end=1015, id_origin='sub_roi_id'), '...'), result=339, iteration_context={'parent_region': Region(id=1, start=4534, end=5549, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=301, id_origin='sub_roi_id'), '...'), result=301, iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=301, end=602, id_origin='sub_roi_id'), '...'), result=301, iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=602, end=904, id_origin='sub_roi_id'), '...'), result=302, iteration_context={'parent_region': Region(id=2, start=9665, end=10569, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=765, id_origin='sub_roi_id'), '...'), result=765, iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=765, end=1530, id_origin='sub_roi_id'), '...'), result=765, iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=1530, end=2296, id_origin='sub_roi_id'), '...'), result=766, iteration_context={'parent_region': Region(id=3, start=12337, end=14633, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=277, id_origin='sub_roi_id'), '...'), result=277, iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=277, end=554, id_origin='sub_roi_id'), '...'), result=277, iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=554, end=831, id_origin='sub_roi_id'), '...'), result=277, iteration_context={'parent_region': Region(id=4, start=20151, end=20982, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=0, start=0, end=250, id_origin='sub_roi_id'), '...'), result=250, iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=1, start=250, end=500, id_origin='sub_roi_id'), '...'), result=250, iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')}),
     TypedIteratorResultTuple(iteration_name='__sub_iter__', input=(Region(id=2, start=500, end=751, id_origin='sub_roi_id'), '...'), result=251, iteration_context={'parent_region': Region(id=5, start=21378, end=22129, id_origin='wb_id')})]




.. GENERATED FROM PYTHON SOURCE LINES 506-508

After the filtering, we only have cases where the value was provided (only inner-iterations in this case).
Based on this we can do further processing.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 4.667 seconds)

**Estimated memory usage:**  9 MB


.. _sphx_glr_download_auto_examples_pipeline__01_gs_iterator.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: _01_gs_iterator.py <_01_gs_iterator.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: _01_gs_iterator.ipynb <_01_gs_iterator.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_