Reshuffling GroupBy

The experimental GroupBy implementation utilizes Modin’s reshuffling mechanism that gives an ability to build range partitioning over a Modin DataFrame.

In order to enable/disable this new implementation you have to specify cfg.ExperimentalGroupbyImpl configuration variable:

In [4]: import modin.config as cfg; cfg.ExperimentalGroupbyImpl.put(True)

In [5]: # past this point, Modin will always use the new reshuffling groupby implementation

In [6]: cfg.ExperimentalGroupbyImpl.put(False)

In [7]: # past this point, Modin won't use reshuffling groupby implementation anymore

The reshuffling implementation appears to be quite efficient when compared to old TreeReduce and FullAxis implementations:

Note

All of the examples below were run on Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (112 cores), 192gb RAM

In [4]: import modin.pandas as pd; import numpy as np

In [5]: df = pd.DataFrame(np.random.randint(0, 1_000_000, size=(1_000_000, 10)), columns=[f"col{i}" for i in range(10)])

In [6]: %timeit df.groupby("col0").nunique() # old full-axis implementation
Out[6]: # 2.73 s ± 28.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [7]: import modin.config as cfg; cfg.ExperimentalGroupbyImpl.put(True)

In [8]: %timeit df.groupby("col0").nunique() # new reshuffling implementation
Out[8]: # 595 ms ± 61.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Although it may look like the new implementation always outperforms the old ones, it’s not actually true. There’s a decent overhead on building the range partitioning itself, meaning that the old implementations may act better on smaller data sizes or when the grouping columns (a key column to build range partitioning) have too few unique values (and thus fewer units of parallelization):

In [4]: import modin.pandas as pd; import numpy as np

In [5]: df = pd.DataFrame({"col0": np.tile(list("abcde"), 50_000), "col1": np.arange(250_000)})

In [6]: %timeit df.groupby("col0").sum() # old TreeReduce implementation
Out[6]: # 155 ms ± 5.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [7]: import modin.config as cfg; cfg.ExperimentalGroupbyImpl.put(True)

In [8]: %timeit df.groupby("col0").sum() # new reshuffling implementation
Out[8]: # 230 ms ± 22.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

We’re still looking for a heuristic that would be able to automatically switch to the best implementation for each groupby case, but for now, we’re offering to play with this switch on your own to see which implementation works best for your particular case.

The new experimental groupby does not yet support all of the pandas API and falls back to older implementation with the respective warning if it meets an unsupported case:

In [14]: import modin.config as cfg; cfg.ExperimentalGroupbyImpl.put(True)

In [15]: df.groupby(level=0).sum()
Out[15]: # UserWarning: Can't use experimental reshuffling groupby implementation because of:
    ...  # Reshuffling groupby is only supported when grouping on a column(s) of the same frame.
    ...  # https://github.com/modin-project/modin/issues/5926
    ...  # Falling back to a TreeReduce implementation.