PERF: Pivot_table Causes a Significant Increase in Memory Usage When Dealing with Dataframes of Over 4000 Rows

[ForeverFerret]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this issue exists on the latest version of pandas.

• I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

def generate_pivot_table(
    dataframe: DataFrame,
    values=None,
    index=None,
    columns=None,
) -> DataFrame:
    """创建数据透视表

    Args:
        dataframe: 需要透视的功率回退值DataFrame数据
        values: 需透视的数据, 默认['SarBackoff']
        index: 行索引, 默认['Mode', 'Band', 'Antenna']
        columns: 列索引, 默认['SAR场景', '部位']

    Returns:
        转化行列后的DataFrame数据格式
    """

    # 设置数据透视表默认值
    if values is None:
        values = ["SarBackoff"]
    if index is None:
        index = ["Mode", "Band", "Antenna"]
    if columns is None:
        columns = ["SAR场景", "部位"]

    # 返回对应数据透视表
    # TODO 约4000行以上数据, 透视表时内存占用45G+
    return dataframe.pivot_table(
        values=values,
        index=index,
        columns=columns,
        sort=False,
        fill_value="/",
    )

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Chinese (Simplified)_China.936
pandas : 2.1.1
numpy : 1.26.1
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.3
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.16.1
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None

Prior Performance

https://stackoverflow.com/questions/77320771/pivot-table-causes-a-significant-increase-in-memory-usage-when-dealing-with-data

[mroeschke]

Thanks for the report but appears the stackoverflow post and this post doesn't have a reproducible example so closing for now, but feel free to open a new issue if reproducible by a triager