#PANDAS
Understanding Predicate Pushdown at the Row-Group Level in Parquet with PyArrow and Python
Posted on
January 19, 2020
in
python ·
pydata ·
parquet ·
arrow ·
pandas
Apache Parquet is a columnar file format to work with gigabytes of data. Reading and writing parquet files is efficiently exposed to python with pyarrow. Additional statistics allow clients to use predicate pushdown to only read subsets of data to reduce I/O. Organizing data by column allows for better compression, as data is more homogeneous. Better compression also reduces the bandwidth required to read the input. Read More
