<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">Peter Hoffmann</title>
  <id>https://peter-hoffmann.com/feed/atom-parquet.xml</id>
  <updated>2020-05-25T00:00:00Z</updated>
  <link href="https://peter-hoffmann.com" />
  <link href="https://peter-hoffmann.com/feed/atom-parquet.xml" rel="self" />
  <subtitle type="text">Atom Feed for parquet related posts peter-hoffmann.com</subtitle>
  <generator>Werkzeug</generator>
  <entry xml:base="https://peter-hoffmann.com/feed/atom-parquet.xml">
    <title type="text">DuckDB vs. Azure Synapse SQL on-demand with Parquet</title>
    <id>https://peter-hoffmann.com/2020/duckdb-vs-azure-synapse-sql-on-demand.html</id>
    <updated>2020-05-25T00:00:00Z</updated>
    <published>2020-05-25T00:00:00Z</published>
    <link href="/2020/duckdb-vs-azure-synapse-sql-on-demand.html" />
    <author>
      <name>Peter Hoffmann</name>
      <uri>https://peter-hoffmann.com</uri>
    </author>
    <content type="html">&lt;p&gt;&lt;strong&gt;Short Disclaimer:&lt;/strong&gt; This post is comparing apples to oranges, because
&lt;a href=&#34;https://www.duckdb.org&#34;&gt;DuckDB&lt;/a&gt; is an embedded database designed to execute
analytical SQL queries on your local machine, whereas &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/on-demand-workspace-overview&#34;&gt;Azure Synapse
SQL-on-Demand&lt;/a&gt;
is Microsoft&#39;s new cloud offering for a serverless SQL on-demand (preview)
endpoint to query data in Azure Blob Storage/Data Lake.&lt;/p&gt;
&lt;p&gt;But in the end I was still triggered by &lt;a href=&#34;https://uwekorn.com/2019/10/19/taking-duckdb-for-a-spin.html&#34;&gt;Uwe Korn&#39;s post: Taking DuckDB for a
spin&lt;/a&gt; to do a
comparison, because even though they are fundamentally different offerings, at
a high level both solutions offer a simple way to run analytical queries on
datasets without needing to install or manage a server.&lt;/p&gt;
&lt;h3&gt;Load the data as Parquet data&lt;/h3&gt;
&lt;p&gt;To get started, convert the yellow tripdata sample data to a Parquet
file and upload it to Azure Blob Storage:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pandas&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pd&lt;/span&gt;
&lt;span class=&#34;kn&#34;&gt;from&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;simplekv.net.azurestore&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;AzureBlockBlobStore&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2016-01.csv&amp;quot;&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;dtype&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;store_and_fwd_flag&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;bool&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;},&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;parse_dates&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_pickup_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_dropoff_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;index_col&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;infer_datetime_format&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;True&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;true_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;false_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;N&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
    &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;to_parquet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2016-01.parquet&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;conn_string&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s1&#34;&gt;&amp;#39;DefaultEndpointsProtocol=https;AccountName=blob;AccountKey=xxx;&amp;#39;&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;store&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;AzureBlockBlobStore&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;conn_string&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;conn_string&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;container&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;benchmark&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;public&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;store&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;putfile&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2016-01.parquet&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2016-01.parquet&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once the data is available in Azure Blob Storage/Data Lake, you can use the &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/develop-openrowset&#34;&gt;openrowset&lt;/a&gt; function
to read the remote dataset in Azure SQL-on-Demand:&lt;/p&gt;
&lt;h2&gt;Count Distinct&lt;/h2&gt;
&lt;p&gt;The minimal COUNT DISTINCT example needs to read all selected columns from Azure Blob Storage once and then execute the count on the data.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/static/2020/azure-sql-on-demand.webp&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
&lt;p&gt;The query roughly takes &lt;code&gt;4s&lt;/code&gt;, which is not too bad and is faster than the &lt;code&gt;5.58s&lt;/code&gt; for DuckDB and the &lt;code&gt;25s&lt;/code&gt; for SQLite.&lt;/p&gt;
&lt;h2&gt;Frequency of events&lt;/h2&gt;
&lt;p&gt;This type of query is a simple way to test aggregation performance.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SELECT&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;MIN&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;cnt&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;),&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AVG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;cnt&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;),&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;c1&#34;&gt;-- MEDIAN(cnt),&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;MAX&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;cnt&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;FROM&lt;/span&gt;
&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SELECT&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;COUNT&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;cnt&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;FROM&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;OPENROWSET&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;BULK&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;https://blob.dfs.core.windows.net/benchmark/yellow_tripdata_2016-01.parquet&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;FORMAT&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;PARQUET&amp;#39;&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;r&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;GROUP&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;BY&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;DATEPART&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;day&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;tpep_pickup_datetime&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;),&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;DATEPART&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;hour&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;tpep_pickup_datetime&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;stats&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;];&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;As the query only needs to read a subset of the columns, it can be executed very fast in &lt;code&gt;&amp;lt;1s&lt;/code&gt;, which again beats DuckDB &lt;code&gt;2.05s&lt;/code&gt; and SQLite &lt;code&gt;10.2s&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;Simple fare regression&lt;/h2&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;with&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;yellow_tripdata_2016_01&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;select&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;FROM&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;OPENROWSET&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;BULK&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;https://blob.dfs.core.windows.net/benchmark/yellow_tripdata_2016-01.parquet&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;FORMAT&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;PARQUET&amp;#39;&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;r&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;
&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;SELECT&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;COUNT&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;))&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SUM&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;COUNT&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;))&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;beta&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AVG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;avg_fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AVG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;avg_trip_distance&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;FROM&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;yellow_tripdata_2016_01&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;        &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;SELECT&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;            &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AVG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;+&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;3&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;STDEV&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;max_fare&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;            &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AVG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;+&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;3&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;*&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;STDEV&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;max_distance&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;FROM&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;yellow_tripdata_2016_01&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AS&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;sub&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;WHERE&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;0&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AND&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;fare_amount&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;sub&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;max_fare&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AND&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;0&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;AND&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;    &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;trip_distance&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;n&#34;&gt;sub&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;max_distance&lt;/span&gt;
&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The result is returned again in less than &lt;code&gt;1s&lt;/code&gt;, which is on par with DuckDB &lt;code&gt;972 ms&lt;/code&gt; and beats SQLite &lt;code&gt;11.7s&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The performance result that SQL-on-Demand beats DuckDB is
irrelevant, because the two database solutions are completely different.&lt;/p&gt;
&lt;p&gt;What is really interesting is that while DuckDB provides the convenience of an
easily embeddable database on your local machine, Azure Synapse
SQL-on-Demand provides similar convenience for running T-SQL statements on Parquet
files as a managed, elastic service in the cloud. You can just click on your SQL
endpoint in Azure and start querying Parquet files without the need to run or manage
a server, and you only pay for the data scanned ($5 per TB). If you are doing
data engineering in Azure, this is a cool new tool in your tool belt.&lt;/p&gt;
&lt;p&gt;And it scales horizontally. While the above examples were done on
a tiny &lt;code&gt;1.7MB&lt;/code&gt; Parquet file, we have done some scaling tests at
&lt;a href=&#34;http://blueyonder.com&#34;&gt;blueyonder&lt;/a&gt; up to terabytes of data and thousands of
Parquet files. Stay tuned for a post with the results and an example of how to use &lt;a href=&#34;https://turbodbc.readthedocs.io&#34;&gt;turbodbc&lt;/a&gt; as a high-performance interface for data pipelines...&lt;/p&gt;
</content>
  </entry>
  <entry xml:base="https://peter-hoffmann.com/feed/atom-parquet.xml">
    <title type="text">Azure Data Explorer and Parquet Files in Azure Blob Storage</title>
    <id>https://peter-hoffmann.com/2020/azure-data-explorer-and-parquet-files-in-from-azure-blob-storage.html</id>
    <updated>2020-02-01T00:00:00Z</updated>
    <published>2020-02-01T00:00:00Z</published>
    <link href="/2020/azure-data-explorer-and-parquet-files-in-from-azure-blob-storage.html" />
    <author>
      <name>Peter Hoffmann</name>
      <uri>https://peter-hoffmann.com</uri>
    </author>
    <content type="html">&lt;h2&gt;Azure Data Explorer&lt;/h2&gt;
&lt;p&gt;With the heavy use of &lt;a href=&#34;https://tech.jda.com/introducing-kartothek/&#34;&gt;Apache Parquet datasets&lt;/a&gt; within my team at Blue Yonder, we are always looking for managed, scalable, elastic query engines on flat files, besides the usual suspects like Drill, Hive, Presto, or Impala.&lt;/p&gt;
&lt;p&gt;For the following tests, I deployed an &lt;a href=&#34;https://azure.microsoft.com/en-us/services/data-explorer/&#34;&gt;Azure Data Explorer cluster&lt;/a&gt; with two instances of Standard_D14_v2 servers, each with 16 vCores, 112 GiB RAM, 800 GiB SSD storage, and an Extremely High network bandwidth class (which corresponds to 8 NICs).&lt;/p&gt;
&lt;h2&gt;Data Preparation NY Taxi Dataset&lt;/h2&gt;
&lt;p&gt;Like in the &lt;a href=&#34;http://peter-hoffmann.com/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html&#34;&gt;understanding parquet predicate pushdown&lt;/a&gt; blog post, we are using the NY Taxi dataset for the tests because it has a reasonable size and useful properties, such as different data types, and it includes some messy data (like all real-world data engineering problems).&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;mkdir&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;csv
mkdir&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;parquet
&lt;span class=&#34;nb&#34;&gt;cd&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;csv
&lt;span class=&#34;k&#34;&gt;for&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;i&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;in&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;01&lt;/span&gt;..12&lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;  &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;do&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;  &lt;/span&gt;wget&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2018-&lt;span class=&#34;nv&#34;&gt;$i&lt;/span&gt;.csv
&lt;span class=&#34;k&#34;&gt;done&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;We can convert the CSV files to Parquet with &lt;a href=&#34;https://arrow.apache.org/docs/python/parquet.html&#34;&gt;pandas and pyarrow&lt;/a&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pandas&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pd&lt;/span&gt;
&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pyarrow&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pa&lt;/span&gt;
&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pyarrow.parquet&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pq&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;months&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;range&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;13&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;k&#34;&gt;def&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nf&#34;&gt;csv_to_parquet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;):&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;csv/yellow_tripdata_2018-&lt;/span&gt;&lt;span class=&#34;si&#34;&gt;{:02d}&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;.csv&amp;quot;&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;format&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;dtype&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;store_and_fwd_flag&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;bool&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;},&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;parse_dates&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_pickup_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_dropoff_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;index_col&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;infer_datetime_format&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;True&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;true_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;false_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;N&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
    &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;tpep_pickup_datetime&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dt&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;year&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;==&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;2018&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;tpep_pickup_datetime&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dt&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;==&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)]&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;parquet/yellow_tripdata_2018-&lt;/span&gt;&lt;span class=&#34;si&#34;&gt;{:02d}&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;.parquet&amp;quot;&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;format&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;to_parquet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;m&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;months&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;csv_to_parquet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;m&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Each CSV file is about 700 MiB; the Parquet files are about 180 MiB; each file has about 10 million rows.&lt;/p&gt;
&lt;h2&gt;Data Ingestion&lt;/h2&gt;
&lt;p&gt;Azure Data Explorer supports control and query commands to interact with the cluster. &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/management/&#34;&gt;Kusto control commands&lt;/a&gt; always start with a dot and are used to manage the service, query information about it, and explore, create, and alter tables. The primary query language is the &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/query/&#34;&gt;Kusto Query Language&lt;/a&gt;, but a subset of &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/api/tds/t-sql&#34;&gt;T-SQL&lt;/a&gt; is also supported.&lt;/p&gt;
&lt;p&gt;The table schema definition supports a number of &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/query/scalar-data-types/&#34;&gt;scalar data types&lt;/a&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;Type&lt;/th&gt;
  &lt;th&gt;Storage Type (internal name)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;bool&lt;/td&gt;
  &lt;td&gt;I8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;datetime&lt;/td&gt;
  &lt;td&gt;DateTime&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;guid&lt;/td&gt;
  &lt;td&gt;UniqueId&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;int&lt;/td&gt;
  &lt;td&gt;I32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;long&lt;/td&gt;
  &lt;td&gt;I64&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;real&lt;/td&gt;
  &lt;td&gt;R64&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;string&lt;/td&gt;
  &lt;td&gt;StringBuffer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;timespan&lt;/td&gt;
  &lt;td&gt;TimeSpan&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;decimal&lt;/td&gt;
  &lt;td&gt;Decimal&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;To create a table for the NY Taxi dataset, we can use the following control command with the table and column names and the corresponding data types:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.create table nytaxi(
    VendorID: int,
    tpep_pickup_datetime: datetime,
    tpep_dropoff_datetime: datetime,
    passenger_count: int,
    trip_distance: real,
    RatecodeID: int,
    store_and_fwd_flag: bool,
    PULocationID: int,
    DOLocationID: int,
    payment_type: int,
    fare_amount: real,
    extra: real,
    mta_tax: real,
    tip_amount: real,
    tolls_amount: real,
    improvement_surcharge: real,
    total_amount: real
)
&lt;/pre&gt;&lt;/div&gt;
&lt;h3&gt;Ingest data into the Azure Data Explorer&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;.ingest into table&lt;/code&gt; command can read data from Azure Blob Storage or Azure Data Lake Storage and import the data into the cluster. This means it ingests the data and stores it locally for better performance. Authentication is done with &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview&#34;&gt;Azure SAS tokens&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Importing one month of CSV data takes about 110 seconds. As a reference, parsing the same CSV file with &lt;code&gt;pandas.read_csv&lt;/code&gt; takes about 19 seconds.&lt;/p&gt;
&lt;p&gt;You should always use obfuscated strings (the &lt;code&gt;h&lt;/code&gt; prefix) to ensure that the SAS token is never recorded or logged:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.ingest into table nytaxi
h&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/csv/yellow_tripdata_2018-01.csv?sp=r&amp;amp;st=2020-02-01T09:20:07Z&amp;amp;se=2020-02-24T20:20:07Z&amp;amp;spr=https&amp;amp;sv=2019-02-02&amp;amp;sr=b&amp;amp;sig=XXX&amp;#39;
with (ignoreFirstRecord=true)
;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Ingesting Parquet data from Azure Blob Storage uses a similar command and determines the file format from the file extension. Besides CSV and Parquet, many more &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/management/data-ingestion/#supported-data-formats&#34;&gt;data formats&lt;/a&gt; like JSON, JSON Lines, ORC, and Avro are supported. According to the documentation, it is also possible to specify the format by appending &lt;code&gt;with (format=&amp;quot;parquet&amp;quot;)&lt;/code&gt;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.ingest into table nytaxi_parquet
h&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-01.parquet?sp=r&amp;amp;st=2020-02-01T09:17:43Z&amp;amp;se=2020-02-27T20:17:43Z&amp;amp;spr=https&amp;amp;sv=2019-02-02&amp;amp;sr=b&amp;amp;sig=xxx&amp;#39;
;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Loading the data from Parquet only took 30 s and already gives a nice speedup. You can also use multiple Parquet files in the blob store to load the data in one run, but I did not see a performance improvement (i.e., better than duration × number of files), which I interpret as no parallel import happening:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.ingest into table nytaxi_parquet(
h&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-01.parquet?sp=r&amp;amp;st=2020-02-01T09:17:43Z&amp;amp;se=2020-02-27T20:17:43Z&amp;amp;spr=https&amp;amp;sv=2019-02-02&amp;amp;sr=b&amp;amp;sig=xxx&amp;#39;,
h&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-02.parquet?sp=r&amp;amp;st=2020-02-01T09:17:43Z&amp;amp;se=2020-02-27T20:17:43Z&amp;amp;spr=https&amp;amp;sv=2019-02-02&amp;amp;sr=b&amp;amp;sig=xxx&amp;#39;,
h&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-03.parquet?sp=r&amp;amp;st=2020-02-01T09:17:43Z&amp;amp;se=2020-02-27T20:17:43Z&amp;amp;spr=https&amp;amp;sv=2019-02-02&amp;amp;sr=b&amp;amp;sig=xxx&amp;#39;
)
;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once the data is ingested, one can query it using Azure Data Explorer, either with the Kusto Query Language or T-SQL:&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/static/2020/azure-data-explorere-query.png&#34; alt=&#34;Kusto Query&#34; /&gt;&lt;/p&gt;
&lt;h3&gt;Query External Tables&lt;/h3&gt;
&lt;p&gt;Loading the data into the cluster gives the best performance, but often you just want to run an ad hoc query on Parquet data in Blob storage. Using &lt;a href=&#34;https://docs.microsoft.com/en-us/azure/kusto/management/externaltables&#34;&gt;external tables&lt;/a&gt; supports exactly this scenario. This time, using multiple files/partitioning helped speed up the query.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.create external table nytaxi_parquet_external(
VendorID: int,
tpep_pickup_datetime: datetime,
tpep_dropoff_datetime: datetime,
passenger_count: int,
trip_distance: real,
RatecodeID: int,
store_and_fwd_flag: bool,
PULocationID: int,
DOLocationID: int,
payment_type: int,
fare_amount: real,
extra: real,
mta_tax: real,
tip_amount: real,
tolls_amount: real,
improvement_surcharge: real,
total_amount: real)
kind=blob
dataformat=parquet (
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-01.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-02.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-03.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-04.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-05.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-06.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-07.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-08.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-09.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-10.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-11.parquet;xxx&amp;#39;,
h@&amp;#39;https://phoffmann.blob.core.windows.net/nytaxi/parquet/yellow_tripdata_2018-12.parquet;xxx&amp;#39;
)
with (docstring = &amp;quot;NyTaxiDataset&amp;quot;)
;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Querying external data looks similar, but has the benefit that you do not have to load the data into the cluster. In a follow-up post, I&#39;ll do some performance benchmarks. Based on my initial experiences, it seems that the query engine is aware of some Parquet properties, such as columnar storage and predicate pushdown, because queries return results faster than loading the full data from Blob storage (with the 30 MB/s limit) would take.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;external_table(&amp;quot;nytaxi_parquet_external&amp;quot;)
    | take 100
;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;img src=&#34;/static/2020/azure-data-explorere-external-table.png&#34; alt=&#34;external table&#34; /&gt;&lt;/p&gt;
&lt;h3&gt;Export Data&lt;/h3&gt;
&lt;p&gt;For the sake of completeness, I&#39;ll just show an example of how to export data from the cluster back to a Parquet file in Azure Blob Storage:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;.export
  to parquet (
    h@&amp;quot;https://phoffmann.blob.core.windows.net/nytaxi/export/test.parquet;xxx&amp;quot;
  )
  &amp;lt;| nytaxi |  limit 100
  ;
&lt;/pre&gt;&lt;/div&gt;
</content>
  </entry>
  <entry xml:base="https://peter-hoffmann.com/feed/atom-parquet.xml">
    <title type="text">Understanding Predicate Pushdown at the Row-Group Level in Parquet with PyArrow and Python</title>
    <id>https://peter-hoffmann.com/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html</id>
    <updated>2020-01-19T00:00:00Z</updated>
    <published>2020-01-19T00:00:00Z</published>
    <link href="/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html" />
    <author>
      <name>Peter Hoffmann</name>
      <uri>https://peter-hoffmann.com</uri>
    </author>
    <content type="html">&lt;h1&gt;Demo Dataset&lt;/h1&gt;
&lt;p&gt;We are using the &lt;a href=&#34;https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page&#34;&gt;NY Taxi
Dataset&lt;/a&gt;
throughout this blog post because it is a real-world dataset, has a
reasonable size, and some useful properties like different data types. It also
includes some messy data (like all real-world data engineering problems).&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;mkdir&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;input
&lt;span class=&#34;nb&#34;&gt;cd&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;input
&lt;span class=&#34;k&#34;&gt;for&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;i&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;in&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;o&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;01&lt;/span&gt;..12&lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;  &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;do&lt;/span&gt;
&lt;span class=&#34;w&#34;&gt;  &lt;/span&gt;wget&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2018-&lt;span class=&#34;nv&#34;&gt;$i&lt;/span&gt;.csv
&lt;span class=&#34;k&#34;&gt;done&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Looking at the first rows of the data gives us some insight into the columns and data format&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;head&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;-n4&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;input/yellow_tripdata_2019-01.csv
VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge
&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt;,2019-01-01&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;00&lt;/span&gt;:46:40,2019-01-01&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;00&lt;/span&gt;:53:20,1,1.50,1,N,151,239,1,7,0.5,0.5,1.65,0,0.3,9.95,
&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt;,2019-01-01&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;00&lt;/span&gt;:59:47,2019-01-01&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;01&lt;/span&gt;:18:59,1,2.60,1,N,239,246,1,14,0.5,0.5,1,0,0.3,16.3,
&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt;,2018-12-21&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;13&lt;/span&gt;:48:30,2018-12-21&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;13&lt;/span&gt;:52:40,3,.00,1,N,236,236,1,4.5,0.5,0.5,0,0,0.3,5.8,
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Each of the files is roughly 700MiB of uncompressed data:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;du&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;-sh&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;input/yellow_tripdata_2018-0*
737M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-01.csv
715M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-02.csv
794M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-03.csv
784M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-04.csv
777M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-05.csv
734M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-06.csv
661M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-07.csv
661M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-08.csv
678M&lt;span class=&#34;w&#34;&gt;	&lt;/span&gt;input/yellow_tripdata_2018-09.csv
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;To convert the data to Parquet, we are going to use
&lt;a href=&#34;https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html&#34;&gt;pandas&lt;/a&gt; to read the CSV and store it in one large Parquet file:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;glob&lt;/span&gt;
&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pandas&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pd&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;files&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;glob&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;glob&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;input/yellow_tripdata_2018-*.csv&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;k&#34;&gt;def&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nf&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;):&lt;/span&gt;
    &lt;span class=&#34;k&#34;&gt;return&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;dtype&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;store_and_fwd_flag&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;bool&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;},&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;parse_dates&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_pickup_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_dropoff_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;index_col&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;infer_datetime_format&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;True&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;true_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;false_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;N&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
    &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;dfs&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;list&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;map&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;files&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;))&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;concat&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dfs&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;to_parquet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2018.parquet&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The resulting Parquet file has a size of &lt;code&gt;2.2GiB&lt;/code&gt;, while the sum of the
original CSV files was &lt;code&gt;11GiB&lt;/code&gt;. Pandas supports two Parquet implementations,
&lt;a href=&#34;https://fastparquet.readthedocs.io/en/latest/&#34;&gt;fastparquet&lt;/a&gt; and
&lt;a href=&#34;https://arrow.apache.org/docs/python/parquet.html&#34;&gt;pyarrow&lt;/a&gt;. They both have
strengths and weaknesses. A comparison should be the topic of another blog post,
and we are going to use PyArrow to analyze the data.&lt;/p&gt;
&lt;p&gt;PyArrow can open a Parquet file without directly reading all the data. It
exposes metadata and only reads the necessary byte ranges of the file to
obtain this information. This is extremely helpful when you are working with
Parquet files that are not available locally and are stored in a remote location
(like Amazon S3 or Azure Blob Storage), because you are only reading a few kB
instead of gigabytes of data to understand your dataset.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pyarrow.parquet&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pq&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;yellow_tripdata_2018.parquet&amp;quot;&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;ParquetFile&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;data&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;[[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;columns:&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;num_columns&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;rows:&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;num_rows&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;row_groups:&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;num_row_groups&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;
        &lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;So we are working with roughly 103 million records, 18 columns, and the file
has 2 row groups.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;  columns:          18
     rows:   102804250
row_groups:          2
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The next step is to have a look at the schema of the Parquet file:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;s&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;schema&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;data&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;[[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;s&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;name&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;s&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;physical_type&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;  &lt;span class=&#34;n&#34;&gt;s&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;logical_type&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;i&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;range&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;len&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;s&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;))]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;Column&lt;/th&gt;
  &lt;th&gt;physical&lt;/th&gt;
  &lt;th&gt;logical&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;VendorID&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;tpep_pickup_datetime&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;TIMESTAMP_MILLIS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;tpep_dropoff_datetime&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;TIMESTAMP_MILLIS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;passenger_count&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;trip_distance&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;RatecodeID&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;store_and_fwd_flag&lt;/td&gt;
  &lt;td&gt;BOOLEAN&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;PULocationID&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;DOLocationID&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;payment_type&lt;/td&gt;
  &lt;td&gt;INT64&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;fare_amount&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;extra&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;mta_tax&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;tip_amount&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;tolls_amount&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;improvement_surcharge&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;total_amount&lt;/td&gt;
  &lt;td&gt;DOUBLE&lt;/td&gt;
  &lt;td&gt;NONE&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Each column has a physical type that defines how the column is stored on disk and an
optional logical type that is used to determine the actual data type. In the case of
&lt;code&gt;tpep_pickup_datetime&lt;/code&gt; and &lt;code&gt;tpep_dropoff_datetime&lt;/code&gt;, the values are stored as &lt;code&gt;INT64&lt;/code&gt;
types on disk but are represented as timestamps in Pandas.&lt;/p&gt;
&lt;p&gt;Logical types are used to extend the types that Parquet can store by
specifying how the primitive types should be interpreted. This keeps the
set of primitive types to a minimum and reuses Parquet&#39;s efficient encodings.
For example, strings are stored as byte arrays (binary) with a UTF8
annotation; the &lt;a href=&#34;https://github.com/apache/parquet-format/blob/master/LogicalTypes.md&#34;&gt;Parquet logical type definitions&lt;/a&gt; provide comprehensive documentation.&lt;/p&gt;
&lt;p&gt;Now let&#39;s dive a little deeper into the file. A Parquet file consists of one
or more &lt;a href=&#34;http://parquet.apache.org/documentation/latest/&#34;&gt;row groups&lt;/a&gt;, which
are a logical horizontal partitioning of the data into rows.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/static/2020/parquet-format.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;s = pq_file.metadata.schema
data = []
for rg in range(pq_file.metadata.num_row_groups):
    rg_meta = pq_file.metadata.row_group(rg)
    data.append([rg, rg_meta.num_rows, sizeof_fmt(rg_meta.total_byte_size)])
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;As we have written the Parquet file with the default values in Pandas, we get
row groups with a size on disk between 512MiB and 1.5GiB.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;row group&lt;/th&gt;
  &lt;th&gt;rows&lt;/th&gt;
  &lt;th&gt;size&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;0&lt;/td&gt;
  &lt;td&gt;67108864&lt;/td&gt;
  &lt;td&gt;1.4GiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;35695386&lt;/td&gt;
  &lt;td&gt;753.0MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;To understand the defaults of the row group sizing, some historical
context is helpful. The Parquet file format was developed as a columnar
data storage format of the Apache Hadoop ecosystem and its underlying
Hadoop Distributed File System (HDFS):&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Larger row groups allow for larger column chunks which makes it possible to
do larger sequential IO. Larger groups also require more buffering in the
write path (or a two pass write). We recommend large row groups (512MB -
1GB). Since an entire row group might need to be read, we want it to
completely fit on one HDFS block. Therefore, HDFS block sizes should also be
set to be larger. An optimized read setup would be: 1GB row groups, 1GB HDFS
block size, 1 HDFS block per HDFS file.
&lt;a href=&#34;https://parquet.apache.org/documentation/latest/&#34;&gt;#&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;When working with Parquet in Python, one typically does not use HDFS as a
storage backend, but either the local file system or a cloud blob storage
like Amazon S3 or Azure Blob Storage. Depending on the read scenarios, different
row group sizes make sense.&lt;/p&gt;
&lt;p&gt;Instead of concatenating the CSV files in Pandas and writing them in one batch,
one can use &lt;a href=&#34;https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html&#34;&gt;pyarrow.ParquetWriter&lt;/a&gt; directly to control how many row groups are written:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pandas&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pd&lt;/span&gt;

&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pyarrow&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pa&lt;/span&gt;
&lt;span class=&#34;kn&#34;&gt;import&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pyarrow.parquet&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;k&#34;&gt;as&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nn&#34;&gt;pq&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;months&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;range&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;13&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;k&#34;&gt;def&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;nf&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;):&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;input/yellow_tripdata_2018-&lt;/span&gt;&lt;span class=&#34;si&#34;&gt;{:02d}&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;.csv&amp;quot;&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;format&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;filename&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;dtype&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;store_and_fwd_flag&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;bool&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;},&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;parse_dates&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_pickup_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;tpep_dropoff_datetime&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;index_col&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;infer_datetime_format&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;True&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;true_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
        &lt;span class=&#34;n&#34;&gt;false_values&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;N&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt;
    &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;k&#34;&gt;return&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;tpep_pickup_datetime&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dt&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;year&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;==&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;2018&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;tpep_pickup_datetime&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dt&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;==&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;month&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)]&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;dfs&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;list&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;map&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;read_csv&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;months&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;))&lt;/span&gt;

&lt;span class=&#34;n&#34;&gt;table&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pa&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;Table&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;from_pandas&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;dfs&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;0&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;],&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;preserve_index&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;writer&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;ParquetWriter&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;yellow_tripdata_2018-rowgroups.parquet&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;table&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;schema&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;

&lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;df&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;dfs&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;table&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pa&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;Table&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;from_pandas&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;df&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;preserve_index&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;False&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;writer&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;write_table&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;table&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;writer&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;close&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;I have also added some data cleansing because, as mentioned earlier, the
taxi dataset includes some messy data, and we only want to have rows within
a monthly dataset with the correct &lt;code&gt;pickup_datetime&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;If we analyze the new Parquet file again, we can see that we now have
a row group for each month of data:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;row group&lt;/th&gt;
  &lt;th&gt;rows&lt;/th&gt;
  &lt;th&gt;total_byte_size&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;0&lt;/td&gt;
  &lt;td&gt;8492076&lt;/td&gt;
  &lt;td&gt;152.0MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;8173231&lt;/td&gt;
  &lt;td&gt;150.5MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;2&lt;/td&gt;
  &lt;td&gt;8040133&lt;/td&gt;
  &lt;td&gt;148.1MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;3&lt;/td&gt;
  &lt;td&gt;9430376&lt;/td&gt;
  &lt;td&gt;169.8MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;4&lt;/td&gt;
  &lt;td&gt;8821105&lt;/td&gt;
  &lt;td&gt;162.4MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;5&lt;/td&gt;
  &lt;td&gt;7849748&lt;/td&gt;
  &lt;td&gt;142.9MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;6&lt;/td&gt;
  &lt;td&gt;9305515&lt;/td&gt;
  &lt;td&gt;168.0MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;7&lt;/td&gt;
  &lt;td&gt;8145164&lt;/td&gt;
  &lt;td&gt;149.8MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;8&lt;/td&gt;
  &lt;td&gt;9224063&lt;/td&gt;
  &lt;td&gt;167.1MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;9&lt;/td&gt;
  &lt;td&gt;7849134&lt;/td&gt;
  &lt;td&gt;142.9MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;10&lt;/td&gt;
  &lt;td&gt;8713831&lt;/td&gt;
  &lt;td&gt;157.8MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;11&lt;/td&gt;
  &lt;td&gt;8759874&lt;/td&gt;
  &lt;td&gt;157.9MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;In addition to the data in the row groups, the Parquet format specifies metadata
that is written per row group:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;rg_meta&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;row_group&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;0&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;rg_meta&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;0&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Per column we can retrieve metadata such as compression, size, and data type, but also
statistical information about the values stored in the row group for the
particular column.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&amp;lt;pyarrow._parquet.ColumnChunkMetaData object at 0x7fa958ab72d0&amp;gt;
  file_offset: 43125536
  file_path:
  physical_type: INT64
  num_values: 8759557
  path_in_schema: tpep_pickup_datetime
  is_stats_set: True
  statistics:
    &amp;lt;pyarrow._parquet.Statistics object at 0x7fa958ab7510&amp;gt;
      has_min_max: True
      min: 2001-01-05 11:45:23
      max: 2018-01-31 23:59:57
      null_count: 0
      distinct_count: 0
      num_values: 8759557
      physical_type: INT64
      logical_type: Timestamp(isAdjustedToUTC=false, timeUnit=microseconds, is_from_converted_type=false, force_set_converted_type=false)
      converted_type (legacy): NONE
  compression: SNAPPY
  encodings: (&amp;#39;PLAIN_DICTIONARY&amp;#39;, &amp;#39;PLAIN&amp;#39;, &amp;#39;RLE&amp;#39;, &amp;#39;PLAIN&amp;#39;)
  has_dictionary_page: True
  dictionary_page_offset: 1312236
  data_page_offset: 2117164
  total_compressed_size: 41813300
  total_uncompressed_size: 68701768
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Looking at the &lt;code&gt;min&lt;/code&gt; and &lt;code&gt;max&lt;/code&gt; statistics of the &lt;code&gt;tpep_pickup_datetime&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;c1&#34;&gt;# tpep_pickup_datetime&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;data&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;[[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;quot;row group&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;min&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;quot;max&amp;quot;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]]&lt;/span&gt;
&lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;rg&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;range&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;num_row_groups&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;):&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;rg_meta&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pq_file&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;metadata&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;row_group&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;rg&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
    &lt;span class=&#34;n&#34;&gt;data&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;append&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;([&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;rg&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;str&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;rg_meta&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;statistics&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;),&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;str&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;rg_meta&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;column&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;statistics&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;max&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)])&lt;/span&gt;
&lt;span class=&#34;n&#34;&gt;print_table&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;data&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The statistics show an interesting property. The values per row group are
disjoint. This means without reading the full data you can know which values
to expect in which row group.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;row group&lt;/th&gt;
  &lt;th&gt;min&lt;/th&gt;
  &lt;th&gt;max&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;0&lt;/td&gt;
  &lt;td&gt;2018-01-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-01-31 23:59:57&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;2018-02-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-02-28 23:59:58&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;2&lt;/td&gt;
  &lt;td&gt;2018-03-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-03-31 23:59:57&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;3&lt;/td&gt;
  &lt;td&gt;2018-04-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-04-30 23:59:58&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;4&lt;/td&gt;
  &lt;td&gt;2018-05-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-05-31 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;5&lt;/td&gt;
  &lt;td&gt;2018-06-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-06-30 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;6&lt;/td&gt;
  &lt;td&gt;2018-07-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-07-31 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;7&lt;/td&gt;
  &lt;td&gt;2018-08-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-08-31 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;8&lt;/td&gt;
  &lt;td&gt;2018-09-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-09-30 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;9&lt;/td&gt;
  &lt;td&gt;2018-10-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-10-31 23:59:58&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;10&lt;/td&gt;
  &lt;td&gt;2018-11-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-11-30 23:59:59&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;11&lt;/td&gt;
  &lt;td&gt;2018-12-01 00:00:00&lt;/td&gt;
  &lt;td&gt;2018-12-31 23:59:58&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;If columns are sorted and/or row groups have disjoint values in a dataset,
readers can take advantage of this through a feature called &lt;code&gt;predicate pushdown&lt;/code&gt;.
To get all taxi trips on a certain day &lt;code&gt;2018-02-20&lt;/code&gt;, the Parquet reader can now
look at the row group statistics, compare the predicate
&lt;code&gt;tpep_pickup_datetime.min &amp;lt;= 2018-02-20 and tpep_pickup_datetime.max &amp;gt;= 2018-02-20&lt;/code&gt; against it, and
only read the parts of the file that potentially include rows for the day.
In our case, one would only have to read &lt;code&gt;row group 1&lt;/code&gt; and by this &lt;code&gt;150MiB&lt;/code&gt;
instead of about &lt;code&gt;2.2GiB&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;In contrast, if we print the statistics for the column &lt;code&gt;trip_distance&lt;/code&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
  &lt;th&gt;row group&lt;/th&gt;
  &lt;th&gt;min&lt;/th&gt;
  &lt;th&gt;max&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
  &lt;td&gt;0&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;189483.84&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;1061.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;2&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;302.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;3&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;943.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;4&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;910.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;5&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;833.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;6&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;7655.76&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;7&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;5381.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;8&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;329.63&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;9&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;302.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;10&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;932.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
  &lt;td&gt;11&lt;/td&gt;
  &lt;td&gt;0.0&lt;/td&gt;
  &lt;td&gt;602.3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Even if readers are only interested in rows with a certain &lt;code&gt;trip_distance&lt;/code&gt;,
one would have to read the whole dataset most of the time. Only for distances
greater than 1000 could one skip some of the row groups.&lt;/p&gt;
&lt;h2&gt;Summary&lt;/h2&gt;
&lt;p&gt;Query engines on Parquet files like Hive, Presto, or Dremio provide predicate
pushdown out of the box to speed up query times and reduce I/O.&lt;/p&gt;
&lt;p&gt;In the Python ecosystem, &lt;a href=&#34;https://github.com/dask/fastparquet/blob/master/fastparquet/api.py#L296-L300&#34;&gt;fastparquet&lt;/a&gt; has support for predicate pushdown at
row-group level. PyArrow
has an &lt;a href=&#34;https://issues.apache.org/jira/browse/ARROW-1796&#34;&gt;open ticket&lt;/a&gt; for an efficient implementation in the Parquet C++ reader.&lt;/p&gt;
&lt;p&gt;Implementing predicate pushdown in Python on top of the exposed statistics is
straightforward. In my team we have done this within &lt;a href=&#34;https://github.com/JDASoftwareGroup/kartothek/blob/6c187ed145a6b8816e5044123ee54f2eedeb33d7/kartothek/serialization/_parquet.py#L91-L179&#34;&gt;kartothek&lt;/a&gt; to speed up reads from large datasets from Azure Blob Storage.&lt;/p&gt;
</content>
  </entry>
</feed>
