Last summer Microsoft has rebranded the Azure Kusto Query engine as Azure Data Explorer. While it does not support fully elastic scaling, it at least allows to scale up and out a cluster via an API or the Azure portal to adapt to different workloads. It also offers parquet support out of the box which made me spend some time to look into it.
#python #pydata #azure #parquet
Apache Parquet is a columnar file format to
work with gigabytes of data. Reading and writing parquet files is efficiently
exposed to python with pyarrow. Additional statistics allow clients to use
predicate pushdown to only read subsets of data to reduce I/O.
Organizing data by column allows for better
compression, as data is more homogeneous. Better compression also reduces the
bandwidth required to read the input.
#python #pydata #parquet #arrow #pandas
Using Infrastructure-as-Code principles with configuration through machine processable definition files in combination with the adoption of cloud computing provides faster feedback cycles in development/testing and less risk in deployment to production.
This talk will give an overview on how to deploy web services on the Azure Cloud with different tools like Azure Resource Manager Templates, the Azure SDK for Python and the Azure module for Ansible and present best practices learned while moving a company into the Azure Cloud.
PyScaffold helps you to easily setup a new Python project.
Apache Spark is a computational engine for large-scale data processing.
PySpark exposes the Spark programming model to Python. It defines an API
for Resilient Distributed Datasets (RDDs) and the DataFrame API.
#python #pydata #spark #talk
cee_syslog_handler is an extension to the python syslog logging handler with support for structured json messages.
When your application grows beyond one machine you need a central space to
log, monitor and analyze what is going on. Logstash and elasticsearch store
your logs in a structured way. Kibana is a web fronted to search and
aggregate your logs.
#python #elasticsearch #talk
There is a resurgence of native-compiled programming languages going on.
Some of this work is in response to Python; we're now part of The
Establishment against which newcomers are measured. Greg Ward gives an overview
of Go, a recent native-compiled language, and how it relates to Python.
Python example to access the twitter API on your behalf, without going
through three legged autorization.