Exasol User Group Karlsruhe

Posted on April 19, 2019 in exasol azure pydata talk

Exasol on Microsoft Azure – automatic deployment in less than 30 minutes

On April 9th the Exasol User Group was hosted by Blue Yonder in Karlsruhe. Exasol meetups are events to share technical ideas and challenges in person with other users and experts from the Exasol community.

Besides the usual networking, we had two talks. Uwe Hesse gave a deep dive into automatic index rebuilds in Exasol, and I talked about Exasol deployment on Microsoft Azure and how we invested in automation to set up new clusters in less than 30 minutes.

The Exasol database has its roots in an on-premises world, and we were among the first customers running the database clusters on Microsoft Azure. Over time, we worked together to automate more and more of the provisioning and deployment.

We now have parameterized Azure Resource Manager templates to provision a cluster with a license server, multiple virtual machines, OS and data disks, and networking. So setting up a cluster is as easy as running an az command:

az group deployment create -g exasol --mode Complete --template-file exasol.json --parameters @exasol.parameters.json

All the configuration is now done in an exasol.parameters.json file, where we can specify the node and storage sizing:

{
  "nodeVmSize": {
  "value": "Standard_E64v3"
},
"numExasolNodes": {
  "value": 3
},
"nodeNdisks": {
  "value": 5
},
"nodeDiskSizeGB": {
  "value": 1024
},
"vnetResourceGroup": {
  "value": "database-network"
},
"vnetName": {
  "value": "database-vnet"
},
"subnetName": {
  "value": "exasol-cluster1"
},
"networkAddressPrefix": {
  "value": "10.3.141."
}
}

The initial provisioning creates one database per cluster. Often, we have more complex requirements. To configure logging, additional storage volumes, and databases, Exasol provides an XML-RPC API. We wrapped the API in an Ansible module so that we can set up databases in a declarative way.

Over time, the duration to provision a new database decreased from multiple hours (many manual steps) to a fully automated 20-minute setup. This not only frees our engineers from tedious work, but also helped us build a much better testing infrastructure, where we now provision multiple new production-like environments per day and have much better end-to-end test coverage.

We plan to open-source the Ansible module in our JDA GitHub account. Until then, if you want to get a sneak peek, just ping me via email.