Tag Archive for: infrastructure

Benjamin Aunkofer

Why using Infrastructure as Code for developing Cloud-based Data Warehouse Systems?

/in , , , , /by

In the contemporary age of Big Data, Data Warehouse Systems and Data Science Analytics Infrastructures have become an essential component for organizations to store, analyze, and make data-driven decisions. With the evolution of cloud computing, many organizations are now migrating their Data Warehouse Systems to the cloud for better scalability, flexibility, and cost-efficiency. Infrastructure as Code (IaC) can be a game-changer in this scenario. By automating the provisioning and management of cloud resources through code, IaC brings a host of advantages to the development and maintenance of Data Warehouse Systems in the cloud.

So why using IaC for Cloud Data Infrastructures?

Of course you – as a human user – can always login into the admin portal of any cloud provider and manually get your resources like SQL databases, ETL tools, Virtual Networks and tools like Synapse, snowflake, BigQuery or Databrikcs in place by clicking on the right buttons….. But here is why you should better follow the idea of having your code explaining which resources are in what order in place in your cloud:

Version Control for your Cloud Infrastructure

One of the primary advantages of using IaC is version control for your Data Warehouse – or Data Lakehouse – Architecture. Whether you’re using Redshift, Snowflake, or any other cloud-based data warehouse solutions, you can codify your architecture settings, allowing you to track changes over time. This ensures a reliable and consistent development environment and makes it easier to identify issues, rollback updates, or replicate the architecture for other projects.

Scalability Tailored for Data Needs

Data Warehouse Systems often require to scale quickly to handle larger datasets or more queries. Traditional manual scaling methods are cumbersome and slow. IaC allows for efficient auto-scaling based on real-time needs. You can write scripts to automatically provision or de-provision resources depending on your data workloads, making your data warehouse highly adaptive to your organization’s changing requirements.

Cost-Efficiency in Resource Allocation

Cloud resources are priced based on usage, so efficient allocation is crucial for managing costs. IaC enables precise control over cloud resources, allowing you to turn them off when not in use or allocate more resources during peak times. For Data Warehouse Systems that often require powerful (and expensive) computing resources, this level of control can translate into significant cost savings.

Streamlined Collaboration Among Teams

Data Warehouse Systems in the cloud often involve cross-functional teams — data engineers, data scientists, and system administrators. IaC allows these teams to collaborate more effectively. Everyone works with the same infrastructure configurations, reducing discrepancies between development, staging, and production environments. This ensures that the data models and queries developed by data professionals are consistent with the underlying infrastructure.

Enhanced Security and Compliance

Data Warehouses often store sensitive information, making security a paramount concern. IaC allows security configurations to be codified and automated, ensuring that every new resource or service deployed complies with organizational and regulatory guidelines. This proactive security approach is particularly beneficial for industries that have to adhere to strict compliance rules like HIPAA or GDPR.

Reliable Environment for Data Operations

Manual configurations are prone to human error, which can compromise the reliability of a Data Warehouse System. IaC mitigates this risk by automating repetitive tasks, ensuring that the infrastructure is consistently provisioned. This brings reliability to data ETL (Extract, Transform, Load) processes, query performances, and other critical data operations.

Documentation and Disaster Recovery Made Easy

Data is the lifeblood of any organization, and losing it can be catastrophic. IaC allows for swift disaster recovery by codifying the entire infrastructure. If a disaster occurs, the infrastructure can be quickly recreated, reducing downtime and data loss.

Most common IaC solutions

The most common tools for creating Cloud Infrastructure as Code are probably Terraform and Pulumi. However, IaC solutions can be very different in their concepts. For example: While Terraform is a pure declarative configuration language that just describes how the infrastructure will look like (execution then by the Terraform-supporting Cloud Provider), Pulumi on the other hand will execute the deployment by a programming language iteratively deploying the wished cloud resources (e.g. using for loops in Python). While executing Pulumi in any supported programming language like Python or C#, Pulumi generates declarative Infrastructure build plans for the Cloud. Any IaC solution is declaring how the infrastrcture looks like.

Terraform

Terraform is one of the most widely used Infrastructure as Code (IaC) tools, developed by HashiCorp. It enables users to define and provision a data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language (HCL).

The following Terraform script will create an Azure Resource Group, a SQL Server, and a SQL Database. It will also output the fully qualified domain name (FQDN) of the SQL Server, which you can use to connect to the database:

The HCL code needs to be placed into the Terrafirm main.tf file. Of course, Terraform and the Azure CLI needs to be installed before.

Pulumi

Pulumi is a modern Infrastructure as Code (IaC) tool that sets itself apart by allowing infrastructure to be defined using general-purpose programming languages like Python, TypeScript, Go, and C#.

Example of a Pulumi Python script creating a SQL Database on Microsoft Azure Cloud:

Running the script will need the installation of Python, Pulumi and the Azure CLI.

Cloud Provider specific IaC Solutions

Cloud providers might come up with their own IaC solutions, here are the probably most common ones:

Microsoft Azure Bicep is an open-source domain-specific language (DSL) developed by Microsoft, aimed at simplifying the process of deploying Azure resources. It serves as a declarative alternative to JSON for writing Azure Resource Manager (ARM) templates. Bicep compiles down to ARM templates, offering a more concise syntax and easier tooling while leveraging the proven, underlying ARM deployment engine.

AWS CloudFormation is a service offered by Amazon Web Services (AWS) that allows you to define cloud infrastructure in JSON or YAML templates.

Google Cloud Deployment Manager is quite similar to AWS CloudFormation but tailored for Google Cloud Platform (GCP), it allows you to define and deploy resources using YAML or Python templates.

IaC Tools for Server Configuration

There are many other IaC solutions and some of them are more focused on configuration of servers. In common they offer software provisioning as well and a lot detailing in regards to micro-configuration of single applications running on the server.

The most common IaC software for Server Configuration might be Ansible, a YAML-based configuration management tool that uses an agentless architecture. It’s easy to set up and widely used for automating tasks like software provisioning and configuration management. Puppet, Chef and SaltStack are further alternatives and master-agent architecture-based.

Other types of IaC Solutions

IaC solutions with a more narrow focus are e.g. Vagrant as a primarily used IaC tool for setting up virtual development environments, especially for the automation of VM (Virtual Machine) provisioning. The widely used Docker Compose is a tool for defining and running multi-container Docker applications, which can be defined using YAML files.

Furthermore we have tools that are working closely together with IaC tooling, e.g. Prometheus as an open-source monitoring toolkit often used in conjunction with other IaC tools for monitoring deployed resources.

Conclusion

Infrastructure as Code significantly enhances the development and maintenance of Cloud-based Data Infrastructures. From versioning your warehouse architecture and scaling resources according to real-time data needs, to facilitating team collaboration and ensuring security compliance, IaC serves as a foundational technology that brings agility, reliability, and cost-efficiency. As organizations continue to realize the importance of data-driven decision-making, leveraging IaC for cloud-based Data Warehouse Systems will likely become a best practice in data engineering and infrastructure management.

Niklas Lang

Kubernetes – der Steuermann für dein Big Data Projekt!

/in , , , , , , /by

Kubernetes ist ein Container-Orchestrierungssystem. Damit lassen sich also Anwendungen auf verschiedene Container aufteilen, wodurch sie effizient und ausfallsicher ausgeführt werden können. Kubernetes ist ein Open-Source-Projekt und wurde erstmals im Jahr 2014 veröffentlicht. Es ist sehr leistungsfähig und kann verteilte Systeme, die über Tausende von Rechnern verstreut sind, verwalten.

In diesem und in vielen anderen Beiträgen zum Thema Kubernetes wird die Abkürzung k8s genutzt. Sie kommt daher, dass das Wort Kubernetes mit k beginnt, mit s endet und dazwischen 8 Buchstaben stehen. Bevor wir beginnen, noch eine kleine Anmerkung, woher der Name Kubernetes eigentlich stammt: Das griechische Wort „Kubernetes“ bedeutet Steuermann und beschreibt genau das, was Kubernetes macht, es steuert. Es steuert verschiedene sogenannte Container und koordiniert deren Ausführung.

Was sind Container und warum brauchen wir sie?

Eines der bestimmenden Merkmale von Big Data oder Machine Learning Projekte ist, dass ein einzelner Computer in vielen Fällen nicht ausreicht, um die gewaltigen Rechenlasten bewältigen zu können. Deshalb ist es notwendig, mehrere Computer zu verwenden, die sich die Arbeit teilen können. Zusätzlich können durch ein solches System auch Ausfälle von einzelnen Computern kompensiert werden, wodurch wiederum sichergestellt ist, dass die Anwendung durchgehend erreichbar ist. Wir bezeichnen eine solche Anordnung von Computern als Computing-Cluster oder verteiltes System für paralleles Rechnen.

Im Mittelpunkt des Open Source Projektes Docker stehen die sogenannten Container. Container sind alleinstehende Einheiten, die unabhängig voneinander ausgeführt werden und immer gleich ablaufen. Docker-Container können wir uns tatsächlich relativ praktisch wie einen Frachtcontainer vorstellen. Angenommen, in diesem Container arbeiten drei Menschen an einer bestimmten Aufgabe (Ich weiß, dass dies wahrscheinlich gegen jedes geltende Arbeitsschutzgesetz verstößt, aber es passt nun mal sehr gut in unser Beispiel).

In ihrem Container finden sie alle Ressourcen und Maschinen, die sie für ihre Aufgabe benötigen. Über eine bestimmte Lucke im Container bekommen sie die Rohstoffe geliefert, die sie benötigen, und über eine andere Lucke geben sie das fertige Produkt heraus. Unser Schiffscontainer kann dadurch ungestört und weitestgehend autark arbeiten. Den Menschen darin wird es nicht auffallen, ob sich das Schiff inklusive Container gerade im Hamburger Hafen, in Brasilien oder irgendwo bei ruhigem Seegang auf offenem Meer befindet. Solange sie kontinuierlich Rohstoffe geliefert bekommen, führen sie ihre Aufgabe aus, egal wo sie sind.

Kubernetes Containers - Foto von Ian Taylor auf Unsplash

Foto von Ian Taylor auf Unsplash

Genauso verhält es sich mit Docker Containern im Softwareumfeld. Es handelt sich dabei um genau definierte, abgeschlossene Applikationen, die auf verschiedenen Maschinen/Rechnern laufen können. Solange sie die festgelegten Inputs kontinuierlich erhalten, können sie auch kontinuierlich weiterarbeiten, unabhängig von ihrer Umgebung.

Was macht Kubernetes?

Wir nutzen Computing-Cluster, um rechenintensive Projekte, wie Machine Learning Modelle, auf mehreren Rechnern zuverlässig und effizient laufen lassen zu können. In Containern wiederum programmieren wir Unteraufgaben, die in sich abgeschlossen sein können und die immer gleich ablaufen, egal ob auf Rechner 1 oder Rechner 2. Das klingt doch eigentlich ausreichend, oder?

Verteilte Systeme bieten gegenüber Einzelrechnern neben Vorteilen auch zusätzliche Herausforderungen, beispielsweise bei der gemeinsamen Nutzung von Daten oder der Kommunikation zwischen den Rechnern innerhalb des Clusters. Kubernetes übernimmt die Arbeit die Container auf das Cluster zu verteilen und sorgt für den reibungslosen Ablauf des Programmes. Dadurch können wir uns auf das eigentliche Problem, also unseren konkreten Anwendungsfall, konzentrieren.

Kubernetes ist also wie der Kapitän, oder Steuermann, auf dem großen Containerschiff, der die einzelnen Container auf seinem Schiff richtig platziert und koordiniert.

Aufbau eines Kubernetes Clusters

Kubernetes wird normalerweise auf einem Cluster von Computern installiert. Jeder Computer in diesem Cluster wird als Node bezeichnet. Auf einem Computer bzw. Node wiederum laufen mehrere sogenannte Pods. Auf den Pods sind die schlussendlichen Container mit den kleineren Applikationen installiert und können in einem lokalen System kommunizieren.

Damit die Pods und die Container darin ohne Komplikationen laufen können, gibt es einige Hilfsfunktionen und -komponenten im Kubernetes Cluster, die dafür sorgen, dass alle Systeme reibungslos funktionieren:

Aufbau Kubernetes Cluster | Abbildung: Kubernetes

Aufbau Kubernetes Cluster | Abbildung: Kubernetes

  • Control Plane: Das ist der Rechner, welcher das komplette Cluster überwacht. Auf diesem laufen keine Pods für die Anwendung. Stattdessen werden den einzelnen Pods die Container zugewiesen, die auf ihnen laufen sollen.
  • Sched: Der Scheduler hält innerhalb des Clusters Ausschau nach neu erstellen Pods und teilt diese zu bestehenden Nodes zu.
  • ETCD: Ein Speicher für alle Informationen, die im Cluster anfallen und aufbewahrt werden müssen, bspw. Metadaten zur Konfiguration.
  • Cloud Controller Manager (CCM): Wenn ein Teil des Systems auf Cloud Ressourcen läuft, kommt diese Komponente zum Einsatz und übernimmt die Kommunikation und Koordination mit der Cloud.
  • Controller Manager (CM): Die wichtigste Komponente im Kubernetes Cluster überwacht das Cluster und sucht nach ausgefallenen Nodes, um dann die Container und Pods neu zu verteilen.
  • API: Diese Schnittstelle ermöglicht die Kommunikation zwischen den Nodes und dem Control Plane.

 

Die Nodes sind deutlich schlanker aufgebaut als das Control Plane und enthalten neben den Pods zwei wesentliche Komponenten zur Überwachung:

  • Kubelet: Es ist das Control Plane innerhalb eines Nodes und sorgt dafür, dass alle Pods einwandfrei laufen.
  • Kube-Proxy (k-proxy): Diese Komponente verteilt den eingehenden Node Traffic an die Pods, indem es das Netzwerk innerhalb des Nodes erstellt.

Fazit

Ein Netzwerk aus verschiedenen Computern wird als Cluster bezeichnet und wird genutzt, um große Rechenlasten auf mehrere Computer aufteilen und dadurch effizienter gestalten zu können. Die kleinste Einheit, in die man eine Applikation aufteilen kann, ist der Docker Container. Dieser beinhaltet eine Unteraufgabe des Programms, die autark, also unabhängig vom System, ausgeführt wird.

Da es in einem Computing-Cluster sehr viele dieser Container geben kann, übernimmt Kubernetes für uns das Management der Container, also unter anderem deren Kommunikation und Koordinierung. Das Kubernetes Cluster hat dazu verschiedene Komponenten die dafür sorgen, dass alle Container laufen und das System einwandfrei funktioniert.

Benjamin Aunkofer

Interview: Does Business Intelligence benefit from Cloud Data Warehousing?

/in , , , , , , , , /by

Interview with Ross Perez, Senior Director, Marketing EMEA at Snowflake

Read this article in German:
“Profitiert Business Intelligence vom Data Warehouse in der Cloud?”

Does Business Intelligence benefit from Cloud Data Warehousing?

Ross Perez is the Senior Director, Marketing EMEA at Snowflake. He leads the Snowflake marketing team in EMEA and is charged with starting the discussion about analytics, data, and cloud data warehousing across EMEA. Before Snowflake, Ross was a product marketer at Tableau Software where he founded the Iron Viz Championship, the world’s largest and longest running data visualization competition.

Data Science Blog: Ross, Business Intelligence (BI) is not really a new trend. In 2019/2020, making data available for the whole company should not be a big thing anymore. Would you agree?

BI is definitely an old trend, reporting has been around for 50 years. People are accustomed to seeing statistics and data for the company at large, and even their business units. However, using BI to deliver analytics to everyone in the organization and encouraging them to make decisions based on data for their specific area is relatively new. In a lot of the companies Snowflake works with, there is a huge new group of people who have recently received access to self-service BI and visualization tools like Tableau, Looker and Sigma, and they are just starting to find answers to their questions.

Data Science Blog: Up until today, BI was just about delivering dashboards for reporting to the business. The data warehouse (DWH) was something like the backend. Today we have increased demand for data transparency. How should companies deal with this demand?

Because more people in more departments are wanting access to data more frequently, the demand on backend systems like the data warehouse is skyrocketing. In many cases, companies have data warehouses that weren’t built to cope with this concurrent demand and that means that the experience is slow. End users have to wait a long time for their reports. That is where Snowflake comes in: since we can use the power of the cloud to spin up resources on demand, we can serve any number of concurrent users. Snowflake can also house unlimited amounts of data, of both structured and semi-structured formats.

Data Science Blog: Would you say the DWH is the key driver for becoming a data-driven organization? What else should be considered here?

Absolutely. Without having all of your data in a single, highly elastic, and flexible data warehouse, it can be a huge challenge to actually deliver insight to people in the organization.

Data Science Blog: So much for the theory, now let’s talk about specific use cases. In general, it matters a lot whether you are storing and analyzing e.g. financial data or machine data. What do we have to consider for both purposes?

Financial data and machine data do look very different, and often come in different formats. For instance, financial data is often in a standard relational format. Data like this needs to be able to be easily queried with standard SQL, something that many Hadoop and noSQL tools were unable to provide. Luckily, Snowflake is an ansi-standard SQL data warehouse so it can be used with this type of data quite seamlessly.

On the other hand, machine data is often semi-structured or even completely unstructured. This type of data is becoming significantly more common with the rise of IoT, but traditional data warehouses were very bad at dealing with it since they were optimized for relational data. Semi-structured data like JSON, Avro, XML, Orc and Parquet can be loaded into Snowflake for analysis quite seamlessly in its native format. This is important, because you don’t want to have to flatten the data to get any use from it.

Both types of data are important, and Snowflake is really the first data warehouse that can work with them both seamlessly.

Data Science Blog: Back to the common business use case: Creating sales or purchase reports for the business managers, based on data from ERP-systems such as Microsoft or SAP. Which architecture for the DWH could be the right one? How many and which database layers do you see as necessary?

The type of report largely does not matter, because in all cases you want a data warehouse that can support all of your data and serve all of your users. Ideally, you also want to be able to turn it off and on depending on demand. That means that you need a cloud-based architecture… and specifically Snowflake’s innovative architecture that separates storage and compute, making it possible to pay for exactly what you use.

Data Science Blog: Where would you implement the main part of the business logic for the report? In the DWH or in the reporting tool? Does it matter which reporting tool we choose?

The great thing is that you can choose either. Snowflake, as an ansi-Standard SQL data warehouse, can support a high degree of data modeling and business logic. But you can also utilize partners like Looker and Sigma who specialize in data modeling for BI. We think it’s best that the customer chooses what is right for them.

Data Science Blog: Snowflake enables organizations to store and manage their data in the cloud. Does it mean companies lose control over their storage and data management?

Customers have complete control over their data, and in fact Snowflake cannot see, alter or change any aspect of their data. The benefit of a cloud solution is that customers don’t have to manage the infrastructure or the tuning – they decide how they want to store and analyze their data and Snowflake takes care of the rest.

Data Science Blog: How big is the effort for smaller and medium sized companies to set up a DWH in the cloud? Does this have to be an expensive long-term project in every case?

The nice thing about Snowflake is that you can get started with a free trial in a few minutes. Now, moving from a traditional data warehouse to Snowflake can take some time, depending on the legacy technology that you are using. But Snowflake itself is quite easy to set up and very much compatible with historical tools making it relatively easy to move over.

Tag Archive for: infrastructure

Nothing Found

Sorry, no posts matched your criteria