What Is Data Lake Architecture?

The volume of information produced by everyone in the world is growing exponentially. To put it in perspective, it’s estimated that by 2023 the big data analytics market will reach $103 billion.

Finding probable solutions for storing big data is a challenge. It’s no easy task to hold enormous amounts of information, clean it and transform it into understandable subsets — it’s best to take one step at a time.

Some reasons why companies access their big data is to:

  • Improve their consumer experience
  • Draw conclusions and make data-driven decisions
  • Identify potential problems
  • Create innovative products

There are ways to help define big data. Combining its characteristics with storage management methods help experts make their clients’ information digestible and understandable. Cue data lakes, which are repositories for big data in its native form.

Think of an actual lake with multiple water sources around the perimeter flowing into it. Picture these as three types of data: structured, semi-structured and unstructured. All this information can remain in a data lake and be accessed in its raw form at any time, making it an attractive storage method.

Here’s how data lakes are created, some of their components and how to avoid common pitfalls.

Creating a Data Lake

One benefit of creating and implementing a data lake is that structuring becomes much more manageable.  Pulling necessary information from a lake allows analysts to compare and contrast data and communicate any connections between datasets to their client.

There are four steps to follow when setting up a data lake:

  1. Choosing a software solution: Microsoft, Amazon and Google are cloud vendors that allow developers to create data lakes without using servers.
  2. Identifying where data is sourced: Where is your information coming from? Once sources are identified, determine how your data will be cleaned or transformed.
  3. Defining process and automation: It’s vital to outline how information should be processed once the data lake ingests it. This creates consistency for businesses.
  4. Establishing retrieval governance: Choosing who has access to what types of information is crucial for companies with multiple locations and departments. It helps with overall organization. Data scientists, for this reason, primarily access data lakes.

The next step would be to determine the extract, transform and load (ETL) process. ETL creates visual interpretations of data to provide context to businesses. When information from a data lake is sent to a warehouse, it can be analyzed.

Components of a Data Lake

Here is what happens to information once a data lake is created:

  • Collection: Data comes in from various sources.
  • Ingestion: Data is processed using management software.
  • Blending: Data is combined from multiple sources.
  • Transformation: Data is analyzed and made sense of.
  • Publication: Data can be used to drive business decisions.

There are other aspects of a data lake to keep in mind. These are the critical components that help provide business solutions:

  • Security: Data lakes require security to protect information — they do not have built-in safety measures.
  • Governance: Determine who can check on the quality of data and perform measurements.
  • Metadata: This provides information about other data to improve understanding.
  • Stewardship: Choose one or more employees to take on the responsibility of managing data.
  • Monitoring: Employ other software to perform the ETL process.

Big data lends itself to incorporating multiple processes to make it usable for companies. The volume of information one company produces is massive — to manage it, experts need to consider these components and steps when building a data lake.

What to Avoid When Using Data Lakes

The last thing people want for their data lake is to see it turn into a swamp. When big data is processed incorrectly, its value decreases, making it useless to the business sourcing it.

The first step in avoiding a common pitfall is to consider the sustainability of the data lake. Planning processes are necessary to ensure it’s secure, and governing and regulating incoming information will allow for long-term use.

A lack of security causes another problem that can arise in data lakes. Safety measures must be implemented. Because enterprises will build data lakes for different purposes, it’s easy for information to become unorganized and vulnerable to hacking. With security, the likelihood of data breaches decreases, and the quality of data remains high.

The most important thing to remember about data lakes is the planning stage. Without proper preparation, they tend to be overwhelming due to their size and complexity. Taking the time and care to establish the processes ahead of time is vital.

Using Data Lake Architecture for Business

Data lakes store massive amounts of information to be used later on to create subsets, analyze metadata and more. Their advantages allow businesses to be flexible, save money and have access to raw information at all times.

Artikelserie: BI Tools im Vergleich – Qlik Sense

Dies ist ein Artikel der Artikel-Serie “BI Tools im Vergleich – Einführung und Motivation“, zu der auch die vorab sehr lesenswerten einführenden Worte und die Ausführungen zur Datenbasis gehören. Auf Grundlage derselben Daten wurde analog zu diesem Artikel hier auch ein Artikel über Microsoft Power BI und einen zu Tableau.

Übrigens gibt es auch Erweiterungen für Qlik Sense, die Process Mining ermöglichen. Eine dieser Erweiterungen ist die von MEHRWERK Process Mining.

Lizenzmodell

Neben Qlik Sense gibt es auch das lang bewährte Qlik View, dass auf der gleichen In-Memory-Kerntechnologie basiert. Qlik Sense wurde im Jahr 2014 vom schwedischen Softwareunternehmen Qlik Tech herausgebracht und bei Qlik Sense liegt auch der Fokus der Weiterentwicklung. Es handelt sich um Self-Service-BI und eine Plattform für Visual Data Analysis. Dabei gibt es die Möglichkeit einer On Premise Server Version (interne Cloud) oder auf die Server von Qlik zu setzen und somit gänzlich auf die Qlik Sense Cloud zu setzen, also die Qlik Sense Cloud als SaaS-Lösung. Dazu gibt es noch Qlik Sense Desktop, das für kleinere Projekte ausreichen kann und ganz ohne die Cloud auskommt, jedoch Ergebnisse bei Bedarf in die Cloud publishen kann. Ähnlich wie bei Tableau und anders als derzeitig bei Power BI, wird für das Editieren von Apps/Dashboards jedoch kein Qlik Sense Desktop benötigt, denn das Erstellen, Bearbeiten und Verwalten von Qlik Sense Reports darf komplett in der Cloud (vom Browser aus) stattfinden.

Der Kunde hat die Wahl zwischen den Lizenzmodellen von Qlik Sense Business (SaaS) und Qlik Sense Enterprise (SaaS oder On Premise). Die Enterprise Variante ist dann noch mal in Enterprise Professional, Enterprise Analyzer und Enterprise Analyzer Capacity eingeteilt, es stehen also insgesamt drei Lizenzen zur Auswahl. Der Preis für Qlik Sense Business beträgt monatlich derzeitig $30 pro Anwender. Das offizielle Preismodell sieht für Enterprise Professionell $70 für einen Benutzer pro Monat vor und für Enterprise Analyzer $40 pro Benutzer pro Monat. Zum Kennenlernen der Business Version gibt eine kostenlose 30-Tage-Testversion.

Die Version Qlik Sense Desktop ist in der Funktionalität an der SaaS Lösung Qlik Sense Enterprise angepasst und steht ihr in nichts Essenziellem nach. Die Desktop Version kann nur auf Windows-Computern ausgeführt werden und die Verwendung mehrerer Bildschirme oder Tablets wird nicht unterstützt. Außerdem werden Sicherheitsfunktionen nicht unterstützt und es gibt keine Funktion zum automatischen Speichern. Mehr zu den Unterschieden hier.

Community & Features von anderen Entwicklern

Wie relevant die Community für Visualisierungstools ist, wurde bereits in den vorherigen Blogartikeln zu Power BI und Tableau beschrieben. Auch Qlik besitzt eine offizielle Community Seite, in der u. a. Diskussionen, Blogs und Support angeboten werden. Auch hier finden sich zu den meisten Problemstellungen eine Menge Lösungsansätze. Zudem bietet Qlik auf den offiziellen Webseiten auch sehr viele Lernvideos an, mit denen sich Neulinge einarbeiten und fortgeschrittene Anwender auch noch einiges erfahren können.

Neben den zahlreichen Visualisierungen können auch weitere Diagramme hinzugefügt werden. Im Qlik Sense Desktop werden bei Arbeitsblatt im Reiter Benutzerdefinierte Objekte zwei Bundles mitgeliefert. Hier können auch Erweiterungen importiert werden. Ein bekanntes Bundle ist die Vizlib, welches hier unterschiedliche Packages zur Verfügung stellt. Diese Erweiterungen können einfach importiert werden, indem die heruntergeladenen Verzeichnisse in den Qlik Sense Extensions Ordner eingefügt werden. Wem auch die Erweiterungen nicht ausreichen, der kann sogenannte Widgets erstellen. Diese werden in HTML und CSS geschrieben, daher ist ein gewisses Grundverständnis vorausgesetzt. Diese Widgets können auf Qlik Sense Funktionalitäten zugreifen und diese per Klick ausführen. So kann bspw. ein Button zum Entfernen aller gesetzten Filter erstellt werden.

Erstellung von Filtern in Qlik Sense

Daten laden & transformieren

Flexibler als die meisten Vergleichstools ist Qlik in der Verknüpfung von Datenquellen. Es werden Hunderte von Datenquellen angeboten, durch die der Anwender Zugriff auf seine Daten erhalten kann. Die von Qlik entwickelte Associate Engine beschleunig die Verarbeitung von verknüpften Daten. Die Anbindung von Cloudanwendungen steht hier im Vordergrund, aber es werden natürlich auch klassische Datenbanken, Textfiles usw. angeboten.

Nachdem die Daten geladen sind, befindet sich im Dateneditor unter dem Reiter auto generated selection eine automatisch generierte Query für den Ladevorgang. Dieses „Datenladeskript“ kann angelegt, bearbeitet und ausgeführt werden. Im Reiter „Main“ befinden sich hier vordefinierte Variableneinstellungen, wie z. B. SET ThousandSep=’.’; wobei auch diese angepasst und erweitert werden können. Zudem gibt es die Möglichkeit, das Datenmodell mit allen Tabellenverbindungen anzeigen zu lassen. Die große Qlik-Community und die Tutorials ermöglicht es jedem Nutzer, die vielen Möglichkeiten mit Qlik Script zügig aus dem Internet zu erlernen.

Daten laden & transformieren: AdventureWorks2017Dataset

Im Reiter Datenmanager werden die empfohlenen Verknüpfungen angezeigt. Diese sind für Einsteiger sehr nützlich. Im Verlauf der Analysen musste jedoch nachjustiert werden. Wenn die ID-Spalten zum Verknüpfen z. B. unterschiedliche Bezeichnungen haben, tut sich der Algorithmus schon mal schwer.

Abbildung eines Datenmodels in Qlik Sense. Zusehen sind die Verbindungen zwischen den Tabellen der Datenbank “AdventureWorks2016”.

Eine vom Tool vordefinierte Detailansicht in Form einer Visualisierung (siehe Screenshot) ermöglicht einen schnellen und einfachen Qualitätscheck der gerade erst geladenen Daten. Hier können die Verbindungen angepasst und neue erstellt werden. Hier können erste Datentransformation durchgeführt werden, z. B. die Ersetzung von Daten oder NULL-Werten.

Datentransformationen mit einfachen Eingabemasken – Hier: Ersetzen von Werten in Tabellen-Spalten.

Zudem können Felder hinzugefügt, also berechnet werden (ähnlich wie in Power BI und Tableau als neues Measure). Z. B. können Textwerte mit dem Operator „&“ verbunden und somit z. B. Vor- und Nachname ganz intuitiv in eine Spalte zusammengefügt werden. Außerdem gibt es mathematische Operatoren für Berechnungen und ein SQL-artiges „like“, um Zeichenfolgen mit Mustern zu vergleichen. Auch an dieser Stelle können Formeln eingegeben werden. Die Formeln umfassen hier: String-, Datums-, numerische, Bedingungs-, mathematische, Verteilungsfunktionen usw. Zu beachten ist hier, dass die Daten neu geladen werden müssen, um die berechneten Spalten zu updaten. Der Umgang mit den Formeln aber erscheint mir einfacher als z. B. mit DAX in Power BI.

Daten visualisieren

Dank einer benutzerfreundlichen Oberfläche sind auch Analysen ohne großes Vorwissen und per Drag and Drop möglich. Individuelle Dashboards sind in wenigen Schritten möglich und erfordern keine besonderen Tricks oder Kniffe um gleich zum Erfolg zu kommen. Die Datenvisualisierung erfolgt in sogenannten Apps, in denen die Dashboards (Seiten in der App) liegen. Diese können von Qlik Sense Desktop nach Qlik Cloud hochgeladen werden und von dort aus mit anderen Usern geteilt werden.

Qlik Sense enthält von Hause aus eine große Anzahl an Visualisierungsmöglichkeiten. „Entdecken Sie neue Einblicke in ihre Daten“ heißt es bei der Funktion namens Einblicke (Insights), denn hier wird der Zugriff auf die Qlik Cognitive Engine gewährt. Dabei kann der Anwender eine Frage an den sogenannten Insight Advisor in natürlicher Sprache formulieren, woraus dann AI-gestützte Dashboard-Vorschläge generiert werden. Auch wenn diese Funktion noch nicht vollkommen ausgereift erscheint, ist dies sicherlich ein Schritt in die Business Intelligence der Zukunft.

Qlik Sense Insights – Einblicke gewinnen mit Stichworten in menschlicher Sprache. Funktioniert mal besser, mal schlechter. Die Titel der Diagramme sind (in Qlik Sense stets per default) die Formeln der Darstellung. Diese lassen sich leicht umbenennen.

Diese Diagrammvorschläge können einen guten ersten Eindruck über verschiedene Dimensionen und Kennzahlen geben und die Diagramme können direkt zu den Arbeitsblättern hinzugefügt werden. Es können auch Fragen gestellt werden, die Berechnungen zur Grundlage haben. So wird im folgenden Beispiel die Korrelation zwischen zwei Kennzahlen ermittelt.

Qlik Sense Insights – Korrelation erstellt mit Anweisung auf Englisch

Den ersten Auftritt hatte die Cognitive Engine im April 2018 und der Insight Advisor im Juni 2018. Über den Insight Adviser werden auch die empfohlenen Verknüpfungen im Datenmanager generiert, diese sollten jedoch vom Anwender (z. B. BI-Developer, Data Analyst oder Data Engineer) jedoch nochmal überblickt werden, da diese nicht unbedingt fehlerfrei abläuft. Gerade in vielen Geschäftsdaten verstecken sich viele “falsche Freunde” unter den ID-Spalten-Benennungen, die einen Zusammenhang herzustellen scheinen – aber es nicht immer tun.

Diagramme können ansonsten auf übliche Weise über eine Paletten ausgewählt werden, um sie dann mit Kennzahlen und Dimensionen zu befüllen. Die Charts können mit vordefinierten Optionen in den Kategorien Daten, Sortieren, Darstellung usw. bearbeitet werden. Unter Darstellung können ggf. verschiedene Designs ausgewählt werden und Beschriftungen, Titel etc. angepasst werden. Die Felder zur Auswahl der Kennzahlen und Dimensionen können nach Tabelle ausgewählt werden, sie sind ansonsten alle in einer Liste und können über eine Suchfunktion schnell gefunden werden, vorausgesetzt die genaue Bezeichnung ist bekannt. Diese Suchfunktion wird auch an anderen Stellen angewandt, immer dann, wenn Felder ausgewählt werden.

Es gibt außerdem die Option „Master-Elemente“, um wieder verwendbare Dimensionen oder Kennzahlen (Measures) zu erstellen.

Hier können Berechnungen für Kennzahlen und Dimensionen hinterlegt und in jedem Arbeitsblatt wiederverwendet werden. Dies gilt auch für Visualisierungen und die damit verbundenen Dateninputs und Einstellungen.

Mit Drag and Drop stößt der Anwender hier schon mal an seine Grenzen, aber dann helfen die Formeln von Qlik Sense Script weiter. Wenn bspw. das Diagramm namens KPI eine Kennzahl mit Filterung nach einer Dimension anzeigen soll, hilft die Formel: Sum({<DimensionName={‘Value’}>} MeasureName. Eine Qlik Sense Formelsammlung ist hier zu finden. Jede Kennzahl und Dimension kann als Formel eingegeben werden. Im Formel bearbeiten – Editor werden auch schon gebräuchliche Berechnungen wie Aggregierungsfunktionen (Sum, Avg, Max usw.) und Distinct, vorgegeben und können auf Knopfdruck und ohne Coding generiert werden, ähnliche wie ein Quick Measure in Power BI.

Fazit

Das Finanzmodell ist auf jede Unternehmensgröße ausgerichtet. Wenn die Datenbereinigung im Vorfeld stattgefunden hat, sind Visualisierungen in wenigen Schritten möglich. Es gibt dabei die Möglichkeit, die Daten in gewissem Rahmen zu transformieren. Für die gewünschte Darstellung der Kennzahlen ist die Verwendung von Qlik Sense Script oftmals erforderlich, jedoch kommen Anfänger auch lange ohne Coding aus. Insgesamt bewerte ich die Nutzerfreundlichkeit auf Grund der intuitiveren Bedienung subjektiv höher als bei Tableau oder Power BI.

Es können Erweiterungen und Widgets zur tiefgründigen Dashboard Erstellung und Analyse genutzt werden. Es gibt viele Drag and Drop Funktionen, um die Dashboards zusammen zu ziehen. Die Erstellung einfacher Berichte erfordert keinen Entwickler oder einen gut ausgebildeten Data Analyst, dennoch werden Unternehmen bei größeren Vorhaben auf Grund der Komplexität von Unternehmensprozessen, die in der Business Intelligence darzustellen versucht werden, nicht um geschultes Personal herum kommen, wofür es viele Angebote an Trainings auch von Qlik-Partnern gibt. Die Schnelligkeit der Datenverarbeitung liegt dank der Associative Engine im Vergleich zu den anderen beiden Tools vorne. AI-gestützte Vorschläge können bei der Dashboard-Erstellung zusätzliche Unterstützung leisten. Die Kombination beider Komponenten, Schnelligkeit und Ai-gestützte Vorschläge des Insight Advisors, grenzt das Qlik Sense Tool zwar nicht so sehr von den anderen Anbietern ab, wie Qlik gerne hätte…. Dennoch ist Qlik Sense auch heute noch ein Tool, dass für Ad-Hoc-Analytics wie Business Intelligence mit Standard Reporting in Erwägung gezogen werden sollte.

How Microsoft Azure Is Impacting Financial Companies

Microsoft Azure has taken a large chunk of the cloud marketplace, transforming companies with the speed and security of the cloud. Microsoft has over the years used Azure to cushion companies against risk, deal with fraud and differentiate their customer experience. 

With Microsoft Cloud App Security, customers experience 75% automatic threat elimination because of increased visibility and automated threat protection. With all these and more amazing benefits of using Azure, its market share is bound to increase even more over the coming years.

https://www.flickr.com/photos/91869083@N05/8493934839/in/photolist-dWzCUp-efhrzk-29k3oWh-9zALPj-9zALPh-9aXgpG-91z6Eo-6pABZ8-2htjpWP-Wrr2UG-aNxVLK-4z3omV-2kEyM6k-9GvMhf-Rf9aM7-4z7CQJ-aS8oqx-ekXUoo-9aU3wz-9aXjnw-aS8HTZ-LPgq61-2kjSEYf-2hamKDd-2h6JfeX-2h7gxoF-Fx6eAM-pQ6Ken-fbNckF-2iMRZSS-2hTUA6v-b8ayve-b8awer-dZwwJ7-2i3mmqV-e1dGQz-2dZwNg6-b8aoSH-b8arkc-6ztgDn-b8asCZ-efwZLM-b8atnM-b8attr-2kGQugq-2iowpX5-6zbcAC-dAQCVY-b8aoq8-517Jxq

Image Source

Financial companies have not been left behind by the Azure bandwagon. The financial industry is using Microsoft Azure to enhance its core functionsinvest money by making informed decisions, and minimize risk while maximizing returns. 

Azure facilitates these core functions by helping with the storage of huge amounts of data—  some dating back to decades ago—, data retrieval and data security. 

It also helps financial companies to keep up with regulatory compliance.

Microsoft Azure is not the only cloud services provider. But here’s why it is the most outstanding when it comes to helping financial companies achieve their business goals.

Azure Offers Hybrid and Multi-Cloud Computing for Financial Companies

The financial services industry is extremely dynamic. Organizations offering financial services have to constantly test the market and come up with new and innovative products and services. 

They are also often under pressure to extend their services across borders. Remember they have to do all of this while at the same time managing their existing customers, containing their risk, and dealing with fraud.

Financial regulations also keep changing. As financial companies increasingly embrace new technology for their services— including intelligent cloud computing— and they have to comply with industry regulations. They cannot afford to leave loopholes as they take on their journey with the cloud.

The financial services industry is highly competitive and keeps up with modernity. These companies have had to resort to the dynamic hybrid, multi-cloud computing, and public cloud strategies to keep up with the trend.

This is how a hybrid cloud model worksit enables existing on-premises applications to be extended through a connection to the public cloud. 

This allows financial companies to enjoy the speed, elasticity, and scale of the public cloud without necessarily having to remodel their entire applications. These organizations are afforded the flexibility of deciding what parts of their application remains in an existing data center and which one resides in the cloud.

Cloud computing with Azure allows financial organizations to operate more efficiently by providing end-to-end protection to information, allowing the digitization of financial services, and providing data security. 

Data security is particularly important to financial firms because they are often targeted by fraudsters and cyber threats. They, therefore, need to protect crucial information which they achieve by authenticating their data centers using Azure.

Here’s why financial companies cannot think of doing without Azure’s hybrid cloud computing even for just a day.

https://unsplash.com/collections/28744506/work?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

Photo by Windows on Unsplash

  • The ability to expand their geographic reach

Azure enables financial companies to establish data centers in new locations to meet globally growing demand. This allows them to open and explore new markets. They can then use Azure DevOps pipelines to maintain their data factories and keep everything consistent.

  • Consistent Infrastructure management

The hybrid cloud model promotes a consistent approach to infrastructure management across all locations, whether it is on-premises, public cloud, or the edge.

  • Increased Elasticity

Financial firms and banks utilizing Azure services can respond with great agility to transactional changes or changes in demand by provisioning or de-provisioning as the situation at hand demands. 

In cases where the organization requires high computation such as complex risk modeling, a hybrid strategy allows it to expand its capacity beyond its data center without overwhelming its servers.

  • Flexibility

A hybrid strategy allows financial organizations to choose cloud services that fall within their budget, match their needs, and suit their features.

  • Data security and enhanced regulatory compliance

Hybrid and multi-cloud strategies are a superb alternative for strictly on-premises strategies when one considers resiliency, data portability, and data security.

  • Reduces CapEx Expenses

Managing on-premises infrastructure is expensive. Financial companies utilizing Azure do not need to spend large amounts of money setting them up and managing them. 

With the increased elasticity of the hybrid system, financial organizations only pay for the resources they actually use, at a relatively lower cost.

Financial Organizations Have Access to an Analytics Platform

As we mentioned earlier, financial companies have the core function of making financial decisions in order to invest money and gain maximum returns at the least possible risk. 

Having been entrusted with their customers’ assets, the best way to ensure success in making profits is by using an analytics system.

Getting the form of analytics that helps with solving this investment problem is the kind of headache that does not go away by taking a tablet of ibuprofen and a glass of waterintegrating data is not an easy task. Besides, building a custom analytics solution from scratch is quite expensive.

Luckily for financial companies, Azure has a dedicated analytics platform for the financial services industry. It is custom-made just for these types of organizations. 

Their system is quite intuitive and easy to use. Companies not only get to save the resources they would have otherwise used to build a custom solution, but they get to learn about their investment risks and get instant results at cloud speed. 

They can mitigate against negatively impactful market occurrences and gain profits even when operating in adverse market conditions.

https://unsplash.com/collections/28744506/work?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

Image by Headway on Unsplash

Financial Companies Get Advanced Data Management

Good analytics goes hand-in-hand with a great data management system. Financial companies need to have good data, create an organized data warehouse, and have a secure data storage system.

In addition to storing your data, Microsoft Azure ensures your storage can be optimized to support advanced applications, for example, machine learning and forecasting. 

Azure even allows you to compress and store documents for long periods of time when you write the data to Microsoft Azure Blob Storage. These documents can be retrieved anytime when the need arises for auditors’, regulators’, and lawyers’ perusal. 

Conclusion

Microsoft has over time managed to gain the trust of many industries, the financial services industry inclusive. Using its cloud computing giant, Azure, it has empowered these companies to carry out their functions efficiently and at the lowest cost and risk possible.

Azure’s hybrid cloud computing strategy has made financial operations flexible, opened doors for financial companies to establish their services in multiple locations, and provided them with consistent infrastructure management, among many other benefits.

With their futuristic model and commitment to growth, it’s only prudent to assume that Microsoft Azure will continue carrying the mantle as the best cloud services provider in the financial services industry.

Zusatz-Studium „Data Science and Big Data“ an der TU Dortmund

Anzeige
Jetzt anmelden für das weiterbildendes Studium „Data Science and Big Data“ an der Technischen Universität Dortmund!

Im Februar 2022 startet das berufsbegleitenden weiterbildende Studium „Data Science and Big Data“ an der Technischen Universität Dortmund zum 6. Mal.
Renommierte Wissenschaftlerinnen und Wissenschaftlern vermitteln Ihnen die neuesten datenwissenschaftlichen Erkenntnisse und zeigen, wie dieses Wissen praxisnah im eigenen Big-Data Projekt umgesetzt werden kann. Von der Analyse über das Management bis zur zielgerichteten Darstellung der Ergebnisse lernen Sie dabei Methoden der Disziplinen Statistik, Informatik und Journalistik kennen.

Das weiterbildende Studium richtet sich an alle Personen, die über einen natur-  oder ingenieurwissenschaftlich/ statistische Studienhintergrund verfügen oder aufgrund ihrer mehrjährigen Berufserfahrung mit Fragestellungen zum Thema Datenanalyse vertraut sind.

Mögliche Berufsgruppen sind:

  • Data Analyst
  • Consultant/ Unternehmensberater
  • Business Analyst
  • Software-Entwickler

Das weiterbildende Studium umfasst 10 Veranstaltungstage über eine Dauer von 10 Monaten (Kursabschluss: November 2022). Die Kosten betragen 6.900 € (zahlbar in 3 Raten). Bewerbungsschluss ist der 29. November 2021. Weitere Informationen und Hinweise zur Anmeldung finden Sie unter: https://wb.zhb.tu-dortmund.de/zertifikatskurse/data-science-and-big-data/

Bewerbungsformular für Zusatzstudium an der TU Dortmund

Bewerbungsformular (Download)

 

Bei Fragen können Sie sich gerne an den zuständigen Bildungsreferenten Daniel Neubauer wenden: daniel.neubauer@tu-dortmund.de oder 0231/755-6632

Rethinking linear algebra part two: ellipsoids in data science

*This is the fourth article of my article series “Illustrative introductions on dimension reduction.”

1 Our expedition of eigenvectors still continues

This article is still going to be about eigenvectors and PCA, and this article still will not cover LDA (linear discriminant analysis). Hereby I would like you to have more organic links of the data science ideas with eigenvectors.

In the second article, we have covered the following points:

  • You can visualize linear transformations with matrices by calculating displacement vectors, and they usually look like vectors swirling.
  • Diagonalization is finding a direction in which the displacement vectors do not swirl, and that is equal to finding new axis/basis where you can describe its linear transformations more straightforwardly. But we have to consider diagonalizability of the matrices.
  • In linear dimension reduction such as PCA or LDA, we mainly use types of matrices called positive definite or positive semidefinite matrices.

In the last article we have seen the following points:

  • PCA is an algorithm of calculating orthogonal axes along which data “swell” the most.
  • PCA is equivalent to calculating a new orthonormal basis for the data where the covariance between components is zero.
  • You can reduced the dimension of the data in the new coordinate system by ignoring the axes corresponding to small eigenvalues.
  • Covariance matrices enable linear transformation of rotation and expansion and contraction of vectors.

I emphasized that the axes are more important than the surface of the high dimensional ellipsoids, but in this article let’s focus more on the surface of ellipsoids, or I would rather say general quadratic curves. After also seeing how to draw ellipsoids on data, you would see the following points about PCA or eigenvectors.

  • Covariance matrices are real symmetric matrices, and also they are positive semidefinite. That means you can always diagonalize covariance matrices, and their eigenvalues are all equal or greater than 0.
  • PCA is equivalent to finding axes of quadratic curves in which gradients are biggest. The values of quadratic curves increases the most in those directions, and that means the directions describe great deal of information of data distribution.
  • Intuitively dimension reduction by PCA is equal to fitting a high dimensional ellipsoid on data and cutting off the axes corresponding to small eigenvalues.

Even if you already understand PCA to some extent, I hope this article provides you with deeper insight into PCA, and at least after reading this article, I think you would be more or less able to visually control eigenvectors and ellipsoids with the Numpy and Maplotlib libraries.

*Let me first introduce some mathematical facts and how I denote them throughout this article in advance. If you are allergic to mathematics, take it easy or please go back to my former articles.

  • Any quadratic curves can be denoted as \boldsymbol{x}^T A\boldsymbol{x} + 2\boldsymbol{b}^T\boldsymbol{x} + s = 0, where \boldsymbol{x}\in \mathbb{R}^D , A \in \mathbb{R}^{D\times D} \boldsymbol{b}\in \mathbb{R}^D s\in \mathbb{R}.
  • When I want to clarify dimensions of variables of quadratic curves, I denote parameters as A_D, b_D.
  • If a matrix A is a real symmetric matrix, there exist a rotation matrix U such that U^T A U = \Lambda, where \Lambda = diag(\lambda_1, \dots, \lambda_D) and U = (\boldsymbol{u}_1, \dots , \boldsymbol{u}_D). \boldsymbol{u}_1, \dots , \boldsymbol{u}_D are eigenvectors corresponding to \lambda_1, \dots, \lambda_D respectively.
  • PCA corresponds to a case of diagonalizing A where A is a covariance matrix of certain data. When I want to clarify that A is a covariance matrix, I denote it as A=\Sigma.
  • Importantly covariance matrices \Sigma are positive semidefinite and real symmetric, which means you can always diagonalize \Sigma and any of their engenvalues cannot be lower than 0.

*In the last article, I denoted the covariance of data as S, based on Pattern Recognition and Machine Learning by C. M. Bishop.

*Sooner or later you are going to see that I am explaining basically the same ideas from different points of view, using the topic of PCA. However I believe they are all important when you learn linear algebra for data science of machine learning. Even you have not learnt linear algebra or if you have to teach linear algebra, I recommend you to first take a review on the idea of diagonalization, like the second article. And you should be conscious that, in the context of machine learning or data science, only a very limited type of matrices are important, which I have been explaining throughout this article.

2 Rotation or projection?

In this section I am going to talk about basic stuff found in most textbooks on linear algebra. In the last article, I mentioned that if A is a real symmetric matrix, you can diagonalize A with a rotation matrix U = (\boldsymbol{u}_1 \: \cdots \: \boldsymbol{u}_D), such that U^{-1}AU = U^{T}AU =\Lambda, where \Lambda = diag(\lambda_{1}, \dots , \lambda_{D}). I also explained that PCA is a case where A=\Sigma, that is, A is the covariance matrix of certain data. \Sigma is known to be positive semidefinite and real symmetric. Thus you can always diagonalize \Sigma and any of their engenvalues cannot be lower than 0.

I think we first need to clarify the difference of rotation and projection. In order to visualize the ideas, let’s consider a case of D=3. Assume that you have got an orthonormal rotation matrix U = (\boldsymbol{u}_1 \: \boldsymbol{u}_2 \: \boldsymbol{u}_3) which diagonalizes A. In the last article I said diagonalization is equivalent to finding new orthogonal axes formed by eigenvectors, and in the case of this section you got new orthonoramal basis (\boldsymbol{u}_1, \boldsymbol{u}_2, \boldsymbol{u}_3) which are in red in the figure below. Projecting a point \boldsymbol{x} = (x, y, z) on the new orthonormal basis is simple: you just have to multiply \boldsymbol{x} with U^T. Let U^T \boldsymbol{x} be (x', y', z')^T, and then \left( \begin{array}{c} x' \\ y' \\ z' \end{array} \right) = U^T\boldsymbol{x} = \left( \begin{array}{c} \boldsymbol{u}_1^{T}\boldsymbol{x} \\ \boldsymbol{u}_2^{T}\boldsymbol{x} \\ \boldsymbol{u}_3^{T}\boldsymbol{x} \end{array} \right). You can see x', y', z' are \boldsymbol{x} projected on \boldsymbol{u}_1, \boldsymbol{u}_2, \boldsymbol{u}_3 respectively, and the left side of the figure below shows the idea. When you replace the orginal orthonormal basis (\boldsymbol{e}_1, \boldsymbol{e}_2, \boldsymbol{e}_3) with (\boldsymbol{u}_1, \boldsymbol{u}_2, \boldsymbol{u}_3) as in the right side of the figure below, you can comprehend the projection as a rotation from (x, y, z) to (x', y', z') by a rotation matrix U^T.

Next, let’s see what rotation is. In case of rotation, you should imagine that you rotate the point \boldsymbol{x} in the same coordinate system, rather than projecting to other coordinate system. You can rotate \boldsymbol{x} by multiplying it with U. This rotation looks like the figure below.

In the initial position, the edges of the cube are aligned with the three orthogonal black axes (\boldsymbol{e}_1,  \boldsymbol{e}_2 , \boldsymbol{e}_3), with one corner of the cube located at the origin point of those axes. The purple dot denotes the corner of the cube directly opposite the origin corner. The cube is rotated in three dimensions, with the origin corner staying fixed in place. After the rotation with a pivot at the origin, the edges of the cube are now aligned with a new set of orthogonal axes (\boldsymbol{u}_1,  \boldsymbol{u}_2 , \boldsymbol{u}_3), shown in red. You might understand that more clearly with an equation: U\boldsymbol{x} = (\boldsymbol{u}_1 \: \boldsymbol{u}_2 \: \boldsymbol{u}_3) \left( \begin{array}{c} x \\ y \\ z \end{array} \right) = x\boldsymbol{u}_1 + y\boldsymbol{u}_2 + z\boldsymbol{u}_3. In short this rotation means you keep relative position of \boldsymbol{x}, I mean its coordinates (x, y, z), in the new orthonormal basis. In this article, let me call this a “cube rotation.”

The discussion above can be generalized to spaces with dimensions higher than 3. When U \in \mathbb{R}^{D \times D} is an orthonormal matrix and a vector \boldsymbol{x} \in \mathbb{R}^D, you can project \boldsymbol{x} to \boldsymbol{x}' = U^T \boldsymbol{x}or rotate it to \boldsymbol{x}'' = U \boldsymbol{x}, where \boldsymbol{x}' = (x_{1}', \dots, x_{D}')^T and \boldsymbol{x}'' = (x_{1}'', \dots, x_{D}'')^T. In other words \boldsymbol{x} = U \boldsymbol{x}', which means you can rotate back \boldsymbol{x}' to the original point \boldsymbol{x} with the rotation matrix U.

I think you at least saw that rotation and projection are basically the same, and that is only a matter of how you look at the coordinate systems. But I would say the idea of projection is more important through out this article.

Let’s consider a function f(\boldsymbol{x}; A) = \boldsymbol{x}^T A \boldsymbol{x} = (\boldsymbol{x}, A \boldsymbol{x}), where A\in \mathbb{R}^{D\times D} is a real symmetric matrix. The distribution of f(\boldsymbol{x}; A) is quadratic curves whose center point covers the origin, and it is known that you can express this distribution in a much simpler way using eigenvectors. When you project this function on eigenvectors of A, that is when you substitute U \boldsymbol{x}' for \boldsymbol{x}, you get f = (\boldsymbol{x}, A \boldsymbol{x}) =(U \boldsymbol{x}', AU \boldsymbol{x}') = (\boldsymbol{x}')^T U^TAU \boldsymbol{x}' = (\boldsymbol{x}')^T \Lambda \boldsymbol{x}' = \lambda_1 ({x'}_1)^2 + \cdots + \lambda_D ({x'}_D)^2. You can always diagonalize real symmetric matrices, so the formula implies that the shapes of quadratic curves largely depend on eigenvectors. We are going to see this in detail in the next section.

*(\boldsymbol{x}, \boldsymbol{y}) denotes an inner product of \boldsymbol{x} and \boldsymbol{y}.

*We are going to see details of the shapes of quadratic “curves” or “functions” in the next section.

To be exact, you cannot naively multiply U or U^T for rotation. Let’s take a part of data I showed in the last article as an example. In the figure below, I projected data on the basis (\boldsymbol{u}_1,  \boldsymbol{u}_2 , \boldsymbol{u}_3).

You might have noticed that you cannot do a “cube rotation” in this case. If you make the coordinate system (\boldsymbol{u}_1, \boldsymbol{u}_2, \boldsymbol{u}_3) with your left hand, like you might have done in science classes in school to learn Fleming’s rule, you would soon realize that the coordinate systems in the figure above do not match. You need to flip the direction of one axis to match them.

Mathematically, you have to consider the determinant of the rotation matrix U. You can do a “cube rotation” when det(U)=1, and in the case above det(U) was -1, and you needed to flip one axis to make the determinant 1. In the example in the figure below, you can match the basis. This also can be generalized to higher dimensions, but that is also beyond the scope of this article series. If you are really interested, you should prepare some coffee and snacks and textbooks on linear algebra, and some weekends.

When you want to make general ellipsoids in a 3d space on Matplotlib, you can take advantage of rotation matrices. You first make a simple ellipsoid symmetric about xyz axis using polar coordinates, and you can rotate the whole ellipsoid with rotation matrices. I made some simple modules for drawing ellipsoid. If you put in a rotation matrix which diagonalize the covariance matrix of data and a list of three radiuses \sqrt{\lambda_1}, \sqrt{\lambda_2}, \sqrt{\lambda_3}, you can rotate the original ellipsoid so that it fits the data well.

3 Types of quadratic curves.

*This article might look like a mathematical writing, but I would say this is more about computer science. Please tolerate some inaccuracy in terms of mathematics. I gave priority to visualizing necessary mathematical ideas in my article series. If you are not sure about details, please let me know.

In linear dimension reduction, or at least in this article series you mainly have to consider ellipsoids. However ellipsoids are just one type of quadratic curves. In the last article, I mentioned that when the center of a D dimensional ellipsoid is the origin point of a normal coordinate system, the formula of the surface of the ellipsoid is as follows: (\boldsymbol{x}, A\boldsymbol{x})=1, where A satisfies certain conditions. To be concrete, when (\boldsymbol{x}, A\boldsymbol{x})=1 is the surface of a ellipsoid, A has to be diagonalizable and positive definite.

*Real symmetric matrices are diagonalizable, and positive definite matrices have only positive eigenvalues. Covariance matrices \Sigma, whose displacement vectors I visualized in the last two articles, are known to be symmetric real matrices and positive semi-defintie. However, the surface of an ellipsoid which fit the data is \boldsymbol{x}^T \Sigma ^{-1} \boldsymbol{x} = const., not \boldsymbol{x}^T \Sigma \boldsymbol{x} = const..

*You have to keep it in mind that \boldsymbol{x} are all deviations.

*You do not have to think too much about what the “semi” of the term “positive semi-definite” means fow now.

As you could imagine, this is just one simple case of richer variety of graphs. Let’s consider a 3-dimensional space. Any quadratic curves in this space can be denoted as ax^2 + by^2 + cz^2 + dxy + eyz + fxz + px + qy + rz + s = 0, where at least one of a, b, c, d, e, f, p, q, r, s is not 0.  Let \boldsymbol{x} be (x, y, z)^T, then the quadratic curves can be simply denoted with a 3\times 3 matrix A and a 3-dimensional vector \boldsymbol{b} as follows: \boldsymbol{x}^T A\boldsymbol{x} + 2\boldsymbol{b}^T\boldsymbol{x} + s = 0, where A = \left( \begin{array}{ccc} a & \frac{d}{2} & \frac{f}{2} \\ \frac{d}{2} & b & \frac{e}{2} \\ \frac{f}{2} & \frac{e}{2} & c \end{array} \right), \boldsymbol{b} = \left( \begin{array}{c} \frac{p}{2} \\ \frac{q}{2} \\ \frac{r}{2} \end{array} \right). General quadratic curves are roughly classified into the 9 types below.

You can shift these quadratic curves so that their center points come to the origin, without rotation, and the resulting curves are as follows. The curves can be all denoted as \boldsymbol{x}^T A\boldsymbol{x}.

As you can see, A is a real symmetric matrix. As I have mentioned repeatedly, when all the elements of a D \times D symmetric matrix A are real values and its eigen values are \lambda_{i} (i=1, \dots , D), there exist orthogonal/orthonormal matrices U such that U^{-1}AU = \Lambda, where \Lambda = diag(\lambda_{1}, \dots , \lambda_{D}). Hence, you can diagonalize the A = \left( \begin{array}{ccc} a & \frac{d}{2} & \frac{f}{2} \\ \frac{d}{2} & b & \frac{e}{2} \\ \frac{f}{2} & \frac{e}{2} & c \end{array} \right) with an orthogonal matrix U. Let U be an orthogonal matrix such that U^T A U = \left( \begin{array}{ccc} \alpha  & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{array} \right) =\left( \begin{array}{ccc} \lambda_1  & 0 & 0 \\ 0 & \lambda_2 & 0 \\ 0 & 0 & \lambda_3 \end{array} \right). After you apply rotation by U to the curves (a)” ~ (i)”, those curves are symmetrically placed about the xyz axes, and their center points still cross the origin. The resulting curves look like below. Or rather I should say you projected (a)’ ~ (i)’ on their eigenvectors.

In this article mainly (a)” , (g)”, (h)”, and (i)” are important. General equations for the curves is as follows

  • (a)”: \frac{x^2}{l^2} + \frac{y^2}{m^2} + \frac{z^2}{n^2} = 1
  • (g)”: z = \frac{x^2}{l^2} + \frac{y^2}{m^2}
  • (h)”: z = \frac{x^2}{l^2} - \frac{y^2}{m^2}
  • (i)”: z = \frac{x^2}{l^2}

, where l, m, n \in \mathbb{R}^+.

Even if this section has been puzzling to you, you just have to keep one point in your mind: we have been discussing general quadratic curves, but in PCA, you only need to consider a case where A is a covariance matrix, that is A=\Sigma. PCA corresponds to the case where you shift and rotate the curve (a) into (a)”. Subtracting the mean of data from each point of data corresponds to shifting quadratic curve (a) to (a)’. Calculating eigenvectors of A corresponds to calculating a rotation matrix U such that the curve (a)’ comes to (a)” after applying the rotation, or projecting curves on eigenvectors of \Sigma. Importantly we are only discussing the covariance of certain data, not the distribution of the data itself.

*Just in case you are interested in a little more mathematical sides: it is known that if you rotate all the points \boldsymbol{x} on the curve \boldsymbol{x}^T A\boldsymbol{x} + 2\boldsymbol{b}^T\boldsymbol{x} + s = 0 with the rotation matrix P, those points \boldsymbol{x} are mapped into a new quadratic curve \alpha x^2 + \beta y^2 + \gamma z^2 + \lambda x + \mu y + \nu z + \rho = 0. That means the rotation of the original quadratic curve with P (or rather rotating axes) enables getting rid of the terms xy, yz, zx. Also it is known that when \alpha ' \neq 0, with proper translations and rotations, the quadratic curve \alpha x^2 + \beta y^2 + \gamma z^2 + \lambda x + \mu y + \nu z + \rho = 0 can be mapped into one of the types of quadratic curves in the figure below, depending on coefficients of the original quadratic curve. And the discussion so far can be generalized to higher dimensional spaces, but that is beyond the scope of this article series. Please consult decent textbooks on linear algebra around you for further details.

4 Eigenvectors are gradients and sometimes variances.

In the second section I explained that you can express quadratic functions f(\boldsymbol{x}; A) = \boldsymbol{x}^T A \boldsymbol{x} in a very simple way by projecting \boldsymbol{x} on eigenvectors of A.

You can comprehend what I have explained in another way: eigenvectors, to be exact eigenvectors of real symmetric matrices A, are gradients. And in case of PCA, I mean when A=\Sigma eigenvalues are also variances. Before explaining what that means, let me explain a little of the totally common facts on mathematics. If you have variables \boldsymbol{x}\in \mathbb{R}^D, I think you can comprehend functions f(\boldysmbol{x}) in two ways. One is a normal “functions” f(\boldsymbol{x}), and the others are “curves” f(\boldsymbol{x}) = const.. “Functions” get an input \boldsymbol{x} and gives out an output f(\boldsymbol{x}), just as well as normal functions you would imagine. “Curves” are rather sets of \boldsymbol{x} \in \mathbb{R}^D such that f(\boldsymbol{x}) = const..

*Please assume that the terms “functions” and “curves” are my original words. I use them just in case I fail to use functions and curves properly.

The quadratic curves in the figure above are all “curves” in my term, which can be denoted as f(\boldsymbol{x}; A_3, \boldsymbol{b}_3)=const or f(\boldsymbol{x}; A_3)=const. However if you replace z of (g)”, (h)”, and (i)” with f, you can interpret the “curves” as “functions” which are denoted as f(\boldsymbol{x}; A_2). This might sounds too obvious to you, and my point is you can visualize how values of “functions” change only when the inputs are 2 dimensional.

When a symmetric 2\times 2 real matrices A_2 have two eigenvalues \lambda_1, \lambda_2, the distribution of quadratic curves can be roughly classified to the following three types.

  • (g): Both \lambda_1 and \lambda_2 are positive or negative.
  • (h): Either of \lambda_1 or \lambda_2 is positive and the other is negative.
  • (i): Either of \lambda_1 or \lambda_2 is 0 and the other is not.

The equations of (g)” , (h)”, and (i)” correspond to each type of f=(\boldsymbol{x}; A_2), and thier curves look like the three graphs below.

And in fact, when start from the origin and go in the direction of an eigenvector \boldsymbol{u}_i, \lambda_i is the gradient of the direction. You can see that more clearly when you restrict the distribution of f=(\boldsymbol{x}; A_2) to a unit circle. Like in the figure below, in case \lambda_1 = 7, \lambda_2 = 3, which is classified to (g), the distribution looks like the left side, and if you restrict the distribution in the unit circle, the distribution looks like a bowl like the middle and the right side. When you move in the direction of \boldsymbol{u}_1, you can climb the bowl as as high as \lambda_1, in \boldsymbol{u}_2 as high as \lambda_2.

Also in case of (h), the same facts hold. But in this case, you can also descend the curve.

*You might have seen the curve above in the context of optimization with stochastic gradient descent. The origin of the curve above is a notorious saddle point, where gradients are all 0 in any directions but not a local maximum or minimum. Points can be stuck in this point during optimization.

Especially in case of PCA, A is a covariance matrix, thus A=\Sigma. Eigenvalues of \Sigma are all equal to or greater than 0. And it is known that in this case \lambda_i is the variance of data projected on its corresponding eigenvector \boldsymbol{u}_i (i=0, \dots , D). Hence, if you project f(\boldsymbol{x}; \Sigma), quadratic curves formed by a covariance matrix \Sigma, on eigenvectors of \Sigma, you get f(\boldsymbol{x}; \Sigma) = ({x'}_1 \: \dots \: {x'}_D) (\lambda_1 {x'}_1 \: \dots \: \lambda_D {x'}_D)^t =\lambda_1 ({x'}_1)^2 + \cdots + \lambda_D ({x'}_D)^2.  This shows that you can re-weight ({x'}_1 \: \dots \: {x'}_D), the coordinates of data projected projected on eigenvectors of A, with \lambda_1, \dots, \lambda_D, which are variances ({x'}_1 \: \dots \: {x'}_D). As I mentioned in an example of data of exam scores in the last article, the bigger a variance \lambda_i is, the more the feature described by \boldsymbol{u}_i vary from sample to sample. In other words, you can ignore eigenvectors corresponding to small eigenvalues.

That is a great hint why principal components corresponding to large eigenvectors contain much information of the data distribution. And you can also interpret PCA as a “climbing” a bowl of f(\boldsymbol{x}; A_D), as I have visualized in the case of (g) type curve in the figure above.

*But as I have repeatedly mentioned, ellipsoid which fit data well isf(\boldsymbol{x}; \Sigma ^{-1}) =(\boldsymbol{x}')^T diag(\frac{1}{\lambda_1}, \dots, \frac{1}{\lambda_D})\boldsymbol{x}' = \frac{({x'}_{1})^2}{\lambda_1} + \cdots + \frac{({x'}_{D})^2}{\lambda_D} = const..

*You have to be careful that even if you slice a type (h) curve f(\boldsymbol{x}; A_D) with a place z=const. the resulting cross section does not fit the original data well because the equation of the cross section is \lambda_1 ({x'}_1)^2 + \cdots + \lambda_D ({x'}_D)^2 = const. The figure below is an example of slicing the same f(\boldsymbol{x}; A_2) as the one above with z=1, and the resulting cross section.

As we have seen, \lambda_i, the eigenvalues of the covariance matrix of data are variances or data when projected on it eigenvectors. At the same time, when you fit an ellipsoid on the data, \sqrt{\lambda_i} is the radius of the ellipsoid corresponding to \boldsymbol{u}_i. Thus ignoring data projected on eigenvectors corresponding to small eigenvalues is equivalent to cutting of the axes of the ellipsoid with small radiusses.

I have explained PCA in three different ways over three articles.

  • The second article: I focused on what kind of linear transformations convariance matrices \Sigma enable, by visualizing displacement vectors. And those vectors look like swirling and extending into directions of eigenvectors of \Sigma.
  • The third article: We directly found directions where certain data distribution “swell” the most, to find that data swell the most in directions of eigenvectors.
  • In this article, we have seen PCA corresponds to only one case of quadratic functions, where the matrix A is a covariance matrix. When you go in the directions of eigenvectors corresponding to big eigenvalues, the quadratic function increases the most. Also that means data samples have bigger variances when projected on the eigenvectors. Thus you can cut off eigenvectors corresponding to small eigenvectors because they retain little information about data, and that is equivalent to fitting an ellipsoid on data and cutting off axes with small radiuses.

*Let A be a covariance matrix, and you can diagonalize it with an orthogonal matrix U as follow: U^{T}AU = \Lambda, where \Lambda = diag(\lambda_1, \dots, \lambda_D). Thus A = U \Lambda U^{T}. U is a rotation, and multiplying a \boldsymbol{x} with \Lambda means you multiply each eigenvalue to each element of \boldsymbol{x}. At the end U^T enables the reverse rotation.

If you get data like the left side of the figure below, most explanation on PCA would just fit an oval on this data distribution. However after reading this articles series so far, you would have learned to see PCA from different viewpoints like at the right side of the figure below.

 

5 Ellipsoids in Gaussian distributions.

I have explained that if the covariance of a data distribution is \boldsymbol{\Sigma}, the ellipsoid which fits the distribution the best is \bigl((\boldsymbol{x} - \boldsymbol{\mu}), \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu})\bigr) = 1. You might have seen the part \bigl((\boldsymbol{x} - \boldsymbol{\mu}), \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu})\bigr) = (\boldsymbol{x} - \boldsymbol{\mu}) \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu}) somewhere else. It is the exponent of general Gaussian distributions: \mathcal{N}(\boldsymbol{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\boldsymbol{\Sigma}|} exp\{ -\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu}) \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu}) \}.  It is known that the eigenvalues of \Sigma ^{-1} are \frac{1}{\lambda_1}, \dots, \frac{1}{\lambda_D}, and eigenvectors corresponding to each eigenvalue are also \boldsymbol{u}_1, \dots, \boldsymbol{u}_D respectively. Hence just as well as what we have seen, if you project (\boldsymbol{x} - \boldsymbol{\mu}) on each eigenvector of \Sigma ^{-1}, we can convert the exponent of the Gaussian distribution.

Let -\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu}) \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu}) be \boldsymbol{y} and U ^{-1} \boldsymbol{y}= U^{T} \boldsymbol{y} be \boldsymbol{y}', where U=(\boldsymbol{u}_1 \: \dots \: \boldsymbol{u}_D). Just as we have seen, (\boldsymbol{x} - \boldsymbol{\mu}) \boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu}) =\boldsymbol{y}^T\Sigma^{-1} \boldsymbol{y} =(U\boldsymbol{y}')^T \Sigma^{-1} U\boldsymbol{y}' =((\boldsymbol{y}')^T U^T \Sigma^{-1} U\boldsymbol{y}' = (\boldsymbol{y}')^T diag(\frac{1}{\lambda_1}, \dots, \frac{1}{\lambda_D}) \boldsymbol{y}' = \frac{({y'}_{1})^2}{\lambda_1} + \cdots + \frac{({y'}_{D})^2}{\lambda_D}. Hence \mathcal{N}(\boldsymbol{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\boldsymbol{\Sigma}|} exp\{ -\frac{1}{2}(\boldsymbol{y}) \boldsymbol{\Sigma}^{-1}(\boldsymbol{y}) \} =  \frac{1}{(2\pi)^{D/2}} \frac{1}{|\boldsymbol{\Sigma}|} exp\{ -\frac{1}{2}(\frac{({y'}_{1})^2}{\lambda_1} + \cdots + \frac{({y'}_{D})^2}{\lambda_D} ) \} =\frac{1}{(2\pi)^{1/2}} \frac{1}{|\boldsymbol{\Sigma}|} exp\biggl( -\frac{1}{2} \frac{({y'}_{1})^2}{\lambda_1} \biggl) \cdots \frac{1}{(2\pi)^{1/2}} \frac{1}{|\boldsymbol{\Sigma}|} exp\biggl( -\frac{1}{2}\frac{({y'}_{D})^2}{\lambda_D} \biggl).

*To be mathematically exact about changing variants of normal distributions, you have to consider for example Jacobian matrices.

This results above demonstrate that, by projecting data on the eigenvectors of its covariance matrix, you can factorize the original multi-dimensional Gaussian distribution into a product of Gaussian distributions which are irrelevant to each other. However, at the same time, that is the potential limit of approximating data with PCA. This idea is going to be more important when you think about more probabilistic ways to handle PCA, which is more robust to lack of data.

I have explained PCA over 3 articles from various viewpoints. If you have been patient enough to read my article series, I think you have gained some deeper insight into not only PCA, but also linear algebra, and that should be helpful when you learn or teach data science. I hope my codes also help you. In fact these are not the only topics about PCA. There are a lot of important PCA-like algorithms.

In fact our expedition of ellipsoids, or PCA still continues, just as Star Wars series still continues. Especially if I have to explain an algorithm named probabilistic PCA, I need to explain the “Bayesian world” of machine learning. Most machine learning algorithms covered by major introductory textbooks tend to be too deterministic and dependent on the size of data. Many of those algorithms have another “parallel world,” where you can handle inaccuracy in better ways. I hope I can also write about them, and I might prepare another trilogy for such PCA. But I will not disappoint you, like “The Phantom Menace.”

Appendix: making a model of a bunch of grape with ellipsoid berries.

If you can control quadratic curves, reshaping and rotating them, you can make a model of a grape of olive bunch on Matplotlib. I made a program of making a model of a bunch of berries on Matplotlib using the module to draw ellipsoids which I introduced earlier. You can check the codes in this page.

*I have no idea how many people on this earth are in need of making such models.

I made some modules so that you can see the grape bunch from several angles. This might look very simple to you, but the locations of berries are organized carefully so that it looks like they are placed around a stem and that the berries are not too close to each other.

 

The programming code I created for this article is completly available here.

[Refereces]

[1]C. M. Bishop, “Pattern Recognition and Machine Learning,” (2006), Springer, pp. 78-83, 559-577

[2]「理工系新課程 線形代数 基礎から応用まで」, 培風館、(2017)

[3]「これなら分かる 最適化数学 基礎原理から計算手法まで」, 金谷健一著、共立出版, (2019), pp. 17-49

[4]「これなら分かる 応用数学教室 最小二乗法からウェーブレットまで」, 金谷健一著、共立出版, (2019), pp.165-208

[5] 「サボテンパイソン 」
https://sabopy.com/