ACID vs BASE Concepts
Understanding databases for storing, updating and analyzing data requires the understanding of two concepts: ACID and BASE. This is the first article of the article series Data Warehousing Basics.
The properties of ACID are being applied for databases in order to fulfill enterprise requirements of reliability and consistency.
ACID is an acronym, and stands for:
- Atomicity – Each transaction is either properly executed completely or does not happen at all. If the transaction was not finished the process reverts the database back to the state before the transaction started. This ensures that all data in the database is valid even if we execute big transactions which include multiple statements (e. g. SQL) composed into one transaction updating many data rows in the database. If one statement fails, the entire transaction will be aborted, and hence, no changes will be made.
- Consistency – Databases are governed by specific rules defined by table formats (data types) and table relations as well as further functions like triggers. The consistency of data will stay reliable if transactions never endanger the structural integrity of the database. Therefor, it is not allowed to save data of different types into the same single column, to use written primary key values again or to delete data from a table which is strictly related to data in another table.
- Isolation – Databases are multi-user systems where multiple transactions happen at the same time. With Isolation, transactions cannot compromise the integrity of other transactions by interacting with them while they are still in progress. It guarantees data tables will be in the same states with several transactions happening concurrently as they happen sequentially.
- Durability – The data related to the completed transaction will persist even in cases of network or power outages. Databases that guarante Durability save data inserted or updated permanently, save all executed and planed transactions in a recording and ensure availability of the data committed via transaction even after a power failure or other system failures If a transaction fails to complete successfully because of a technical failure, it will not transform the targeted data.
ACID Databases
The ACID transaction model ensures that all performed transactions will result in reliable and consistent databases. This suits best for businesses which use OLTP (Online Transaction Processing) for IT-Systems such like ERP- or CRM-Systems. Furthermore, it can also be a good choice for OLAP (Online Analytical Processing) which is used in Data Warehouses. These applications need backend database systems which can handle many small- or medium-sized transactions occurring simultaneous by many users. An interrupted transaction with write-access must be removed from the database immediately as it could cause negative side effects impacting the consistency(e.g., vendors could be deleted although they still have open purchase orders or financial payments could be debited from one account and due to technical failure, never credited to another).
The speed of the querying should be as fast as possible, but even more important for those applications is zero tolerance for invalid states which is prevented by using ACID-conform databases.
BASE Concept
ACID databases have their advantages but also one big tradeoff: If all transactions need to be committed and checked for consistency correctly, the databases are slow in reading and writing data. Furthermore, they demand more effort if it comes to storing new data in new formats.
In chemistry, a base is the opposite to acid. The database concepts of BASE and ACID have a similar relationship. The BASE concept provides several benefits over ACID compliant databases asthey focus more intensely on data availability of database systems without guarantee of safety from network failures or inconsistency.
The acronym BASE is even more confusing than ACID as BASE relates to ACID indirectly. The words behind BASE suggest alternatives to ACID.
BASE stands for:
- Basically Available – Rather than enforcing consistency in any case, BASE databases will guarantee availability of data by spreading and replicating it across the nodes of the database cluster. Basic read and write functionality is provided without liabilityfor consistency. In rare cases it could happen that an insert- or update-statement does not result in persistently stored data. Read queries might not provide the latest data.
- Soft State – Databases following this concept do not check rules to stay write-consistent or mutually consistent. The user can toss all data into the database, delegating the responsibility of avoiding inconsistency or redundancy to developers or users.
- Eventually Consistent –No guarantee of enforced immediate consistency does not mean that the database never achieves it. The database can become consistent over time. After a waiting period, updates will ripple through all cluster nodes of the database. However, reading data out of it will stay always be possible, it is just not certain if we always get the last refreshed data.
All the three above mentioned properties of BASE-conforming databases sound like disadvantages. So why would you choose BASE? There is a tradeoff compared to ACID. If databases do not have to follow ACID properties then the database can work much faster in terms of writing and reading from the database. Further, the developers have more freedom to implement data storage solutions or simplify data entry into the database without thinking about formats and structure beforehand.
BASE Databases
While ACID databases are mostly RDBMS, most other database types, known as NoSQL databases, tend more to conform to BASE principles. Redis, CouchDB, MongoDB, Cosmos DB, Cassandra, ElasticSearch, Neo4J, OrientDB or ArangoDB are just some popular examples. But other than ACID, BASE is not a strict approach. Some NoSQL databases apply at least partly to ACID rules or provide optional functions to get almost or even full ACID compatibility. These databases provide different level of freedom which can be useful for the Staging Layer in Data Warehouses or as a Data Lake, but they are not the recommended choice for applications which need data environments guaranteeing strict consistency.
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/data/enterprise-bi-adfQuelle: 
https://docs.microsoft.com/en-us/azure/architecture/solution-ideas/articles/enterprise-data-warehouseQuelle: 
Quelle: https://docs.microsoft.com/de-de/azure/event-hubs/event-hubs-about 
, and it includes words from “a”, “ablation”, “actually” to “zombie”, “?”, “!”
, and the others are 
. When you want to choose the No. i word, which is “indeed” in the example below, its corresponding one-hot vector is 
, where only the No. i element is 
dimensional vector, whose dimension is fewer than the vocabulary size 
. When the inputs are “You’ve got the touch” or “You’ve got the power” , you put the one-hot vector corresponding to “You’ve”, “got”, “the”, “touch” or “You’ve”, “got”, “the”, “power” sequentially every time step 
.
, 
.
, a list of vectors. The vectors are usually embedding vectors, and the 
is the index of the order of tokens. For example the sentence “You’ve go the power.” can be expressed as 
, where 
denote “You’ve”, “got”, “the”, “power”, “.” respectively. In this case 
.
usually includes two tokens 
and 
at the beginning and the end of the sentence. They mean “Beginning Of Sentence” and “End Of Sentence” respectively. Thus in many cases 
. 
and 
are also both vectors, at least in the Tensorflow tutorial.
is the probability of incidence of the sentence. But it is easy to imagine that it would be very hard to directly calculate how likely the sentence 
as a product of the probability of incidence or a certain word, given all the words so far. When you’ve got the words 
so far, the probability of the incidence of 
, given the context is 
. 
is a probability of the the sentence 
, and the probability of 
can be decomposed this way: 
.
.
be 
be ![P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})](https://data-science-blog.com/wp-content/ql-cache/quicklatex.com-9b85b225f0635a7627a99481018f6166_l3.png "Rendered by QuickLaTeX.com")
be 
, then ![P(\boldsymbol{X}) = P(\boldsymbol{x}^{(0)})\prod_{t=0}^{\tau}{P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})}](https://data-science-blog.com/wp-content/ql-cache/quicklatex.com-59e2d215f0fea11ec20386dfef220a30_l3.png "Rendered by QuickLaTeX.com")
. Language models calculate which words to come sequentially in this way.
![P(\boldsymbol{x}^{(0)})\prod_{t=0}^{4}{P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})}](https://data-science-blog.com/wp-content/ql-cache/quicklatex.com-4d263b8554a322d3ab8a8841552bc75b_l3.png "Rendered by QuickLaTeX.com")
. Given a context 
is 
. In the figure below, the distribution at the left side is less confident because probabilities do not spread widely, on the other hand the one at the right side is more confident that next word is “got” because the distribution concentrates on “got”.
is 
.
gets higher. Thus 
gets lower, where usually 
or 
.
, which is composed of 
sentences in total. Each sentence 
![(\boldsymbol{x}^{(0)})\prod_{t=0}^{\tau ^{(n)}}{P(\boldsymbol{x}_{n}^{(t+1)}|\boldsymbol{X}_{n, [0, t]})}](https://data-science-blog.com/wp-content/ql-cache/quicklatex.com-ccc74ac2aee4f90a6b035dcfd2632927_l3.png "Rendered by QuickLaTeX.com")
has 
tokens in total excluding 
. And let 
, where ![z = \frac{-1}{|\mathcal{V}|}\sum_{n=1}^{|\mathcal{D}|}{\sum_{t=0}^{\tau ^{(n)}}{log_{b}P(\boldsymbol{x}_{n}^{(t+1)}|\boldsymbol{X}_{n, [0, t]})}](https://data-science-blog.com/wp-content/ql-cache/quicklatex.com-c1ccb76fcad455ad635fc1ec8b2f6f81_l3.png "Rendered by QuickLaTeX.com")
. The 
is usually 
or 
.
is vocabulary {“the”, “You’ve”, “Walberg”, “touch”, “power”, “Nights”, “got”, “Mark”, “Boogie”}. Also assume that the evaluation data set for perplexity of a language model is 
, where 
. In this case 
. I have already showed you how to calculate the perplexity of the sentence “You’ve got the touch.” above. You just need to do a similar thing on another sentence “You’ve got the power”, and then you can get the perplexity of the language model.
is a certain type of 
matrix, the formula of a D-dimensional ellipsoid whose center is identical to the origin point is as follows: 
, where 
. As is always the case with formulas in data science, you can visualize such ellipsoids if you are talking about 1, 2, or 3 dimensional data like in the figure below, but in general D-dimensional space, it is theoretical/imaginary stuff on blackboards.
, where 
or 
, where 
. These are special cases of the equation 
. In this case the axes of ellipsoids the same as those of the coordinate system. Thus in this simple case, 
or 
.
as below (The samples are plotted in purple). Intuitively, the data “swell” the most along the vector 
. Also it is clear that 
is the only vector orthogonal to 
expresses the data in a better way, and you you can get new coordinate points of the samples by projecting them on new axes as done with yellow lines below.
as below, the data “swell” the most also along 
.
on the axis 
.
is the mean of data in the original coordinate, then the deviation of 
on the axis 
, as shown in the figure. Hence the variance, I mean the mean of the deviation on is 
, where 
is the total number of data points. After some deformations, you get the next equation 
, where 
. 
is known as a covariance matrix.
, and for mathematical derivation we need some college level calculus, so if that is too much for you, you can skip reading this part till the next section.
including 
. Introducing a 
. In conclusion 
satisfies 
. If you have read my last article on eigenvectors, you wold soon realize that this is an equation for calculating eigenvectors, and that means 
. We have seen that 
is a the variance of data when projected on a vector 
is the biggest variance possible when the data are projected on a vector.
, and it it the second biggest variance possible, and in this case the date are projected on 
matrix 
, and in fact the matrix is just a constant multiplication of this covariance matrix. I think now you understand that PCA is calculating the orthogonal eigenvectors of covariance matrix of data, that is diagonalizing covariance matrix with orthonormal eigenvectors. Hence we can guess that covariance matrix enables a type of linear transformation of rotation and expansion and contraction of vectors. And data points swell along eigenvectors of such matrix.
denote Japanese, Math, English scores respectively. The mean of the data is 
, and the covariance matrix of data in the original coordinate system is 
. The eigenvalues of 
are 
, and 
, and their corresponding unit eigenvectors are 
respectively. 
is an orthonormal matrix, where 
. As I explained in the last article, you can diagonalize 
: 
.
.
means. Each element of 
denotes coordinate of the data point 
, and 
). 
enables a rotation of a rigid body, which means the shape or arrangement of data will not change after the rotation, and 
, and 
is the coordinate of 
denotes the coordinate point of the purple point in the red coordinate system. 
denotes the coordinates of data projected on new axes 
, which are unit eigenvectors of 
, which means you can express deviations of the original data as linear combinations of the three factors 
, and 
. We expect that those three factors contain keys for understanding the original data more efficiently. If you concretely write down all the equations for the factors: 
, 
, and 
. If you examine the coefficients of the deviations 
, and 
, we can observe that 
almost equally reflects the deviation of the scores of all the subjects, thus we can say 
is 
. You can see 
, where 
. The variance of data projected on new D-dimensional coordinate system is 
. This means that in the new coordinate system after PCA, covariances between any pair of variants are all zero.
.
from the reduced two dimensional coordinate system 
. Then it mathematically clearer that we can express the data with two factors: “how smart the student is” and “whether he is at scientific side or liberal art side.”
is a statistic which indicates how much the corresponding 
is called the contribution ratio of eigenvector 
and 
are respectively 
, 
, 
. You can decide how many degrees of dimensions you reduce based on this information.
, which most people would have seen in compulsory education, is a part of mapping. If you get a value x, you get a value y corresponding to the x.
, and machine translation is nothing but mappings between these sequence data.
Herr Dr. Frank Block ist Head of IT Data Science bei Roche Diagnostics mit Sitz in der Schweiz. Zuvor war er Chief Data Scientist bei der Ricardo AG nachdem er für andere Unternehmen die Datenanalytik verantwortet hatte und auch 20 Jahre mit mehreren eigenen Data Science Consulting Startups am Markt war. Heute tragen ca. 50 Mitarbeiter bei Roche Diagnostics zu Data Science Projekten bei, die in sein Aktivitätsportfolio fallen: 
