Cloud, on-premises and where the data lives

Why this matters

Where data is stored shapes almost everything an analyst can do with it: how fast a query runs, what it costs, who can reach it, and whether it may legally be moved at all. The same question asked of data in an on-premises warehouse, a cloud data lake and a departmental data mart can have three different answers in speed, cost and permission.

The exam asks you to recognise infrastructure models and storage architectures, and to choose between them for a scenario. This lesson covers the models, the main architectures for analytical data, and the trade-offs that decide where an analysis should run.

The lesson

On-premises, cloud and hybrid, and what each means for an analyst

On-premises infrastructure is owned and run by the organisation in its own data centre or server room. It gives full control over hardware, security and location, and a predictable cost once bought. It also means buying capacity in advance: a warehouse sized for today may be too small in two years, and expanding it takes procurement and installation.

Cloud infrastructure is rented from a provider and delivered over the network. Its strengths for data work are elasticity, the ability to add storage and computing power in minutes and release it afterwards, and managed services, where the provider runs the database or warehouse so the organisation does not have to. Cloud services are usually described by how much the provider manages: infrastructure as a service (IaaS) provides virtual machines and storage; platform as a service (PaaS) provides a managed platform such as a database; software as a service (SaaS) provides a finished application, such as a hosted BI tool.

Hybrid combines the two, typically keeping some systems on-premises while using the cloud for others, and multi-cloud uses more than one provider. Most organisations are hybrid in practice, which is why analysts so often need to combine data from systems that live in different places.

For the analyst, the practical differences are access and cost. On-premises data may only be reachable from inside the corporate network. Cloud data may be reachable from anywhere with the right credentials, which is convenient and also a larger responsibility.

Data warehouse, data lake, lakehouse and data mart

Four architectures for analytical data come up repeatedly.

A data warehouse stores structured, cleaned data organised for reporting, usually in a dimensional model such as a star schema. Data is transformed before it is loaded, so its structure is defined in advance. This is called schema-on-write. Warehouses are reliable and fast for known questions, and less suited to raw or unstructured data.

A data lake stores data in its raw form, structured or not, usually as files in cheap object storage. Structure is applied when the data is read rather than when it is written, which is schema-on-read. Lakes are flexible and inexpensive and suit exploration and data science, but without governance they decay into a data swamp: files nobody can find, understand or trust.

A lakehouse combines the two: data stays in open file formats in lake storage, while a management layer adds the warehouse features of tables, schemas, transactions and fast queries. It aims to give one copy of the data for both reporting and data science.

A data mart is a smaller, focused subset of warehouse data for one business area, such as sales or finance. It gives that team a simpler model and faster queries, at the cost of another copy that must be kept in step with the warehouse.

Cost and performance: storage is cheap, moving data is not

In the cloud, the cost of data work comes from a few sources, and they do not behave alike.

Storage is relatively cheap, and cheaper still in tiers designed for data that is rarely read, known as cool or archive storage. The trade-off is retrieval time and retrieval charges.

Compute, the processing power that runs queries, is charged by use. A badly written query that scans an entire table every hour costs money every hour. Many cloud warehouses charge by the amount of data scanned, so reading only the columns and partitions you need is both faster and cheaper.

Data transfer is often the surprise. Moving data into a cloud is usually free, but moving it out, or between regions, is typically charged as egress. An analysis that downloads a large data set to a laptop every morning may cost more in transfer than in storage.

The general principle is to move the computation to the data rather than the data to the computation: summarise in the warehouse and download the small result, instead of downloading the raw rows and summarising locally.

Latency, data residency and who controls the copy

Latency is the delay between a request and a response. Data far from the people or systems using it answers more slowly, and a dashboard that queries a warehouse in another continent will feel it. Latency also covers data freshness: a warehouse loaded nightly is up to a day behind the systems it copies.

Data residency is the requirement that data be stored in a particular country or region, usually because a law or contract says so. Personal data about people in one jurisdiction may have to stay there, or may only be moved with safeguards. Cloud providers let customers choose the region where data is stored, which makes compliance possible but does not make it automatic.

Every copy of data also raises the question of control. Copying a data set from the warehouse to a cloud storage bucket, a laptop or a BI tool creates another place where it can be accessed, lost or left out of date. Fewer copies, each with a known owner, are easier to secure and easier to trust.

Choosing where an analysis should run

Given these trade-offs, a few questions decide where an analysis belongs.

• Where is the data already? Running the analysis close to it avoids transfer cost and latency, and avoids creating copies.
• How large is it? A few thousand rows fit on a laptop; billions of rows need a warehouse or a distributed platform.
• Is it sensitive or restricted? Personal or regulated data may be permitted only in approved systems and regions.
• How often will it run? A one-off exploration can use whatever is convenient; a daily production report needs a scheduled, monitored home.
• Who needs the results? A shared dashboard needs a platform everyone can reach; a private investigation does not.

Answering these usually points to one of three places: the warehouse or lake itself for large or regulated work, a managed analytics or BI platform for shared reporting, or a local workbench, like the one this course uses, for exploration on small, non-sensitive extracts.