Storage and containerization for data work

Why this matters

Two analysts can run the same query on the same data and get different experiences: one waits seconds, the other waits an hour, because of how the data was laid out on storage. Two analysts can run the same script and get different answers, because one had a newer version of a library than the other. Storage and environment are infrastructure topics, but their effects land directly on the analyst.

The exam asks you to recognise storage types and containerization and to know why they matter for data work. This lesson covers both from the analyst's side: not how to run the infrastructure, but what it changes about the work.

The lesson

File, block and object storage, from the analyst's side

Storage systems present data in three main ways.

File storage organises data as files in folders, as on a laptop or a shared network drive. It is familiar and easy to browse, and it suits documents and small shared data sets. Shared file storage becomes awkward at very large scale, and a shared drive full of files with names such as final_v3_really.csv is a common source of confusion about which version is current.

Block storage presents raw disk volumes to a server, which formats them with its own file system. Databases usually run on block storage because it is fast and consistent. An analyst rarely deals with it directly; it sits underneath the database being queried.

Object storage stores each item as an object with a unique key and metadata, in a flat namespace called a bucket or container, reached over HTTP. It is cheap, effectively unlimited, and durable, which is why data lakes are built on it. Objects are written whole rather than edited in place, so object storage suits data that is written once and read many times, which describes most analytical data.

For an analyst, the storage type mainly shows up in how data is addressed: a file path on a drive, a table in a database, or an object key in a bucket.

Compression, partitioning and why file layout changes query speed

Two layout choices change how fast analytical data can be read.

Compression shrinks data so less has to be stored and moved. Columnar formats such as Parquet compress especially well, because the values in one column resemble each other. Less data read from storage usually means faster queries and, where storage reads are charged, lower cost. The trade-off is the processing needed to decompress, which is almost always worth it for analysis.

Partitioning splits a large data set into parts by the value of a column, most often a date. A year of orders might be stored as one folder per month. A query that asks only about March can then skip the other eleven folders entirely, which is called partition pruning. Choosing the partition column well matters: partition by the column most queries filter on, and avoid creating thousands of tiny partitions, because each file carries overhead.

Databases apply the same idea with indexes, which let a query find rows matching a condition without scanning the whole table. Indexes speed up reads and slow down writes slightly, which is why transactional and analytical systems index differently.

The lesson for the analyst is that the same question can be cheap or expensive depending on whether it aligns with how the data is laid out. Filtering on the partition column is fast; filtering on anything else may read everything.

Containers: packaging an analysis so it runs the same everywhere

An analysis depends on more than its code: a language version, libraries at particular versions, system tools, and settings. Move the code to another machine and any of these may differ, which is the source of the classic complaint that something "works on my machine".

A container packages an application together with everything it needs to run: libraries, dependencies and configuration. It runs in isolation on a container engine, most commonly Docker, and behaves the same on a laptop, a server or a cloud platform. Containers share the host's operating system kernel, which makes them much lighter and faster to start than virtual machines, each of which runs a complete operating system of its own.

For data work, containers are used to run a scheduled pipeline in exactly the same environment every time, to share an analysis environment across a team, and to deploy a model or data application consistently. When many containers must be run and scaled together, an orchestration platform such as Kubernetes manages them.

Images, volumes and where a container's data actually lives

A container is started from an image: a read-only template describing the environment, built from a recipe file, often called a Dockerfile. The same image can start many identical containers.

A container's own file system is temporary. When the container is removed, anything written inside it is lost. That is a feature for reproducibility, because every run starts clean, and a trap for data: an analysis that saves its results inside the container loses them when the container goes.

Data that must outlast a container is kept outside it, in a volume, a storage location mounted into the container, or in an external system such as a database or object storage. The practical rule is to keep code and environment in the image, and data outside it. Images should never contain credentials or sensitive data, because anyone who can obtain the image can read what is inside.

Reproducible environments: pinned versions and a requirements file

A container is one way to make an environment reproducible. A lighter way, enough for much analytical work, is to record and pin versions.

In Python, a virtual environment keeps a project's libraries separate from every other project's, so upgrading a library for one analysis does not break another. A requirements file, conventionally requirements.txt, lists the libraries a project needs. Pinning means recording exact versions, such as pandas==2.2.2 rather than just pandas, so the environment can be recreated precisely later. Other ecosystems have equivalents, such as lock files in R.

Pinning matters because libraries change their behaviour between versions: a default changes, a function is deprecated, a calculation is corrected. An unpinned analysis rerun a year later may produce different numbers with no change to its own code. Recording the environment alongside the code, and both in version control, means an analysis can be rerun and checked, which is the reproducibility this course keeps returning to.