Amazon S3: object storage, storage classes and lifecycle policies

Why this matters

Amazon S3 is one of the oldest AWS services and one of the most widely used. Product photos on shopping sites, backups, log files, video libraries and the raw data behind analytics projects very often live in S3. If you work anywhere near AWS, you will hear "put it in a bucket" within your first week.

Storage is also where cloud bills quietly grow. Data piles up, nobody deletes it, and it sits in the most expensive storage class for years. A manager who knows that S3 offers cheaper classes for data that is rarely read, and that rules can move or delete data automatically, can cut a storage bill without anyone touching the application. A salesperson needs to match a customer's need ("we keep records for years and almost never look at them") to the right class. A junior engineer needs to know which kind of storage a workload needs before choosing S3 at all. The exam tests exactly these judgements.

The lesson

AWS storage services

AWS offers three basic kinds of storage, and choosing between them is the first decision:

• Object storage keeps each piece of data as a whole object with its own name and metadata, reached over the web. Amazon S3 is the object store, and the S3 Glacier storage classes are its archive tiers.
• Block storage splits data into fixed-size blocks on a volume that acts like a disk attached to one server. Amazon EBS and instance store provide it.
• File storage presents a shared file system with folders that many servers can mount at once. Amazon EFS and Amazon FSx provide it.

Alongside these are AWS Storage Gateway, which connects on-premises systems to AWS storage, AWS Backup, which manages backups across services, and AWS Elastic Disaster Recovery. Block, file and hybrid storage and AWS Backup are covered in lesson 19. This lesson covers S3. Prices for each class are covered in lesson 23.

Identifying the uses for object storage

In Amazon S3 (Simple Storage Service), you store objects in buckets. An object is the data itself (a photo, a PDF, a log file) together with metadata describing it, and it is identified by a key, its unique name within the bucket. A bucket is the container. Each bucket is created in an AWS Region you choose, and its name must be globally unique.

The features that matter for choosing it:

• No capacity planning. Storage is virtually unlimited; you pay for what you store and never provision a size.
• Reached over HTTPS from anywhere, through the S3 API, with permissions deciding who can read what. Buckets are private by default.
• Durability. Most storage classes keep data redundantly across several Availability Zones, so losing one zone does not lose the data.
• Whole-object writes. You upload or replace an object as a unit; you do not edit a few bytes in the middle of it as you would on a disk.

That last point decides what S3 is good for. Typical uses are:

• static website content: HTML pages, images, scripts and downloads;
• backups and archives;
• data lakes, central stores of raw data for analytics;
• media libraries, user uploads and log files;
• software and file distribution.

What S3 is not for: an instance's boot volume, the files of a transactional database, or a low-latency scratch disk. These need block storage, attached to the instance, such as Amazon EBS or instance store.

A few S3 features appear as answer options, so know what each does:

• S3 Versioning keeps earlier versions of an object when it is overwritten or deleted, which protects against mistakes but adds stored data.
• S3 Object Lock prevents objects from being deleted or overwritten for a retention period, which suits compliance records.
• S3 Replication copies objects automatically to another bucket, in another Region (Cross-Region Replication) or the same one (Same-Region Replication).
• S3 Transfer Acceleration speeds up uploads over long distances using AWS edge locations.
• Bucket policies and access points grant access, including to other accounts.

How the Amazon S3 storage classes differ

Every object in S3 has a storage class. The classes trade storage price against how quickly and how cheaply you can read the data back. The general rule: the less often you read data, the cheaper it is to keep, but the more you pay, or wait, when you do read it.

• S3 Standard: for frequently accessed data. Low latency, high throughput and no retrieval fee. The default, and the right choice for website images requested all day.
• S3 Intelligent-Tiering: for data whose access patterns are unknown or change over time. S3 monitors each object and moves it between access tiers automatically, with no retrieval fees; you pay a small monitoring charge instead.
• S3 Standard-Infrequent Access (S3 Standard-IA): for data read rarely but needed in milliseconds when it is. Storage costs less than Standard, but each retrieval is charged, so data read constantly costs more here.
• S3 One Zone-Infrequent Access (S3 One Zone-IA): like Standard-IA, but kept in a single Availability Zone, so it is cheaper and is lost if that zone is. Use it only for data you can re-create, such as copies or derived files.
• S3 Glacier Instant Retrieval: for long-lived archive data accessed rarely, perhaps once a quarter, that must still come back in milliseconds.
• S3 Glacier Flexible Retrieval: for archives where waiting minutes to hours for a restore is acceptable. Storage is cheaper than Instant Retrieval.
• S3 Glacier Deep Archive: the lowest-cost S3 class, for data kept for years and almost never read, where a restore taking hours is fine. Long-term compliance records are the classic case.

Apart from One Zone-IA, these classes store data across multiple Availability Zones. The infrequent-access and Glacier classes also charge for a minimum storage duration, so they suit data that will stay put, not data that is deleted after a few days.

To pick a class from a scenario, find two clues: how often the data is read, and how fast it must come back. "Constantly" means Standard. "Unknown or changing" means Intelligent-Tiering. "Rarely, but immediately" means Standard-IA, or Glacier Instant Retrieval for archives. "Can re-create it" and "single zone" mean One Zone-IA. "Hours is fine" means Glacier Flexible Retrieval, and "lowest cost, years, almost never" means Glacier Deep Archive.

Understanding use cases for lifecycle policies

An S3 Lifecycle configuration is a set of rules that S3 applies to objects automatically according to their age. A rule can apply to a whole bucket or only to objects with a certain prefix (folder-like name) or tag. Rules take two kinds of action:

• Transition actions move objects to a cheaper storage class after a set number of days, for example from S3 Standard to S3 Standard-IA, then on to S3 Glacier Deep Archive.
• Expiration actions delete objects once they reach a set age.

Typical uses:

• Logs that cool with age. Application logs read often for the first month, rarely after that and kept for years for audit can move to Standard-IA, then to Glacier Deep Archive, then be deleted when the retention period ends. Nobody has to remember to do it.
• Temporary files. Files that should disappear a fixed number of days after creation can be expired automatically.
• Old versions. In a versioned bucket, rules can move or delete earlier versions so they do not build up indefinitely.
• Unfinished uploads. Rules can clean up large uploads that were started but never completed.

Know where lifecycle rules stop. They do not encrypt objects, cache content near users or grant other accounts access. And they work by age on a schedule you set, so they suit known patterns. When nobody knows how data will be used, S3 Intelligent-Tiering is the better fit, because it reacts to actual access instead of age. And if the requirement is to stop objects being deleted, that is S3 Object Lock, the opposite of an expiration rule.

What the exam is really asking

Storage questions in this area turn on a handful of clues. "Objects", "buckets", "from anywhere over HTTPS" or "virtually unlimited" mean S3; "boot volume" or "attached disk" mean block storage, not S3. For classes, read how often and how fast the data is needed, and whether it can be re-created. For lifecycle, "automatically", "after N days", "move to cheaper storage" or "delete" mean an S3 Lifecycle rule, while "unknown access patterns" means Intelligent-Tiering.