Analytics patterns
Building an S3 data lake with Glue, Athena and Lake Formation, and choosing between Athena, Redshift, EMR and OpenSearch, with Firehose ingestion, QuickSight dashboards and AppSync as a GraphQL layer.
Exam tasks: 2.5 (design solutions that meet performance objectives: analytics and data access)
The decision: where does the data land, who queries it and how often, and do they need ad hoc SQL, a warehouse, big-data processing, text search or an API?
Reference architecture
Choosing a query engine
| Athena | Redshift | EMR | OpenSearch Service | |
|---|---|---|---|---|
| What it is | Serverless SQL on S3 | Columnar data warehouse | Managed Spark, Hive, Presto, Flink | Search and log analytics engine |
| Pricing shape | Per TB scanned (or provisioned capacity) | Per node-hour or Serverless RPU-hour | Per instance-second, plus EMR fee | Per instance-hour and storage, or Serverless |
| Best for | Ad hoc queries, infrequent reports, querying logs in S3 | Frequent, complex BI with many concurrent users | Custom large-scale processing, ML feature engineering, existing Hadoop jobs | Full-text search, log exploration, near-real-time dashboards |
| Data lives in | S3 | Redshift managed storage, plus S3 via Spectrum | S3 (EMRFS) or HDFS | OpenSearch indexes |
| Ops effort | None | Low (Serverless) to medium | Medium: clusters, versions | Medium: domain sizing, shards |
Exam signal
"Query CSV or JSON logs in S3 occasionally, no infrastructure" means Athena. "Hundreds of analysts running dashboards all day over years of data" means Redshift. "Existing Spark or Hadoop jobs" means EMR. "Search and visualize application logs in near real time" means OpenSearch Service.
The S3 data lake
Glue Data Catalog and crawlers
- The Data Catalog is a Hive-compatible metastore: databases, tables, schemas and partitions for data in S3. Athena, Redshift Spectrum, EMR and Glue jobs all share it.
- Crawlers infer schemas and add partitions on a schedule. Alternatives are partition projection in Athena or adding partitions from the ETL job.
- Glue ETL jobs (Spark or Python shell) convert and clean data. Job bookmarks process only new files.
Making Athena fast and cheap
Athena bills by data scanned, so less data scanned means lower cost and faster queries.
- Convert to a columnar format (Parquet or ORC) and compress it. Queries then read only the columns they need.
- Partition by the fields you filter on most, such as date or Region, so queries skip whole prefixes.
- Avoid many tiny files. Combine them into objects of roughly 128 MB or more.
- Use workgroups to separate teams, set per-query data scan limits and track cost.
- Federated queries reach DynamoDB, RDS and other sources through Lambda connectors.
- Apache Iceberg tables add updates, deletes and time travel to data in S3.
Crawling everything every hour
Running crawlers constantly over a large lake is slow and costly. For predictable partition layouts, partition projection calculates partitions at query time and needs no crawler at all.
Lake Formation
- Central permissions on catalog resources: database, table, column, row and cell-level access.
- Tag-based access control (LF-Tags) scales permissions across thousands of tables.
- Cross-account sharing of catalog tables without copying data, through AWS RAM.
- Enforced by Athena, Redshift Spectrum, EMR and Glue when they read through the catalog.
Exam signal
"Analysts in the marketing account may see only non-PII columns of a table in the data lake account" points to Lake Formation column-level permissions and cross-account sharing, not bucket policies. S3 permissions can't filter columns.
QuickSight
- Serverless BI dashboards over Athena, Redshift, RDS, S3, OpenSearch and SaaS sources.
- SPICE is an in-memory engine that imports data for fast dashboards and fewer source queries.
- Row-level security filters the data each reader sees. Dashboards can be embedded in apps.
- QuickSight is now the BI part of Amazon Quick (Quick Sight). Existing QuickSight features and APIs still apply.
Redshift and Redshift Spectrum
- RA3 nodes separate compute from managed storage. Redshift Serverless bills only while queries run.
- Spectrum queries S3 through external tables in the Data Catalog, so you can join hot warehouse tables with cold history in the lake without loading it.
- Concurrency scaling adds capacity for bursts of queries. Data sharing gives other clusters live, read-only access without copying.
- Zero-ETL integrations replicate Aurora, RDS and DynamoDB data into Redshift without building pipelines.
- Federated queries read live data in RDS or Aurora PostgreSQL and MySQL.
OpenSearch Service for log analytics
- Ingest with Amazon Data Firehose, OpenSearch Ingestion (a managed Data Prepper pipeline that replaces Logstash), or a CloudWatch Logs subscription.
- Visualize with OpenSearch Dashboards (the successor to Kibana).
- Tier storage: hot nodes for recent data, UltraWarm for older read-only indexes, and cold storage on S3 for rarely queried data. Index State Management moves indexes automatically.
- OpenSearch Serverless removes cluster sizing for search and time-series collections.
- Run dedicated master nodes and three AZs for production domains.
Legacy: use Amazon OpenSearch Service instead
Amazon Elasticsearch Service was renamed OpenSearch Service in 2021. It still runs legacy Elasticsearch versions up to 7.10, but new domains use OpenSearch. An "ELK stack on AWS" today means OpenSearch Service with OpenSearch Dashboards, fed by OpenSearch Ingestion or Firehose.
Amazon Data Firehose
- Fully managed delivery to S3, Redshift, OpenSearch, Splunk, Apache Iceberg tables and HTTP endpoints. No shards or consumers to manage.
- Buffers by size and time, so it's near real time (seconds to minutes), not millisecond streaming.
- Transforms with Lambda, converts JSON to Parquet or ORC with a Glue table schema, and dynamically partitions S3 output by fields in the records.
- Loads Redshift by staging in S3 and issuing
COPY. - Sources include direct PUTs, Kinesis Data Streams, MSK, CloudWatch Logs and AWS WAF logs.
Legacy: use Amazon Data Firehose and Amazon Managed Service for Apache Flink instead
Kinesis Data Firehose was renamed Amazon Data Firehose in 2024, and Kinesis Data Analytics became Managed Service for Apache Flink. Older questions use the old names for the same services.
EMR
- Choose EMR on EC2 for full control, EMR on EKS to share a Kubernetes cluster, or EMR Serverless to run Spark and Hive jobs without clusters.
- Store data in S3, not HDFS, so clusters can be transient: start, run the job, terminate.
- Run core nodes On-Demand and task nodes on Spot, because task nodes hold no HDFS data.
AppSync as a GraphQL layer
- One managed GraphQL endpoint whose resolvers read from DynamoDB, Aurora, OpenSearch, Lambda and HTTP APIs, so a client gets data from several sources in a single request.
- Real-time subscriptions over WebSockets push changes to clients. Offline sync comes through Amplify client libraries.
- Auth modes: API key, IAM, Cognito user pools, OIDC and Lambda authorizers, combinable per field.
- Server-side caching and merged APIs (combine several teams' APIs behind one endpoint).
Exam signal
"Mobile and web clients need one API that combines data from several back ends, with real-time updates" points to AppSync. REST with a single back end fits API Gateway better.
Scenarios
Athena charges by data scanned. Columnar, compressed Parquet reads only the needed columns, and partitioning on the filter fields skips most of the data. A provisioned warehouse for a few weekly queries costs more. Transfer Acceleration speeds uploads and doesn't affect query cost. Frequent crawling adds cost without reducing scans.
Lake Formation enforces column-level permissions for Athena, and LF-Tags scale to hundreds of tables without writing a grant for each. Copying data creates drift and extra storage. Bucket policies work at the object level and can't hide a column inside a file. A Redshift view solves one table but not the Athena requirement, and doesn't scale to the whole lake.
Further reading
Compute and containers
Choosing between EC2, Lambda, ECS, EKS, Fargate and Elastic Beanstalk, plus ECS network modes, Lambda concurrency and VPC access, and edge compute.
End-user computing and contact centers
Choosing between WorkSpaces, WorkSpaces Applications (formerly AppStream 2.0) and WorkSpaces Secure Browser, and building a contact center on Amazon Connect with Lex, Polly, Transcribe and Contact Lens.