Governed Data Lake and Analytics Platform on AWS
Streaming ingestion via Kinesis and Firehose, Lambda and Glue ETL into raw and curated S3 zones governed by Lake Formation, queried by Athena and Redshift, visualized in QuickSight.
Open editable diagramArchitecture diagram
Edit this exact diagram
Open on the canvasAWS data lake and analytics architecture diagram with Kinesis, Firehose, Lambda, Glue, raw and curated S3 zones, Lake Formation, Athena, Redshift, and QuickSight. The image links to a fully editable BuildPlane starter.
Overview
An ingest API sends records to Amazon Kinesis, which feeds both a real-time Lambda path and a Firehose delivery path into a raw Amazon S3 zone. AWS Glue transforms raw data into curated, query-optimized datasets cataloged for Athena and Amazon Redshift. Lake Formation governs data access, and Amazon QuickSight presents analytical dashboards.
Components
- Ingestion: An ingest API feeds Kinesis, the real-time buffer for all downstream processing.
- Processing: A stream ETL Lambda and a Firehose batch path populate the lake; a Glue job curates raw into query-ready data.
- Data Lake: Raw and curated S3 zones plus the Glue catalog are governed centrally by Lake Formation.
- Warehouse + Analytics: Athena queries the lake directly, Redshift powers heavy BI, and QuickSight visualizes both.
- Governance: Lake Formation applies fine-grained access control across every query engine.
Data Flow
- External sources push records to the ingest API, which writes to the Kinesis stream.
- Kinesis feeds a real-time ETL Lambda and a Firehose delivery that lands raw data in S3.
- A Glue job curates raw data into the query-optimized zone and refreshes the catalog.
- Athena and Redshift query the curated zone under Lake Formation governance.
- QuickSight renders dashboards over Athena and Redshift.
Customize First
- Adopt a medallion (bronze/silver/gold) layout and add Iceberg/Hudi table formats if you need upserts.
- Replace or complement Redshift with EMR/Spark for large transformations.
- Tighten Lake Formation policies to your data classification and PII handling rules.
Related AWS guides
Understand the decisions behind this diagram
Use these practical explanations to compare services, failure boundaries, and operating tradeoffs before adapting the architecture.
Design rationale
Decisions that shape this architecture
Preserve raw data before heavy transformation
Firehose lands an immutable raw record in S3. Teams can replay transformations, investigate defects, and build new curated datasets without depending on the current ETL implementation.
Separate real-time and curated paths
Lambda handles latency-sensitive transformations while Glue performs larger curation jobs. The two paths should publish clear contracts rather than competing to own the same output.
Apply governance across query engines
Lake Formation centralizes permissions for data accessed through Athena, Redshift integrations, and analytics users. Catalog ownership and classification remain essential.
Before production
Operational checks
Define schema evolution and malformed-record handling before producers change.
Partition curated data around real query filters and file-size targets.
Track freshness, completeness, duplication, and lineage for every dataset.
Separate producer, data-engineering, analyst, and sensitive-data permissions.
Scope and tradeoffs
What this diagram does not solve
A data lake does not create trustworthy data automatically
Quality contracts, ownership, lineage, retention, privacy classification, and incident procedures determine whether analysts can rely on the output.
Redshift is optional, not a default duplication target
Use a warehouse when workload, concurrency, modeling, and performance needs justify it. Athena may be enough for intermittent lake queries.
Common questions
Frequently asked questions
Why keep both raw and curated S3 zones?
The raw zone preserves source records for replay and audit. The curated zone contains validated, organized, and query-optimized datasets for consumers.
When should Athena be used instead of Redshift?
Athena fits serverless, intermittent queries directly over S3. Redshift fits sustained warehouse workloads that need predictable performance, concurrency, and modeled data.
What does Lake Formation add?
Lake Formation provides centralized data-lake permissions and governance across supported services, complementing IAM and the Glue Data Catalog.
Keep exploring
Related AWS architectures
IoT Telemetry and Analytics Platform on AWS
This architecture authenticates a device fleet, buffers telemetry, preserves raw events, derives searchable and time-series views, raises operational alerts, and presents fleet trends. Each storage path serves a distinct access pattern instead of treating one database as universal.
Explore the architecture
Machine Learning Batch Inference Platform on AWS
This architecture runs scheduled or on-demand scoring as a durable workflow. Step Functions coordinates jobs, AWS Batch supplies container compute, ECR pins the runtime, an approved SageMaker model supplies the artifact, and S3 preserves inputs and predictions.
Explore the architecture
Event-Driven Order Processing Architecture on AWS
This design accepts an order through an authenticated API, publishes a durable domain event, and uses a saga to coordinate payment, inventory, fulfillment, and notification. It separates request latency from business processing while preserving explicit recovery paths.
Explore the architectureFrom reference to working draft
Adapt this architecture with your AWS AI Agent
Start from the exact diagram in this guide. Change requirements, challenge a decision, or ask what fails first, while keeping the architecture editable.