Asynchronous File Processing Pipeline on AWS
This design accepts uploads immediately and lets workers process them at a controlled rate. S3 protects the source file, SQS absorbs bursts and retries, Lambda performs bounded work, and separate result and status stores make completion observable.
Published August 18, 2026
Reference architecture
Architecture diagram
Open on the canvasAsynchronous AWS file processing architecture diagram with S3 input, SQS queue, Lambda processor, S3 results, and DynamoDB status. The image links to a fully editable BuildPlane starter.
Overview
An asynchronous pipeline that accepts files quickly and processes them at a controlled pace. S3 provides durable input, SQS absorbs bursts, Lambda performs the work, and separate result and status stores make completion easy to inspect.
Components
- Incoming Files: An S3 landing bucket durably accepts uploads before any processing begins.
- Processing Queue: SQS separates upload volume from worker throughput and keeps pending jobs retryable.
- File Processor: Lambda validates and transforms each queued file without a permanently running server.
- Results and Status: S3 stores processed output while DynamoDB records the state of each file.
Processing Flow
- A user or application uploads a file to the incoming S3 bucket.
- The upload produces a job message in the SQS processing queue.
- Lambda receives a queued job and processes the referenced file.
- The worker stores output in S3 and updates processing status in DynamoDB.
Customize First
- Add a dead-letter queue before handling untrusted or repeatedly failing files.
- Define object-size limits, retry behavior, and idempotency for the worker.
- Add notifications when callers need completion events instead of polling status.
Design rationale
Decisions that shape this architecture
Store the file before scheduling work
S3 becomes the durable source of truth. Queue messages carry an object reference rather than the file payload, keeping messages small and retries safe.
Use SQS as the pressure boundary
The queue decouples upload rate from worker concurrency. Lambda can scale within downstream limits while excess jobs wait durably.
Track status separately from binary output
DynamoDB records job state and errors while S3 holds the processed file. Callers can inspect progress without listing or parsing storage objects.
Before production
Operational checks
Configure a dead-letter queue and an explicit redrive procedure.
Make processing idempotent for duplicate S3 events and SQS delivery.
Set visibility timeout longer than the expected processing duration.
Validate file type, size, and content before trusted downstream use.
Scope and tradeoffs
What this diagram does not solve
Lambda is not ideal for every file
Large, memory-heavy, GPU-based, or very long jobs may fit ECS, AWS Batch, MediaConvert, or another specialized service better.
The starter does not include caller notification
Add SNS, EventBridge, WebSocket delivery, or a callback mechanism when clients should not poll the status table.
Common questions
Frequently asked questions
Why put SQS between S3 and Lambda?
SQS buffers bursts, controls retry behavior, and lets worker concurrency be tuned independently from upload volume.
Should the file itself be placed in SQS?
No. Store the file in S3 and place the bucket, object key, version, and required job metadata in the message.
How should failed files be handled?
Record a terminal status, retain diagnostic context, move poison messages to a dead-letter queue, and provide an explicit replay or operator-review path.
Keep exploring
Related AWS architectures
Secure File Upload and Malware-Scanning Pipeline on AWS
This architecture keeps large upload bytes out of the API, lands every object in an encrypted trust boundary, scans before promotion, quarantines suspicious files, and exposes processing status to the application. It is a security-focused workflow rather than a generic upload shortcut.
Explore the architecture
Serverless REST API Architecture on AWS
This architecture is the smallest useful serverless API baseline: a managed HTTPS entry point, stateless function compute, on-demand data storage, and observable execution. It is intentionally compact so the first production decisions remain easy to see.
Explore the architecture
Event-Driven Order Processing Architecture on AWS
This design accepts an order through an authenticated API, publishes a durable domain event, and uses a saga to coordinate payment, inventory, fulfillment, and notification. It separates request latency from business processing while preserving explicit recovery paths.
Explore the architectureFrom reference to working draft
Adapt this architecture with your AWS AI Agent
Start from the exact diagram in this guide. Change requirements, challenge a decision, or ask what fails first, while keeping the architecture editable.