Building a Serverless E-commerce Analytics Pipeline on AWS

From Real-Time Orders to Actionable Insights with DynamoDB, Kinesis, Athena & QuickSight

📅 October 2025

⏱️ 15 min read

🏷️ AWS, Analytics, E-commerce

The Challenge: You're running a successful e-commerce platform on AWS with DynamoDB handling thousands of orders daily. Your business stakeholders need real-time insights into sales performance, customer behavior, and revenue trends. But DynamoDB isn't designed for complex analytical queries—and setting up a traditional data warehouse feels like overkill.

The Solution: A fully serverless analytics pipeline that costs less than $100/month, requires zero infrastructure management, and delivers insights in seconds. Here's how we built it.

Why This Architecture?

When building analytics for e-commerce, you face several competing requirements: real-time data freshness, cost efficiency, query performance, and scalability. Traditional approaches—like ETL jobs or dedicated data warehouses—often sacrifice one for the other.

Our serverless pipeline leverages AWS managed services to deliver all these benefits simultaneously, with minimal operational overhead.

💰 Cost-Effective

Pay only for what you use. No always-on infrastructure. Estimated at $80-100/month for 100K orders/day.

⚡ Real-Time

Data flows continuously from DynamoDB to S3 in under 5 minutes with no batch processing delays.

🔄 Zero Maintenance

Fully managed services mean no servers to patch, no clusters to scale, no infrastructure to babysit.

📈 Scalable

Automatically handles traffic spikes during Black Friday sales or product launches without manual intervention.

The Architecture

Our pipeline consists of five key components, each handling a specific part of the data flow:

DynamoDB Orders Table
▼ (Stream Enabled)
Kinesis Data Stream
▼ (Native Integration)
Kinesis Firehose
▼ (Lambda Transform)
S3 (Parquet Format)
▼ (Glue Catalog)
Amazon Athena
▼ (SPICE Import)
Amazon QuickSight

Component Breakdown

1. DynamoDB with Kinesis Streams

Instead of using DynamoDB Streams directly, we leverage the native Kinesis Data Streams integration. This gives us better scalability and enables multiple consumers without impacting database performance.

OrdersTable:
  Type: AWS::DynamoDB::Table
  Properties:
    StreamSpecification:
      StreamViewType: NEW_AND_OLD_IMAGES
    KinesisStreamSpecification:
      StreamArn: !GetAtt KinesisDataStream.Arn
                

2. Lambda Transformation Function

DynamoDB stores data in a typed JSON format (e.g., {"S": "value"} for strings). Our Lambda function flattens this into standard JSON and handles complex types like nested objects and arrays.

Key transformations:

Converts DynamoDB typed format to native Python types
Serializes complex objects (billing info, product lines) as JSON strings for Athena
Adds event metadata (INSERT/MODIFY/REMOVE, timestamps)
Handles missing fields with sensible defaults

3. Kinesis Firehose with Parquet Conversion

Here's where the magic happens. Firehose automatically:

Buffers data into optimal batch sizes (128MB or 5 minutes)
Converts to Parquet using AWS Glue schema for 90% storage reduction
Compresses with Snappy for fast decompression during queries
Partitions by date (year/month/day) for efficient query pruning

                    💡 Pro Tip: Parquet with Snappy compression reduces S3 costs by 10x compared to JSON and makes Athena queries 10-20x faster. This single optimization saves hundreds of dollars monthly on large datasets.
                

4. AWS Glue Catalog with Partition Projection

Instead of running expensive Glue Crawlers to discover partitions, we use partition projection. Athena automatically calculates partition locations based on query predicates.

# No crawler needed - Athena knows where to look
Parameters:
  projection.enabled: 'true'
  projection.year.type: integer
  projection.year.range: '2024,2030'
  storage.location.template: 
    's3://bucket/orders/year=${year}/month=${month}/day=${day}/'
                

5. Amazon Athena & QuickSight

Athena provides SQL interface over S3 data. QuickSight imports data into SPICE (in-memory engine) for sub-second dashboard performance. We set up incremental refresh to only load new partitions daily.

Implementation Highlights

Handling Complex E-commerce Data

Our Orders table contains nested structures like product lines and billing information. Rather than flattening everything (which creates wide tables), we store complex objects as JSON strings and use Athena's JSON functions for analysis:

-- Extract product information from JSON
SELECT 
  order_id,
  json_extract_scalar(product_line, '$.product_id') as product_id,
  json_extract_scalar(product_line, '$.quantity') as quantity,
  total_price / 100.0 as revenue_dollars
FROM ecommerce_analytics.orders
WHERE year = '2025' AND month = '10'
  AND event_name IN ('INSERT', 'MODIFY');
                

Financial Precision

E-commerce requires exact calculations. We store prices as integers (cents) in BIGINT columns to avoid floating-point errors, then divide by 100 in queries for display.

Cost Breakdown

Based on 100,000 orders per day (3M orders/month):

Service	Usage	Monthly Cost
DynamoDB Streams	Included with table	$0
Kinesis Data Stream	2 shards × 730 hours	$45
Lambda	3M invocations, 512MB	$5
Kinesis Firehose	~30GB processed/day	$25
S3 Storage	100GB Parquet (compressed)	$3
Athena	$5/TB scanned (~1TB/month)	$5
QuickSight	5 users, capacity pricing	$90
Total Monthly Cost		~$173

                    💰 Cost Optimization: Using SPICE in QuickSight dramatically reduces Athena costs. Instead of querying S3 hundreds of times daily, we refresh SPICE once per hour—cutting query costs by 95%.
                

Real-World Use Cases

1. Revenue Dashboard

Track daily revenue by storefront, product category, and region. Identify top-performing products and underperforming regions in real-time.

2. Customer Lifetime Value

Analyze purchase patterns, repeat customer rates, and average order values. Segment customers by behavior for targeted marketing.

-- High-value repeat customers
SELECT 
  user_id,
  COUNT(DISTINCT order_id) as order_count,
  SUM(total_price) / 100.0 as lifetime_value,
  AVG(total_price) / 100.0 as avg_order_value
FROM ecommerce_analytics.orders
WHERE event_name IN ('INSERT', 'MODIFY')
  AND year = '2025'
GROUP BY user_id
HAVING order_count > 3
ORDER BY lifetime_value DESC
LIMIT 100;
                

3. Discount Code Effectiveness

Measure which promotional codes drive the most revenue and identify discount abuse patterns.

4. Partner Performance Reports

If you're running a marketplace, generate automated reports for each partner showing their sales, commissions, and trending products.

Key Takeaways

Start with the right format: Parquet isn't optional—it's essential for cost-effective analytics at scale
Use partition projection: Eliminates Glue Crawler costs and latency
Leverage SPICE: QuickSight's in-memory engine transforms query performance and slashes Athena bills
Separate concerns: Keep infrastructure code (CloudFormation) separate from application code (Lambda)
Monitor continuously: Set up CloudWatch alarms for Firehose delivery failures and Lambda errors

What's Next?

This architecture is production-ready, but there are several enhancements you might consider:

Add Amazon EventBridge: Trigger alerts when revenue drops below thresholds
Implement data retention policies: Archive old data to Glacier for compliance
Create aggregation tables: Pre-compute daily/monthly summaries for faster dashboards
Add machine learning: Use SageMaker to predict customer churn or forecast demand
Enable cross-region replication: Replicate analytics data for disaster recovery