Cost optimization strategies in system design

Why cost optimization is a systems problem

Consider a web application whose cloud bill has reached $200,000 per month and whose owners want a 40% reduction without degrading the user experience. The objective is not to make every resource smaller. It is to find waste, quantify the effect of each change, and preserve the availability, durability, latency, and delivery guarantees the product depends on.

Cloud spend usually comes from several interacting buckets: compute, storage, data transfer, managed services, and observability. A cheaper instance can be irrelevant if cross-AZ traffic or log ingestion is the real driver. Conversely, a low-cost change can be expensive operationally if it removes a redundancy or adds a new failure mode.

Scope and assumptions

The examples below use AWS-style services and illustrative prices for a $200,000 monthly bill. Actual rates vary by region, account discounts, data-transfer path, workload shape, and pricing changes, so treat the figures as a way to reason about relative impact rather than as a quote. The scenario includes both quick wins and changes that require architectural work; production databases and availability controls still need explicit risk review.

30-second mental model

Cost optimization is a feedback loop:

1. Measure: Attribute spend to services, teams, environments, and requests; correlate it with utilization and customer-facing SLOs.
2. Classify: Separate waste (idle capacity, orphaned storage, unnecessary transfer) from intentional spend (redundancy, headroom, compliance).
3. Change the cheapest safe lever first: Right-size, expire, compress, cache, or reroute before making a structural commitment.
4. Verify and guard: Compare the bill and operational metrics after the change, then add budgets, tags, and infrastructure-as-code checks so the saving persists.

The useful question is not "what is the cheapest architecture?" It is "what is the lowest total cost that still meets the required reliability and performance?"

The review usually proceeds through these five areas:

1. Observe first: Use Cost Explorer, billing tags, and resource utilization metrics to understand where the money actually goes. Never optimize blind.
2. Right-size compute: Match instance types to actual utilization, not worst-case provisioning.
3. Optimize storage lifecycle: Move cold data to cheaper tiers automatically.
4. Eliminate data transfer traps: Cross-AZ, cross-region, and NAT gateway fees are the biggest surprise costs.
5. Tune managed services and observability: High-cardinality metrics, over-provisioned databases, and verbose logging silently drain budgets.

The key principle: measure, then cut. Every dollar saved has a risk attached. The job is to find the optimizations where the savings are large and the risk is acceptable.

The Architecture

Here is a typical cloud architecture with cost annotations showing where money goes and where waste accumulates:

The bill is easier to understand when each cost category is tied to a concrete request path.

Compute (35%): Twenty c5.2xlarge instances running at 22% average CPU utilization. That may indicate over-provisioning, but average CPU alone is not enough: memory, P95/P99 CPU, request latency, and scaling behavior also matter. A common waste pattern is sizing for a short peak and leaving that capacity running 24/7.

Storage and Database (30%): The S3 bucket has 85TB of data, but 90% of it is older than 90 days and rarely accessed. That cold data is sitting in S3 Standard at $0.023/GB when it could be in Infrequent Access at $0.0125/GB or Glacier at $0.004/GB. The RDS instance is multi-AZ (good for durability) but sized for a traffic pattern from two years ago.

Data Transfer (20%): This is the line item that surprises people. NAT Gateway charges alone can run $5,000-10,000/month if all outbound traffic from private subnets passes through it. Cross-AZ traffic between microservices adds $0.01/GB in each direction, and when services communicate frequently, this adds up.

Observability (7%): CloudWatch log ingestion at 500GB/month is expensive. Most of that volume is debug-level logging that nobody reads. Datadog's custom metrics bill scales with cardinality: 800 metrics sounds small, but if each metric has high-cardinality tags, the effective metric count can be 80,000+.

Cost visibility is most useful before a large bill arrives. Tag every resource with team, environment, and service; group Cost Explorer views by those tags; and connect spend to a unit such as cost per request, customer, or gigabyte. When the bill changes, the owner and the workload responsible should be obvious.

Five-minute end-to-end review

1. Build a baseline: Export the last three to six billing periods, group by service and allocation tag, and record request volume, latency, error rate, availability, and data-retention requirements alongside spend.
2. Find waste: Look for idle or low-utilization compute, unattached volumes and snapshots, cold objects in hot storage tiers, traffic through NAT or across AZs, and logs or metrics with unnecessary volume or cardinality.
3. Choose a safe sequence: Apply reversible changes first (log sampling, lifecycle policies, endpoint routing, instance tests), then make commitments such as Savings Plans only after the new baseline is stable.
4. Change one dimension at a time: Canary the change, watch SLOs and saturation signals, and keep a rollback path. Do not attribute a bill reduction to a change until the billing data has caught up.
5. Make the saving durable: Add budget alerts, mandatory tags, expiration policies, and IaC limits. Review cost per unit with the same regularity as latency and error rate.

Compute Cost Optimization: Right-Sizing and Spot Instances

Compute is usually the first place teams look for savings, and there are three levers: right-sizing, pricing model (reserved vs spot vs on-demand), and architectural changes like serverless.

Let me walk through the right-sizing decision.