The Databricks cost optimization checklist: 23 fixes that cut DBU spend
Every point below comes from bills I have worked on in production roles. Work through them in order — the compute section alone usually pays for the hour it takes to read this.
Databricks bills grow the same way everywhere: a workspace starts small, teams multiply, nobody owns the number, and eighteen months later finance asks why the platform costs six figures a month. The good news is that DBU waste is rarely exotic. It is the same fifteen or twenty mistakes, repeated across clusters, jobs, tables, and warehouses.
This checklist covers all of them, grouped into five areas: compute, storage and data layout, workflows, governance and FinOps, and SQL warehouses. Each checkpoint is one imperative and the reasoning behind it. No tooling to buy, no framework to adopt — just changes you can make this week.
Compute (checkpoints 1–7)
Compute is where most of the money goes and where most of it is wasted. Start here.
1. Move every scheduled workload off all-purpose clusters and onto job clusters.
All-purpose compute is billed at roughly two to three times the DBU rate of job compute, cloud and tier depending. Any notebook running on a schedule against an all-purpose cluster is paying an interactive premium for a batch workload. This is the single most common finding in cost audits, and often the largest.
2. Set autoscaling minimums to what the job needs at rest, not at peak.
A cluster scaling 8–20 workers holds eight machines even when the job needs two. Autoscaling only saves money if the floor is honest — set the minimum to the smallest count that keeps the job healthy and let the range do its job. For short, predictable jobs, a fixed-size cluster with no autoscaling is frequently cheaper than paying the scale-up lag.
3. Use spot or preemptible instances for batch workers.
Spot capacity cuts the cloud-VM side of the bill by 60–90% for interruption-tolerant work. Keep the driver on-demand so an eviction does not kill the whole run, and enable decommissioning so shuffle data migrates before a node is reclaimed. Skip spot only for tight-SLA jobs where a restart genuinely hurts.
4. Turn Photon on only where it pays for itself.
Photon roughly doubles the DBU rate, so it must at least halve the runtime to break even. It usually clears that bar on SQL-heavy, wide-scan, join-and-aggregate workloads — and usually fails it on Python UDF-heavy code and small jobs dominated by cluster startup. Benchmark per workload; never enable it fleet-wide by default.
5. Do the serverless math before defaulting to it.
Serverless compute removes idle time and startup lag, which is a genuine saving for spiky, short, or infrequent workloads. But the per-DBU rate is higher, so a long-running, well-utilized classic job cluster on spot instances can still beat it comfortably. Compare on total cost per run, including the idle minutes classic compute wastes, not on the headline rate.
6. Right-size from cluster metrics, not from fear.
Open the cluster metrics UI (or export hardware metrics) and look at actual CPU and memory utilization over a week. Clusters sitting at 20% CPU are oversized; memory pressure and disk spill in the Spark UI tell you when they are undersized. Most teams provision for the worst run they can remember and then never look again.
7. Set aggressive auto-termination on every interactive cluster.
An all-purpose cluster idling overnight burns interactive-rate DBUs producing nothing. Set auto-termination to 10–30 minutes via cluster policy so nobody can create a cluster without it, and treat the old 120-minute default as a bug. The re-start friction is real but small; the idle spend is real and large.
Storage & data layout (checkpoints 8–11)
Bad table layout does not just cost storage — it makes every downstream job read more data and burn more compute.
8. Put OPTIMIZE and compaction on a cadence tied to write patterns.
Streaming and frequent small writes fragment Delta tables into thousands of small files, and every query pays the file-listing and open-file tax. Schedule OPTIMIZE daily on hot tables and weekly on the rest, or enable predictive optimization on Unity Catalog managed tables and let the platform decide. Either way, stop letting small files accumulate silently.
9. Prefer liquid clustering over static partitioning for new tables.
Static partitioning on the wrong column — high cardinality, or one that stops matching the query pattern — creates tiny-file sprawl you cannot undo without a rewrite. Liquid clustering (CLUSTER BY) gives data-skipping benefits without hard directory boundaries and can be re-keyed as access patterns change. Reserve classic partitioning for genuinely huge tables with a stable, low-cardinality filter column.
10. Tighten VACUUM retention deliberately.
Every MERGE and OVERWRITE leaves dead files behind, and until VACUUM runs you pay cloud storage for all of them — on churny tables that can be several times the live data size. Run VACUUM on a schedule and set retention to what time travel actually requires, typically the 7-day default. Check downstream readers and any deep-clone or streaming dependencies before shortening below that.
11. Enable deletion vectors on tables with frequent updates and deletes.
Without deletion vectors, deleting one row rewrites the entire Parquet file containing it — brutal on wide tables with GDPR deletes or CDC merges. Deletion vectors mark rows as removed instead, cutting write amplification dramatically for MERGE, UPDATE, and DELETE workloads. Purge them periodically with REORG TABLE ... APPLY (PURGE) so read overhead stays low.
Want the audit done for you?
Book a cost-focused Power Hour: we query your billing system tables live, rank your top cost drivers, and you leave with the three fixes that matter most for your bill. For heavy ongoing FinOps, the managed cost optimization service at databricks.support owns the number month over month.
Book a cost teardown1:1 consulting is delivered through RapidData. No discovery phase.
Workflows (checkpoints 12–14)
Orchestration decisions quietly multiply compute costs across every run, every day.
12. Match your DLT/Lakeflow tier to the features you actually use.
Lakeflow Declarative Pipelines bill at different DBU rates per tier, and the advanced tier costs meaningfully more than core. If a pipeline uses no expectations and no CDC processing, it should not be paying the advanced rate. Audit each pipeline's feature usage against its tier — downgrading is a settings change, not a rewrite.
13. Share one job cluster across tasks in a workflow — unless their shapes differ.
A five-task job that spins up five separate job clusters pays five startup penalties, five to ten minutes of billable-but-useless time per run. Declare a shared job cluster and point every similar task at it. Split clusters only when tasks genuinely need different shapes — a memory-heavy aggregation next to a small API call — or when parallel branches would fight for the same executors.
14. Stop letting retries mask waste.
A job that fails after 50 minutes and succeeds on the second attempt costs nearly double every run, and the green tick hides it forever. Query job run history for high retry counts and long failed-attempt durations, fix the underlying flakiness — usually memory pressure, spot eviction on the driver, or an upstream dependency — and set retry policies with backoff instead of instant full-cost re-runs.
Governance & FinOps (checkpoints 15–19)
You cannot optimize a bill you cannot attribute. This section is boring and it is where savings become permanent. If nobody owns the number internally and you are weighing outside help, my guide on how to hire a Databricks consultant covers when to bring in an expert and what that should cost.
15. Enforce cost tags through cluster policies, not goodwill.
Custom tags on clusters, jobs, and warehouses flow through to billing records — but only if they exist. Make team, project, and environment tags mandatory fields in cluster policies so untagged compute cannot be created. Retrofitting attribution three months later is guesswork; enforcing it up front is a policy JSON change.
16. Build your spend reporting on the system billing tables.
The system.billing.usage table gives per-workload, per-SKU, per-tag DBU consumption, and joining it to system.billing.list_prices turns that into dollars. This is the ground truth the platform itself bills from — more granular than any cloud cost dashboard. A single Databricks SQL dashboard on these two tables replaces most third-party FinOps tooling for a single-account estate.
17. Set budgets and anomaly alerts before you need them.
Databricks budgets in the account console alert when workspace or tag-scoped spend crosses a threshold you set. Pair that with a scheduled SQL alert on system.billing.usage for week-over-week jumps per job, so a runaway stream or a fat-fingered cluster size pages someone within a day instead of appearing on next month's invoice.
18. Publish per-team showback monthly.
The month teams see their own number is the month behavior changes — clusters shrink and forgotten jobs get switched off with no mandate needed. Use the tags from checkpoint 15 to cut spend by team and share the ranking openly. You do not need formal chargeback through finance; visibility alone does most of the work.
19. Move your predictable baseline onto committed-use pricing.
Databricks commit contracts discount DBUs meaningfully versus on-demand once you can forecast a floor — and after a year of billing data, you can. Commit to the baseline you are confident in, keep the spiky remainder on-demand, and never commit to hoped-for growth: an unused commitment is the most expensive DBU there is. Do the optimization work in checkpoints 1–18 first, then size the commit on the leaner bill.
SQL warehouses (checkpoints 20–23)
BI traffic is bursty and human-driven — the perfect shape for waste if warehouse settings are left at defaults.
20. Set auto-stop as low as your users can tolerate.
A warehouse idling between dashboard refreshes bills the whole time. Serverless warehouses restart in seconds, so a 5-minute auto-stop (or lower via the API) costs users almost nothing; classic warehouses restart slower, so 10 minutes is a fair compromise. The 45-minute-plus settings I still find in audits are pure waste.
21. Size for the typical query, not the worst one.
Each T-shirt size up doubles the DBU rate, so an oversized warehouse doubles the cost of every query, all day. Start one size smaller than feels safe and check the query history profile for disk spill — spill is the signal to go up a size, and its absence at low utilization is the signal to come down. One heavy weekly job does not justify sizing the shared warehouse for it; give that job its own.
22. Use the scaling policy for concurrency, not size.
Scaling out (min/max cluster count) handles many simultaneous small queries; scaling up (warehouse size) handles individually heavy ones — confusing the two is expensive in both directions. Keep the minimum cluster count at one and let queueing absorb brief spikes; raise the maximum only if the monitoring tab shows sustained queueing during business hours.
23. Let caching do the cheap work.
Warehouses cache query results and remote data locally, so repeated dashboard queries can cost near zero — but only when traffic lands on a warm warehouse. Route BI tools at one shared warehouse per audience instead of giving every team its own cold singleton, and align dashboard refresh schedules so they hit warm cache instead of each triggering a fresh start.
What to do with this list
Run checkpoints 1, 7, and 20 today — they are settings changes with instant payback. Then work through one section per week, verifying each change against system.billing.usage so the savings are measured, not assumed. If the bill is big enough that a percentage point matters, get a second pair of eyes on it before you negotiate any commit.
FAQ
Databricks cost optimization questions
How much can I realistically save on my Databricks bill?
Industry FinOps write-ups commonly report 20–40% first-pass savings on workspaces that have never been audited, mostly from moving scheduled work off all-purpose clusters, fixing auto-termination and warehouse auto-stop, and right-sizing from real cluster metrics. For mature, already-tuned platforms, treat 5–15% as a realistic target from storage layout, Photon selectivity, and commit pricing. The variance is real: a single all-purpose-to-job-cluster fix can halve a team's compute line.
How long does a Databricks cost audit take?
A focused Power Hour is enough to query the system.billing.usage tables, rank your top ten cost drivers, and identify the two or three fixes worth doing first. A full audit — every job, cluster policy, warehouse, and table layout, with a written remediation plan — takes one to two weeks part-time, because you need at least one weekly billing cycle to verify changes actually moved the number.
Can I do this myself, or do I need an expert?
Most of this checklist is DIY: auto-stop, auto-termination, tagging, and job clusters need no outside help. Bring in an expert when the monthly bill is large enough that 20% pays for the engagement many times over, when nobody owns the number internally, or when the fixes touch contested ground like shared clusters, DLT tiers, or commit negotiations. An experienced outside voice also settles internal arguments quickly because the recommendations come with benchmarks, not opinions.
Cut your DBU bill with a certified expert on the call.
One cost-focused Power Hour: your billing system tables, my benchmarks, a ranked fix list you can hand to the team the same day. Need someone to own the number continuously? The ongoing FinOps service at databricks.support does exactly that.
Book a cost teardown1:1 consulting is delivered through RapidData. Reply within one business day.