Databricks cost optimization for production workloads

Databricks cost optimization starts with ownership and enforced defaults, not a heroic round of Spark tuning. If engineers can launch any node type, leave interactive compute running, and charge jobs to anonymous shared clusters, your invoice is recording governance failures as infrastructure spend. Fix the control plane first, then optimize the workloads that remain.

I treat each workload as an economic unit: it has an owner, a purpose, a service target, a Databricks SKU, cloud VM cost, and a measured result. DBUs alone are not cost, and a faster query is not automatically cheaper. The useful comparison is dollars per successful run, per refreshed table, or per dashboard service window. That framing prevents the two mistakes I see most often: celebrating a lower DBU graph while cloud VM charges rise, and buying a larger cluster because one poorly shaped query occasionally spills.

The sequence matters. Establish cost data you trust, block configurations you already know are wasteful, separate interactive work from scheduled production, test Photon against complete runs, and put spot capacity only where interruption is acceptable. The SQL later in this article gives you a repeatable way to find owners, SKUs, idle service windows, and expensive queries without pretending that every system table has perfect attribution.

Measure dollars per outcome before changing compute

A cost baseline must combine Databricks usage with the cloud bill and a workload outcome. system.billing.usage records DBUs and other billable units, while system.billing.list_prices supplies historical Databricks list prices. Neither table contains the VM, disk, network, or cloud service charges from your provider account. If you optimize only the Databricks line item, you are looking at half the machine.

Use a weekly grain for the first pass. Daily numbers overreact to retries and month-end batches, while monthly numbers hide the day a new job doubled spend. For each production workload, record these fields:

• Owner and cost center, taken from enforced tags rather than a cluster name
• Successful runs and the business output, such as partitions refreshed or reports served
• Databricks list cost, contracted cost if finance can provide it, and cloud infrastructure cost
• Runtime percentiles, failure count, and retry cost
• The service target that constrains how aggressively you can scale down or use spot capacity

List cost is useful for comparison even when your contract has discounts. Do not present it as an invoice. Finance should map effective contract rates separately, and your cloud cost export should allocate VM and storage charges with the same workspace, environment, owner, and workload tags. If a workload lacks those tags, classify it as unallocated rather than distributing it evenly. Equal allocation rewards the team that failed to label its compute.

Choose a denominator that the team cannot improve by doing less useful work. Cost per hour encourages shorter availability windows but says nothing about output. Cost per successful job run works for stable batch sizes; cost per terabyte processed is better when input volume changes; cost per dashboard service hour fits a shared SQL warehouse. Keep latency and failure rate beside the cost figure so nobody can claim savings by missing a service target.

The first report should separate all-purpose compute, jobs compute, SQL warehouses, pipelines, and serverless products by billing_origin_product and sku_name. That split exposes architecture choices. A scheduled notebook running on an all-purpose cluster is not a Spark tuning problem. It is a workload-placement problem with a predictable premium and poor shutdown behavior.

Cluster policies should make the cheap path ordinary

A cluster policy is a cost control only when it restricts expensive choices. A policy full of optional defaults is a suggestion that disappears the moment someone copies an old cluster. Databricks policy rules can fix, allow, forbid, or bound compute attributes, including autotermination_minutes, autoscale.max_workers, runtime_engine, cluster_type, cloud availability settings, and the virtual dbus_per_hour attribute. Attributes absent from the policy remain unrestricted.

Build separate policies for interactive analysis, scheduled jobs, and exceptional workloads. The interactive policy should use a low worker ceiling and forced auto-termination. The job policy should permit only jobs compute, require ownership tags, and allow a wider autoscaling range. GPU, memory-heavy, or fixed-size exceptions deserve their own policies with named owners. One giant policy becomes a negotiation document and gives every user the most expensive exception.

This AWS example is intentionally strict. Replace the node types, DBU ceiling, and tag vocabulary with values tested in your account:

{
  "cluster_type": {"type": "fixed", "value": "job"},
  "autotermination_minutes": {"type": "fixed", "value": 15, "hidden": true},
  "autoscale.min_workers": {"type": "fixed", "value": 1, "hidden": true},
  "autoscale.max_workers": {"type": "range", "minValue": 1, "maxValue": 12, "defaultValue": 4},
  "runtime_engine": {"type": "allowlist", "values": ["PHOTON", "STANDARD"], "defaultValue": "PHOTON"},
  "dbus_per_hour": {"type": "range", "maxValue": 40},
  "aws_attributes.availability": {"type": "fixed", "value": "SPOT_WITH_FALLBACK"},
  "aws_attributes.first_on_demand": {"type": "fixed", "value": 1},
  "custom_tags.owner": {"type": "unlimited", "isOptional": false},
  "custom_tags.cost_center": {"type": "unlimited", "isOptional": false},
  "custom_tags.environment": {"type": "allowlist", "values": ["dev", "stage", "prod"]}
}

This policy keeps the driver on on-demand capacity by setting first_on_demand to 1 and lets workers use spot with fallback. It does not prove that twelve workers or Photon are economical. It caps the blast radius while preserving a controlled test. Azure and Google Cloud use different provider attributes, so copy the intent, not the AWS field names.

Policy rollout has an awkward edge: changing a definition does not repair every running cluster or every job definition that points elsewhere. Inventory policy IDs on active compute, terminate or edit noncompliant interactive clusters, and update job clusters through deployment code. Then remove unrestricted creation permissions. A dashboard that finds violations but leaves the creation path open merely documents tomorrow's waste.

Photon earns its place in a controlled comparison

Photon should win on total run cost, not on a screenshot of one faster stage. It replaces supported Spark SQL execution with a native vectorized engine, while Catalyst still plans the query. Unsupported operations can fall back to the standard Spark runtime. Python and Scala UDFs, RDD work, Dataset APIs, and stateful streaming can limit or remove the benefit, so enabling Photon on a UDF-heavy pipeline may raise the DBU rate without shortening the expensive part.

Run an A/B comparison on the same input snapshot, cluster shape, runtime family, cache state, and output validation. Test at least several representative runs after startup effects settle. Record wall time, DBUs, cloud instance hours, bytes read, shuffle, spill, output row count, and failures. Calculate both components:

Databricks cost = DBUs consumed * effective DBU price
Cloud cost      = instance hours * effective instance price + attached services
Run cost        = Databricks cost + cloud cost
Unit cost       = Run cost / successful business units produced

Suppose the standard engine consumes 18 DBUs over 24 minutes and Photon consumes 15 DBUs over 11 minutes. That is promising, but the decision still depends on the SKU rate and VM lifetime. If a job cluster sits through the same long initialization or the task waits on an external API, the shorter SQL stage may not move total cost enough. Conversely, a higher DBU rate can still produce a cheaper run when the cluster terminates much sooner.

Inspect the Spark UI SQL/DataFrame view on classic compute and the query profile on SQL warehouses. Photon operators appear distinctly from standard operators, which lets you locate fallback boundaries. Do not rewrite an entire application because one operator falls back. First attack the portion with the highest task time, scan volume, shuffle, or spill. Replacing a row-by-row UDF with built-in SQL expressions often improves portability and lets Photon process more of the plan.

SQL warehouses already use Photon. For interactive SQL, compare serverless, pro, and classic warehouse behavior against your networking requirements and demand pattern rather than trying to disable the engine. Serverless adds faster startup and workload management where available, but its invoice still needs a service window and concurrency denominator. A warehouse that responds quickly and stays available for unused hours can remain expensive.

Put spot workers behind a recovery plan

Spot capacity fits retryable, distributed workers; it is a poor default for a fragile driver or a job with a narrow completion deadline. On AWS, SPOT_WITH_FALLBACK can obtain on-demand instances when spot capacity is unavailable, and first_on_demand: 1 keeps the driver on on-demand capacity. That pattern removes a common single point of interruption while retaining worker savings. Azure spot VMs and Google Cloud Spot VMs have different eviction behavior and policy fields, but the placement decision stays the same.

Classify jobs before changing availability. A good spot candidate reads durable input, writes idempotently to Delta tables, checkpoints long work, tolerates worker loss, and has enough schedule slack for a retry. A bad candidate holds essential state only in memory, writes to an external system without idempotency, runs one huge task on a single executor, or must finish minutes before a customer deadline.

Test interruption instead of trusting the Spark label "fault tolerant." Cancel workers or run in a pool with real spot churn, then observe whether stages recover, output remains correct, and the retry stays inside the service target. Track these numbers for the spot cohort:

• Spot share of worker hours and fallback share
• Evictions, failed runs, and automatic retries
• Extra runtime and recomputation after loss
• Net cloud savings after retry cost
• Missed completion targets

A popular recommendation is to put every worker on spot and let autoscaling handle it. It is popular because the discount is visible before the failure modes are measured. It fails when autoscaling asks for a scarce instance family, fallback silently raises the on-demand share, or repeated evictions turn a long shuffle into paid recomputation. Allow two or three tested worker families where flexible node types are supported, and alert on fallback and retry rates.

Keep production and development strategies separate. Development can accept smaller clusters, faster termination, and more interruption. Production often deserves an on-demand driver, a minimum amount of stable capacity, and spot workers above that floor. The cheapest theoretical cluster has no value if an engineer reruns it manually three times.

Separate scheduled work from human waiting

Scheduled production work belongs on jobs compute or an appropriate serverless product, not on a shared all-purpose cluster. Jobs compute gives each run a bounded lifecycle and clearer metadata; multitask jobs can reuse compute across tasks when startup dominates. Interactive compute exists for exploration, but it needs forced auto-termination and a small default shape because humans leave browser tabs open.

Autoscaling does not fix an oversized minimum. A cluster configured for 8 to 40 workers spends at least eight workers even when the workload can run on two. Set the minimum from observed steady work, then cap the maximum from concurrency and memory tests. Review scale-up delay as well as peak size. A job that ends before new workers become useful pays for a broad range it never exploits.

Pools reduce launch latency by keeping instances ready, but idle pool instances still create cloud cost. Use pools when measured startup latency affects a service target or when many short jobs can share the warm capacity. Set a low minimum idle count and a short idle termination period. Do not create one pool per team for visual tidiness; fragmented idle capacity is still idle capacity.

For SQL warehouses, match auto-stop to the actual query pattern. Five-minute gaps between dashboard refreshes do not justify a stop-start cycle if startup hurts users, while an overnight warehouse with no queries should not remain available from habit. Compare query arrival times with warehouse events. If a warehouse runs for ten hours but receives queries in two compact windows, schedule or auto-stop around those windows and confirm that dashboard refreshes still meet their target.

Cluster size is the last tuning knob, not the first. Before increasing it, check for an exploding join, missing filters, skew, small files, disk spill, and Python UDFs. More workers can make a bad plan cost more without making its longest task faster. A smaller cluster with a corrected join usually beats an expensive cluster carrying the same mistake in parallel.

Billing queries should expose owners and corrections

The first cost query should reconcile usage by owner, SKU, and workload identifier. System billing records can contain ORIGINAL, RETRACTION, and RESTATEMENT rows. Sum usage_quantity; do not count rows or discard negative corrections. The query below joins each usage record to the price effective at its end time and reports list cost. It keeps unallocated spend visible.

WITH priced_usage AS (
  SELECT
    u.workspace_id,
    u.usage_date,
    u.sku_name,
    u.billing_origin_product,
    u.usage_metadata.cluster_id AS cluster_id,
    u.usage_metadata.job_id AS job_id,
    u.usage_metadata.warehouse_id AS warehouse_id,
    u.identity_metadata.run_as AS run_as,
    u.custom_tags['owner'] AS owner_tag,
    u.usage_quantity,
    u.usage_quantity * p.pricing.effective_list.default AS list_cost
  FROM system.billing.usage u
  LEFT JOIN system.billing.list_prices p
    ON u.cloud = p.cloud
   AND u.sku_name = p.sku_name
   AND u.usage_start_time >= p.price_start_time
   AND (u.usage_end_time < p.price_end_time OR p.price_end_time IS NULL)
  WHERE u.usage_date >= date_sub(current_date(), 30)
)
SELECT
  coalesce(owner_tag, run_as, 'UNALLOCATED') AS owner,
  billing_origin_product,
  sku_name,
  coalesce(job_id, warehouse_id, cluster_id, 'NO_RESOURCE_ID') AS resource_id,
  round(sum(usage_quantity), 2) AS usage_units,
  round(sum(list_cost), 2) AS databricks_list_cost
FROM priced_usage
GROUP BY ALL
HAVING abs(sum(usage_quantity)) > 0
ORDER BY databricks_list_cost DESC;

Expect output shaped like owner | billing_origin_product | sku_name | resource_id | usage_units | databricks_list_cost. Cast pricing.effective_list.default if your SQL environment does not coerce it as expected. The price join is temporal because a current price applied to old usage rewrites history. Contract discounts and cloud VM charges still sit outside this result.

Tags can be absent or misleading, so use system metadata as a second attribution path. identity_metadata.run_as identifies the run identity for jobs compute, while SQL warehouse billing can provide ownership metadata. For classic all-purpose and jobs clusters, join usage_metadata.cluster_id to the regional system.compute.clusters history and select the configuration version whose change_time was in effect when usage occurred. A simple join on cluster ID can duplicate records whenever the cluster configuration changed.

Create an exception report, not just a top-ten chart. Flag rows with missing owner or cost center tags, all-purpose SKUs used by scheduled identities, resource IDs with no matching inventory, and week-over-week cost increases beyond a threshold chosen from your own variability. Route each exception to a named owner. Cost data without a decision path becomes another dashboard nobody opens.

Query history finds waste that billing cannot name

system.query.history gives query-level execution evidence for SQL warehouses and supported serverless workloads, but it does not assign an exact invoice amount to every statement. Warehouse compute is shared across concurrent queries, startup, idle time, and scaling events. Treat per-query dollar allocation as an estimate unless you have defined and documented an allocation method. Use query history first to rank engineering work by task time, bytes read, queue time, and repeated execution.

This query groups repeated SQL text by a hash so you can find patterns that consume the most aggregate task time. It also exposes high scan-to-result ratios and queueing:

SELECT
  sha2(statement_text, 256) AS query_hash,
  any_value(left(regexp_replace(statement_text, '\s+', ' '), 180)) AS sample_sql,
  count(*) AS executions,
  round(sum(total_task_duration_ms) / 3600000.0, 2) AS task_hours,
  round(sum(execution_duration_ms) / 3600000.0, 2) AS wall_hours,
  round(sum(waiting_at_capacity_duration_ms) / 60000.0, 1) AS queued_minutes,
  round(sum(read_bytes) / pow(1024, 4), 3) AS tebibytes_read,
  sum(read_rows) AS rows_read,
  sum(produced_rows) AS rows_returned
FROM system.query.history
WHERE start_time >= date_sub(current_timestamp(), 14)
  AND execution_status = 'FINISHED'
GROUP BY query_hash
ORDER BY task_hours DESC
LIMIT 50;

One execution with a large scan may be legitimate. A dashboard query that scans the same cold data hundreds of times is usually a better target because fixing it saves money every day. Compare read_files with pruned_files, inspect filter columns, and check whether the table layout supports the predicates people actually use. An enormous rows_read to rows_returned ratio is a clue, not a conviction: an aggregation may correctly read many rows to return one.

Queue time and execution time require opposite reactions. High waiting_at_capacity_duration_ms during a business window can justify more warehouse concurrency or a larger maximum cluster count. Low queue time with long execution points to the query or data layout. Increasing capacity for a query that spills, explodes a join, or fails to prune files buys more hardware for unchanged SQL.

To find service-window waste, pair query arrival timestamps with system.compute.warehouse_events. Calculate running minutes before the first query, gaps between query bursts, and minutes after the last query. Then compare those idle minutes with the warehouse's auto-stop and startup behavior. Do not allocate all idle time to the first query or the warehouse owner without saying so. Shared capacity needs an explicit rule, such as allocation by task time plus a separate unallocated idle bucket.

Test one workload through the whole bill

A complete optimization test follows one workload from trigger to output and includes the capacity that waits around it. It also counts failed attempts, because a retry that disappears from the success dashboard still consumed DBUs, virtual machines, operator time, and schedule slack. Stage-level improvements are useful diagnostics, but they miss cluster startup, library installation, autoscaling delay, retries, idle time between tasks, and termination lag. Those edges often explain why a technically faster Spark plan barely changes the invoice.

Take a nightly ingestion job as an example. It launches a fresh eight-worker cluster, spends nine minutes installing libraries and listing source files, runs transformations for twenty minutes, writes a Delta table, and remains alive for another fifteen minutes because a downstream task waits on an external service. The Spark UI makes the transformation look like the expensive part. The lifecycle shows that compute runs for more than twice that period. Turning on Photon may shrink the transformation, but packaging dependencies in the runtime, replacing the external wait with a separate task, and lowering termination delay can save more.

Capture a run record before changing anything. Use the job run ID to join billing usage where that metadata is populated, record cluster start and termination times, and collect task durations from the job timeline. Save the input version or immutable date range and validate the output row count plus a business checksum. A cost result without output validation can reward a run that skipped data.

Change one variable at a time. First move the driver to on-demand and workers to spot, or first enable Photon, or first change the worker shape. If you change all three and the run becomes cheaper but less reliable, you will not know which choice caused either outcome. Repeat the candidate across normal volume, a heavy day, and at least one failure or interruption test. Compare medians for routine cost and the expensive tail for capacity planning.

Keep a plain decision record for each candidate. Record successful runs, median run cost, the cost of a slow run, median duration, retries, and whether output validation passed. Put the baseline, Photon candidate, and spot-worker candidate on separate rows in your experiment ledger.

Fill the cells with your own billing and run data rather than a borrowed benchmark. Five runs in the table are an experiment outline, not a universal sample size. High-variance workloads need a longer window, while deterministic batch jobs may show a stable difference quickly.

Watch for cost shifting between teams and systems. Moving preparation into an upstream service can make the Databricks job look cheaper while raising another bill. Replacing a shared all-purpose cluster with job clusters can improve attribution but increase repeated startup for many tiny tasks. Bundle tasks that share dependencies and data locality when doing so preserves failure isolation; do not bundle unrelated jobs merely to keep a cluster warm.

Approve the change only after it meets the service target and lowers the chosen unit cost. Then update the policy or deployment definition so the candidate becomes the normal path, and retain the baseline data. Runtime upgrades, data growth, and new query shapes can reverse an old result. Retest when unit cost or duration crosses a practical threshold, not on an arbitrary calendar alone.

Savings survive only when someone owns the regression

Cost optimization is finished when the cheaper configuration becomes the default and a regression produces an alert. A one-time resize has a short half-life: job definitions get copied, data volume grows, a runtime changes, and an engineer creates an unrestricted cluster to meet a deadline. Put policies and job definitions in version control, review exceptions with expiry dates, and keep the cost queries on a schedule.

Use a weekly operating review with four decisions: delete or terminate unused resources, assign unallocated spend, approve or reject policy exceptions, and select the next workload experiment. Keep the experiment ledger small. Each entry needs a baseline period, one change, cost per outcome, latency, failures, and a decision. If a team cannot state what changed, it cannot explain the saving or reproduce it.

Set alerts on conditions that point to action: missing mandatory tags, an all-purpose cluster running past its approved window, a job retry spike, spot fallback above your tested range, a warehouse with long idle windows, or a repeated query whose aggregate task hours jump. Avoid a single account-budget alarm as the only control. It arrives after several unrelated causes have accumulated and tells the operator where to look only after money is gone.

At oleg.is, I use a Team & AI Audit to connect infrastructure waste with the engineering process that created it; the fixed engagement is $5,000 over five business days, with at least $50,000 per year in identified savings or it is free. That offer is useful when nobody inside the company owns the cross-team investigation, but the controls in this article do not require an outside advisor.

The hardest cut is often organizational. Retire the shared cluster that everyone fears touching, move its scheduled notebooks into owned jobs, and force new compute through policies. Once every dollar has an owner and every exception expires, Photon, spot capacity, and query tuning become ordinary engineering decisions instead of invoice archaeology.