Databricks began its journey as a cutting-edge solution for machine learning, offering advanced AI/ML capabilities. Over time, it has evolved into a comprehensive lakehouse platform—seamlessly blending the scalability of data lakes with the reliability and performance of data warehouses.
Today, Databricks is not only capable of running large-scale ETL pipelines but also supports advanced analytics through both its native tools and external integrations. Its strong foundation in Apache Spark, paired with innovations like Delta Lake, allows for efficient real-time and batch data processing.
More recently, Databricks has started enhancing its support for transactional workloads and low-latency reporting by introducing a managed PostgreSQL database as part of its platform offering. This move further reinforces Databricks’ goal of being a truly end-to-end platform—enabling transactional processing, real-time streaming, batch analytics, and AI/ML workloads all in one place.
To support such a wide array of use cases, flexible and scalable compute options are critical. A single compute type cannot meet the diverse performance and workload demands of streaming, analytics, and machine learning tasks.
Recognizing this, Databricks has made significant advancements in providing multiple compute options, allowing users to choose the right infrastructure based on their workload requirements.
|
Compute Types |
Detail |
Use Cases |
|
All-purpose cluster |
- Multi-purpose, persistent compute - Supports multiple users -Ability to fine grain the compute, memory, nodes |
- Collaborative data science |
|
Job Cluster |
- Ephemeral clusters created by jobs -Ability to fine grain the compute, memory, nodes |
- Scheduled ETL/ELT pipelines |
|
SQL Warehouse Serverless |
- Optimized for SQL workloads - Built in Photon engine to optimize SQL queries - Has T-shirt sizing options from 2x-small to 4x-large |
- BI dashboards (e.g., Power BI) |
|
Serverless Compute |
- Databricks manages the infrastructure
|
- Interactive dashboards |
|
Model Serving Compute |
- Scalable REST API endpoints for models |
- Deploying ML models |
|
Instance Pools |
- Compute with idle, ready-to-use instances, used to reduce start and autoscaling times |
- Used for ETL workloads where it is required to reduce the time taken to provision compute |
|
Databricks Apps capacity |
- Compute for run Databricks apps
|
- Compute for applications that are built natively on Databricks |
|
Serverless OLTP Database |
- Serverless Postgres build into the Databricks platform - comes in capacity units of 1,2 and 4 |
- Used for transactional databases for low latency read/write |
Real-World Example: Choosing the Right Compute in a Modern Data and Analytics Landscape
To understand how Databricks' diverse compute options come into play, let’s walk through a real-world example. Imagine an enterprise that operates a Databricks platform to power its end-to-end data, analytics, and AI use cases.
This organization ingests data from multiple external sources to build a layered lakehouse architecture—starting with raw bronze, moving to cleansed silver, and finally creating domain-specific gold data products. These products power AI/ML use cases, analytical dashboards, and even support business applications that require low-latency, transactional access.
Here’s how Databricks' various compute options are utilized across this end-to-end data journey:
General Purpose Compute
Used for developing and testing ETL and AI/ML pipelines, this compute type is ideal for interactive workloads. Data is ingested from external sources via JDBC or preferably through the Databricks Lakehouse Connector, then transformed into structured gold-layer assets and models.
Job Clusters
To run production pipelines, the organization leverages job clusters, which are 30% more cost-effective than general-purpose clusters. These are ephemeral, created for the duration of a job, helping control costs in production environments.
Cluster Pools
Since job clusters take 4–5 minutes to provision, cluster pools are used to minimize job startup time. By pre-warming instances, jobs begin executing almost instantly, improving efficiency and meeting tighter SLA requirements.
Serverless Jobs Compute
Given the variable nature of ETL workloads, a serverless compute option is adopted for elasticity. Databricks automatically provisions the right-sized infrastructure, optimizing resource utilization and minimizing overhead, especially during sudden spikes in data processing
SQL Serverless Warehouses
To support data analysts and reporting workloads, SQL Serverless Warehouses are provisioned. These come with Photon, an advanced vectorized query engine, to accelerate SQL performance for dashboards and ad-hoc querying.
Model Serving Compute
As part of the downstream consumption layer, a model serving endpoint is set up to host and serve AI/ML models. This enables real-time predictions, allowing web applications and business services to interact with models seamlessly via REST APIs.
OLTP Apps Compute
For management reporting and business-facing web apps, the company utilizes Databricks OLTP compute. This compute type accesses data stored in the managed PostgreSQL OLTP database, enabling low-latency transactions and reporting through Databricks-native apps or external services.
Choosing the Right Compute for the Right Task
As illustrated, each Databricks compute option has a well-defined role within the data and analytics ecosystem. By aligning compute capabilities with workload requirements—from ingestion to batch and real-time inference—organizations can optimize for cost, performance, and scalability.
Comparing Databricks Compute Flexibility with Other Leading Platforms
To appreciate the versatility of Databricks’ compute offerings, it’s worth comparing them with similar capabilities from other major data platforms. While many vendors have evolved to support broader workloads across data and AI, Databricks stands out for its fine-grained control, workload-optimized compute types, and truly unified platform.
Here’s how it stacks up:
Snowflake
Snowflake has emerged as a top-tier cloud data warehouse and recently expanded into the AI/ML space through Snowpark. It offers:
- Standard warehouses with T-shirt sizing
- Serverless compute through Snowflake-managed clusters
- Snowpark-optimized warehouses for AI/ML
However, Snowflake lacks the ability to customize the underlying cluster infrastructure—such as VM types, worker counts, or configuration tuning—something Databricks offers extensively through general-purpose and job cluster compute.
GCP BigQuery
BigQuery is known for its fast, serverless query engine and excellent performance on analytical workloads. Key characteristics:
- Fully serverless compute model
- Option to purchase dedicated slots for cost optimization
However, it does not support T-shirt sizing or fine-grained compute configuration, making it less flexible for advanced tuning or multi-modal workloads.
GCP Dataproc
Dataproc offers Apache Spark-as-a-service with strong support for batch and streaming data processing:
- Full control over compute (VM types, node counts, etc.)
- Serverless mode available
While compared to Databricks in terms of raw compute control, it lacks a native analytics or AI layer. For complete functionality, it must be paired with BigQuery (analytics) and Vertex AI (ML)—adding complexity to the stack.
GCP Dataflow
Dataflow is GCP’s serverless data ingestion and transformation service, ideal for streaming and batch pipelines:
- Prebuilt templates and integrations
- Scales automatically based on load
However, it does not offer compute customization and is limited in advanced tuning or use for complex data processing workloads.
AWS Glue
AWS Glue wraps Apache Spark for data ingestion and transformation and provides:
- Serverless execution of Spark jobs
- Integration with other AWS services
Like Dataflow, Glue is optimized for ingestion and processing, but lacks the compute customization and flexibility found in Databricks compute type around analytics and SQL specific workloads.
AWS EMR
EMR is AWS’s equivalent of Dataproc—offering Apache Spark and Hadoop workloads on cloud infrastructure:
- Full control over compute configuration
- Now includes serverless mode
However, like Dataproc, EMR must be coupled with AWS Redshift (for analytics) and Bedrock or SageMaker (for AI/ML) to deliver the full spectrum of capabilities that Databricks provides natively.
AWS Redshift
Redshift is AWS's data warehouse offering:
- Offers both provisioned and serverless clusters
- Supports fine-grained VM and node configuration
While strong in the analytics space, Redshift’s compute options are less flexible and less unified compared to Databricks, particularly for AI/ML workloads.
Microsoft Fabric
Fabric introduces workspace-level capacity provisioning, similar to Power BI Premium:
- Allows T-shirt sizing at the workspace level
- No granular control over VM types or compute resources
Final Thoughts
Databricks leads the pack when it comes to offering a unified, scalable, and highly customizable compute architecture. With options tailored for ETL, streaming, AI/ML, SQL analytics, and transactional workloads, Databricks allows enterprises to:
- Optimize costs
- Scale intelligently
- Serve multiple personas and use cases
Its ability to fine-tune compute per workload, combined with a lakehouse architecture, positions it as a true end-to-end data and AI platform.
AUTHOR - FOLLOW
Maulik Divakar Dixit
Director, Data Engineering, <br>Databricks Champion<br>Databricks MVP
