Kafka Cluster Types in Confluent Cloud¶
Confluent offers different types of Apache Kafka® clusters in Confluent Cloud. The cluster type you choose determines the features, capabilities, and price of the cluster. Use the information in this topic to find the cluster with the features and capabilities that best meets your needs.
Cluster types¶
Confluent Cloud offers these Kafka cluster types:
- Basic clusters - Used for experimentation, early development and basic use cases.
- Standard clusters - Used for production-ready features and functionality.
- Enterprise clusters - Used for production-ready functionality that requires private networking capabilities.
- Dedicated clusters - Used for critical production workloads with high traffic or private networking requirements.
- Freight clusters - Used for high-throughput, relaxed latency workloads that are less expensive than self-managed open source Kafka.
Freight cluster considerations
- Freight clusters are currently available in select AWS regions. For more information, see Cloud Providers and Regions for Confluent Cloud.
- You must optimize the clients you use to connect to Freight clusters. For more information, see Freight Clients for Confluent Cloud.
- Freight clusters do not support the following:
- Idempotent producer
- Transactions
- Freight clusters do not support the following metrics and functions:
DELETE_RECORDSREPLICA_STATUS- Topic cleanup policy is fixed to delete
Tip
If you need a region that is not yet available, submit a request by providing your requested AWS region on this Freight cluster site.
Cluster provisioning and scaling¶
Confluent uses billing units to provision and scale clusters.
- Elastic scaling
- Basic, Standard, Enterprise, and Freight clusters are elastic, shrinking and expanding automatically based on load. You don’t resize these clusters (unlike Dedicated clusters). When you need more capacity, your cluster expands up to the fixed ceiling. If you’re not using capacity above the minimum, you’re not paying for it. If you’re at zero capacity, you don’t pay for anything.
Freight clusters scaling considerations
Freight clusters are elastic, scaling automatically based on load, so you only pay for the eCKUs you use. However, Freight clusters may not scale as quickly as other eCKU clusters. In general, Freight clusters can support an additional 4 eCKU of capacity (240 MBps of ingress / 720 MBps of egress) every 10 minutes. If you workload grows faster than this, you may experience higher latency or failed requests. As your workload decreases, you pay for your actual eCKU usage.
In scenarios where you expect a large, rapid increase in traffic, consider contacting your account team to get Confluent to work with you to meet your request.
- Manual scaling
- Dedicated clusters are provisioned and billed in terms of Confluent Unit for Kafka (CKU). CKUs are a unit of horizontal scalability in Confluent Cloud that provide a preallocated amount of resources. How much you can ingest and stream per CKU depends on a variety of factors including client application design and partitioning strategy. For more information, see Monitor Dedicated Clusters in Confluent Cloud and Dedicated Cluster Performance and Expansion in Confluent Cloud.
Features¶
All clusters have the following features:
- Kafka ACLs
- Fully-managed replica placement
- User interface to manage consumer lag
- Topic management
- Fully-Managed Connectors
- View and consume Connect logs
- Stream Governance
- Stream Catalog
- Stream Lineage
- Encryption-at-rest
- Encryption-in-transit
- Role-based Access Control (RBAC) (Basic clusters do not support RBAC roles for resources within the Kafka cluster)
Feature comparison table¶
The tables below offer comparisons of the features supported by only some Kafka cluster types.
| Feature | Basic | Standard | Enterprise | Dedicated | Freight |
|---|---|---|---|---|---|
| Exactly Once Semantics | Yes | Yes | Yes | Yes | No |
| Key based compacted storage | Yes | Yes | Yes | Yes | No |
| Custom Connectors | Yes | Yes | No | Yes | No |
| Flink | Yes | Yes | Yes | Yes | No |
| ksqlDB | Yes | Yes | No | Yes | No |
| Public networking | Yes | Yes | No | Yes | No |
| Private networking | No | No | Yes | Yes | Yes |
| OAuth | No | Yes | Yes | Yes | Yes |
| Audit logs | No | Yes | Yes | Yes | Yes |
| Self-managed encryption keys | No | No | Yes | Yes | No |
| Automatic Elastic scaling | Yes | Yes | Yes | No | Yes |
| Stream Sharing | Yes | Yes | No | Yes but all private networking options are not supported | No |
| Client Quotas | No | No | No | Yes | No |
Cluster linking capabilities¶
The table below offers a comparison of cluster linking capabilities by cluster type.
| Cluster type | Basic | Standard | Enterprise | Dedicated | Freight |
|---|---|---|---|---|---|
| Supports source clusters | Yes | Yes | Yes (*) | Yes (*) | No |
| Supports destination clusters | No | No | Yes (*) | Yes (*) | No |
* Capability dependent on the networking type and the other cluster involved. To learn more, see Supported cluster types in the Cluster Linking documentation.
Uptime service level agreement options¶
The table below offers a comparison of uptime service level agreements (SLA) options by cluster type. For more information, see Confluent Cloud Service Level Agreement.
Considerations:
- To obtain a higher uptime SLA, you can upgrade from Basic to a Standard cluster at any time using the Cloud Console.
- Standard and Enterprise clusters require 2 eCKU minimums for the 99.99% SLA.
- Dedicated clusters require Multi-Zone deployments for the 99.99% SLA.
| Cluster type | 99.5% | 99.9% | 99.95% | 99.99% |
|---|---|---|---|---|
| Basic | Yes | No | No | No |
| Standard | No | Yes | No | Yes (Requires 2 eCKU) |
| Enterprise | No | Yes | No | Yes (Requires 2 eCKU ) |
| Dedicated | No | No | Yes (SZ) | Yes (MZ) |
| Freight | No | No | No | Yes |
eCKU/CKU comparison¶
Use the table below to compare limits for a single billing unit for each cluster type.
| Dimension | Basic eCKU | Standard eCKU | Enterprise eCKU | Dedicated CKU | Freight eCKU |
|---|---|---|---|---|---|
| Ingress (MBps) | 5 | 25 | 60 | 60 | 60 |
| Egress (MBps) | 15 | 75 | 180 | 180 | 180 |
| Partitions (pre-replication) | 30 | 250 | 3,000 | 4,500 | 3,000 |
| Number of partitions that you can compact (pre-replication) | 30 | 250 | 360 | 4,500 | None |
| Total client connections | 20 | 1000 | 4,500 | 18,000 | 18,000 |
| Connection attempts (per second) | 5 | 80 | 250 | 500 | 500 |
| Requests (per second) | 100 | 1,500 | 7,500 | 15,000 | 15,000 |
| Kafka REST Produce v3 - Max throughput (MBps): | N/a | N/a | N/a | 50 | N/a |
| Kafka REST Produce v3 - Max connection requests (per second): | N/a | N/a | N/a | 300 | N/a |
| Kafka REST Produce v3 - Max streamed requests (per second): | N/a | N/a | N/a | 3000 | N/a |
| Kafka REST Admin v3 - Max connection requests (per second): | N/a | N/a | N/a | 300 | N/a |
Minimum/maximum eCKU requirements¶
The table below lists minimum and maximum requirements for elastic cluster types.
| Cluster type SKU | Minimum eCKU/CKU | Maximum eCKU/CKU |
|---|---|---|
| Basic | 1 | 50 |
| Standard | 1 (99.9% SLA), 2 (99.99% SLA) | 10 |
| Enterprise | 1 (99.9% SLA), 2 (99.99% SLA) | 10 |
| Freight | 2 | 152 |
CKU purchase limits¶
Dedicated clusters can be purchased in any whole number of CKUs up to the limit.
- For organizations with credit card billing, the upper limit is 4 CKUs per Dedicated cluster. Clusters up to 152 * CKUs are available by request.
- For organizations with integrated cloud provider billing or payment using an invoice, the upper limit is 24 CKUs per Dedicated cluster. Clusters up to 152 * CKUs are available by request.
For clusters that can scale to 152 * CKU, contact Confluent Support to discuss the onboarding process and product considerations.
Single-zone clusters can have 1 or more CKUs, whereas multi-zone clusters, which are spread across three availability zones, require a minimum of 2 CKUs. Zone availability cannot be changed after the cluster is created.
* AWS and Google Cloud support Kafka clusters to 152 CKUs. Azure supports Kafka clusters to 100 CKUs.
Fixed limits and recommended guidelines¶
CKUs determine the capacity of your cluster. For a Confluent Cloud cluster, the expected performance for any given workload is dependent on a variety of dimensions, such as message size and number of partitions.
There are two categories of CKU dimensions:
- Dimensions with a fixed limit that cannot be exceeded.
- Dimensions with a more flexible guideline that may be exceeded depending on the overall cluster load.
The recommended guideline for a dimension is calculated for a workload optimized across the dimensions, enabling high levels of CKU utilization as measured by the cluster load metric. You may exceed the recommended guideline for a dimension, and achieve higher performance for that dimension, usually only if your usage of other dimensions is less than the recommended guideline or fixed limit.
Also note that usage patterns across all dimensions affect the workload and you may not achieve the suggested guideline for a particular dimension. For example, if you reach the partition limit, you will not likely reach the maximum CKU throughput guideline.
You should monitor the cluster load metric for your cluster to see how your usage pattern correlates with cluster utilization.
When a cluster’s load metric is high, the cluster may delay new connections and/or throttle clients in an attempt to ensure the cluster remains available. This throttling would register as non-zero values for the producer client produce-throttle-time-max and produce-throttle-time-avg metrics and consumer client fetch-throttle-time-max and fetch-throttle-time-avg metrics.
Dimensions with fixed limits¶
The following dimensions have fixed limits that you cannot exceed:
- Storage (pre-replication)
- Partitions (pre-replication)
- Connection attempts
- Kafka REST Produce v3
- Kafka REST Admin v3
Dimensions with recommended guidelines¶
These dimensions provide guidelines for capacity planning. The ability to fully utilize these dimensions depend on the workload and utilization of other dimensions. See more about measuring load in cluster load metric and for the maximum bandwidth for each cloud provider (AWS, Google Cloud, Azure), are available in Benchmark Your Dedicated Apache Kafka Cluster on Confluent Cloud.
The following dimensions come with recommended guidelines:
- Ingress
- Egress
- Total client connections
- Requests
Cluster limit comparison¶
Use the table below to compare cluster limits across cluster types.
| Dimension | Basic | Standard | Enterprise | Dedicated | Freight |
|---|---|---|---|---|---|
| Ingress (MBps) * † | 250 | 250 | 600 | 9,120 | 9,120 |
| Egress (MBps) * † | 750 | 750 | 1800 | 27,360 | 27,360 |
| Partitions (pre-replication) * † | 4096 | 4096 | 30,000 | 100,000 | 50,000 |
| Number of partitions you can compact * † | 4096 | 4096 | 3,600 | 100,000 | None |
| Total client connections * † | 1000 | 10,000 | 45,000 | 2,736,000 | 2,736,000 |
| Connection attempts (per second) * † | 80 | 800 | 2500 | 76,000 | 76,000 |
| Requests (per second) * † | 15,000 | 15,000 | 75,000 | 2,280,000 | 2,280,000 |
| Message size (MB) | 8 | 8 | 20 | 20 | 20 |
| Client version (minimum) | 0.11.0 | 0.11.0 | 0.11.0 | 0.11.0 | 0.11.0 |
| Request size (MB) | 100 | 100 | 100 | 100 | 100 |
| Fetch bytes (MB) | 55 | 55 | 55 | 55 | 55 |
| API keys | 50 | 100 | 500 | 2,000 | 500 |
| Partition creation and deletion (per five minute period) | 250 | 500 | 500 | 5,000 | 500 |
| Connector tasks per Kafka cluster | 250 | 250 | 250 | 250 | N/A |
| ACLs | 1,000 | 1,000 | 4,000 | 10,000 | 10,000 |
| Kafka REST Produce v3 - Max throughput (MBps): | 10 | 10 | 10 | 7,600 † | 10 |
| Kafka REST Produce v3 - Max connection requests (per second): | 25 | 25 | 25 | 45,600 † | 25 |
| Kafka REST Produce v3 - Max streamed requests (per second): | 1000 | 1000 | 1000 | 456,000 † | 1000 |
| Kafka REST Produce v3 - Max message size for Kafka REST Produce API (MB): | 8 | 8 | 8 | 20 | 8 |
| Kafka REST Admin v3 - Max connection requests (per second): | 25 | 25 | 25 | 45,600 † | 25 |
* Limit based on Elastic Confluent Unit for Kafka (eCKU). You only pay for the capacity you use up to the limit. For more information, see Elastic Confluent Unit for Kafka.
† Limit based on a Dedicated Kafka cluster with 152 CKU. For more information, see CKU purchase limits and Confluent Unit for Kafka.
The capabilities provided in this topic are for planning purposes, and are not a guarantee of performance, which varies depending on each unique configuration.
Migrate from open source Kafka¶
If you are currently self-managing Kafka, use the following information to help choose which cluster type best suits your use-cases.
- Ingress: Use producer outgoing-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec metrics to understand your throughput.
- Egress: the consumer incoming-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec to understand your throughput.
- Storage (pre-replication): Use the amount of disk space your cluster is using to understand your storage needs.
- Partitions (pre-replication): Use the
kafka.controller:type=KafkaController,name=GlobalPartitionCountmetric to understand your partition usage. Find details in the Broker section. - Total client connections: Use the broker kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metrics to understand how many connections you are using. This value may not have a 1:1 ratio to connections in Confluent Cloud, depending on the number of brokers, partitions, and applications in your self-managed cluster.
- Connection attempts: Use the rate of change for the
kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metric and
the Consumer
connection-creation-ratemetric to understand how many new connections you are creating. For details, see Broker Metrics and Global Connection Metrics. - Requests: Use the broker kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce FetchConsumer FetchFollower} metrics and client request-rate metrics to understand your request volume.
Partition limits¶
The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.
All clusters offer:
- Unlimited storage per partition
- Unlimited storage per partition for compacted topics
Use the table below to compare partition limits across cluster types.
| Dimension | Basic | Standard | Enterprise | Dedicated | Freight |
|---|---|---|---|---|---|
| Ingress per partition | ~5 MBps | ~5 MBps | ~6 MBps | ~12 MBps (aggregate producer throughput) | ~6 MBps |
| Egress per partition | ~15 MBps | ~15 MBps | ~18 MBps | ~36 MBps (aggregate producer throughput) | ~18 MBps |