Architectural Considerations for Confluent Cloud Clients

While a discussion on different types of architectures deserves much more than this section provides, this page discusses three topics:

  1. Serverless architectures
  2. Stateless microservices
  3. Cloud-native applications

This topic briefly describes the Confluent Cloud client architectural considerations for serverless architectures, stateless microservices, and cloud-native application architectures.

Serverless architectures

Serverless architectures rely extensively on either ephemeral functions reacting to events (FaaS or Lambda) or third party services that are exposed only through API calls. Applications require serverless architectures to be elastic, with usage-based pricing and zero operational burden. As such, Confluent Cloud Basic, Standard, and Enterprise clusters are elastic. They have usage-based pricing that is based on throughput and storage. Other than calling an API or configuring the function, there is no user involvement in failure recovery or upgrades; Confluent is responsible for the availability, reliability, and uptime of your Kafka clusters. Serverless offerings in Confluent Cloud include not only core Kafka broker services but also event stream processing with ksqlDB and data migration into and out of end systems with fully managed connectors. At a high level, this achieves an ETL pipeline: move data into Confluent Cloud (extract), create long-running, auto-scaling streams transformations by publishing SQL to a REST API (transform), and persist this data (load).

Stateless microservices

You also may build your application to speak to Confluent Cloud with stateless microservices. Microservices architectures build applications as a collection of distributed, loosely coupled services, which works well in a cloud environment where the cloud providers themselves give you access to distributed services. Data storage in its many forms is typically handled by external services, whether it’s fully managed Kafka with Confluent Cloud or any of the cloud provider services. This means that the microservices that make up your cloud-native application can be stateless and rely on other cloud services to handle their state. Being stateless also allows you to build more resilient applications, since loss of a service instance doesn’t result in a loss of data because processing can instantly move to another instance.

Cloud-native applications

When using Confluent Cloud, your Kafka client applications may be cloud-native such that they are also running in the cloud. While new applications can be developed on Confluent Cloud from inception, it may be the case that some of your legacy applications migrate to the cloud over time. The path to cloud may take different forms:

  1. Application is cloud native, and is also running in the cloud
  2. Application runs in an on-premise Kafka cluster, and then you have the bridge-to-cloud pattern in which Confluent Replicator streams data between your Kafka cluster and Confluent Cloud
  3. Application runs on prem to Confluent Cloud, and then you migrate the application to cloud over time

Developers who run applications in the cloud for the first time are often surprised by the volatility of the cloud environment. IP addresses can change, certificates can expire, servers are restarted, entire instances are sometimes decommissioned, and network packets going across the WAN are lost more frequently than in most data centers. While it is always a good idea to plan for change, when you are running applications in the cloud, it is mandatory to plan for it. Since cloud environments are built for frequent change, a cloud-native design lets you use the volatile environment to your advantage. The key to successful cloud deployments is to build applications that handle the volatility of cloud environments gracefully, which results in more resilient applications.