Kafka Client Quick Start for Confluent Cloud

You can write Kafka client applications to connect to Confluent Cloud in any supported language. You just need to configure the clients using the Confluent Cloud cluster credentials.

Confluent’s official clients are available for:

This document covers how to properly configure a client. If you have already configured a client and are looking for examples on writing a client application, see the “Hello, World!” code examples that produce to and consume from any Kafka cluster, including Confluent Cloud clusters.

For more information about Kafka clients in general, see Kafka Clients.

Note

All clients that connect to Confluent Cloud must support SASL_PLAIN or SASL_OAUTHBEARER (with OAuth-OIDC configured) and TLS 1.2 encryption.

Warning

Avoid intermediate certificate pinning because it can introduce connection failures. Confluent might change certificates and their properties at any time. If the intermediate certificate you pinned is replaced, then your applications will fail to connect.

If your organization requires pinning, then pin to the Let’s Encrypt ISRG Root X1 certificate instead of any intermediate certificate. Be aware that if Confluent switches CA providers, connections to your applications might be impacted.

Configure clients

Clients must be configured properly to connect to a Confluent Cloud cluster. For a subset of client languages, you can also configure clients to connect to a Confluent Cloud Schema Registry cluster.

You can obtain a configuration file prefilled with necessary credentials either from the Confluent Cloud Console or on the Confluent CLI.

Configure clients from the Confluent Cloud Console

The easiest way to get started connecting your client apps to Confluent Cloud is to copy and paste the configuration file from the Confluent Cloud Console.

  1. Log in to Confluent Cloud.

  2. Select an environment.

  3. Select a cluster.

  4. Select Clients from the navigation menu.

  5. (Optional) Click + New client button.

  6. Select the language you are using for your client application.

    ../_images/cloud-client-languages.png

  7. Once you have selected a language, create or use existing API keys for your Kafka cluster and Schema Registry cluster as needed. Then, copy and paste the displayed configuration into your client application source code.

    ../_images/cloud-client-configuration-example.png

    Tip

    For first time users, you can click on Get started with our example project and tutorial right below the displayed configuration, which walks through not only client configuration but also “Hello, World!” client code example that produces to and consumes from a Confluent Cloud cluster.

Configure clients from the Confluent CLI

For Confluent CLI frequent users, once you have set up context in the CLI, you can use one-line command confluent kafka client-config create to create a configuration file for connecting your client apps to Confluent Cloud.

The following table lists client languages, corresponding language ID, and whether the language supports Confluent Cloud Schema Registry configuration. For languages that support Confluent Cloud Schema Registry configuration, you can optionally configure it for your client apps by passing Schema Registry information via the flags to the command.

Language Language ID Support for Confluent Cloud Schema Registry Notes
Clojure clojure No  
C/C++ cpp No See examples: C/C++ examples (librdkafka)
C# csharp No  
Go go Yes See examples: confluent-kafka-go/examples
Groovy groovy No  
Java java Yes  
Kotlin kotlin No  
Ktor ktor Yes  
Node.js nodejs No Node.js is a compatible client language, but is not supported by Confluent.
Python python Yes See examples: confluent-kafka-python//examples
REST API restapi Yes  
Ruby ruby No  
Rust rust No  
Scala scala No  
Spring Boot springboot Yes  

Tip

If you already have CLI context set up, you can skip directly to step 6 to create the client configuration.

Prerequisites:

  1. Log in to your cluster using the confluent login command with the cluster URL specified.

    confluent login

    Enter your Confluent Cloud credentials:
Email: susan@myemail.com
Password:

  2. Set the Confluent Cloud environment.

    1. Get the environment ID.

      confluent environment list

      Your output should resemble:

            Id       |        Name
+--------------+--------------------+
  * t2703      | default
    env-abc123 | demo-env-102893
    env-xyz123 | ccloud-demo
    env-wxy123 | data-lineage-demo
    env-abc12d | my-new-environment

    2. Set the environment using the ID (<env-id>).

      confluent environment use <env-id>

      Your output should resemble:

      Now using "env-xyz123" as the default (active) environment.

  3. Set the cluster to use.

    1. Get the cluster ID.

      confluent kafka cluster list

      Your output should resemble:

            Id      |   Name    | Type  | Cloud    |  Region  | Availability | Status
+-------------+-----------+-------+----------+----------+--------------+--------+
    lkc-oymmj | cluster_1 | BASIC | gcp      | us-east4 | single-zone  | UP
  * lkc-7k6kj | cluster_0 | BASIC | gcp      | us-east1 | single-zone  | UP

    2. Set the cluster using the ID (<cluster-id>). This is the cluster where the commands are run.

      confluent kafka cluster use <cluster-id>

      To verify the selected cluster after setting it, type confluent kafka cluster list again. The selected cluster will have an asterisk (*) next to it.

  4. Create an API key and secret, and save them.

    You can generate the API key on the Confluent CLI or from the Confluent Cloud Console. Be sure to save the API key and secret.

    1. Run the following command to create the API key and secret, using the ID (<cluster-id>).

      confluent api-key create --resource <cluster-id>

      Your output should resemble:

      It may take a couple of minutes for the API key to be ready.
Save the API key and secret. The secret is not retrievable later.
+---------+------------------------------------------------------------------+
| API Key | ABC123xyz                                                        |
| Secret  | 123xyzABC123xyzABC123xyzABC123xyzABC123xyzABC123xyzABC123xyzABCx |
+---------+------------------------------------------------------------------+

    For more information, see Use API Keys to Authenticate to Confluent Cloud.

  5. Set the API key to use for Confluent CLI commands, using the ID (<cluster-id>).

    confluent api-key use <api-key> --resource <cluster-id>

  6. Create a client configuration file for the language of your choice, using language ID (<language-id>). Then, copy and paste the displayed configuration into your client application source code.

    See Client Language Table for a list of language IDs and whether the language supports Schema Registry configuration.

    Tip

    For the output of command confluent kafka client-config create <language-id>, the client configuration file is printed to stdout and the errors or warnings are printed to stderr.

    You can redirect the command output to separate files by doing the following:

    confluent kafka client-config create <language-id> 1> config-file.config 2> errors-warnings-file.err

    Tip

    If you do not want to use the CLI context, you can alternatively pass all the necessary information via flags:

    confluent kafka client-config create <language-id> \
  --environment <env-id> \
  --cluster <cluster-id> \
  --api-key <api-key> --api-secret <api-secret> \
  --schema-registry-api-key <api-key> --schema-registry-api-secret <api-secret> # only for languages that support Schema Registry configuration

    • For languages that do NOT support Schema Registry configuration, run the following command:

      confluent kafka client-config create <language-id>

    • For languages that support Schema Registry configuration, run the following command:

      confluent kafka client-config create <language-id> \
  --schema-registry-api-key <api-key> \
  --schema-registry-api-secret <api-secret>