How to Use Kafka Tools With Confluent Cloud

There are times when you may want to use one of the CLI tools that Apache Kafka® provides with your Confluent Cloud cluster.

You can do this by creating a configuration file that contains basic connectivity details such as the bootstrap server and a cluster API key and secret. You can use this file with any Kafka tool that accepts a configuration file. Some of the tools that provide a configuration option, and the option to specify the configuration file are listed in the table that follows:

Kafka Tool Config property option
kafka-configs.sh command-config
kafka-console-producer.sh consumer.config
kafka-console-consumer.sh producer.config
kafka-consumer-groups.sh command-config

Note that you might not be able to use every Kafka tool with every Confluent Cloud cluster type.

Tip

If you install Confluent Platform, you can use all of the Kafka CLI tools in the $CONFLUENT_HOME/bin folder in the same way as the CLI tools that install with Kafka. Confluent has dropped the .sh extension, so you don’t need this when you call the Confluent versions of these tools. For more information, see CLI Tools for Confluent Platform.

Create a configuration file

  1. Create a file named cloud.properties and save it in a secure location. You will populate this file with credentials to access your Confluent Cloud account so you must keep it in a safe place. Add the following content to the file:

    ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
request.timeout.ms=20000
bootstrap.servers=<cloud-bootstrap-server>
retry.backoff.ms=500
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<cloud-key>" password="<cloud-password>";
security.protocol=SASL_SSL

Next, you will populate the file with your Confluent Cloud cluster information:

  1. Sign in to your Confluent Cloud account.

  2. In the Cloud Console, navigate to the cluster you want to work with.

  3. Navigate to the Cluster Settings page and copy the Bootstrap server to the bootstrap server entry in the config file. You also should save the bootstrap server for use with the Kafka tools.

    ../../_images/ccloud-bootstrap-server.png

  4. Next, create an API key and add the key and password to the configuration file. In the Cloud Console, navigate to API Keys, Add key and copy the key to the Username field, and the secret to the Password field in the config file and save the file. For more about how to create a API key for your cluster, including with the Confluent CLI, see Create a resource API key.

Call the tool

After you have set up the configuration file that references your cluster, you can use it with some of the Kafka tools. You will also need the bootstrap server of the Confluent Cloud cluster when you call a tool.

The following example shows you how to write messages to a Confluent Cloud cluster topic named test with the kafka-console-producer tool. In this example, the configuration file and bootstrap server are specified as options. Make sure the topic already exists on the cluster.

./bin/kafka-console-producer.sh --producer.config ../path/to/secure/cloud.properties --bootstrap-server pkc-12345.us-west4.gcp.confluent.cloud:9092 --topic test

Now enter some messages at the prompt (>):

> first event message
> second event message

To test your calls, navigate to the Cloud Console, select the topic, and click the Messages tab. You should see the messages you just produced.

../../_images/ccloud-topic-messages.png

Related content