Quick Start for Confluent Cloud Provider Integration

Confluent Cloud Provider Integration offers Identity and Access Management (IAM) Role-based authorization that lets you adopt the temporary security credentials of an IAM role, which acts as a set of permission policies. Trusted entities, such as IAM users, applications, or cloud services can assume this role. Using this approach, you can create a secure access connection between source or sink resources on AWS and Confluent Cloud for data ingestion or transfer.

This quick start walks you through the Confluent Cloud Console, CLI and APIs to get you up and running with IAM role-based authorization using Confluent Provider Integration. The quick start assumes that you already have a working Confluent Cloud environment and an AWS account.

Supported connectors

The following connectors support IAM role-based authorization using Confluent Provider Integration:

Source connectors

The following fully-managed source connector supports provider integration setup:

• Amazon DynamoDB CDC Source connector

Sink connectors

The following fully-managed sink connector supports provider integration setup:

• Amazon S3 Sink connector

Prerequisites

• Access to AWS Management Console.
• Access to Confluent Cloud.
• The OrganizationAdmin or EnvironmentAdmin role to setup provider integration. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin.
• In addition to the above role, the ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role can create a connector using the provider integration resource. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin. Use the following example CRN "crn_pattern": "crn://confluent.cloud/organization=org-ID/environment=env-ID/provider-integration=cspi-123j1" to identify the provider integration resources.
• An environment ID for your organization.
• A Confluent Cloud API key to authenticate with the Provider Integration API. For information about how to create a Confluent Cloud API key, see Manage API Keys.
• cURL and jq installed to use the API request examples in this document.

Quota and limits

By default, the resource quota for number of integrations per Organization is set to 100. Contact Confluent Support if you want to increase the quota limit.

Using the Confluent Cloud Console

Configure provider integration

Sign in to your Confluent Cloud account and go through the following instructions to configure your first provider integration in Confluent Cloud.

Step 1: Select a role in AWS

You can create a new role or select an existing role to create a provider integration. In this step, you choose to create a new role to configure your first provider integration.

Note

If you select an existing IAM role, you can proceed directly to mapping the role in Confluent Cloud.

1. Go to the Environments page at https://confluent.cloud/environments and select the environment in which you want to create your first provider integration.
2. Click the Provider integrations tab.
3. Click Add integration. The role screen appears.
4. Select New role to create a new role in your AWS account.
5. Click Continue.

Step 2: Create permission policy in AWS

In this step, you will create an IAM permission policy in your AWS account.

1. Open the AWS console at https://console.aws.amazon.com/iam/ and create a new permission policy. For more information, see Create AWS IAM Policies.
2. In Confluent Cloud console, select a Confluent resource to view the permission policy needed for AWS interaction. Copy the Permission-policy.json and paste it into the policy editor in the AWS console. Save your new IAM permission policy.
3. Click Continue.

Step 3: Create a new role in AWS and map in Confluent

In this step, you will create a new IAM role in your AWS account and map the role in Confluent.

Create a new role in AWSMap the role in Confluent

1. Open the AWS console at https://console.aws.amazon.com/iam/ and create a new IAM role. For more information, see Create AWS IAM Roles.

2. In the Trusted entity type screen, select Custom trust policy. In Confluent Cloud console, copy the Trust-policy.json and paste it into the policy editor in the AWS console. Save your new IAM role.

   Note

   You will need to update the policy with the Confluent-specific configurations such as Confluent ARN and ExternalID after mapping the role in Confluent.

3. Copy the ARN of the IAM role you just created for use in mapping the role in Confluent.

Step 4: Update trust policy in AWS

In this step, you will update the trust policy in the AWS account with Confluent IAM role configurations. This allows the Confluent IAM role to assume the role in your AWS account.

1. In Confluent Cloud console, copy the Trust-policy.json file that includes iam_role_arn, external_id and TagSession for each IAM role association.
2. Open the AWS console at https://console.aws.amazon.com/iam/ and go the trust policy you created in the Step 3. For more information, see Edit trust policies.
3. Paste the updated Trust-policy.json into the trust policy editor in the AWS console.

Create a connector with an IAM role

At the Add Source/Sink connector screen, complete the following in the Authentication step:

1. Under AWS credentials, select IAM Roles in the Authentication method field.
2. Select your provider integration name in the Provider integration name field.

Validate IAM assume role authorization in Confluent

After successful provisioning, your new connector is processing messages. Use the Confluent Cloud Console to view these messages and the resources related to the provider integration.

View topic messagesView provider integration resources

1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click Connectors.
3. Click the connector you just created above.
4. View the messages produced to a topic.

This validates the successful IAM role authorization through Confluent Provider Integration.

Delete resources

When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.

Delete a connectorDelete a provider integration

1. Sign in to your Confluent Cloud account.
2. Click the connector you created and choose the Settings tab.
3. Click Delete connector, enter the connector name, and click Confirm.

Using the Confluent CLI

Step 1: Create an IAM policy and role in AWS

Follow the AWS instructions to create an IAM policy and role in your AWS account.

Step 2: Create an IAM role mapping in Confluent

Enter the following command to set up a provider integration and create an IAM role mapping in Confluent:

confluent provider-integration create <name> [flags]

For example, to create a provider integration named s3-provider-integration, add AWS ARN in --customer-role-arn and environment ID in --environment string.

confluent provider-integration create s3-provider-integration --cloud aws --customer-role-arn arn:aws:iam::000000000000:role/my-test-aws-role --environment env-abcdef

Example output:

+-------------------+--------------------------------------------------------+
| ID                | cspi-12345                                             |
| Name              | s3-provider-integration                                |
| Provider          | aws                                                    |
| Environment       | env-abcdef                                             |
| IAM Role ARN      | arn:aws:iam::990301199020:role/cspi-12345              |
| External ID       | 123456-2312-21bc-n2xx-oo1a7b7890                       |
| Customer Role ARN | arn:aws:iam::037803949979:role/test-iam-role9999999    |
+-------------------+--------------------------------------------------------+

Step 3: Update trust policy in AWS

Follow the AWS instructions to update the trust policy in the AWS account with Confluent IAM role configurations. This allows the Confluent IAM role to assume the role in your AWS account.

Step 4: Create a connector configuration file with provider integration properties in Confluent

Create a JSON file that contains the connector configuration properties with provider integration. Add authentication.method and provider.integration.id in the JSON file. Set authentication.method as IAM Roles in this case.

{
   "name":"Test-S3-Sink-Connector",
   "connector.class":"S3_SINK",
   "kafka.auth.mode":"KAFKA_API_KEY",
   "kafka.api.key":"<my-kafka-api-key>",
   "kafka.api.secret":"<my-kafka-api-secret>",
   "authentication.method":"IAM Roles",
   "provider.integration.id":"dlz-f3a90de",
   "s3.region":"<s3-region>",
   "s3.bucket.name":"<my-iam-s3-bucket>",
   "input.data.format":"JSON",
   "output.data.format":"JSON",
   "time.interval":"DAILY",
   "tasks.max":"1",
   "topics": "<topic-1>, <topic-2>"
}

Step 5: Load the properties file and create the connector in Confluent

Enter the following command to load the JSON and start the connector:

confluent connect cluster create --config-file <file-name>.json

For example:

confluent connect cluster create --config-file s3-sink-config.json

Example output:

Created connector confluent-s3-sink lcc-ix4dl

Step 6: Validate IAM assume role authorization in Confluent

After successful provisioning, verify messages are populating your Kafka topic and view resources related to the provider integration.

Verify messagesView provider integration resources

1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click Connectors.
3. Click the connector you just created above.
4. View the messages produced to a topic.

Step 7: Delete resources

When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.

Delete a connectorDelete a provider integration

Enter the following command to delete the connector:

confluent connect cluster delete --config-file <file-name>.json

Using the Confluent APIs

Step 1: Create an IAM policy and role in AWS

Follow the instructions to create an IAM policy and role in your AWS account.

Create an IAM PolicyCreate an IAM Role

1. Open the AWS console at https://console.aws.amazon.com/iam/.

2. Navigate to Policies, and then click Create policy.

3. In the Policy editor screen, select the JSON option.

4. Copy and paste the S3 or DynamoDB permission policy document in the editor screen.

5. Click Next to review the permissions.

6. In the Review and create page, enter a Policy name and a Description.

7. Click Create policy to save your new IAM policy.

   Note

   For more information about AWS IAM Policies, see Create AWS IAM Policies.

Now you can get started creating IAM roles in your AWS account.

Step 2: Create an IAM role mapping in Confluent

Use Register a provider integration API to set up a provider integration and create an IAM role mapping in Confluent.

1. Add environment ID and customer_iam_role_arn in the POST API request.

   curl --request POST \
    --url https://api.confluent.cloud/pim/v1/integrations \
    --header 'Authorization: Basic <base64-encoded-key-and-secret>' \
    --header 'content-type: application/json' \
    --data '{
      "display_name":"s3_provider_integration",
      "provider":"AWS",
      "config":{
            "customer_iam_role_arn":"arn:aws:iam::000000000000:role/my-test-aws-role",
            "kind":"AwsIntegrationConfig"
        },
      "environment":{
      "id":"env-00000"
     }
   }'
   

2. Make a note of the iam_role_arn, external_id and TagSession generated in the output for each IAM role association. These configurations are used by Confluent when assuming the IAM role in your AWS account.

Note

Use Read a provider integration API to fetch Confluent IAM role configurations in your environment.

Step 3: Update trust policy in AWS

Follow the steps to update the trust policy with Confluent IAM role configurations in the AWS account. This allows the Confluent IAM role to assume the role in your AWS account.

1. Open the AWS console at https://console.aws.amazon.com/iam/

2. Navigate to Roles, and then open the IAM role you created in the Step 1.

3. In Trust relationships tab, click Edit trust policy and update the following configurations:

   • Change Effect to Allow.
   • Add Confluent IAM ARN from Step 2.
   • Add ExternalId from Step 2.
   • Add TagSession from Step 2.

   {
      "Version": "2012-10-17",
      "Statement": [
       {
          "Effect": "Allow",
          "Principal": {
             "AWS": "arn:aws:iam::058264372625:role/cspi-devc4j3k8"
            },
          "Action": "sts:AssumeRole",
          "Condition": {
             "StringEquals": {
                "sts:ExternalId": "19fa38ea-43bb-4307-89e1-2072a694a8ec"
               }
            }
       },
       {
          "Effect": "Allow",
          "Principal": {
             "AWS": "arn:aws:iam::000000000000:role/my-test-aws-role"
            },
          "Action": "sts:TagSession"
        }
      ]
   }
   

4. Click Update policy to update the trust policy.

Step 4: Create a connector with an IAM role in Confluent

Now that you’ve successfully configured a Confluent IAM role that can assume the role in your AWS account, you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.

Make a note of the following:

• You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
• Use Read a provider integration API to fetch provider.integration.id in your Confluent environment.

For example, to create a S3 sink connector, add authentication.method and provider.integration.id in the Create a connector API request. Set authentication.method as IAM Roles in this case.

curl --request POST \
 --url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
 --header 'Authorization: Basic <base64-encoded-key-and-secret>' \
 --header 'content-type: application/json' \
 --data '{
    "name":"Test-S3-Sink-Connector",
    "config":{
    "connector.class":"S3_SINK",
    "name":"Test-S3-Sink-Connector",
    "kafka.auth.mode":"KAFKA_API_KEY",
    "kafka.api.key":"<my-kafka-api-key>",
    "kafka.api.secret":"<my-kafka-api-secret>",
    "s3.region":"<s3-region>",
    "s3.bucket.name":"<my-iam-s3-bucket>",
    "output.data.format":"JSON",
    "time.interval":"DAILY",
    "tasks.max":"1",
    "topics": "<topic-1>, <topic-2>",
    "authentication.method":"IAM Roles",
    "provider.integration.id":"dlz-f3a90de"
    },
 }'

Step 5: Validate IAM assume role authorization in Confluent

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.
3. View the messages produced to a topic, for example, from the AWS S3 bucket.

This validates the successful IAM role authorization through Provider Integration feature.

Step 6: Delete resources

When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.

Delete a connectorDelete a provider integration

1. Sign in to your Confluent Cloud account.
2. Click the connector you created and choose the Settings tab.
3. Click Delete connector, enter the connector name, and click Confirm.

Related content

• Learn about Provider Integration APIs.
• Know more about Provider Integration CLI.
• Learn about Provider Integration resource in the Terraform registry.