Use AWS VPC Peering with Confluent Cloud¶

AWS VPC peering enables you to route traffic using private IPv4 addresses between your AWS virtual private cloud (VPC) and Confluent Cloud. Your VPC can communicate with Confluent Cloud as if they are within the same network.

Confluent Cloud is available through AWS Marketplace or directly from Confluent.

For more information about VPC peering with AWS, see Introduction to Amazon VPC.

Managed connectors created in a VPC-peered cluster can access data sources and sinks hosted in all peered VPCs, if the firewall rules allow connector traffic to and from the peered VPCs.

When you set up a VPC peering connection between AWS VPC and Confluent Cloud using /27 CIDR blocks, the following Confluent Cloud feature is supported:

Fetch from Follower, a cost optimization to allow clients to consume from the nearest follower, instead of the leader

The high-level workflow to set up a VPC peering connection to Confluent Cloud:

Identify a Confluent Cloud network you want to use, or set up a new Confluent Cloud network.
In Confluent Cloud, create a VPC peering connection.
In AWS, accept the peering request.
In AWS, add the new connection to the route table.
To support outbound connections from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

Requirements and considerations¶

A Confluent Cloud network of the “VPC Peering” type and the “AWS” provider.

Pay special attention that the CIDR blocks you select satisfy the requirements described in Confluent Cloud network CIDR blocks and block size for peering and Transit Gateway.
All AWS availability zones, except use1-az3 in the us-east-1 region, are supported.
Transitive VPC peering is not supported.

If you peer Network A to Network B, and peer Network B to Confluent Cloud, applications running in Network A will not be able to access Confluent Cloud. Although they don’t provide transitive routing, shared AWS VPCs can be leveraged to enable Confluent Cloud connectivity. For more information, see AWS Working with Shared VPCs.

To achieve transitivity, you can link an AWS Transit Gateway to a Confluent Cloud cluster in AWS.
You can have multiple VPC peering connections. For information about limits, see Network quotas in Confluent Cloud.
You can colocate multiple Confluent Cloud Dedicated clusters in the same Confluent Cloud network, but this is limited by the expected number and size of the clusters. The applicable limits are specified in Networks.
Cross-region peering is not supported through the Confluent Cloud Console. Contact Confluent Support to see if your regions are supported and to request configuration.
You might need to increase your route quota when you use VPC peering because the Confluent Cloud and AWS routes are shared.
If you have custom DNS, your DNS servers must be able to access the authoritative DNS servers for Confluent Cloud, which are hosted by Confluent on the internet.
Access to Confluent Cloud serverless products

Connections established for use with Dedicated Kafka clusters may also be used to connect to some serverless products. For service-specific information, see:
- Flink

Create a VPC peering connection¶

This section describes how to create an AWS VPC peering connection to a Confluent Cloud network.

Step 1: Create a VPC peering connection in Confluent Cloud¶

You need to gather the following information from the Amazon VPC Console:

The AWS account ID associated with the VPC you are peering to Confluent Cloud network.
The AWS VPC ID you are peering with Confluent Cloud network.
The CIDR block of the AWS VPC you are peering with the Confluent Cloud network.

Follow the steps below to create a VPC peering connection in Confluent Cloud.

In the Network Management tab of the desired Confluent Cloud environment, click the For dedicated cluster tab.
Click the Confluent Cloud network to which you want to add the peering connection.
In the Ingress connections tab, click + VPC Peering.
Specify the following field values.
- Name: The name of this connection.
- AWS Account Number: The AWS account ID associated with the VPC you are peering to Confluent Cloud network.
- AWS VPC ID: The AWS VPC ID you are peering with Confluent Cloud network.
- AWS VPC CIDR: The CIDR block of the AWS VPC you are peering with the Confluent Cloud network.
Click Add.

Peering connection provisioning will take a few minutes to complete. Your peering connection status will transition from “Provisioning” to “Waiting for connection” in the Confluent Cloud Console.

A peering connection must be created from your VPC to the Confluent Cloud network in order to access Confluent Cloud clusters and services in the Confluent Cloud network.

REST request

POST https://api.confluent.cloud/networking/v1/peerings

REST authentication

See Authentication.

REST request body

{
  "spec":{
   "display_name":"<connection name>",
   "cloud":{
        "kind":"AwsPeering",
        "account":"<AWS account ID>",
        "vpc":"<AWS VPC ID>",
        "routes":[
           "<AWS VPC CIDR>"
        ],
        "customer_region":"<AWS VPC region>"
     },
     "environment":{
        "id":"<environment id>"
     },
     "network":{
        "id":"<Confluent Cloud networkd id>"
     }
  }
}

routes: The CIDR blocks of the VPC you are peering with the Confluent Cloud network. This is used by Confluent Cloud network to route traffic back to your network. The CIDR block must be a private range and cannot overlap with the Confluent Cloud CIDR block.

Use the confluent network peering create Confluent CLI command to create a peering connection:

confluent network peering create aws-peering <flags>

The following command-specific flags are supported:

--network: Required. Confluent Cloud network ID.
--cloud: Required. The cloud provider. Set to aws.
--cloud-account: Required. AWS account ID associated with the VPC that you are peering with Confluent Cloud network.
--virtual-network: Required. AWS VPC ID that you are peering with Confluent Cloud network.
--customer-region: Cloud region ID of the AWS VPC that you are peering with Confluent Cloud network.
--aws-routes: Required. A comma-separated list of CIDR blocks of the AWS VPC that you are peering with Confluent Cloud network.

The CIDR blocks cannot not be identical and not completely within the Confluent Cloud network CIDRs.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a VPC peering:

confluent network peering create aws-peering \
  --network n-123456 \
  --cloud aws \
  --cloud-account 123456789012 \
  --virtual-network vpc-1234567890abcdef0 \
  --aws-routes 172.31.0.0/16,10.108.16.0/21

Step 2: Accept the peering connection request in AWS¶

When the connection status is “Waiting for connection” in the Confluent Cloud Console, go to the Amazon VPC Console and accept the peering request.

You have seven days to accept the request before it expires. For details on accepting peering connections, refer to Create and accept VPC peering connections in Amazon Virtual Private Cloud in the AWS documentation.

If your request has expired, contact Confluent to resend the request. After you have accepted the peering request, the status of the peering connection will change to “Ready”.

In the Amazon VPC Console, click Peering connections in the navigation pane.
Select the pending VPC peering connection (the status is Pending acceptance).

If there is no pending VPC peering connection, verify that you selected the Region of the accepter VPC.
Click Actions, and click Accept request.
When prompted for confirmation, choose Accept request.

Save the connection id. The value is prefixed with pcx-. You need to input this value in the next step when you add the connection to the route table.

When the connection request is accepted, the connection status becomes “Ready” in the Confluent Cloud Console.

Step 3: Add the new connection to the route table in AWS¶

Add the new peering connection that you accepted in Step 2: Accept the peering connection request in AWS, to the route table for your VPC. For details on updating route tables, see the AWS documentation.

For the routing to become effective, the route table must be associated with subnet(s).

Go to the Amazon VPC Console, and click Route Tables.
Select the route table for your VPC and click Edit routes.
Click Add route, and specify the following.

Add routes for all three /27 CIDR blocks to all your VPC route tables, not just the zone-aligned routes.
- Destination: Specify the Confluent Cloud network VPC CIDR block.
  
  You can find the Confluent Cloud network VPC CIDR blocks in the Network Overview of your Confluent Cloud network, in the Confluent Cloud CIDR field.
- Target:
  1. Select Peering connection.
  2. Specify the Peering connection ID of the new connection you accepted in Step 2: Accept the peering connection request in AWS. The value is prefixed with pcx-.
Click Save changes.
Update the network ACLs on your VPCs.

If required, update the security group rules that are associated with your EC2 instance to ensure that traffic can flow between your VPCs and the Confluent Cloud network.

Configure DNS forwarding¶

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VPC or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VPC.

DNS forwarding requires VPC peering or TGW connection where there is bi-directional network access between your network and Confluent Cloud clusters.

Step 1: Get DNS resolver IP addresses¶

To use the DNS forwarding feature with your AWS VPC, you can set up AWS Inbound Endpoints or use your own DNS server:

If you wish to forward DNS requests from Confluent Cloud to a Route53 hosted zone, create Inbound Endpoints for Confluent Cloud network to access your DNS servers.

AWS recommends deploying multiple endpoints in different availability zones for availability reasons.

For details, see Configuring inbound endpoints.

Once the endpoints are created, input the IP addresses of the Inbound Endpoints to which to forward requests as described in the next step.
If you want to use your self-hosted DNS server, use the IP address of that DNS server in Confluent Cloud in the next step.

Step 2: Create a DNS Forwarder in Confluent Cloud¶

Set up DNS forwarding in Confluent Cloud:

In Confluent Cloud, navigate to the DNS Forwarding tab on the Network Detail page.
Input the following information:
- DNS server IPs: Up to 3 IP addresses of your DNS servers to which we should forward DNS requests.
- Domain list: Up to 10 domains to which you wish to route the DNS requests.
Wait until provisioning is complete and DNS is propagated.

Send a request to create a DNS Forwarder resource:

REST request

POST https://api.confluent.cloud/networking/v1/dns-forwarders

REST request body

{
  "spec":
  {
    "display_name": "<The Custom name for the DNS Resolver>",
    "environment":
    {
      "id": "<The Environment ID where the DNS Resolver belongs to>"
    },
    "config":
    {
      "kind": "ForwardViaIp",
      "dns_server_ips": "<A list of IP address(es), up to 3, of DNS server(s) from your VPC>"
    },
    "domains": "<A list of domains, up to 10, for the DNS forwarder to use>",
    "gateway":
    {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

Use the confluent network dns forwarder create Confluent CLI command to set up a DNS forwarder:

confluent network dns forwarder create <dns-forwarder-name> <flags>

The following command-specific flags are supported:

--dns-server-ip: Required. A comma-separated list of IP addresses for the DNS server.
--gateway: Required. Gateway ID. To get the gateway id, run the following CLI command:
```
confluent network describe
```
--domains: A comma-separated list of domains for the DNS forwarder to use.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment and --output.

The following is an example Confluent CLI command to create a DNS forwarder:

confluent network dns forwarder create \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

The following is an example Confluent CLI command to create a named DNS forwarder:

confluent network dns forwarder create  my-dns-forwarder \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

Use the confluent_dns_forwarder Confluent Terraform Provider resource to set up a DNS forwarder.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_forwarder" "main" {
  display_name = "dns_forwarder"
  environment {
    id = confluent_environment.development.id
  }
  domains = ["example.com", "domainname.com"]
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  forward_via_ip {
    dns_server_ips = ["10.200.0.0", "10.200.0.1"]
  }
}

Next steps¶

Try Confluent Cloud on AWS Marketplace with $1000 of free usage for 30 days, and pay as you go. No credit card is required.