Use Google Cloud Private Service Connect for Serverless Products on Confluent Cloud¶

Confluent Cloud offers private connectivity for serverless products, specifically Enterprise Kafka clusters. In this private network setup, your Enterprise clusters are only accessible from tenant-specific private endpoints through PrivateLink Attachment.

PrivateLink Attachments on Google Cloud are powered by Google Cloud Private Service Connect.

Confluent Cloud uses the following private networking resources for serverless Confluent Cloud products. These resources are regional and do not have a mapping to specific availability zones.

PrivateLink Attachment

The PrivateLink Attachment (PrivateLinkAttachment) resource represents a reservation to establish a Private Service Connect connection from your virtual private cloud (VPC) to regional services in a Confluent Cloud environment.

A PrivateLink Attachment belongs to an Environment in the Confluent resource hierarchy.

This resource is referred to as a gateway in the Confluent Cloud Console.

PrivateLink Attachment Connection

A PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) is a registration of a specific Private Service Connect endpoint that is allowed to connect to your Confluent Cloud Environment. PrivateLink Attachment Connections belong to a specific PrivateLink Attachment.

This resource is referred to as an access point in the Confluent Cloud Console.

You can use the Confluent Cloud Console, Confluent REST API, Confluent CLI, or Terraform to establish Private Service Connect connectivity to Confluent serverless products, such as Enterprise Kafka clusters.

The high-level workflow is:

In Confluent Cloud, create a PrivateLink Attachment.

PrivateLink Attachments are powered by Private Service Connect.
In Google Cloud, create a Private Service Connect endpoint to the service attachment URI you get in Step 1.

If you use the Confluent Cloud Console for configuration, this step is merged into the next step and shows up as the first and second steps in access point creation.
In Confluent Cloud, create a PrivateLink Attachment Connection for the Private Service Connect endpoint you created.

A PrivateLink Attachment Connection is required for each Private Service Connect endpoint.
Set up DNS resolution.

Requirements and considerations¶

Review the following requirements and considerations before you decide to use a PrivateLink Attachment in Confluent Cloud:

You can connect to only one environment in a region from a VPC or from an on-premises network. A single VPC cannot have PrivateLink Attachment connections to multiple Confluent Cloud environments.
Each PrivateLink Attachment on Google Cloud can only hold up to 10 Private Service Connect endpoints and thus up to 10 PrivateLink Attachment Connections in Confluent Cloud.
Confluent Cloud Console components, such as topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Private Service Connect, see Use the Confluent Cloud Console with Private Networking.
For the regions supported for PrivateLink Attachment on Google Cloud, see Cloud Providers and Regions for Confluent Cloud.

Create a PrivateLink Attachment in Confluent Cloud¶

The PrivateLink Attachment resource, referred as the gateway in the Confluent Cloud Console, represents a reservation to establish a Private Service Connect connection from your VPC to regional services in your Confluent Cloud environment.

When you create a PrivateLink Attachment in an environment and in a region, the PrivateLink Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.

In the Confluent Cloud Console, the PrivateLink Attachment resources are labeled and referred to as gateways.

In the Confluent Cloud Console, select an environment for the PrivateLink Attachment.
In the Network management tab in the environment, click For serverless products.
Click + Add gateway configuration.
In Add Network Configuration page, click PrivateLink and then click Continue.
On the From your VPC or VNet to Confluent Cloud pane, click + Create configuration.
On the Configure gateway sliding panel, enter the following information.
- Gateway name
- Cloud provider: Select Google Cloud.
- Region
Click Submit.
You can continue to create a Private Service Connect endpoint.

Alternatively, you can create an access point at a later time by navigating to this gateway in the Network management tab.

The PrivateLink Attachment will be provisioned and will move to the Waiting for connection state.

A PrivateLink Attachment can be in one of the following states:

WAITING FOR CONNECTION: The PrivateLink Attachment is waiting for a connection to be created.
READY: Google Cloud Private Service Connect connectivity is ready to be used.
EXPIRED: A valid connection has not been provisioned within the allotted time of 7 days. Expired attachments cannot be recovered, and a new PrivateLink Attachment must be provisioned.

Send a request to create a PrivateLink Attachment resource:

REST request

POST https://api.confluent.cloud/networking/v1/private-link-attachments

REST request body

{
  "spec":
  {
    "display_name": "<name of this resource>",
    "cloud": "GCP",
    "region": "<The cloud service provider region of the PrivateLink Attachment>",
    "environment":
    {
      "id": "<environement id>"
    }
  }
}

In the REST response, status.phase should be set to PROVISIONING.

Check the status of the new PrivateLink Attachment:

REST request

GET https://api.confluent.cloud/networking/v1/private-link-attachments/<platt-id>

Use the confluent network private-link attachment create Confluent CLI command to create a PrivateLink Attachment:

confluent network private-link attachment create <attachment-name> <flags>

The following command-specific flags are supported:

--cloud: Required. The cloud provider. Set to gcp.
--region: Required. Google Cloud region where the resources to be accessed using the PrivateLink Attachment.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a private link attachment:

confluent network private-link attachment create my-private-link-attachment \
  --cloud gcp \
  --region us-central1

Use the confluent_private_link_attachment resource to create a PrivateLink Attachment.

An example snippet of Terraform configuration for PrivateLink Attachment:

resource "confluent_private_link_attachment" "main" {
  cloud = "GCP"
  region = "us-central1"
  display_name = "staging-platt"
  environment {
    id = "env-3732nw"
  }
}

See Terraform configuration example for creating a PrivateLink Attachment with ACLs using Terraform.

Provision a Private Service Connect endpoint in Google Cloud¶

In the Google Cloud console, create a Private Service Connect endpoint to the PrivateLink Attachment.

To set up a Private Service Connect endpoint:

In the Google Cloud console, go to the VPC Dashboard on the Private Service Connect page.
In the CONNECTED ENDPOINTS tab, click + CONNECT ENDPOINT for each endpoint you want to create and complete the next two steps.

For details of the process, see Access published services through endpoints.
Specify the following values in the fields:
- Target: Select Published service.
- Target service: Enter the Service attachment URI shown in Step 2 on the Create access point sliding panel in Confluent Cloud.
  
  You can also use the value of the Service Attachment URI from your Network overview of the PrivateLink Attachment gateway in the Confluent Cloud Console.
- Endpoint name: Give the name for this Private Service Connect endpoint.
- Network: Specify the network to place this endpoint.
- Subnetwork: Because subnets are regional resources in Google Cloud, you must select a subnet in the same region and availability zone as your Confluent Cloud network from the Networking tab in the Confluent Cloud Console.
- IP address: Specify the IP address to assign this endpoint.
Click ADD ENDPOINT.

Even if the new endpoint shows as accepted in Google Cloud, you still need to create a PrivateLink Attachment Connection for this endpoint in Confluent Cloud as described in the next section. Without the PrivateLink Attachment Connection, requests to Confluent Cloud will be dropped.

Create a PrivateLink Attachment Connection in Confluent Cloud¶

Create a PrivateLink Attachment Connection resource in Confluent Cloud. A PrivateLink Attachment Connection represents a private endpoint in your VPC.

In the Confluent Cloud Console, the PrivateLink Attachment Connection resources are labeled and referred to as access points.

Confluent will only allow connections to your Environment from endpoints that are registered against your PrivateLink Attachment as connections.

In the Network Management tab of the desired Confluent Cloud environment, click the For serverless products tab.

Make sure the PrivateLink Attachment is in the correct region of the private endpoint.
Click the gateway to which you want to add the Private Service Connect Endpoint.
Click + Create access point in the Access points tab.
In Google Cloud, create a Private Service Connect endpoint to the service attachment URI shown in Step 2 on the sliding panel.
Specify the Private Service Connect connection ID.

The connection ID is the Google Cloud-specific ID of the endpoint you created in the previous step.
Specify the access point name.
Click Create access point to finish the process.

The PrivateLink Attachment and PrivateLink Attachment Connection should now move to the READY state once the connection is accepted.

Send a request to create a PrivateLink Attachment Connection resource:

REST request

POST https://api.confluent.cloud/networking/v1/private-link-attachment-connections

REST request body

{
  "spec":
  {
    "display_name": "<PrivateLinkAttachmentEndpoint name>",
    "cloud":
    {
      "kind": "GcpPrivateLinkAttachmentConnection",
      "private_service_connect_connection_id": "<ID of the private endpoint you created in Google Cloud>",
    },
    "environment":
    {
      "id": "<Environment ID>",
    },
    "private_link_attachment":
    {
      "id": "<PrivateLinkAttachment ID>",
    }
  }
}

REST response example

{
  "api_version": "networking/v1",
  "kind": "PrivateLinkAttachmentConnection",
  "id": "plattc-xyzuvw",
  "status": {
    "phase": "PROVISIONING",
    "error_code": "",
    "error_message": "",
  }
}

status.phase is PROVISIONING because a private endpoint connection has not yet been accepted.

Check the status of the new PrivateLink Attachment Connection:

REST request

GET https://api.confluent.cloud/networking/v1/private-link-attachment-connections/<platt-id>

Use the confluent network private-link attachment connection create Confluent CLI command to create a PrivateLink Attachment Connection:

confluent network private-link attachment connection create <connection-name> <flags>

The following command-specific flags are supported:

--cloud: Required. The cloud provider. Set to gcp.
--endpoint: Required. ID of an Google Cloud private endpoint that is connected to the Google Cloud Private Service Connect service.
--attachment: Required. PrivateLink Attachment ID.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a private link attachment connection:

confluent network private-link attachment connection create gcp-private-link-attachment-connection \
  --cloud gcp \
  --endpoint  1234567890123456 \
  --attachment platt-123456

Use the confluent_private_link_attachment_connection resource to create a PrivateLink Attachment Connection.

An example snippet of Terraform configuration for PrivateLink Attachment Connection:

resource "confluent_private_link_attachment_connection" "gcp" {
  display_name = "my_endpoint"
  environment {
    id = "env-8gv0v5"
  }
  gcp {
    private_service_connect_connection_id = "1234567890123456"
  }
  private_link_attachment {
    id = "platt-plyvyl"
  }
}

In the Google Cloud console, check the status of the Private Service Connect endpoint.

Go to the private endpoint resource in the Google Cloud console and verify that the private endpoint connection status is “Approved”.

Important

If you delete the Private Service Connect endpoint you created in Google Cloud, you should also delete the corresponding PrivateLink Attachment Connection in Confluent Cloud to avoid the security risk arising from Google Cloud reusing the Private Service Connect Connection ID.

Set up DNS resolution in Google Cloud¶

Set up DNS resolution with a DNS zone and a DNS record in Google Cloud.

In Confluent Cloud, verify that the status of the PrivateLink Attachment Connection is “READY”.
In Confluent Cloud, open the newly created PrivateLink Attachment to get the DNS domain value of Confluent Cloud.

The value is in the <region>.gcp.private.confluent.cloud pattern.

To create the DNS zone and a DNS record for the Private Service Connect:

In the Google Cloud console, browse to the Cloud DNS page.
Create a DNS zone.
1. Click + CREATE ZONE.
  - Zone Type: Select Private.
  - Zone name: Specify a name for this DNS zone.
  - DNS name: Specify the Confluent Cloud DNS Domain. You can find this value in the Confluent Cloud Console on the Confluent Cloud network detail page.
2. Click CREATE.
Create a DNS record.
1. Go to the DNS zone you created in the previous step.
2. In the RECORD SETS section, click + ADD STANDARD. The Create record set page appears.
3. Create a DNS record for your Confluent Cloud network.
  - DNS name: *
  - Resource record type: A
  - TTL: 1
  - TTL unit: minutes
  - IPv4 Address: The IP address you specified for the Private Service Connect endpoint in Create a PrivateLink Attachment Connection in Confluent Cloud.
4. Click CREATE.
5. Verify the Private Zone is in use by the desired Network(s).