Stream Governance Packages, Features, and Limits in Confluent Cloud¶
This page provides an overview of Stream Governance packages, including how to upgrade or delete them, regions mappings, and feature support available with each package type.
Getting started with Always On Stream Governance¶
Ready to get started using Stream Governance in your Confluent Cloud environments? Skip to the Quick Start for Schema Management on Confluent Cloud, which begins with explaining how to choose a Stream Governance package when you add a cloud environment and enable Schema Registry, Stream Catalog, and Stream Lineage.
If you first want to learn more about package options, pricing, features, and how they work, read on! This page goes in depth on Stream Governance packages and features.
Governance Package Types¶
Confluent offers two types of governance packages in Confluent Cloud that are applied to your cloud environments. The governance package type you choose determines the features, capabilities, limits, and price of the environment package. Use the information in this topic to find the governance package with the features and capabilities that best meets your needs.
- Use the Essentials package to help you get started with the governance fundamentals.
- For enterprise grade data governance in production workloads, choose the Advanced package.
The table below offers a high-level comparison of features across the governance package types.
| Feature | Essentials | Advanced |
|---|---|---|
| Schema Registry | ✔ (Yes) | ✔ (Yes) |
| Schema Registry SLA | 99.5% | 99.99% |
| Schema Registry calls per second | Read: 75 Write: 25 | Read: 75 Write: 25 |
| Data rules | No | ✔ (Yes) |
| Number of data contracts included [1] | 100 [2] | 20,000 |
| Stream catalog tags | ✔ (Yes) | ✔ (Yes) |
| Stream catalog business metadata | (No) | ✔ (Yes) |
| Stream catalog REST API | ✔ (Yes) | ✔ (Yes) |
| Stream catalog GraphQL API | (No) | ✔ (Yes) |
| Data portal (powered by the Stream Catalog) | ✔ (Yes) | ✔ (Yes) |
| Stream lineage (point in time: last 10 minutes) | ✔ (Yes) | ✔ (Yes) |
| Stream lineage (point in time: last 7 days) | No | ✔ (Yes) |
| AsyncAPI specification export and import | ✔ (Yes) | ✔ (Yes) |
| [1] | Number of data contracts include all schema versions created by users and Confluent components like Kafka Connect and Apache Flink®. If a schema version is soft-deleted but not hard-deleted, it counts towards the number of data contracts. |
| [2] | Data contracts created above this free limit are charged. |
For information on pricing, see Stream Governance packages pricing.
Manage Governance Packages: Use “Essentials” or upgrade to “Advanced”¶
A Stream Governance package is tied to a Confluent Cloud environment. By default, an Essentials package is enabled at cloud environment creation time, but you can also choose the Advanced package at this point. If you choose Essentials, you can upgrade to Advanced later.
The cloud provider and region for the environment Schema Registry and Stream Catalog will be the cloud provider and region of the first cluster you create on the cloud environment.
Note
- Deleting a cloud environment will delete the governance package associated and the corresponding Schema Registry and Stream Catalog.
- You can upgrade to an Advanced governance package, but it is not possible to downgrade from Advanced to Essentials.
Create a cloud environment with a Governance package selection¶
You must select a Stream Governance package as a part of adding a cloud environment. You can choose either Essentials or Advanced. If you choose Essentials, you can upgrade to Advanced later, if needed.
To create an environment with the Cloud Console:
- Log in to Confluent Cloud at https://confluent.cloud.
- Select Environments on the left panel, choose Add cloud environment, provide an environment name in the dialog, and click Create.
- Choose a Stream Governance package to enable Schema Registry, Stream Catalog, and Stream Lineage, either upgrade to Advanced
or accept the Essentials package:
- Upgrade to Stream Governance Advanced starting at $1/hour
- Now, continue with Stream Governance Essentials for free
(Note that the full workflow for the Cloud Console is also provided in Add a cloud environment in the Quick Start for Schema Management on Confluent Cloud.)
To create an environment with the Confluent CLI, use the confluent environment create command.
For example, to create an environment called “my-other-stuff” with the “ESSENTIALS” governance package:
confluent environment create my-other-stuff --governance-package ESSENTIALS
Your output should resemble:
+---------------------------+----------------+
| Current | false |
| ID | env-vnnpnz |
| Name | my-other-stuff |
| Stream Governance Package | ESSENTIALS |
+---------------------------+----------------+
To create an environment with the Confluent Cloud APIs, use the Create an environment API call, and indicate the environment name and governance package you want (ESSENTIALS or ADVANCED).
You will need a Confluent Cloud API key and secret to make these calls. In the examples, these have been stored shell environment variables. (Note that since you are working at the Confluent Cloud environment level, you do not need Schema Registry API keys and secrets for these particular calls.)
For example, to create an environment called “my-other-stuff” with the “ESSENTIALS” governance package:
me$ curl --silent -u $APIKEY_CLOUD:$APISECRET_CLOUD --url https://api.confluent.cloud/org/v2/environments --header 'content-type: application/json' --data '{"display_name":"my-other-stuff","stream_governance_config":{"package":"ESSENTIALS"}}' | jq .
Your output should resemble:
{
"api_version": "org/v2",
"display_name": "my-other-stuff",
"id": "env-2jjwwq",
"kind": "Environment",
"metadata": {
"created_at": "2024-05-01T22:25:26.42437Z",
"resource_name": "crn://confluent.cloud/organization=7ec51216-6104-4493-bf5b-50b11910f748/environment=env-2jjwwq",
"self": "https://api.confluent.cloud/org/v2/environments/env-2jjwwq",
"updated_at": "2024-05-01T22:25:26.42437Z"
},
"stream_governance_config": {
"package": "ESSENTIALS"
. . .
View a cloud environment governance package¶
To view the governance package associated with an environment on the Confluent Cloud Console, navigate to the environment, and view the package indicated under “Stream Governance package” on the right panel.
To view the governance package associated with an environment using the Confluent CLI, use either of the commands, confluent environment list or confluent environment describe:
Use confluent environment list:
me$ confluent environment list
Your output should resemble:
Current | ID | Name | Stream Governance Package ----------+------------+---------------------+---------------------------- | env-1wrq56 | new-destination-env | ESSENTIALS | env-3w2p50 | new-source-env | ESSENTIALS * | env-97wz20 | my-stuff | ADVANCED
Use confluent environment describe:
me$ confluent environment describe env-97wz20
Your output should resemble:
+---------------------------+------------+ | Current | true | | ID | env-97wz20 | | Name | my-stuff | | Stream Governance Package | ADVANCED | +---------------------------+------------+
To view the governance package associated with an environment with the Confluent Cloud REST APIs, see List environments or Read an environment. in the Confluent Cloud API reference.
You will need a Confluent Cloud API key and secret to make these calls. In the examples, these have been stored shell environment variables. (Note that since you are working at the Confluent Cloud environment level, you do not need Schema Registry API keys and secrets for these particular calls.)
For example, use List environments:
curl --silent -u $APIKEY_CLOUD:$APISECRET_CLOUD --request GET --url https://api.confluent.cloud/org/v2/environments | jq .
Your output should resemble the following. Note that listing environments shows all environment IDs:
"api_version": "org/v2",
"data": [
{
"api_version": "org/v2",
"display_name": "my-stuff",
"id": "env-97wz20",
"kind": "Environment",
"metadata": {
"created_at": "2023-02-21T21:03:04.884544Z",
"resource_name": "crn://confluent.cloud/organization=7ec51216-6104-4493-bf5b-50b11910f748/environment=env-97wz20",
"self": "https://api.confluent.cloud/org/v2/environments/env-97wz20",
"updated_at": "2024-04-29T18:35:54.69138Z"
},
"stream_governance_config": {
"package": "ADVANCED"
}
},
{
"api_version": "org/v2",
"display_name": "new-source-env",
"id": "env-3w2p50",
"kind": "Environment",
"metadata": {
"created_at": "2024-03-31T13:30:12.042014Z",
"resource_name": "crn://confluent.cloud/organization=7ec51216-6104-4493-bf5b-50b11910f748/environment=env-3w2p50",
"self": "https://api.confluent.cloud/org/v2/environments/env-3w2p50",
"updated_at": "2024-03-31T13:30:12.042014Z"
},
"stream_governance_config": {
"package": "ESSENTIALS"
}
...
Or, use Read an environment and provide the environment ID:
curl --silent -u $APIKEY_CLOUD:$APISECRET_CLOUD --request GET --url https://api.confluent.cloud/org/v2/environments/env-97wz20 | jq .
Your output should resemble:
"api_version": "org/v2",
"display_name": "my-stuff",
"id": "env-97wz20",
"kind": "Environment",
"metadata": {
"created_at": "2023-02-21T21:03:04.884544Z",
"resource_name": "crn://confluent.cloud/organization=7ec51216-6104-4493-bf5b-50b11910f748/environment=env-97wz20",
"self": "https://api.confluent.cloud/org/v2/environments/env-97wz20",
"updated_at": "2024-04-29T18:35:54.69138Z"
},
"stream_governance_config": {
"package": "ADVANCED"
Upgrade a cloud environment from Essentials to Advanced¶
Once you choose a package for an environment, you cannot downgrade, but you can upgrade from Essentials to Advanced. You can do so, from the Cloud Console, the Confluent CLI, or the Confluent Cloud REST APIs.
To upgrade an environment from Essentials to Advanced on the Confluent Cloud Console, navigate to the environment, and choose Upgrade now under “Stream Governance package” on the right panel.
Features that specifically require the Advanced Stream Governance package provide an upgrade option when you hover over the feature on the right panel. (For example, Business metadata requires the Advanced package.)
To upgrade to an environment from Essentials to Advanced, use confluent environment update.
For example, this command upgrades the the “my-other-stuff” environment from the previous Confluent CLI example (environment ID env-vnnpnz) from Essentials to the Advanced package:
confluent environment update env-vnnpnz --governance-package advanced
Your output should resemble:
+---------------------------+----------------+
| Current | false |
| ID | env-vnnpnz |
| Name | my-other-stuff |
| Stream Governance Package | ADVANCED |
+---------------------------+----------------+
To upgrade from Essentials to Advanced using the Confluent Cloud REST APIs, see Update an Environment in the Confluent Cloud API reference.
You will need a Confluent Cloud API key and secret to make these calls. In the examples, these have been stored shell environment variables. (Note that since you are working at the Confluent Cloud environment level, you do not need Schema Registry API keys and secrets for these particular calls.)
For example, to upgrade the environment “my-other-stuff” from ESSENTIALS to ADVANCED:
curl --silent -u $APIKEY_CLOUD:$APISECRET_CLOUD --request PATCH \
> --url 'https://api.confluent.cloud/org/v2/environments/{env-2jjwwq}' \
> --header 'content-type: application/json' \
> --data '{"display_name":"my-other-stuff","stream_governance_config":{"package":"ADVANCED"}}'
Your output should resemble the following, showing the associated governance package as upgraded to ADVANCED:
{
"api_version": "org/v2",
"display_name": "my-other-stuff",
"id": "env-2jjwwq",
"kind": "Environment",
"metadata": {
"created_at": "2024-05-01T22:25:26.42437Z",
"resource_name": "crn://confluent.cloud/organization=7ec51216-6104-4493-bf5b-50b11910f748/environment=env-2jjwwq",
"self": "https://api.confluent.cloud/org/v2/environments/env-2jjwwq",
"updated_at": "2024-05-01T22:45:17.119634Z"
},
"stream_governance_config": {
"package": "ADVANCED"
}
}
Resilience in Schema Registry and Stream Governance¶
- The Schema Registry cluster is a multi-zone cluster where all nodes are set to leader eligible.
- All Schema Registry nodes are behind a Confluent Cloud Load Balancer.
- The storage used by the Schema Registry cluster is also a multi-zone store, protecting schema data from zonal failures.
Cloud providers and region support¶
Stream Governance Essentials and Advanced packages are available in all Confluent Cloud regions.
You select a Stream Governance package as a part of adding a cloud environment. The Schema Registry cluster is automatically assigned to the same region as the first Kafka cluster deployed in an environment. This Schema Registry region does not change, it persists regardless of clusters created in other regions or deleted. The Schema Registry region determines where Schema Registry and Stream Catalog services will run, and corresponding metadata is stored.
Limitations¶
- You can upgrade to an Advanced governance package, but it is not possible to downgrade from Advanced to Essentials. The only option is to delete the cluster, and start over.
- There are rate limits on API requests to search the Catalog API, which is not designed for continuous requests at upper rate limits. To learn more about alternative strategies to list subjects, schemas, and topics, see Catalog API usage limitations and best practices.
Delete a Schema Registry cluster¶
You cannot delete a Schema Registry cluster directly. The Schema Registry cluster is auto-deleted if you delete an environment.
Deprecation of SRCM v2 clusters and regions APIs and upgrade guide¶
The Schema Registry cluster management (SRCM) v2 regions API and v2 clusters API will no longer be supported after February 2025. Similarly, any related Confluent CLI commands and Confluent Terraform Provider resources and data sources that rely on this API will no longer be supported.
Guidelines on how to upgrade to SRCM API v3, and details on the v2 deprecation timeframes are given below.
Information on the associated Confluent CLI upgrade is provided in Update to Confluent CLI v4.
With regard to Confluent Terraform, see the Confluent Provider 2.0.0: Upgrade Guide.
Deprecation notice for SRCM v2 regions and clusters APIs¶
Support for the SRCM v2 clusters and regions API will end on February 28, 2025. Schema Registry is automatically provisioned in the same region as the first Kafka cluster provisioned in the environment. To learn more about region support, see Cloud providers and region support.
Upgrade to SRCM v3 clusters API¶
You should upgrade from the SRCM v2 to v3 clusters API by February 28, 2025.
- For
GetandListoperations, upgrade to srcm/v3: - For
CreateandDeleteoperations, Schema Registry will automatically be provisioned after the first Kafka cluster in the environment, and deleted during environment deletion. These operations are no longer supported. - For
Updateoperations, update the Stream Governance package using the org/v2 Environments API