Unlocking the Full Potential of CloudEvents

CloudEvents is an open-source initiative launched by the Serverless Working Group of the CNCF. Work on CloudEvents began in 2017, and since then, the initiative has evolved into a significant standard for exchanging events between cloud applications and services. The idea behind CloudEvents is to create a unified standard for the format and transmission of events, regardless of the underlying technology and cloud platform. The specification is continuously being developed and improved to ensure that it meets the needs of developers and businesses.

As it is a fairly well-known specification, it is already being used by various well-known platforms and technologies, including Google Cloud’s Eventarc, which relies on CloudEvents to enable seamless event-based communication and processing. Similarly, open-source projects like Knative use CloudEvents to build and orchestrate serverless applications and workflows. Additionally, Microsoft Azure has also integrated CloudEvents into its event processing services, further enhancing collaboration and data exchange between different cloud platforms and services. This underscores the relevance and widespread use of CloudEvents as a standard for event data description in the cloud landscape.

Looking at this example using an IoT device, it is immediately apparent that several steps need to be taken before the information reaches a data lake or an analysis tool. Upon closer inspection, one also recognizes different protocols. The transformation costs time and computing power each time. To prevent this, CloudEvents was developed to provide a standardized structure while still using different protocols.

Event Definitions

Definition of Event Groups

Multiple events can be defined within a definition group. A definition is an event. The attributes “id”, “description”, and “format” are important here. Instead of the format “CloudEvent”, one of the following formats can be used alternatively: AMQP, MQTT, Protobuf, HTTP. If an event requires additional attributes, these can be defined via “attributes”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


    "definitionGroups": {
        "Contoso.CRM.Events": {
            "id": "Contoso.CRM.Events",
            "format": "CloudEvents/1.0",
            "definitions": {
                "Contoso.CRM.Events.CustomerCreated": {
                    "id": "Contoso.CRM.Events.CustomerCreated",
                    "description": "An order has been placed",
                    "format": "CloudEvents/1.0",
                    "metadata": {
                        "attributes": {
                            "id": {
                                "type": "string",
                                "required": true
                            },
                            "type": {
                                "type": "string",
                                "value": "Contoso.CRM.Events.CustomerCreated",
                                "required": true
                            },
                            "time": {
                                "type": "datetime",
                                "required": true
                            },
                            "source": {
                                "type": "uritemplate",
                                "value": "/crm/customers",
                                "required": true
                            }
                        }
                    },
                    "schemaFormat": "JSONSchema/draft-07",
                    "schemaUrl": "#/schemaGroups/Contoso.CRM.Events/schemas/customerCreatedEventData"
                },

            }
        }
    },

Endpoints

Endpoints are key components in the world of cloud communication and event processing. They act as interfaces or “destinations” where events are generated, sent, or received. Endpoints represent a superset of definition groups and serve as an extension. This provides additional attributes that allow for a more detailed definition of how to interact with CloudEvents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


"endpoints": {
    "Contoso.CRM.Eventing.Http": {
        "id": "Contoso.CRM.Eventing.Http",
        "usage": "producer",
        "config": {
            "protocol": "HTTP",
            "strict": false,
            "endpoints": [
                "https://erpsystem.com/events"
            ]
        },
        "definitionGroups": [
            "#/definitionGroups/Contoso.CRM.Events"
        ],
        "format": "CloudEvents/1.0"
    }
}

These attributes include:

usage

The usage attribute specifies how the endpoint is used. It can act as a Producer, Consumer, or Subscriber.

Event producers send events to specific endpoints, a process often referred to as the push model. Examples include publishing messages to MQTT or AMQP topics, where events are actively sent to subscribed topics. Another example is posting events to an HTTP webhook, which transmits information to predefined URL endpoints. This push model is essential for real-time communication and event processing in technical applications.

Event consumers consume events from various endpoints. This process is occasionally referred to as the pull model because consumers actively retrieve events from the sources. A clear example of this is the use of protocols such as MQTT or AMQP, where event consumers actively retrieve messages from the corresponding endpoints to process them in real-time. Another approach is polling using HTTP (GET), where event consumers periodically query the endpoints to check for new events. This approach is crucial for implementing real-time data in applications and seamless communication between different components.

A subscriber endpoint provides a subscriber API. Here, the subscriber defines which events should be sent to a specific producer endpoint. This concept enables the targeted selection and transmission of events to desired recipients and is a central element in managing event streams and controlling event processing.

channel

This attribute allows multiple endpoints of the same channel to be related to each other. More specifically, the mapping for a channel is defined.

definitionGroups

Endpoints can refer to other DefinitionGroups. This allows for the creation of a hierarchical structure of CloudEvents definitions.

config

This attribute allows for the configuration of connectivity settings.

CloudEvent Modes in Detail

Binary-Mode

The information about an event is stored in the message body itself. This has the advantage that the transmission can be very efficient without the need for conversion. This is particularly useful when recipients are not familiar with CloudEvents as they can simply ignore the metadata.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


POST /event HTTP/1.0
Host: example.com
Content-Type: application/json
ce-specversion: 1.0
ce-type: com.bnova.item 
ce-source: http: //bonva.com/repo
ce-id: 610b6dd4-c85d-417b-b58f-3771e532
{
"action": "item",
"itemID": "31"
｝

Structured-Mode

In Structured-Mode, CloudEvents are transmitted as structured messages that use a specific format such as JSON or XML. These messages contain both the event data and metadata that provides additional information about the event. This allows the recipient to receive detailed information about the event and perform appropriate actions based on this information. However, transmission in Structured-Mode is slightly slower compared to Binary-Mode because the structured format causes more overhead and thus more data needs to be transmitted. This CloudEvent format, consisting of event metadata and data within the payload, allows for easy forwarding of the same event across multiple routing hops and protocols.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


"specversion": "1.0",
"type": "com.bnova.ittem",
"source": "http://bnova.com/repo",
"id": "610b6dd4-c85d-417b-b58f-3771e532",
"datacontenttype": "application/json",
"data"：
  ｛
    "action": "item",
    "itemID": "31"
  }
｝

Batch-Mode

Batch mode allows multiple CloudEvents to be transmitted in a single message, reducing the number of network connections required and thus enabling more efficient transmission. Another advantage is that the events in the message can be independent of each other and yet be combined in a single transmission. This results in higher scalability. Additionally, batch mode offers the ability to control the order of events in the message and retransmit individual events if necessary. The individual events are encoded according to a specific event format in a single message body, which can be “null” or more events.

General Information on Using CloudEvents

Minimum Set of Required Attributes

The minimum set of data is called context attributes. This is necessary to send the event to the right place.

Required:

id –> Unique identifiability using ID and Source
source –> URI-Reference / Source of the event
specversion –> Determines the CloudEvent version
type –> Determines the type

Optional:

subject
time
datacontenttype
dataschema

Another option is to extend the specification, which makes it possible to pass additional attributes to a message. The prerequisite for this is to comply with the naming convention. Existing extensions are:

The most up-to-date listing can be found here: https://github.com/cloudevents/spec/blob/main/cloudevents/documented-extensions.md.

Event Formats

The serialization of the CloudEvent is defined by the event format, which can include JSON, Protobuf, Avro, and XML, among others. However, not all formats are supported by every SDK. JSON is the most commonly supported format, while other formats such as Avro or Protobuf may not be available in certain SDKs. For example, the Javascript SDK supports JSON but not Avro or Protobuf.

Protocols

In addition, protocol bindings define how the CloudEvent is bound to an application protocol. Supported protocols include HTTP, AMQP, Kafka, MQTT, NATS, and WebSockets. Again, it is important to note that not all protocol bindings are available in every SDK.

SDKs

To write CloudEvents in different languages, the most common SDKs are available:

Use Cases

CloudEvents can be applied in various technology stacks.

AWS Lambda: CloudEvents is a supported format for events emitted by AWS Lambda. This means that Lambda functions can emit CloudEvents to pass events to other services.
Kubernetes: Kubernetes can use CloudEvents to react to cluster events such as scaling or errors.
Knative: Knative, a serverless platform built on Kubernetes. It uses CloudEvents to describe events within the platform.
Azure Event Grid: Azure Event Grid can accept CloudEvents as a supported event format and allows events to be mediated between different Azure services.
Apache Camel: Apache Camel, an open-source integration system. It supports CloudEvents in its latest version to improve the interaction of events flowing through different systems.

Schema Registry

If you now consider all the information as a registry, xRegistry can be used here. This is a project that serves to manage metadata about resources. It provides a REST-based interface to create, edit, delete, and retrieve such resources. All the information we have seen before can be stored in a large repository. Such an example also exists on GitHub.

Here is a practical example of what you can also do with the xRegistry tool.

First, the CLI needs to be installed.

1

pip install git+https://github.com/clemensv/cloudevents-registry-cli.git

This gives you the ability to read code from all the definitions you have created.

The following languages are currently supported. These can also be output using xregistry list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


--languages options:
styles: 
├── py: Python 3.9+
│   └── producer: 
├── asaql: Azure Stream Analytics
│   ├── dispatch: Azure Stream Analytics: Dispatch to outputs by CloudEvent type
│   └── dispatchpayload: Azure Stream Analytics: Dispatch to outputs by CloudEvent type
├── ts: JavaScript/TypeScript
│   └── producerhttp: JavaScript/TypeScript HTTP Producer
├── asyncapi: Async API 2.0
│   └── producer: Async API 2.0 Producer/Publisher
├── openapi: Open API 3.0
│   ├── producer: Open API 3.0 Producer
│   └── subscriber: Open API 3.0 Subscriber
├── java: Java 13+
│   ├── producer: Java CloudEvents SDK endpoint producer class
│   └── consumer: Java CloudEvents SDK endpoint consumer class
└── cs: C# / .NET 6.0+
    ├── azfunctionhttp: C# Azure Function with HTTP trigger
    ├── azfunctioneventgrid: C# Azure Function with Azure Event Grid trigger
    ├── azfunctioneventhubs: C# Azure Function with Azure Event Hubs trigger
    ├── azfunctionservicebus: C# Azure Function with Azure Service Bus trigger
    ├── producer: C# CloudEvents SDK endpoint producer class
    └── consumer: C# CloudEvents SDK endpoint consumer class

The following options are used:

style –> This can be used to specify what type should be created. This available selection can be obtained via xregistry list
language –> This defines the language in which the code should be created. The available options can also be identified using xregistry list.
output
projectname
definition

Example - Java

1

xregistry generate --style producer --language java --output tmp/testJava --projectname crm --definition samples/message-definitions/contoso-crm.cereg

This creates the following files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


.
└── testJava
    ├── main
    │   ├── contoso
    │   │   └── crm
    │   │       └── events
    │   │           ├── EventsEventProducer.java
    │   │           ├── ExceptionMessages.java
    │   │           └── RegexPatterns.java
    │   └── crm
    │       ├── CustomerAddressUpdatedEventData.java
    │       ├── CustomerContactAddedEventData.java
    │       ├── CustomerContactUpdatedEventData.java
    │       ├── CustomerCreatedEventData.java
    │       ├── CustomerDeletedEventData.java
    │       ├── CustomerNoteAddedEventData.java
    │       ├── CustomerNoteDeletedEventData.java
    │       ├── CustomerNoteUpdatedEventData.java
    │       ├── CustomerStatusUpdatedEventData.java
    │       └── CustomerUpdatedEventData.java
    └── pom.xml

Example - OpenAPI

1

xregistry generate --style producer --language openapi --output tmp/testOpenApi --projectname crm --definition samples/message-definitions/contoso-crm.cereg

Here the following definition is created:

1
2


└── testOpenApi
    └── Crm.yml

Conclusion

It can be seen that CloudEvents is gaining increasing importance in the cloud landscape by pursuing the goal of establishing a uniform standard for the transmission of events. The central problem is that producers and cloud providers use different formats, which hinders interoperability across different platforms. CloudEvents attempts to solve this dilemma by providing a specification that facilitates communication between different services, platforms, and systems. Numerous integrations on well-known platforms such as Google Cloud or Azure already underscore the success of this initiative. The use of CloudEvents enables the use of different protocols and formats without compromising compatibility.

This techup has been translated automatically by Gemini