Enriching events with Amazon EventBridge and AWS Lambda

This year I have been diving deep into the relationship between producers and consumers, and also trying to understand the impacts of event design itself (video) and how it impacts our architecture.

Event design and producer/consumer responsibilities are often over looked when implementing event-driven applications, it’s easy to jump in, start building, and see immediate business value.

Event driven architectures allow us to create decoupled architectures and separate the producer from its consumers. The contracts defined on your events are important, and without contract consideration consumers can often ask the producer for more information, and you end up with the ever growing contract between producer and its consumers.

One way to keep producers contracts simple whilst giving consumers what they want is to use the content enrichment pattern.

In this blog post we will cover why you might want to enrich your events, and how you can get started using AWS Lambda and Amazon EventBridge to enrich events for downstream consumers.

Grab a drink, and let’s get started.

Why enrich events?

Early stages of event-driven applications, the producer may not have many consumers listening to its events. The contracts and relationships between producer and consumer is simple, and if you are lucky enough you may of agreed on some contract between the two (recommended EventStorming or Event Modeling).

Simple contracts between producer and consumer

As time goes on the contract may change, and more consumers may come and go as they please (after-all this is the benefit of event-driven applications). As more consumers are added, the need for more / different data may increase.

Producer with many consumers, consumers unfufilled with data in event.

In the example above, the producer may raise an event but the downstream services need more information to do something useful with this information. Normally they go back to some database or API to get the information required.

Consumers going back to producer to get more information as data not in the event.

If you are happy with this architecture and design, then this may be OK, but what if we could introduce a sort of middleware / enrichment pattern that can help downstream consumers get what they want, and also keep the producer contract simple.

Enrichment pattern enriches data for downstream consumers, keeping contracts simple.

With the enrichment pattern we can keep the producer contract simple and give consumers what they want without the need for direct consumer callbacks. The enricher also abstracts the data fetching requirement away from any consumers or producers keeping the responsibilities of producer or consumer relatively simple.

So how can we implement an enrichment pattern with EventBridge with Lambda?

Let’s dive into the pattern.

Enriching events with Amazon EventBridge and AWS Lambda

Like many things, there are hundreds of ways you can enrich your events (which I explore in other blog posts coming soon).

This pattern uses metadata in the event and custom EventBridge rules to pick up all events that need enriching, once enriched the event is sent back onto the bus for downstream consumers.

Enrichment pattern example with EventBridge and Lambda (enricher)

1. Producer raises an events with enrich:trueinside it’s metadata.

import { PutEventsCommand, EventBridgeClient } from '@aws-sdk/client-eventbridge'
const client = new EventBridgeClient({})

export async function handler(event: any) {
  await client.send(
    new PutEventsCommand({
      Entries: [
        {
          EventBusName: process.env.EVENT_BUS_NAME,
          DetailType: 'OrderCreated',
          Detail: JSON.stringify({
            metadata: {
              // enriched flag set
              enrich: true,
            },
            data: {
              user: {
                // notification event, just the id sent
                id: 'd9df3e81-eafd-4872-b960-adc84d49812e',
              },
            },
          }),
          Source: 'myapp.orders',
        },
      ],
    })
  )
}

2. Custom event bus consumes event

3. Rule is setup on the EventBus listening to all events that have an enrichment flag

// Event Pattern
detail: {
  metadata: {
    // if the event needs enriching then forward it to the enricher
    enrich: [{ exists: true }],
  },
}

4. Lambda function (enricher) takes the event and enriches it talking to external database or API. Once enriched, the enriched flag is removed from the metadata and the event is thrown back onto the event bus. The enricher is opinionated and knows how to enrich based on the event and schema of the event.

export async function handler(event: any) {
  // Take off the enrichment flag first, downstream consumers dont need this.
  delete event?.detail?.metadata?.enrich

  // custom enrich code
  const enrichedEventDetail = await enrich(event)

  await client.send(
    new PutEventsCommand({
      Entries: [
        {
          EventBusName: process.env.EVENT_BUS_NAME,
          DetailType: event['detail-type'],
          Source: event['source'],
          Detail: JSON.stringify(enrichedEventDetail),
        },
      ],
    })
  )
}

5. Another rule is setup to listen to events that do not have the enriched flag. Event is back onto the bus and downstream consumers get the event with enriched data (in this example the user information).

{
    // listen to OrderCreated events
    detailType: ['OrderCreated'],
    detail: {
      metadata: {
        // note: you have to make sure you only consume events without `enrich` flag on.
        enrich: [{ exists: false }],
      },
    }
}

That’s it. We have a pattern that allows us to enrich data for downstream consumers from the schema contract itself.

Things to consider with this pattern

This pattern is mainly a proof of concept, and hopefully shares with you the reasons why you might want to enrich your events and a way you can achieve this, but this pattern is quite opinionated about how you structure your events and rules, so it’s worth considering that.

• Producers have to know to add the enrich flag onto the event
• EventBridge rules have to filer the enrich flag to make sure you don’t pick up events that need enriching
• Extra events and enrichment compute leads to extra cost.
• When you enrich your events, should consumers get all this information? Any sensitive information in there?

Summary

When starting with event-driven architectures the producer and consumer relationship and contract may be simple. Over time this contract changes and requirements from consumers will change. Producers may get pressure to change contracts to suit downstream consumers, or consumers go back to producers (callbacks through API) to get the information they need.

The content enrichment pattern allows us to inject middleware between our producer and consumers. Enrichment allows us to keep our producer contracts simple (leaning towards notification events) and keep our consumer contracts stable with the information they require.

The content enrichment pattern stops the need for consumers to request for information from external sources and can simplify your consumer code, but the enrichment comes with design considerations to think about.

The enrichment pattern adds more process into your producer and consumer flow, and could increase your costs. The pattern shown in this blog post also is opinionated about how you structure your events and rules, which is OK, but something you should consider if using.

Hopefully this blog post gave you insight into the content enrichment pattern, and maybe it’s something you can take into your event-driven applications and start using today.

You can find all the code for this pattern, and more details over at serverlessland.com

I would love to hear your thoughts on the pattern or if you have any other ideas for similar patterns, you can reach me on Twitter.

If you are interested in EventBridge patterns I have more coming out soon, and will be covering patterns like:

• Claim check pattern with EventBridge (Out now)
• Outbox pattern with EventBridge (Out now)
• Enrichment with Step Functions
• Enrichment with multiple event buses
• And more….

Extra resources

• Enterprise integration patterns: Content enrichment pattern (website)
• EventBridge enrichment with AWS Lambda : ServerlessLand Pattern (code/website)
• Code for pattern shown on blog post (code)
• Serverless EventBridge patterns (80+)
• Talk I done around Best practices to design your events in event-driven applications (video)
• Over 40 resources on EventBridge (GitHub)

Until next time.