Data Lake

Collect, store, and index all events in their raw format in perpetuity with complete fidelity and high durability to support auditing, replay, and analytics.

Context, problem, and forces

In our cloud-native system, we have chosen to leverage value-added cloud services to implement our event streaming and our databases. This empowers self-sufficient, full-stack teams to focus their efforts on the requirements of their components and delegate the complexity of operating these services to the cloud provider. We have also architected a topology of multiple event streams to connector our producer components to our consumer components in a well-reasoned manner, which provides proper bulkheads to ensure that a disruption in one stream does not impact the entire system. One side effect of using a cloud-streaming service is that these services only retain events in the stream for a short period of time, usually one to seven days.

In a reactive, cloud-native system, we are effectively turning...