event-store | Event Sourcing Library for PHP | Microservice library

I see there's some confusion between the various Buses and the Event Store. Before attempting to implement an Event Bus, you need to answer one important question that lies at the foundation of any Event Sourcing implementation:

• How to preserve the Event Store as the Single Source of Truth?

That is, your Event Store contains the complete state of the domain. This also means that the consumers of the Event Bus (whatever it ends up being - a message queue, a streaming platform, Redis, etc.) should only get the events that are persisted. Therefore, the goals become:

• Only deliver events on the Bus that are persisted to the Store (so if you get an error writing to the Store, or maybe a Concurrency Exception, do not deliver via bus!)
• Deliver all events to all interested consumers, without losing any events

These two goals intuitively translate to "I want atomic commit between the Event Store and the Event Bus". This is simplest to achieve when they're the same thing!

So, instead of thinking about how to connect an "Event Bus" to command handlers and send events back and forth, think about how to retrieve already persisted events from the Event Store and subscribe to that. This also removes any dependency between command handlers and event subscribers - they live on different sides of the Event Store (writer vs. reader), and could be in different processes, on different machines.

You can't. The process() method is a terminal operation that does not allow you to emit data downstream. Instead, you can use transform() though (it's basically the same a process() but allows you to emit data downstream); or depending on your app, transformValues() or flatTransform() etc.

Using transform() you get KStream back, that you can write into a topic.

How do you solve this issue?

It's known issue, and of course nor simple timestamps, nor global sequence, nor event naïve methods will not help.
Use vector clock with weak timestamp to enumerate your events and vector cursor to read them. That guarantees some stable deterministic order to intermix events between aggregates. This will work even if each thread has clock synchronization gap, which is regular use case for database clusters, because perfect timestamp synchronization is impossible.
Also this automatically gives possibility to seamless mix reading events from event store and event bus later, and excludes any database locks inter different aggregates events.

Algorithm draft:
1) Determine real quantity of simultaneous transactions in your database, e.g. maximum number of workers in cluster.
Since every event had been written in only one transaction in one thread, you can determine it's unique id as tuple (thread number, thread counter), where thread counter is amount of transactions processed on current thread.
Calculate event weak timestamp as MAX(thread timestamp, aggregate timestamp), where aggregate timestamp is timestamp of last event for current aggregate.

2) Prepare vector cursor for reading events via thread number boundary. Read events from each thread sequentially until timestamp gap exceed allowed value. Allowed weak timestamp gap is trade between event reading performance and preserving native events order.
Minimal value is cluster threads synchronization time delta, so events are arrived in native aggregate intermix order. Maximum value is infinity, so events will be spitted by aggregate. When using RDBMS like postgres, that value can be automatically determined via smart SQL query.

You can see referent implementation for PostgreSQL database for saving events and loading events. Saving events performance is about 10000 events per second for 4GB RAM RDS Postgres cluster.

You could study Konstantin Knauf's Queryable Billing Demo as an example of how to approach some of the issues involved. The central, relevant ideas used there are:

1. Trigger the windows after every event, so that their results are being continuously updated
2. Make the results queryable (using Flink's queryable state API)

This was the subject of a Flink Forward conference talk. Video is available.

Rather than making the results queryable, you could instead stream out the window updates to a dashboard or database.

Also, note that you can cascade windows, meaning that the results of the hourly windows could be the input to the daily windows, etc.

With other answers mentioned the connection and loading using the traditional db2. I have to add some more details that are required specifically for Db2 Event Store.

Assuming we are using a Db2 Client container, which can be found at docker hub with tag ibmcom/db2. Basically we have to go through following steps:

1/ establish a remote connection from db2 client container to the remote db2 eventstore database

2/ use db2 CLP commands to load the csv file using the db2's external table load feature, which will load csv file from db2 client container to the remote eventstore database.

Step 1: Run the following commands, or run the it in a script. Note that the commands need to be run as the db2 user in the db2 client container. The db2 user name is typically db2inst1

I had a very similar issue. Firstly delete (rename) your .babelrc file which you use (remove where ever you set it).

If storybook can't find that file, then it will use its own settings. This worked for me to prove that it was that file that caused the issue.

If this is the same for you then create a new .babelrc file, and place it into the storybook folder. Storybook will now use this and your project can continue to use the existing one.

The tricky part is finding the config setting in you existing .babelrc file that is breaking storybook - for me it was the react-hot-load/babel, but you don't have that listed.

My file ended up with only @babel/plugin-proposal-class-properties and @babel/plugin-proposal-rest-spread for plugins.

Actually, I did not create the stream. Just followed this documentation, it's okay now

https://github.com/prooph/proophessor-do/blob/master/docs/installation/manual.md

if i want to retrieve the users profile given a certain email-address, should i query the read side for the users id, and then query the event-store using this id for the profile data? or should the read side already keep all profile information?

You should use a specially designed ReadModel for searching profiles using the email address. You should query the Event-store only to rehydrate the Aggregates, and you rehydrate the Aggregates only to send them commands, not queries. In CQRS an Aggregate may not be queried.

If the read side has all information, what is the reason for the event-store? If its truly write-only, it's not really useful is it?

The Event-store is the source of truth for the write side (Aggregates). It is used to rehydrate the Aggregates (they rebuild their internal & private state based on the previous emitted events) before the process commands and to persist the new events. So the Event-store is append-only but also used to read the event-stream (the events emitted by an Aggregate instance). The Event-store ensures that an Aggregate instance (that is, identified by a type and an ID) processes only a command at a time.

if the user-model changes (for instance, the profile now includes a description of the profile) and i use a read-side that contains all profile data, how do i update this read side in lagom to now also contain this description?

I don't use any other framework but my own but I guess that you rewrite (to use the new added field on the events) and rebuild the ReadModel.

Following that question, should i keep different read-side tables for different fields of the profile instead of one table containing the whole profile

You should have a separate ReadModel (with its own table(s)) for each use case. The ReadModel should be blazing fast, this means it should be as small as possible, only with the fields needed for that particular use case. This is very important, it is one of the main benefits of using CQRS.

if a different service needs access to the data, should it always ask the user-service, or should it keep its own read side as needed? In case of the latter, doesn't that violate the CQRS principle that the service that owns the data should be the only one reading and writing that data?

Here depends on you, the architect. It is preferred that each ReadModel owns its data, that is, it should subscribe to the right events, it should not depend on other ReadModels. But this leads to a lot of code duplication. In my experience I've seen a desire to have some canonical ReadModels that own some data but also can share it on demand. For this, in CQRS, there is also the term query. Just like commands and events, queries can travel in your system, but only from ReadModel to ReadModel.

Queries should not be sent during a client's request. They should be sent only in the background, as an asynchronous synchronization mechanism. This is an important aspect that influences the resilience and responsiveness of your system.

I've use also live queries, that are pushed from the authoritative ReadModels to the subscribed ReadModels in real time, when the answer changes.

In case of the latter, doesn't that violate the CQRS principle that the service that owns the data should be the only one reading and writing that data?

No, it does not. CQRS does not specify how the R (Read side) is updated, only that the R should not process commands and C should not be queried.

Can microservice has his own database and store data inside it too?

Definitely, microservice can have its own database. But let's use terms from ES/CQRS. Database can represent Event Store (append-only log of immutabale events) and Read Model - some database used to answer queries which is populated by proseccing events.

So, microservice can have its own Read model, populated from events from other microservices.

Or microservice can process commands and save events to the shared Event Store.

Or microservice can process commands and save events to its own Event store.

Choice is yours, and it depends on degree of separation you want to achieve among microservices.

I would put all events that usually consumed together into same Event store. Which means I should be able to query for these events and have a single ordered stream as a result.

when I execute command in microservice and need some data for validation purposes do I need call ReadModel or what?

Command is executed by Aggregate, that has its own state. This state is built by processing all events for this aggregate, and this state should be used to validate a command.

You cannot/should not talk to Read Models in the command handler, primarily because those read models are not consistent with aggregate state. Aggregate state is consistent.

You can query Read Model before sending a command (to make sure it can be sent). But in command handler you need to rely on aggregate state only.

There is a famous case of registering user with requirement of a unique name. As a primary validation, in your UI code you can query read model and tell user that entered name is taken. If name is not taken, UI lets user issue a command. I'm assuming your Aggregate root is user.

But when processing this command ({id:123, type:CREATE_USER, name:somename}) you cannot check that "somename" is taken, because aggregate state for user 123 does not contain a list of taken names. You can potentially query some AllUsernames read model, but it can be milliseconds old, and some other user could take this "somename" already. So in this scenario, you will find a duplication during adding names to read model. And at that point you can do some compensation action - usually issue a command to suspend a user with duplicated name and ask him to re-register or change his name somehow.

It may seems strange, but if you have a really distributed system with several replicas of user list, you'll have the same problem, so why not just embrace the fact that data is always not fully consistent, and just deal with it?