concurrency-control | provides easy concurrency control of any promise | Reactive Programming library

You have to choose between two types of isolation levels for your table and the weaker one is the default, so there is no running away from isolation levels. https://docs.databricks.com/delta/optimizations/isolation-level.html

Delta Lake has OCC (Optimistic Concurrency Control) this means that the data you want to write to your table is validated against all of the data that the other 99 processes want to write. This means that 100*100=10000 validations are being made. https://en.wikipedia.org/wiki/Optimistic_concurrency_control

Please also bear in mind that your data processing architecture will finish when the last notebook of the 100 finishes. Maybe one or multiple of the 100 notebooks takes 3 hours to finish and the insert is not to blame?

If long running notebooks is not the case I would suggest you try to store your result data from each notebook in some sort of data structure (e.g. store it in 100 files from each notebook) and then batch insert the data of the data structure (e.g. files) to the destination table.

The data processing will be parallel, the insert will not be parallel.

It's a great distributed system question. Let me break down the problem into sub-parts for readability and even before explain what is _seq_no and _primary_term as there isn't much explanation of those on the ES site.

1. _seq_no is the incremental counter which is assigned to ES document for each operation(update, delete, index), for example:- the first time you index a doc, it will have value 1, next update will have 2, next delete operation will have three and so on. Read operation doesn't update it.
2. _primary_term is the also an incremental counter, but change only when a replica shard is promoted as primary, due to network or any other failure, so if everything is excellent in your cluster it will not be changed, but in case of some failure and other replica promoted to primary then it would be increased.

Coming to the first question,

Q:- What happens to the document in the default case where I don't send the _seq_no and _primary_term values?

Ans:- you can have lost update issue, suppose you have a counter which you are updating, simultaneously 2 requests read the counter value to 1 and trying to increment by 1. now when you don't specify these above terms explicitly, then it's calculated by ES. Now both the requests reach simultaneously to ES, then ES(primary shard) will process them one by one by increasing the sequence number, so at the end, your counter will have value 2, instead of 3. to make sure this doesn't happen, you pass these term values explicitly, and when ES tries to update them will see different sequence number and will reject your request. To prevent such lost updates, use-cases, its always recommended sending explicit version number.

Q:- I'm trying to figure out if I need to send these values to get eventual consistency or if I get it for free without sending those values..

Answer:- These are related to concurrency control and nothing to deal with eventual consistency. In ES, write always happens to primary shards, but read can happen to any replicas(may contain obsolete data), which makes ES eventual consistent.

Important read

https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0

Change your code to have Authorities in authRequest Authentication token like below;

I remember stumbling across the very same problem in a project I was working on. As I could not figure out the solution using the Spring documentation - and other answers on Stack Overflow were not working for me - I ended up creating a workaround.

The trick is essentially to force the application to authenticate the user on a WebSocket connection request. To do that, you need a class which intercepts such events and then once you have control of that, you can call your authentication logic.

Create a class which implements Spring's ChannelInterceptorAdapter. Inside this class, you can inject any beans you need to perform the actual authentication. My example uses basic auth:

I'm new to Spring but looks like your spring application-context file needs something like this-

I figure it out... If I added ?${_csrf.parameterName}=${_csrf.token} to the form action then everything works.

I don't know if it is the best approach, but it works!

Sometime ago I got to some resoneable solution, at some extent. I publish it now, cauze I think it may be usefull to someone else. (Obs.: the solution was implemented using SpringSecurity2.x .)

I resolved this problem finally:

1. Link to similar problem: Spring Security: invalid-session-url versus logout-success-url
2. Official documentation of Spring security: https://docs.spring.io/spring-security/site/docs/3.1.x/reference/springsecurity-single.html#ns-session-mgmt
3. Looking note[8] https://docs.spring.io/spring-security/site/docs/3.1.x/reference/springsecurity-single.html#ftn.d0e1047

[8] If you are running your application behind a proxy, you may also be able to remove the session cookie by configuring the proxy server.