go-cs | Concurrent ssh client -

Function that generates timestamp based guid

So I'm building an ASP.NET-Core API connecting to a mongoDB instance. I was reading through the official Microsoft tutorial regarding this topic and came across the linked code sample.

Basically they instantiate a BookService and create a new instance of MongoClient in the scope of the constructor.

My goal for this app is to create logic that monitors the database and will trigger actions when a document is added to the database (like sending an email). However, since this application may not be started when a database is first populated, how can I manually create a ResumeToken that points to the very first document that was added into a collection, so that on the first run, I can start at the beginning and iterate through the changes until I reach the end. I recognize that I'll need to store the ResumeToken from the lastChangeStreamDocument for future restarts, but I'm interested in the "first run" scenario. I though enumerator.Reset(); was the correct option, but it threw an exception indicating that it wasn't supported.

I've followed the test provided in https://github.com/mongodb/mongo-csharp-driver/blob/master/tests/MongoDB.Driver.Examples/ChangeStreamExamples.cs and have successfully configured a Change Stream with the following code

We have the following MongoDB setup:

Infrastructure

3 replica set running in AWS. At this point, the nodes are all in the same availability zone and are all i3.large instances. 2 of the nodes are hosting the DB data on the NVME drives, and 1 is hosting it on an EBS volume with Provisioned IOPS.

Data

The data set up is a bit dubious, but should work fine according to my understanding of the documentation.

We have a database per customer - about 55 thousand of them.

Each database contains a few collections with account-specific data. There is nothing particularly fancy about the data but some collections do have indices in addition to ones that are there on _id by default.

The Writes

Most of the data that is written are events associated with customer accounts. These are collected into a queue (SQS) and are currently written by a single thread. The writer buffers about 15 minutes worth of data, then determines which database to write the data to, and flushes it. The process is a C# Windows service.

Some write operations bypass the queue (considered high priority). These include account creations and other high priority events.

The Problem

For the most part, this setup works fine. The problem comes when we need to perform an operation on all customer accounts - e.g. delete events that are older than X, or add certain data to every account. In both of these scenarios, the process operates on a list of account IDs, and does its thing.

Both scenarios have about the same problem manifestation. The process starts up fast and goes through many accounts quickly. As it keeps going past ~20k accounts, it starts slowing down. The slowdowns get longer and longer and start affecting DB reads. As of last run, it became unresponsive after ~41k accounts have been processed (but already caused read failures at that point).

The DB itself does response. From the terminal, I am able to get the rs.status() and am able to get rs.printSlaveReplicationInfo(), which is now showing a growing gap between PRIMARY and SECONDARIES.

Connecting to the database from a remote client get stuck on retrieving replica sets.

There is nothing in the logs that stands out on either the PRIMARY or SECONDARIES. A snippet of dump from primary is below.

Any thoughts or ideas?

Thanks!

2018-12-18T18:46:43.238+0000 I NETWORK [conn39304] received client metadata from 172.30.1.180:52756 conn39304: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.8" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:46:44.059+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after locks: 0, after logicalSessionRecordCache: 0, after network: 0, after opLatencies: 0, after opcounters: 0, after opcountersRepl: 0, after repl: 0, after security: 0, after storageEngine: 0, after tcmalloc: 0, after transactions: 0, after transportSecurity: 0, after wiredTiger: 1058, at end: 1058 } 2018-12-18T18:46:44.498+0000 I NETWORK [conn39305] received client metadata from 172.30.1.193:58142 conn39305: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.7" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:46:44.500+0000 I ACCESS [conn39305] Successfully authenticated as principal __system on local 2018-12-18T18:46:44.540+0000 I ACCESS [conn39304] Successfully authenticated as principal __system on local 2018-12-18T18:46:45.758+0000 I COMMAND [PeriodicTaskRunner] task: UnusedLockCleaner took: 243ms 2018-12-18T18:47:22.360+0000 I NETWORK [listener] connection accepted from 172.30.1.180:52758 #39306 (245 connections now open) 2018-12-18T18:47:22.399+0000 I NETWORK [conn39306] received client metadata from 172.30.1.180:52758 conn39306: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.8" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:47:22.401+0000 I ACCESS [conn39306] Successfully authenticated as principal __system on local 2018-12-18T18:47:22.465+0000 I NETWORK [listener] connection accepted from 172.30.1.193:58144 #39307 (246 connections now open) 2018-12-18T18:47:22.539+0000 I NETWORK [conn39307] received client metadata from 172.30.1.193:58144 conn39307: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.7" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:47:22.579+0000 I ACCESS [conn39307] Successfully authenticated as principal __system on local 2018-12-18T18:47:35.372+0000 I ACCESS [conn137] Successfully authenticated as principal __system on local 2018-12-18T18:47:35.374+0000 I ACCESS [conn137] Successfully authenticated as principal __system on local 2018-12-18T18:47:35.377+0000 I ACCESS [conn137] Successfully authenticated as principal __system on local 2018-12-18T18:47:35.554+0000 I ACCESS [conn137] Successfully authenticated as principal __system on local 2018-12-18T18:47:37.797+0000 I ACCESS [conn137] Successfully authenticated as principal __system on local 2018-12-18T18:47:46.685+0000 I NETWORK [listener] connection accepted from 172.30.1.187:33484 #39308 (247 connections now open) 2018-12-18T18:47:46.699+0000 I NETWORK [conn39308] received client metadata from 172.30.1.187:33484 conn39308: { driver: { name: "mongo-csharp-driver", version: "0.0.0.0" }, os: { type: "Windows", name: "Microsoft Windows NT 6.2.9200.0", architecture: "x86_64", version: "6.2.9200.0" }, platform: ".NET Framework 4.5" } 2018-12-18T18:47:46.770+0000 I ACCESS [conn39308] Successfully authenticated as principal tdservice on admin 2018-12-18T18:48:02.362+0000 I NETWORK [listener] connection accepted from 172.30.1.180:52760 #39309 (248 connections now open) 2018-12-18T18:48:02.419+0000 I NETWORK [conn39309] received client metadata from 172.30.1.180:52760 conn39309: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.8" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:48:02.421+0000 I ACCESS [conn39309] Successfully authenticated as principal __system on local 2018-12-18T18:48:02.470+0000 I NETWORK [listener] connection accepted from 172.30.1.193:58146 #39310 (249 connections now open) 2018-12-18T18:48:02.489+0000 I NETWORK [conn39310] received client metadata from 172.30.1.193:58146 conn39310: { driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.7" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } } 2018-12-18T18:48:02.510+0000 I ACCESS [conn39310] Successfully authenticated as principal __system on local

In MongoDB the struct (valuetype) serialization and Deserialization is not possible, because MongoDB throws an Exception: BsonClassMapSerializer.cs line: 84.

But I want to solve this problem generally.

We want to create a library lets call it PolyglotPersistence.Lib. My "clients" use this library to save his datastuctures into Database, which can be MongoDB Azure CosomosDB, or own implemented MemoryDB and some other solution.

But the MongoDB cannot save all kind of data structure, because of the struct problem.

I found some question/answer already in Stackoverflow but these solution are not general solution.

Example 1 How do you serialize value types with MongoDB C# serializer? This is not general at all, When I apply this solution, I have to create a Serialize/Deserializer for every struct. That is ok, it can be done with a generic StructSerializer<>, but still the "clients" have to register it for all struct. Which is not acceptable, because they are don't know where the data will be serialized (Cosmos/Mongo/Memory/etc...).

Example 2 Serializing Immutable Value types with Mongo C# Driver It is nearly the same solution. Must register a special Serializer by "client".

Example 3 Deserialize a nested struct with MongoDB C# driver They change to class, which is not a good way for us.

Possible Solution 1 We create a new rule: When the "client" uses a struct in his data-structure he must be inhertit from a special base class, lets say "IStruct". And we register a serializer for this type, and the problem solved.

But it is littebit uncomfortable for our clients, and not a bulletproof solution.

Possible Solution 2 When the user add a new type for our library (PolyglotPersistence.Lib) then we have to walk-trough on this type recursively, and detect is there the structure in it. When we found it, then we have to register a serializer for this type, when it is not registered yet.

But for this solution we have to find all the struct in clients datastructure.

Possible Solution 3 Register a Serializer for basetype of struct. I don't know is it exists or not. But this will be best solution. The ultimate base class for struct :)

So the questions are:

1. Is there ultimate base class or interface for all struct originally built in c#?
2. If I have System.Type how can I detect it is a struct, with hundred percent safe?

Thank you all answer, and please do not mark this question to duplicate, because the already answered solution are not fit to our problem. And Please, please read the question before mark it. Thank you

P.S. All comment will be appreciated :)

Driver Language - C#

Driver Version - 2.7.0

DB Version - 4.0.0

.NET Framework 4.6.1

Exception message (credentials and server removed)

The connection string 'mongodb+srv://USER:PASS@uat-xxxx.mongodb.net/test?retryWrites=true' is not valid.

Code

I have to convert a log file into a json file to train a unsupervised model. The log file is in format -

I'm going through the mongoDB Driver Documentation Quick Tour for the first time. Specifically the 2.4 version.

I've created a fresh mongodb instance at the 192.168.1.50 address, and it appears to be running correctly.

The MongoDB documentation gives the following example:

This is a follow-up question to this one. What I am trying to achieve is to substitue [JsonConstructor] from my classes in order to get rid of the dependency to Json.Net in projects where I am not doing any serialization. With what I have gathered from other sources, using a ContractResolver is the way to go to implement some kind of "for type Foo use delegate Bar to create an instance of it"-mapping (see MongoDB's class map). What's important to me is, that I want to stick with my readonly fields in my entities and therefore I must use some factory methods with parameters to create instances while the given arguments get assigned to the readonly fields during the creation process.

I have come up with the following solution (I went from having a public parameterless constructor, to a attributed constructor, to a private parameterless constructor, to no attributed constructor at all and with every step I explored the behavior in the overriden methods of my contract resolver):