Document Database

Explore all libraries in Document Database

Explore all Document Database open source software, libraries, packages, source code, cloud functions and APIs.

Trending Discussions on Document Database

Why does async IO block in C#?

Construct an object that has member functions from an object that has no member function with their properties are the same?

How to store and organize swagger files?

DbContext in ResolveWith, HotChocolate GraphQL

TypeScript, private properties with getters and json

How To: Get latest (newest) document from Cosmos DB?

Non-Relational Database Design For Big Data Warehouse

Would Elasticsearch work better with a document database like MongoDB or a relational database like MySQL?

How do I update a MongoDB document with new value using reactors Mono? (Kotlin)

How do I store a picture in azure blob storage in asp.net mvc application?

Why does async IO block in C#?

Construct an object that has member functions from an object that has no member function with their properties are the same?

How to store and organize swagger files?

DbContext in ResolveWith, HotChocolate GraphQL

TypeScript, private properties with getters and json

How To: Get latest (newest) document from Cosmos DB?

Non-Relational Database Design For Big Data Warehouse

Would Elasticsearch work better with a document database like MongoDB or a relational database like MySQL?

How do I update a MongoDB document with new value using reactors Mono? (Kotlin)

How do I store a picture in azure blob storage in asp.net mvc application?

QUESTION

Why does async IO block in C#?

Asked 2022-Mar-20 at 20:14

I've created a WPF app that targets a local document database for fun/practice. The idea is the document for an entity is a .json file that lives on disk and folders act as collections. In this implementation, I have a bunch of .json documents that provide data about a Video to create a sort of an IMDB clone.

I have this class:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19

Way up the call stack, though the API layer and into the UI layer, I have an ICommand in a ViewModel:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31

The goal is to show the user a message 'Querying...' while the query is being processed. However, that message is never shown and the UI locks up until the query is complete, at which point the result message shows.

In VideoRepository, if I comment out the foreach loop and uncomment the two lines below it, the UI doesn't lock up and the 'Querying...' message gets shown for 5 seconds.

Why does that happen? Is there a way to do IO without locking up the UI/blocking?

Fortunately, if this were behind a web API and hit a real database, I probably wouldn't see this issue. I'd still like the UI to not lock up with this implementation though.

EDIT: Dupe of Why File.ReadAllLinesAsync() blocks the UI thread?

Turns out Microsoft didn't make their async method very async. Changing the IO line fixes everything:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31//var json = await File.ReadAllTextAsync(file.FullName); // Bad
32var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName)); // Good
33

ANSWER

Answered 2022-Mar-20 at 19:18

You are probably targeting a .NET version older than .NET 6. In these old versions the file-system APIs were not implemented efficiently, and were not even truly asynchronous. Things have been improved in .NET 6, but still the synchronous file-system APIs are more performant than their asynchronous counterparts. Your problem can be solved simply by switching from this:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31//var json = await File.ReadAllTextAsync(file.FullName); // Bad
32var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName)); // Good
33var json = await File.ReadAllTextAsync(file.FullName);
34

to this:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31//var json = await File.ReadAllTextAsync(file.FullName); // Bad
32var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName)); // Good
33var json = await File.ReadAllTextAsync(file.FullName);
34var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName));
35

If you want to get fancy, you could also solve the problem in the UI layer, by using a custom LINQ operator like this:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31//var json = await File.ReadAllTextAsync(file.FullName); // Bad
32var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName)); // Good
33var json = await File.ReadAllTextAsync(file.FullName);
34var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName));
35public static async IAsyncEnumerable&lt;T&gt; OnThreadPool&lt;T&gt;(
36    this IAsyncEnumerable&lt;T&gt; source,
37    [EnumeratorCancellation] CancellationToken cancellationToken = default)
38{
39    var enumerator = await Task.Run(() =&gt; source
40        .GetAsyncEnumerator(cancellationToken)).ConfigureAwait(false);
41    try
42    {
43        while (true)
44        {
45            var (moved, current) = await Task.Run(async () =&gt;
46            {
47                if (await enumerator.MoveNextAsync())
48                    return (true, enumerator.Current);
49                else
50                    return (false, default);
51            }).ConfigureAwait(false);
52            if (!moved) break;
53            yield return current;
54        }
55    }
56    finally
57    {
58        await Task.Run(async () =&gt; await enumerator
59            .DisposeAsync()).ConfigureAwait(false);
60    }
61}
62

This operator offloads to the ThreadPool all the operations associated with enumerating an IAsyncEnumerable<T>. It can be used like this:

1public class VideoRepository : IVideoRepository
2{
3    public async IAsyncEnumerable&lt;Video&gt; EnumerateEntities()
4    {
5        foreach (var file in new DirectoryInfo(Constants.JsonDatabaseVideoCollectionPath).GetFiles())
6        {
7            var json = await File.ReadAllTextAsync(file.FullName); // This blocks
8            var document = JsonConvert.DeserializeObject&lt;VideoDocument&gt;(json); // Newtonsoft
9            var domainObject = VideoMapper.Map(document); // A mapper to go from the document type to the domain type
10            yield return domainObject;
11        }
12        // Uncommenting the below lines and commenting out the above foreach loop doesn't lock up the UI.
13        //await Task.Delay(5000);
14        //yield return new Video();
15    }
16
17    // Rest of class.
18}
19QueryCommand = new RelayCommand(async (query) =&gt; await SendQuery((string)query));
20
21private async Task SendQuery(string query)
22    {
23        QueryStatus = &quot;Querying...&quot;;
24        QueryResult.Clear();
25
26        await foreach (var video in _videoEndpoints.QueryOnTags(query))
27            QueryResult.Add(_mapperService.Map(video));
28
29        QueryStatus = $&quot;{QueryResult.Count()} videos found.&quot;;
30    }
31//var json = await File.ReadAllTextAsync(file.FullName); // Bad
32var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName)); // Good
33var json = await File.ReadAllTextAsync(file.FullName);
34var json = await Task.Run(() =&gt; File.ReadAllText(file.FullName));
35public static async IAsyncEnumerable&lt;T&gt; OnThreadPool&lt;T&gt;(
36    this IAsyncEnumerable&lt;T&gt; source,
37    [EnumeratorCancellation] CancellationToken cancellationToken = default)
38{
39    var enumerator = await Task.Run(() =&gt; source
40        .GetAsyncEnumerator(cancellationToken)).ConfigureAwait(false);
41    try
42    {
43        while (true)
44        {
45            var (moved, current) = await Task.Run(async () =&gt;
46            {
47                if (await enumerator.MoveNextAsync())
48                    return (true, enumerator.Current);
49                else
50                    return (false, default);
51            }).ConfigureAwait(false);
52            if (!moved) break;
53            yield return current;
54        }
55    }
56    finally
57    {
58        await Task.Run(async () =&gt; await enumerator
59            .DisposeAsync()).ConfigureAwait(false);
60    }
61}
62await foreach (var video in _videoEndpoints.QueryOnTags(query).OnThreadPool())
63    QueryResult.Add(_mapperService.Map(video));
64

Source https://stackoverflow.com/questions/71549393

QUESTION

Construct an object that has member functions from an object that has no member function with their properties are the same?

Asked 2021-Dec-17 at 04:43

I am trying to build a javascript app with a document database that can store and retrieve object data (provided as api) without function members.

Now I have a class which have many properties and some functions as prototype.

1Project: function(){
2  this.a= 'abc';
3  this.f = function(){
4    console.log(this.a);
5  }
6}
7//object from the databse
8p0 = {
9  a: 'abc';
10}
11

I want to convert a plain object to an object with member function usable.

When I try something like this, it won't work:

1Project: function(){
2  this.a= 'abc';
3  this.f = function(){
4    console.log(this.a);
5  }
6}
7//object from the databse
8p0 = {
9  a: 'abc';
10}
11// It won't work:
12// for it needs a map that have many properties such as writable etc.
13var pobj = Object.create(new Project(), p0);
14

I tried to search this question with different keywords on the internet, but I didn't find one related.

ANSWER

Answered 2021-Dec-17 at 04:43

The function you probably want to use is Object.assign.

So you can use Object.create to create an instance of the class (using the prototype), then assign the values from the data onto the new instance and then call the constructor (make sure to not override the values).

1Project: function(){
2  this.a= 'abc';
3  this.f = function(){
4    console.log(this.a);
5  }
6}
7//object from the databse
8p0 = {
9  a: 'abc';
10}
11// It won't work:
12// for it needs a map that have many properties such as writable etc.
13var pobj = Object.create(new Project(), p0);
14function Project() {
15  if (this.foo === undefined) this.a = 'foobar';
16  return this;
17}
18
19Project.prototype.print = function() {
20  console.log(this.foo);
21};
22
23var data = {
24  foo: 'bar'
25};
26
27var obj = Object.create(Project.prototype); // empty instance
28obj = Object.assign(obj, data); // copy data
29obj = obj.constructor(); // start constructor
30
31obj.print();

Alternatively you could create a new instance using the new operator and then assign the data.

1Project: function(){
2  this.a= 'abc';
3  this.f = function(){
4    console.log(this.a);
5  }
6}
7//object from the databse
8p0 = {
9  a: 'abc';
10}
11// It won't work:
12// for it needs a map that have many properties such as writable etc.
13var pobj = Object.create(new Project(), p0);
14function Project() {
15  if (this.foo === undefined) this.a = 'foobar';
16  return this;
17}
18
19Project.prototype.print = function() {
20  console.log(this.foo);
21};
22
23var data = {
24  foo: 'bar'
25};
26
27var obj = Object.create(Project.prototype); // empty instance
28obj = Object.assign(obj, data); // copy data
29obj = obj.constructor(); // start constructor
30
31obj.print();function Project() {
32  this.a = 'foobar';
33  return this;
34}
35
36Project.prototype.print = function() {
37  console.log(this.foo);
38};
39
40var data = {
41  foo: 'bar'
42};
43
44var obj = Object.assign(new Project(), data);
45
46obj.print();

Unrelated note It's usually a good idea to declare public functions that don't require closure outside the function body using class.prototype.functionName = function(){...}

Source https://stackoverflow.com/questions/70388410

QUESTION

How to store and organize swagger files?

Asked 2021-Mar-29 at 09:02

Suppose I need to store and organize API documentation for ~1000 REST APIs. Each API provides a swagger file in JSON format and text files with images for guides, tutorials, and code examples. Users may update the documentation, browse and query it by API names, URLs, request fields, etc., and also search the text files.

I can probably store the text files and images in some cloud storage (e.g. S3) and create a full-text search index. I wonder how to store the swagger files. Should I store them just as the text files in a cloud storage or store them in a document database (e.g. Mongo) ?

What do you think ?

ANSWER

Answered 2021-Mar-28 at 18:01

what do you mean by "swagger files" exactly? the generated output from openAPI schema files?

I'm wondering why would you want to host that in a database? wouldn't be so much easier to have an S3 bucket per "REST API" and make that bucket a static web server instead? making it automatically accessible and never dependent on a connection to a secondary service?

you might still want to use the DB to organize user/editors/admins, but ultimately I would simply host everything in a CDN and make sure to override when a new update is generated and clear the cache (CloudFront), you can even have "folders" to host a version in that bucket so user could easily revert back to a previous version if any mistake...

won't you agree?

Source https://stackoverflow.com/questions/66844273

QUESTION

DbContext in ResolveWith, HotChocolate GraphQL

Asked 2021-Jan-29 at 09:22

So I just get started, I have an entity which keeps the data from other entities not by a direct relation but with keeping the EntityId and EntityType(Enum). When I read these records from GraphQL I expect to resolve a field with a resolver as follow,

1        public class AssignmentResolver
2        {
3            public object GetEntity( Assignment assignment, AppDbContext context)
4            {
5                        if(assignment.EntityType == AssignmentEntityType.PERSON) 
6                       {
7                                return context.People.FirstOrDefault(x =&gt; x.Id == assignment.EntityId);
8                       }
9                       // And more checks
10                      return null;
11            }
12        }
13

Then I can say

1        public class AssignmentResolver
2        {
3            public object GetEntity( Assignment assignment, AppDbContext context)
4            {
5                        if(assignment.EntityType == AssignmentEntityType.PERSON) 
6                       {
7                                return context.People.FirstOrDefault(x =&gt; x.Id == assignment.EntityId);
8                       }
9                       // And more checks
10                      return null;
11            }
12        }
13    public class AssignmentQueryType: ObjectType&lt;Assignment&gt;
14    {
15        protected override void Configure(IObjectTypeDescriptor&lt;Assignment&gt; descriptor)
16        {
17            descriptor.Field(&quot;entity&quot;).ResolveWith&lt;AssignmentResolver&gt;(x =&gt; x.GetEntity(default!, default!));
18        }
19    }
20

I wanna know if this is right or is there a better way... I mean the better way would be using a document database for this but that's not an option for now. I also maybe instead of putting the EntityType and EntityId can simply set an actual relation to those other entities but I wanna see if this current way is possible.

ANSWER

Answered 2021-Jan-29 at 09:22

Well that was fast.

I found my problem. It seems that in the resolver I cannot just return an object because the schema should be clear when being read.

So from the resolver if I return a viewModel which is shared between all those entities then we are good to go.

So the GetEntity code will change to

1        public class AssignmentResolver
2        {
3            public object GetEntity( Assignment assignment, AppDbContext context)
4            {
5                        if(assignment.EntityType == AssignmentEntityType.PERSON) 
6                       {
7                                return context.People.FirstOrDefault(x =&gt; x.Id == assignment.EntityId);
8                       }
9                       // And more checks
10                      return null;
11            }
12        }
13    public class AssignmentQueryType: ObjectType&lt;Assignment&gt;
14    {
15        protected override void Configure(IObjectTypeDescriptor&lt;Assignment&gt; descriptor)
16        {
17            descriptor.Field(&quot;entity&quot;).ResolveWith&lt;AssignmentResolver&gt;(x =&gt; x.GetEntity(default!, default!));
18        }
19    }
20            public EntityViewModel GetEntity( Assignment assignment, [Service] AppDbContext context)
21            {
22                        if(assignment.EntityType == AssignmentEntityType.PERSON) 
23                       {
24                                var entity = context.People.FirstOrDefault(x =&gt; x.Id == assignment.EntityId);
25                                return new EntityViewModel(entity);
26                       }
27                       // And more checks
28                      return null;
29            }
30

Source https://stackoverflow.com/questions/65951533

QUESTION

TypeScript, private properties with getters and json

Asked 2020-Aug-25 at 13:22

I'm pretty new to JavaScript/Typescript overall and I've run into an issue with serialization of classes that I don't know how to handle.

I create objects from REST or database requests requests like this:

1export interface ILockerModel {
2    id: string
3    lockerId: string
4    ownerId: string
5    modules: IModuleModel[]
6}
7

1export interface ILockerModel {
2    id: string
3    lockerId: string
4    ownerId: string
5    modules: IModuleModel[]
6}
7export class LockerModel implements ILockerModel {
8    private _id: string
9    private _lockerId: string
10    private _ownerId: string
11    private _modules: ModuleModel[]
12
13    constructor(document: ILockerModel) {
14        this._id = document.id
15        this._lockerId = document.lockerId
16        this._ownerId = document.ownerId
17        this._modules = document.modules.map(m =&gt; new ModuleModel(m))
18    }
19    // Utility methods
20}
21

I then have multiple utility methods that make it easier to work with the model, adding and removing things from lists and so on.

When I'm done I want to persist the object to a document database or return it in a REST response, so I call JSON.stringify(objectInstance). However, this gives me the class but with all properties underscored (_), not my getter values. This breaks the deserializtion in other parts of my application.

Serializing the interface gives me what I want, but I haven't found a straightforward way to go from the class to the interface representation. The issue gets tougher because I deserialize the data in a hierarchy (see the modules mapping in the constructor).

How do you usually solve this issue?

ANSWER

Answered 2020-Aug-25 at 13:19

As far as I can see you do not really implement the ILockerModel. Shouldnt this throw an error?

When I run it, I get the following:

Type 'LockerModel' is missing the following properties from type 'ILockerModel': id, lockerId, ownerId, modules

The other thing is that JSON.strigify() just takes your object and makes a string representation of all its properties. It does not care about your getters. If you want it to transform it to the right format, you should give it an object in the correct format.

One solution would be to just remove the '_' from all the keys, by using a combination of map and reduce:

1export interface ILockerModel {
2    id: string
3    lockerId: string
4    ownerId: string
5    modules: IModuleModel[]
6}
7export class LockerModel implements ILockerModel {
8    private _id: string
9    private _lockerId: string
10    private _ownerId: string
11    private _modules: ModuleModel[]
12
13    constructor(document: ILockerModel) {
14        this._id = document.id
15        this._lockerId = document.lockerId
16        this._ownerId = document.ownerId
17        this._modules = document.modules.map(m =&gt; new ModuleModel(m))
18    }
19    // Utility methods
20}
21const input = {
22  _test: 123,
23  _hello: 'world'
24};
25
26console.log(input);
27console.log(JSON.stringify(input));
28
29const convertToJson = (obj) =&gt; {
30  return Object.entries(obj) // Create array from object
31    .map(([key, value]) =&gt; [  // change key to remove '_'
32      key.startsWith('_') ? key.substring(1) : key, 
33      value
34    ])
35    .reduce((acc, [key, value]) =&gt; { // Transform back to object
36      acc[key] = value;
37      return acc;
38    }, {});
39}
40
41const output = convertToJson(input);
42
43
44console.log(output);
45console.log(JSON.stringify(output));

Or if you are allowed to use ES10:

1export interface ILockerModel {
2    id: string
3    lockerId: string
4    ownerId: string
5    modules: IModuleModel[]
6}
7export class LockerModel implements ILockerModel {
8    private _id: string
9    private _lockerId: string
10    private _ownerId: string
11    private _modules: ModuleModel[]
12
13    constructor(document: ILockerModel) {
14        this._id = document.id
15        this._lockerId = document.lockerId
16        this._ownerId = document.ownerId
17        this._modules = document.modules.map(m =&gt; new ModuleModel(m))
18    }
19    // Utility methods
20}
21const input = {
22  _test: 123,
23  _hello: 'world'
24};
25
26console.log(input);
27console.log(JSON.stringify(input));
28
29const convertToJson = (obj) =&gt; {
30  return Object.entries(obj) // Create array from object
31    .map(([key, value]) =&gt; [  // change key to remove '_'
32      key.startsWith('_') ? key.substring(1) : key, 
33      value
34    ])
35    .reduce((acc, [key, value]) =&gt; { // Transform back to object
36      acc[key] = value;
37      return acc;
38    }, {});
39}
40
41const output = convertToJson(input);
42
43
44console.log(output);
45console.log(JSON.stringify(output));const input = {
46  _test: 123,
47  _hello: 'world'
48};
49
50console.log(input);
51console.log(JSON.stringify(input));
52
53const convertToJson = (obj) =&gt; {
54  return Object.fromEntries( // Create Object from array
55    Object.entries(obj) // Create array from object
56      .map(([key, value]) =&gt; [ // change key to remove '_'
57        key.startsWith('_') ? key.substring(1) : key, 
58        value
59      ])
60  );
61}
62
63const output = convertToJson(input);
64
65
66console.log(output);
67console.log(JSON.stringify(output));

Source https://stackoverflow.com/questions/63579375

QUESTION

How To: Get latest (newest) document from Cosmos DB?

Asked 2020-Jul-25 at 20:27

I have a document database that has various properties including one named "SensorCaptureTime". I would like to retrieve only the newest document based on this property. So far, all I have figured out is how to retrieve all the documents sorted by SensorCaptureTime and then loop through them taking the last document in the list. Here's my current code:

1            SensorData mostRecentSensorReadingDocument = null;
2
3            // TODO: There has to be a better way to do this... just not sure what it is yet :-)
4            var queryable = documentClient.CreateDocumentQuery&lt;SensorData&gt;(
5                collectionUri,
6                new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true })
7                .OrderBy(b =&gt; b.SensorCaptureTime)
8                .AsDocumentQuery();
9
10            while (queryable.HasMoreResults)
11            {
12                foreach (SensorData sensorReadingsDocument in await queryable.ExecuteNextAsync&lt;SensorData&gt;())
13                {
14                    mostRecentSensorReadingDocument = sensorReadingsDocument;
15                }
16            }
17
18            return mostRecentSensorReadingDocument;
19

I am assuming there is a way to request a single document, based on something like an aggregate function (e.g. Max), from a (Cosmos) document database without having to loop through the entire result set. Unfortunately, I haven't been able to find a working example.

Thanks!

ANSWER

Answered 2020-Jul-25 at 20:27

Is there only one of these for the entire database or is there one for each device that has data written?

In either case, the problem with this approach is that over time the query will get increasingly more expensive and slow, especially if this is a cross partition query. A better approach may be to use Change Feed and update another collection that only contains a single record for every device that contains the latest reading. Whenever a new reading is inserted into Cosmos DB, Change Feed gets fired, you would then read change feed and do an Upsert on a second collection to update the relevant record with the latest reading. This way, each time you need that value it is simply a point-read which is super efficient.

Another option that may work is to use a composite index with the device id and SensorCaptureTime with Desc order and then do a Top 1 on the collection to get the last item inserted.

Source https://stackoverflow.com/questions/63092001

QUESTION

Non-Relational Database Design For Big Data Warehouse

Asked 2020-Jul-12 at 06:56

Suppose I need to design a table for Spotify where I need to quickly retrieve what items (song or album) a user already purchased so it can play for the user. The scenario is straightforward: when users click to buy the song, the database needs to quickly update a particular song being purchased to the user account.

Since it really requires near real-time response and the table could be increased exponentially, on the other hand, the access format is quite simple and fix, a non-relational database is designed for this use case. That's why I am thinking about using HBase, Cassandra, or MongoDB.

I would like to use UserId as the primary key for this Purchase Table, will Wide Column Stores like (HBase or Cassandra) or Document databases like MongoDB work better for this scenario?

The input is just a user_id and the database table response with all available purchased items. What is the best database table design strategy?

1{user_id:int
2   {purchased_item: item1
3                    item2
4                    item3
5   }
6}
7

The second table will be used for searching for specific artists, albums, genres, and songs that are available for purchase.

Appreciate if you can share any examples of best practice from the real-world application. Or any good article/document/blogs I can read.

ANSWER

Answered 2020-Jul-12 at 06:56

If you are considering near real-time I would definitely consider using Cassandra especially for history detailed storage!

What I would do using Cassandra is the folowing:

CREATE TABLE purchases( user_id uuid, purchase_id uuid, item_id uuid, item_details text, item_name text, time_of purchase timestamp, PRIMARY KEY((user_id), purchase_id, item_id));

This will let you cluster the data in several ways first using the user_id then using the purchase_id to keep all items recorded per purchase!

By having the the Primary key formed of Partition key the user_id the clustering key the purchase_id and item_id we are able to group the items in the purchase_id and then in the user_id.

https://cassandra.apache.org/doc/latest/data_modeling/intro.html https://docs.datastax.com/en/landing_page/doc/landing_page/current.html

Source https://stackoverflow.com/questions/62849389

QUESTION

Would Elasticsearch work better with a document database like MongoDB or a relational database like MySQL?

Asked 2020-May-04 at 04:02

I am trying to decide which database to build my site on, it will have a similar structure to sites like IMDb. I have decided I would like to use Elaticsearch (for searches) in combination with another database (as a primary datastore). What type of database would work better with Elasticsearch, a NoSQL (probably document database) like MongoDB or an SQL relational database like MySQL?

ANSWER

Answered 2020-May-04 at 04:02

It totally depends on your use-cases primary datastore, if you want to have a

Key-based lookup (mentioning on key-value, document-based NoSQL)
Simple queries on a few columns
Ok to not have ACID properties
The data structure is changing frequently, Hence difficult to use RDBMS
you want to scale rapidly and ok to have eventual consistency

And much more difference between SQL and NoSQL mentioned in this blog, but higher-level above points are critical to determining whether to choose SQL or NoSQL database.

Elasticsearch in your case is used only for searching and used as secondary storage, hence doesn't add much value to the decision but yeah if you use NoSQL, then how you sync your data can be important factors to provide near real-time search(as NoSQLs are eventual consistent system and might delay the sync operation).

Source https://stackoverflow.com/questions/61581643

QUESTION

How do I update a MongoDB document with new value using reactors Mono? (Kotlin)

Asked 2020-Mar-13 at 20:08

So the context is that I require to update a value in a single document, I have a Mono, the parameter Object contains values such as username (to find the correct user by unique username) and an amount value.

The problem is that this value (due to other components of my application) is the value by which I need to increase/decrease the users balance, as opposed to passing a new balance. I intend to do this using two Monos where one finds the user, then this is combined to the other Mono with the inbound request, where I can then perform a simple sum (i.e balance + changeRequest.amount) then return this to the document database.

1override fun increaseBalance(changeRequest: Mono&lt;ChangeBalanceRequestResource&gt;): Mono&lt;ChangeBalanceResponse&gt; {
2
3    val changeAmount: Mono&lt;Decimal128&gt; = changeRequest.map { it.transactionAmount }
4
5    val user: Mono&lt;User&gt; = changeRequest.flatMap { rxUserRepository.findByUsername(it.username)
6
7
8    val newBalace = user.map {
9        val r = changeAmount.block()
10        it.balance = sumBalance(it.balance!!, r!!)
11            rxUserRepository.save(it)
12    }
13    .flatMap { it }
14    .map { it.balance!! }
15
16    return Mono.just(ChangeBalanceResponse("success", newBalace.block()!!))
17 }
18

Obviously I'm trying to achieve this in a non-blocking fashion. I'm also open to using only a single Mono if that's possible/optimal. I also appreciate I've truly butchered the example and used .block as a placeholder to illustrate what I'm trying to achieve.

P.S this is my first post, so any tips on how to express my problem clearer would be useful.

ANSWER

Answered 2020-Mar-13 at 20:08

Here's how I would do this in Java (Using Double instead of Decimal128):

1override fun increaseBalance(changeRequest: Mono&lt;ChangeBalanceRequestResource&gt;): Mono&lt;ChangeBalanceResponse&gt; {
2
3    val changeAmount: Mono&lt;Decimal128&gt; = changeRequest.map { it.transactionAmount }
4
5    val user: Mono&lt;User&gt; = changeRequest.flatMap { rxUserRepository.findByUsername(it.username)
6
7
8    val newBalace = user.map {
9        val r = changeAmount.block()
10        it.balance = sumBalance(it.balance!!, r!!)
11            rxUserRepository.save(it)
12    }
13    .flatMap { it }
14    .map { it.balance!! }
15
16    return Mono.just(ChangeBalanceResponse("success", newBalace.block()!!))
17 }
18public Mono&lt;ChangeBalanceResponse&gt; increaseBalance(Mono&lt;ChangeBalanceRequestResource&gt; changeRequest) {
19
20    Mono&lt;Double&gt; changeAmount = changeRequest.map(a -&gt; a.transactionAmount());
21    Mono&lt;User&gt; user = changeRequest.map(a -&gt; a.username()).flatMap(RxUserRepository::findByUsername);
22
23    return Mono.zip(changeAmount,user).flatMap(t2 -&gt; {
24       Double changeAmount = t2.getT1();
25       User user = t2.getT2();
26       //assumes User is chained
27       return rxUserRepository.save(user.balance(sumBalance(changeAmount,user.balance())));
28   }).map(res -&gt; new ChangeBalanceResponse("success",res.newBalance()))
29
30}
31

Source https://stackoverflow.com/questions/60668823

QUESTION

How do I store a picture in azure blob storage in asp.net mvc application?

Asked 2020-Mar-11 at 06:06

I am currently working on an ASP.Net application that stores student information. I am required to store the following information:

Student number
Name
Email address
Home address
Contact no.
Upload photo of the student
isActive flag to state whether the student is active or not

This information is being stored in a document database and the photo of the student needs to be uploaded to Azure blob storage while returning the link of the image to the document database

How do I do this?

I currently have a class for the student information which looks like this :

1public class Student
2{
3    [Key]
4    public int StudentId { get; set; }
5
6    [Required]
7    [StringLength(8, MinimumLength = 8)]
8    [DisplayName("Student Number")]
9    public string StudentNo { get; set; }
10
11    [Required]
12    [StringLength(50, MinimumLength = 1)]
13    [DisplayName("First Name")]
14    public string FirstName { get; set; }
15
16    [Required]
17    [StringLength(50, MinimumLength = 1)]
18    [DisplayName("Last Name")]
19    public string LastName { get; set; }
20
21    [Required]
22    [DataType(DataType.EmailAddress)]
23    [DisplayName("Email Address")]
24    public string Email { get; set; }
25
26    [Required]
27    [StringLength(100, MinimumLength = 5)]
28    [DataType(DataType.MultilineText)]
29    [DisplayName("Home Address")]
30    public string HomeAddress { get; set; }
31
32    [Required]
33    [DataType(DataType.PhoneNumber)]
34    [StringLength(10)]
35    [DisplayName("Mobile No.")]
36    public string Mobile { get; set; }
37
38    public bool IsActive { get; set; }
39}
40

I have a separate view model for the blob as I was still experimenting with blobs:

1public class Student
2{
3    [Key]
4    public int StudentId { get; set; }
5
6    [Required]
7    [StringLength(8, MinimumLength = 8)]
8    [DisplayName("Student Number")]
9    public string StudentNo { get; set; }
10
11    [Required]
12    [StringLength(50, MinimumLength = 1)]
13    [DisplayName("First Name")]
14    public string FirstName { get; set; }
15
16    [Required]
17    [StringLength(50, MinimumLength = 1)]
18    [DisplayName("Last Name")]
19    public string LastName { get; set; }
20
21    [Required]
22    [DataType(DataType.EmailAddress)]
23    [DisplayName("Email Address")]
24    public string Email { get; set; }
25
26    [Required]
27    [StringLength(100, MinimumLength = 5)]
28    [DataType(DataType.MultilineText)]
29    [DisplayName("Home Address")]
30    public string HomeAddress { get; set; }
31
32    [Required]
33    [DataType(DataType.PhoneNumber)]
34    [StringLength(10)]
35    [DisplayName("Mobile No.")]
36    public string Mobile { get; set; }
37
38    public bool IsActive { get; set; }
39}
40 public class BlobViewModel
41{
42
43    public string BlobContainerName { get; set; }
44    public string StorageUri { get; set; }
45    public string ActualFileName { get; set; }
46    public string PrimaryUri { get; set; }
47    public string fileExtension { get; set; }
48
49    public string FileNameWithoutExt
50    {
51        get
52        {
53            return Path.GetFileNameWithoutExtension(ActualFileName);
54        }
55    }
56
57    public string FileNameExtensionOnly
58    {
59        get
60        {
61            return System.IO.Path.GetExtension(ActualFileName).Substring(1);
62        }
63    }
64

How do I combine these 2 so that i can upload an image for the student to be stored in a blob while returning the URI to the student class?

ANSWER

Answered 2020-Mar-10 at 14:52

Using Azure CosmosDB for the storage could help. With CosmosDB, you can easily store the image as an attachment to the student document: https://docs.microsoft.com/en-us/rest/api/cosmos-db/attachments

If you don't plan to use CosmosDB, be more precise please about the database storage.

Source https://stackoverflow.com/questions/60617190

Community Discussions contain sources that include Stack Exchange Network