scalability | Issue tracker for the Scalability team | Continous Integration library

I would like to ask a question about a possible solution for an e-commerce database design in terms of scalability and flexibility.

We are going to use MongoDB and Node on the backend.

I included an image for you to see what we have so far. We currently have a Products table that can be used to add a product into the system. The interesting part is that we would like to be able to add different types of products to the system with varying attributes.

For example, in the admin management page, we could select a Clothes item where we should fill out a form with fields such as Height, Length, Size ... etc. The question is how could we model this way of structure in the database design?

What we were thinking of was creating tables such as ClothesProduct and many more and respectively connect the Products table to one of these. But we could have 100 different tables for the varying product types. We would like to add a product type dynamically from the admin management. Is this possible in Mongoose? Because creating all possible fields in the Products table is not efficient and it would hit us hard for the long-term.

Database design snippet

Maybe we should just create separate tables for each unique product type and from the front-end, we would select one of them to display the correct form?

Could you please share your thoughts?

Thank you!

I'm not sure if the title makes sense, it was the best I could come up with, so here's my scenario.

I have an ASP.NET Core app that I'm using more as a shell and for DI configuration. In Startup it adds a bunch of IHostedServices as singletons, along with their dependencies, also as singletons, with minor exceptions for SqlConnection and DbContext which we'll get to later. The hosted services are groups of similar services that:

1. Listen for incoming reports from GPS devices and put into a listening buffer.
2. Parse items out of the listening buffer and put into a parsed buffer.

Eventually there's a single service that reads the parsed buffer and actually processes the parsed reports. It does this by passing the report it took out of the buffer to a handler and awaits for it to complete to move to the next. This has worked well for the past year, but it appears we're running into a scalability issue now because its processing one report at a time and the average time to process is 62ms on the server which includes the Dapper trip to the database to get the data needed and the EF Core trip to save changes.

If however the handler decides that a report's information requires triggering background jobs, then I suspect it takes 100ms or more to complete. Over time, the buffer fills up faster than the handler can process to the point of holding 10s if not 100s of thousands of reports until they can be processed. This is an issue because notifications are delayed and because it has the potential for data loss if the buffer is still full by the time the server restarts at midnight.

All that being said, I'm trying to figure out how to make the processing parallel. After lots of experimentation yesterday, I settled on using Parallel.ForEach over the buffer using GetConsumingEnumerable(). This works well, except for a weird behavior I don't know what to do about or even call. As the buffer is filled and the ForEach is iterating over it it will begin to "chunk" the processing into ever increasing multiples of two. The size of the chunking is affected by the MaxDegreeOfParallelism setting. For example (N# = Next # of reports in buffer):

• N3 = 1 at a time
• N6 = 2 at a time
• N12 = 4 at a time
• ...

• N6 = 1 at a time
• N12 = 2 at a time
• N24 = 4 at a time
• ...

• N12 = 1 at a time
• N24 = 2 at a time
• N48 = 4 at a time
• ...

• N24 = 1 at a time
• N48 = 2 at a time
• N96 = 4 at a time
• ...

This is arguably worse than the serial execution I have now because by the end of the day it will buffer and wait for, say, half a million reports before actually processing them.

Is there a way to fix this? I'm not very experienced with Parallel.ForEach so from my point of view this is strange behavior. Ultimately I'm looking for a way to parallel process the reports as soon as they are in the buffer, so if there's other ways to accomplish this I'm all ears. This is roughly what I have for the code. The handler that processes the reports does use IServiceProvider to create a scope and get an instance of SqlConnection and DbContext. Thanks in advance for any suggestions!

Long story short, I have been developing a Discord Bot that requires a query to the database every time a message is sent in a server. It will then perform an action depending on the message etc. The query is asynchronous, therefore it will not block another message from being handled.

However in terms of scalability, I do not believe querying a database every time a message is sent is very speedy and could become a problem. Is there a better solution? I am unaware of a way to store data within a particular discord server, which would likely solve my issue.

My main idea is to have heap storage, where the most recently active servers (ie sent messages recently), their data is queried into the heap, and when they are inactive, it is removed from the heap. Is this a good solution? Or is it better to just keep querying every time?

Im sure this has been come across pretty often but all of the resources I can find detail ways to handle this as an individual situation and require a manipulation strategy that lacks scalability.

Problem:

I have two data pipelines; say a production pipeline and a development pipeline. I need to verify that the tables produced at the end of each pipeline are the same. These tables may be in different databases, in different data centers, and may each contain anywhere from hundreds of lines to a billion lines.

I need to provide all rows that each table is missing, and all rows that are mismatched. In the case where there are mismatched rows, I need to provide the specific rows. For example:

I have found this question that talks about the difference between partition and replica, and the answers seem to mention that the Kafka partition is needed for scalability. But I don't get why it's "mandatory" in order to scale your infrastructure? I feel like you could simply add a new node and increase the replication value of the topic?

When I deploy a serverless framework codebase to AWS, I am curious about what method will be better. For now, there are 2 options.

• Use Nest.js or Express.js so I deploy one function to Lambda and this function will handle all API endpoints
• Deploy number of functions so each of them represents a single API endpoint

Regarding scalability, which option is a good approach?

When hosting an Azure Function in an App Service Plan, are there any significant differences compared with using an App Service (EDIT: for a restful API) associated with the same App Service Plan? I assume the only difference is that the Function offers additional out of the box triggers. Any differences I'm missing that would lead to preferring one over the other?

I'd also like to confirm that hosting Azure Functions in an App Service Plan can actually limit scalability if scaling is not configured on the App Service Plan. As I understand it, Functions automatically scale as needed when using Consumption or Premium hosting without additional configuration.

Im am trying to code this .conf file with more scalability, and my idea is to, in order to have multi index in elasticsearch, split the path and get the last position to have the csv name and set it to the type and index in elasticsearch.