DataflowEx | NET dataflow and etl framework built upon Microsoft TPL

by gridsum C# Version: Current License: MIT

X-Ray Key Features Code Snippets Community Discussions(5)Vulnerabilities Install Support

kandi X-RAY | DataflowEx Summary

DataflowEx is a C# library. DataflowEx has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Code tells a lot. Let's migrate the above example to DataflowEx and see what it looks like:. Though there seems to be more code, it is quite clear. We have a class AggregatorFlow representing our flow finally, which inherits from Dataflow with type parameter string. This means the AggregatorFlow class reprensents a dataflow graph itself and accepts strings as input. In this form, dataflow blocks and data become class members. Block behaviors become class methods (which allows the outside to override!). We also implemented the abstract InputBlock property of Dataflow and exposes our internal data as an extra Result property. There are two important calls to RegisterChild() in the constructor. We will come back to this later.

Support

Quality

Security

License

Reuse

Support

DataflowEx has a low active ecosystem.

It has 206 star(s) with 42 fork(s). There are 31 watchers for this library.

It had no major release in the last 6 months.

There are 8 open issues and 10 have been closed. On average issues are closed in 258 days. There are 4 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of DataflowEx is current.

Quality

DataflowEx has 0 bugs and 0 code smells.

Security

DataflowEx has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

DataflowEx code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

DataflowEx is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

DataflowEx releases are not available. You will need to build from source code and install.

Installation instructions are not available. Examples and code snippets are available.

DataflowEx saves you 1178 person hours of effort in developing the same functionality from scratch.

It has 2657 lines of code, 0 functions and 93 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of DataflowEx

Get all kandi verified functions for this library.

DataflowEx Key Features

No Key Features are available at this moment for DataflowEx.

DataflowEx Examples and Code Snippets

No Code Snippets are available at this moment for DataflowEx.

Community Discussions

Trending Discussions on DataflowEx

Encapsulating with DataflowEx

DataflowEx never completes

Avoid shutting down an entire data flow network when one block is faulted

How to configure NLog in a .NET Core app

ReSharper 8.2.3 crashes during startup of Visual Studio 2008

QUESTION

Encapsulating with DataflowEx

Asked 2018-Jan-08 at 16:39

I want to use DataflowEx to encapsulate this test. I am not sure that my design is correct as I have tried a lot to do it my self but I failed to make the Dataflow to complete as in my test and is not clear how I should post the server, IP combination from the outside.

...

ANSWER

Answered 2018-Jan-08 at 16:39

Overriding the Complete method to complete the additional BufferBlock makes the test pass.

Source https://stackoverflow.com/questions/48105215

QUESTION

DataflowEx never completes

Asked 2017-Dec-26 at 19:24

I am trying to use the open source lib DataflowEx with the next Dataflow declaration.

...

ANSWER

Answered 2017-Dec-26 at 16:57

Your flow cannot complete since the last block is a TransformBlock. In your first example you await completion of the Input block which does in fact complete. The Output block cannot complete as the items in its output buffer have nowhere to go. The DataflowEx library is correctly awaiting on the final block in the flow. You can add an ActionBlock or NullTarget to the end to realize completion.

In terms of DataflowEx the final flow should be implmeneting

Source https://stackoverflow.com/questions/47981322

QUESTION

Avoid shutting down an entire data flow network when one block is faulted

Asked 2017-Nov-07 at 19:36

I am using DataFlowEx and am wondering how I can avoid shutting down an entire DataFlow if an exception is thrown.

I have a system where tasks will come in at random times, and I want the network to log failures, abandon that particular task and continue with execution of the others.

In reading the documentation on both TPL and DataFlowEx, specifically things like

It [a faulted block] should decline any further incoming messages. Here

DataflowEx takes a fast-fail approach on exception handling just like TPL Dataflow. When an exception is thrown, the low-level block ends to the Faulted state first. Then the Dataflow instance who is the parent of the failing block gets notified. It will immediately propagate the fatal error: notify its other children to shutdown immediately. After all its children is done/completed, the parent Dataflow also comes to its completion, with the original exception wrapped in the CompletionTask whose status is also Faulted. Here

It almost seems like a block moving on from a failure is not intended...

My flows include a lot of File IO and i am expecting the occasional exception to occur (network volumes going offline during read/write, connection failures, permission issues...)

I don't want the entire pipeline to die.

Here is an example of the code I'm working with:

...

ANSWER

Answered 2017-Nov-07 at 19:36

A block will become faulted if it throws an Exception. If you do not want the pipeline to fail you can either not propagate completion or handle the Exception. Handling the exception can take many forms but it sounds like all you need is a simple retry. You could use a try/catch and implement your own retry loop or use something like Polly. A simple example is shown below.

Source https://stackoverflow.com/questions/47147341

QUESTION

How to configure NLog in a .NET Core app

Asked 2017-Nov-07 at 00:25

I am using a library called DataFlowEx and it requires a configuration for NLog in order to output it's debug and other information.

The tutorial shows the configuration using xml files.

I thought of using Microsoft.Extensions.Configuration, and pointing that to the XML but it doesn't seem to work...

Here is what I have so far:

...

ANSWER

Answered 2017-Nov-06 at 23:32

Maybe this can help:

https://github.com/net-commons/common-logging/issues/153

Stop using the app.config, but instead use a dedicated nlog.config (Remember to configure the nlog.config-file to "Copy Always")

Source https://stackoverflow.com/questions/47146834

QUESTION

ReSharper 8.2.3 crashes during startup of Visual Studio 2008

Asked 2017-Nov-01 at 07:15

When I start up my Visual Studio 2008 with ReSharper 8.2.3 using the /Log switch, I can see in the produced log-file, that several exceptions are thrown by ReSharper. I cannot use VS to change any files (typing is not deactivated, but it ignores any non-ascii typing). As soon as I deactivate ReSharper, everything is back to normal.

devenv.exe /Resharper.LogFile /Resharper.LogLevel Verbose output:

[snipped - startup info and threads logging verbose....]

...

ANSWER

Answered 2017-Nov-01 at 07:15

Who would have thought, that the solution is so simple. I vaguely remembered setting a new language locale on my system prior to going on holiday. After returning from said holiday, the phenomenon described in the question started.

Solution

Remove the language locale en-CH (english Switzerland) and set the default en-US.
Restart the PC.
Problem solved.

Source https://stackoverflow.com/questions/47038830

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install DataflowEx

You can download it from GitHub.

Support

The very 1st feature/component in DataflowEx we strongly recommend is the DbBulkInserter which enables bulk insertion to SQL Server. Though there are many ORM solutions out there but when it comes to insertion, they are just not designed to use the most efficient way: bulk insert. Using SqlBulkCopy internally, DbBulkInserter is born to solve the problem in a speedy way. And it nicely fits in dataflow style programming. Inheriting from Dataflow , DbBulkInserter is a generic class and accepts a generic parameter T, which is the type of your domain objects. Just connect your output flows to DbBulkInserter. Bulk insertion magic will then happen internally in DbBulkInserter. The only extra thing to do is to set up column mapping from your domain object properties to database table columns, taking advantage of C# attributes.

Find more information at: