data-factory | 🏭Auto generate mock data for java test(便于 Java 测试自动生成对象信息) | Mock library

by houbb Java Version: 1.2.0 License: Non-SPDX

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | data-factory Summary

data-factory is a Java library typically used in Testing, Mock, Kafka applications. data-factory has no bugs, it has no vulnerabilities, it has build file available and it has low support. However data-factory has a Non-SPDX License. You can download it from GitHub, Maven.

Auto generate mock data for java test.(便于 Java 测试自动生成对象信息)

Support

Quality

Security

License

Reuse

Support

data-factory has a low active ecosystem.

It has 75 star(s) with 14 fork(s). There are 4 watchers for this library.

It had no major release in the last 12 months.

There are 6 open issues and 3 have been closed. On average issues are closed in 134 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of data-factory is 1.2.0

Quality

data-factory has 0 bugs and 0 code smells.

Security

data-factory has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

data-factory code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

data-factory has a Non-SPDX License.

Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

Reuse

data-factory releases are not available. You will need to build from source code and install.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

data-factory saves you 894 person hours of effort in developing the same functionality from scratch.

It has 2043 lines of code, 190 functions and 101 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed data-factory and discovered the below as its top functions. This is intended to give you an instant insight into data-factory implemented functionality, and help decide if they suit your requirements.

Build data value
Get data class
Get generic interfaces
Init data factory
Builds the data object
Gets the data annotation
Get data value
Build context
Build a String
Get length of random length
Validate parameters check
Build a collection
Create an Iterable instance
Generate random int array
Generate a random size
Build a map
Generate random float array
Build short array
Generate long array
Build boolean array
Generate a random double array
Build char array
Generate random byte array
Build a random array
Build a random enum

Get all kandi verified functions for this library.

data-factory Key Features

No Key Features are available at this moment for data-factory.

data-factory Examples and Code Snippets

No Code Snippets are available at this moment for data-factory.

Community Discussions

Trending Discussions on data-factory

Copy Data pipeline on Azure Data Factory from SQL Server to Blob Storage

what's the property in the body of Getbearer Token web activity mean?

Where to get the " Copy from REST or HTTP using OAuth template" to use Oauth?

How to use two file extensions as wildcard (*.csv or *.xml) in Azure data factory?

Using Azure Data Factory utilities to generate ARM Template does not generate the location tag for the Data Factory resource

Redirect to login with AAD in Azure App Service Authentication vs Authentication (Classic)

Getting Data Lake Metadata in Azure Data Factory's Data Flow

how to set variable for"set variable" activity in data factory

Data Factory does not like A.U.S. Eastern Standard Time

mapping data flow column pattern type =='decimal' not changing decimal columns

QUESTION

Copy Data pipeline on Azure Data Factory from SQL Server to Blob Storage

Asked 2021-Jun-12 at 10:33

I'm trying to move some data from Azure SQL Server Database to Azure Blob Storage with the "Copy Data" pipeline in Azure Data Factory. In particular, I'm using the "Use query" option with the ?AdfDynamicRangePartitionCondition hook, as suggested by Microsoft's pattern here, in the Source tab of the pipeline, and the copy operation is parallelized by the presence of a partition key used in the query itself.

The source on SQL Server Database consists of two views with ~300k and ~3M rows, respectively. Additionally, the views have the same query structure, e.g. (pseudo-code)

...

ANSWER

Answered 2021-Jun-10 at 06:24

When there's a copy activity performance issue in ADF and the root cause is not obvious (e.g. if source is fast, but sink is throttled, and we know why) -- here's how I would go about it :

Start with the Integration Runtime (IR) (doc.). This might be a jobs' concurrency issue, a network throughput issue, or just an undersized VM (in case of self-hosted). Like, >80% of all issues in my prod ETL are caused by IR-s, in one way or another.
Replicate copy activity behavior both on source & sink. Query the views from your local machine (ideally, from a VM in the same environment as your IR), write the flat files to blob, etc. I'm assuming you've done that already, but having another observation rarely hurts.
Test various configurations of copy activity. Changing isolationLevel, partitionOption, parallelCopies and enableStaging would be my first steps here. This won't fix the root cause of your issue, obviously, but can point a direction for you to dig in further.
Try searching the documentation (this doc., provided by @Leon is a good start). This should have been a step #1, however, I find ADF documentation somewhat lacking.

N.B. this is based on my personal experience with Data Factory.
Providing a specific solution in this case is, indeed, quite hard.

Source https://stackoverflow.com/questions/67909075

QUESTION

what's the property in the body of Getbearer Token web activity mean?

Asked 2021-Jun-11 at 07:52

Currently, I'm following this doc to use Oauth to copy data from Rest connector. I applied the suggested temple ,when I configure this web activity, as for the body content, it show I should provide below parameters. I wonder where to get this parameters?

screenshot2:

...

ANSWER

Answered 2021-Jun-11 at 05:46

These are app registration ID and password. You need to register an app in Azure AD.

Below MSFT docs provides details about the same:

https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

Source https://stackoverflow.com/questions/67930326

QUESTION

Where to get the " Copy from REST or HTTP using OAuth template" to use Oauth?

Asked 2021-Jun-10 at 02:49

Currently I'm following this docs to use Oauth to copy data from REST connector into Azure Data Lake Storage in JSON format using OAuth. but I cannot find below temple in my temple gallery. I wonder where and how to get this temple ?

screenshot2:

...

ANSWER

Answered 2021-Jun-10 at 02:19

As the following screenshot shows, you can click Add new resource button, then click pipeline from template and you will find it.

Source https://stackoverflow.com/questions/67913714

QUESTION

How to use two file extensions as wildcard (*.csv or *.xml) in Azure data factory?

Asked 2021-Jun-09 at 09:06

I have a path in ADLS that has a range of different files including *.csv and *.xml (which is not true xml, it's just a csv with xml extension).

I want to copy only *.csv and *.xml files from this path to another using copy activity in ADF. Right now I only can specify one of them as wildcard in the file name of copy activity and not both. Is there any way to specify two wildcards, like for example, .csv or .xml.

BTW, I might be able to use filter activity with get meta data, but this is too much if there is other ways. This documentation didn't help much too:

As I said, filtering won't work (without forEach), and that's not optimized:

...

ANSWER

Answered 2021-Jun-09 at 09:06

No, there isn't a way can specify two wildcards path.

According my experience, the easiest way is that you can create two copy active in one pipeline:

Copy active1: copy the files end with *.csv.
Copy active2: copy the files end with *.xml.

For your another question,there are many ways can achieve it. You could add an if condition to filter the condition: only copy active 1 and 2 both true/succeeded:

You also could do like@Nandan said:

Source https://stackoverflow.com/questions/67898934

QUESTION

Using Azure Data Factory utilities to generate ARM Template does not generate the location tag for the Data Factory resource

Asked 2021-Jun-08 at 20:33

When using the "publish" on the Azure Data Factory the ARM Template is generated

...

ANSWER

Answered 2021-Jun-08 at 20:33

I am not able to reproduce the issue but would suggest not including the factory in the ARM template as documented here: https://docs.microsoft.com/en-us/azure/data-factory/author-global-parameters#cicd

Including the factory will cause other downstream issues when using the automated publish flow for CI/CD such as removing the git configuration on the source factory, so deploying global parameters with PowerShell is the recommended approach. By not including the factory in the ARM template, this error will not occur. Feel free to continue the discussion here: https://github.com/Azure/Azure-DataFactory/issues/285

Source https://stackoverflow.com/questions/67834746

QUESTION

Redirect to login with AAD in Azure App Service Authentication vs Authentication (Classic)

Asked 2021-Jun-08 at 08:10

I'm trying to set up a system where an Azure DataFactory can call an Azure function through its managed identity. Good example here: Authorising Azure Function App Http endpoint from Data Factory

However, this was using the old(er) Authentication/Authorization tool for Azure functions, which has now been renamed Authentication (Classic). Setting the system up through this is fine, I can make the call and get a response, but upgrading to Authorization causes this to break. It seems like the key thing missing is the option of "Action to take when the request is not authenticated", which I cannot seem to set with the new Authorization tool but should be set to "Login with Azure AD"

In summary, how do I set this setting with the new Authorization tool so that a MSI can make a cool to the function and authenticate with AAD.

Image with classic

Image with new Authorization (no visible way to redirect to AAD)

In summary, how do I set this setting with the new Authorization tool so that a MSI can make a cool to the function and authenticate with AAD.

...

ANSWER

Answered 2021-Jun-08 at 08:10

To make it work with the new Authentication, follow the steps below.

1.Edit the Authentication settings in the portal or set it when creating the app as below.

2.Edit the Identity provider, make sure the Issuer URL is https://sts.windows.net/(without /v2.0) and Allowed token audiences include the App ID URI.

For the App ID URI, you could check it in your AD App of the function app -> Expose an API, if you use the old Authentication before, maybe it is your function app URL, it does matter, just make sure Allowed token audiences include it.

3.Then in the datafactory web activity, also make sure the resource is the App ID URI.

Then it will work fine.

Update:

You could refer to my configuration.

Function app:

AD App:

AD App manifest:

Source https://stackoverflow.com/questions/67878998

QUESTION

Getting Data Lake Metadata in Azure Data Factory's Data Flow

Asked 2021-Jun-07 at 12:09

I want to add the timestamp of copying parquet files to my dataframe in data flow as a derived column.
In source module I can filter parquet files by last modified which makes me think that it should be possible to access files' metadata including copied timestamp through derived column transformations, but I couldn't find anything for it in Microsoft documentation.

...

ANSWER

Answered 2021-Jun-07 at 12:09

There is no function can get the last modified time in the data flow expression.

As a workaround, you can create a Get Metadata activity to get that and then pass it's value to a parameter in your data flow.

The expression:@activity('Get Metadata1').output.lastModified

Source https://stackoverflow.com/questions/67870890

QUESTION

how to set variable for"set variable" activity in data factory

Asked 2021-Jun-03 at 09:07

firstly I create a web activity to get keyvault,and then create a "set variable" activity. when I try to create variable in the "set variable" activity, it shows "no results found". BTW I cannot attach screenshot due to less reputation . I refer to this doc to do execution

attached the screenshot

...

ANSWER

Answered 2021-Jun-03 at 09:07

Update:
You should declare a variable first by click blank, then you can select a variable at step3:

After you added ADF managed identity permissions to Get and List secrets.

Add a secret to the key valut. Here my secret name is mysecret.
So your URL should looks like https://your-keyvault-name.vault.azure.net/secrets/mysecret?api-version=7.0
Add dynamic content @activity('Web1').output.value to your Set variable1 activity.

Source https://stackoverflow.com/questions/67818142

QUESTION

Data Factory does not like A.U.S. Eastern Standard Time

Asked 2021-May-30 at 11:02

I am trying to use convertTimeZone to get to my local time in Sydney.

Data Factory is happy with with other conversion like

@convertTimeZone(utcnow() , 'UTC' , 'GMT Standard Time') but when I try for my location @convertTimeZone(utcnow() , 'UTC' , 'A.U.S. Eastern Standard Time')

I get an error

In the function 'convertTimeZone', the value provided for the time zone id 'A.U.S. Eastern Standard Time' was not valid.

it is in the list here https://docs.microsoft.com/en-us/previous-versions/windows/embedded/ms912391(v=winembedded.11)

Which is provided in the documentation here. https://docs.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions#convertTimeZone

Any ideas ? Thanks

...

ANSWER

Answered 2021-May-30 at 11:02

Please see the description of destinationTimeZone in convertTimeZone function:

The name for the source time zone. For time zone names, see Microsoft Time Zone Index Values, but you might have to remove any punctuation from the time zone name.

So remove . in A.U.S. Eastern Standard Time and then have a try this expression:

Source https://stackoverflow.com/questions/67758054

QUESTION

mapping data flow column pattern type =='decimal' not changing decimal columns

Asked 2021-May-21 at 07:43

I'm using a column pattern to catch nulls. My logic is very simple.

Matching condition

...

ANSWER

Answered 2021-May-20 at 20:51

Try this instead - Decimal has precision and scale

startsWith(type, 'decimal')

Source https://stackoverflow.com/questions/67621965

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install data-factory

You can download it from GitHub, Maven.
You can use data-factory like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the data-factory component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: