Data-Engineering | REST API for storing and retrieving documents info

Here you have a working example:

Your URL is correct. Just check the following and then it should work:

1. Add the MSI of the workspace to the workspace resource itself with Role = Contributor

2. In the web activity, set the Resource to "https://dev.azuresynapse.net/" (without the quotes, obviously) This was a bit buried in the docs, see last bullet of this section here: https://docs.microsoft.com/en-us/rest/api/synapse/#common-parameters-and-headers

NOTE: the REST API is unable to cancel pipelines run in DEBUG in Synapse (you'll get an error response saying pipeline with that ID is not found). This means for it to work, you have to first publish the pipelines and then trigger them.

Can produce a single file utilising the copy tool but it works out a bit better using the external table and then merging the files after.

the second stack fails to find it

This is because nested CloudFormation stacks are created in parallel by default.

This means that if one of your child stacks - e.g. the stack which contains KinesisFirehoseRole - is importing the output from another child stack - e.g. the stack which contains KinesisStream - then the stack creation will fail.

This is because as they're created in parallel, how does CloudFormation ensure that the export value has been exported by the time another child stack created is importing it?

To fix this, use the DependsOn attribute on the stack which contains KinesisFirehoseRole.

This should point to the stack which contains KinesisStream as KinesisFirehoseRole has a dependency on it.

DependsOn makes this dependency explicit and will ensure correct stack creation order.

Something like this should work:

According to this example, Secret is a special class that will handle creating volume mounts automatically. Looking at your code, seems that your own volume with mount /credentials is overriding /credentials mount created by Secret, and because you provide empty configs={}, that mount is empty as well.

Try supplying just secrets=[secret_jira_user,secret_storage_credentials] and removing manual volume_mounts.

• Code that generates secret volume mounts under the hood

That syntax will not work on Azure Synapse Analytics dedicated SQL pools and you will receive the following error(s):

Msg 103010, Level 16, State 1, Line 1 Parse error at line: 2, column: 40: Incorrect syntax near 'WITH'.

Msg 104467, Level 16, State 1, Line 1 Enforced unique constraints are not supported. To create an unenforced unique constraint you must include the NOT ENFORCED syntax as part of your statement.

The way to write this syntax would be using ALTER TABLE to add a non-clustered and non-enforced primary key, eg

Great question. As you say, there are no hard and fast rules here and opinions do vary, but let me share my perspective as a QB data scientist and kedro maintainer who has used the layering convention you referred to several times.

For a start, let me emphasise that there's absolutely no reason to stick to the data engineering convention suggested by kedro if it's not suitable for your needs. 99% of users don't change the folder structure in data. This is not because the kedro default is the right structure for them but because they just don't think of changing it. You should absolutely add/remove/rename layers to suit yourself. The most important thing is to choose a set of layers (or even a non-layered structure) that works for your project rather than trying to shoehorn your datasets to fit the kedro default suggestion.

Now, assuming you are following kedro's suggested structure - onto your questions:

When is a dataset a feature rather than a primary dataset? The distinction seems vague...

In the case of simple features, a feature dataset can be very similar to a primary one. The distinction is maybe clearest if you think about more complex features, e.g. formed by aggregating over time windows. A primary dataset would have a column that gives a cleaned version of the original data, but without doing any complex calculations on it, just simple transformations. Say the raw data is the colour of all cars driving past your house over a week. By the time the data is in primary, it will be clean (e.g. correcting "rde" to "red", maybe mapping "crimson" and "red" to the same colour). Between primary and the feature layer, we will have done some less trivial calculations on it, e.g. to find one-hot encoded most common car colour each day.

Is it OK for a primary dataset to consume data from another primary dataset?

In my opinion, yes. This might be necessary if you want to join multiple primary tables together. In general if you are building complex pipelines it will become very difficult if you don't allow this. e.g. in the feature layer I might want to form a dataset containing composite_feature = feature_1 * feature_2 from the two inputs feature_1 and feature_2. There's no way of doing this without having multiple sub-layers within the feature layer.

However, something that is generally worth avoiding is a node that consumes data from many different layers. e.g. a node that takes in one dataset from the feature layer and one from the intermediate layer. This seems a bit strange (why has the latter dataset not passed through the feature layer?).

Is it good practice to build a feature dataset from the INT layer? or should it always pass through Primary?

Building features from the intermediate layer isn't unheard of, but it seems a bit weird. The primary layer is typically an important one which forms the basis for all feature engineering. If your data is in a shape that you can build features then that means it's probably primary layer already. In this case, maybe you don't need an intermediate layer.

The above points might be summarised by the following rules (which should no doubt be broken when required):

1. The input datasets for a node in layer L should all be in the same layer, which can be either L or L-1
2. The output datasets for a node in layer L should all be in the same layer L, which can be either L or L+1

If anyone can offer any further advice or blogs\docs talking about Kedro Data Modelling that would be awesome!

I'm also interested in seeing what others think here! One possibly useful thing to note is that kedro was inspired by cookiecutter data science, and the kedro layer structure is an extended version of what's suggested there. Maybe other projects have taken this directory structure and adapted it in different ways.

Entire sbt file is showing in red including the name, version, scalaVersion

This is likely caused by some missing configuration in IntelliJ, you should have some kind of popup that aks you to "configure Scala SDK". If not, you can go to your module settings and add the Scala SDK.

when i compile following is the error which i am getting now

If you look closely to the error, you should notice this message:

I recommend you execute the COPY INTO command from within Airflow to load the files directly from S3, instead. There isn't a great way to get files to internal stage from S3 without hopping the files to another machine (like the Airflow machine). You'd use SnowSQL to GET from S3 to local, and the PUT from local to S3. The only way to execute a PUT to Internal Stage is through SnowSQL.