engineer | Powerful plugin system | Plugin library

You can try put header=None into .read_csv. Then pandas will throw ParserError if number of columns won't match length of rows. For example:

you start your loop by defining a running variable 'i'. But all over the loop, you redefine it to be other integers and unrelated objects. Such as in line 83, line 155, and others. It's difficult to understand your intentions from the question. but if I understand correctly, the problem can be solved by changing every 'i' in the loop to a differently named temporary variable. A simpler solution would be to change the 'i' variable at the beginning of the for loop to smth else. I suggest you adopt a habit of using variable names that have meaning and not just single or double letters.

You were close to solving it. The short answer is: just add this line to your hibernate.cfg.xml config file (add empty catalog):

Generally using curl -sSL get.docker.com | sh will set you up (it's a shell script that will pick up your OS flavor and install docker automatically). I'm almost fully confident that you'll have no trouble using this method to install it on Kubuntu. Just make sure you have curl installed (which is probably true by default).

As far as I'm aware, Kubuntu is just Ubuntu with a different GUI. Internally it's identical to Ubuntu, which should not bother your Docker installation at all.

The easiest thing you can do is set up a virtual machine with Kubuntu and just test it out to see for yourself.

Check your while conditions (it is always true) , and what you wrote

"loop is when the response is not equal to "y" or "n""

means in pseudo code

NOT (answer = "y" OR answer="n")

When calculating an eigenvector you may change its sign and the solution will also be a valid one.

So any PCA axis can be reversed and the solution will be valid.

Nevertheless, you may wish to impose a positive correlation of a PCA axis with one of the original variables in the dataset, inverting the axis if needed.

I have had an update rejected before. I appealed and the appeal was accepted. It took a few days. You should have details about how to appeal in your email. But in case you don't, here is the link: https://support.google.com/googleplay/android-developer/contact/protectappeals

Don't worry, if it is fine they will reinstate your app.

In this case you do not need Include but SelectMany with custom projection:

Great question. As you say, there are no hard and fast rules here and opinions do vary, but let me share my perspective as a QB data scientist and kedro maintainer who has used the layering convention you referred to several times.

For a start, let me emphasise that there's absolutely no reason to stick to the data engineering convention suggested by kedro if it's not suitable for your needs. 99% of users don't change the folder structure in data. This is not because the kedro default is the right structure for them but because they just don't think of changing it. You should absolutely add/remove/rename layers to suit yourself. The most important thing is to choose a set of layers (or even a non-layered structure) that works for your project rather than trying to shoehorn your datasets to fit the kedro default suggestion.

Now, assuming you are following kedro's suggested structure - onto your questions:

When is a dataset a feature rather than a primary dataset? The distinction seems vague...

In the case of simple features, a feature dataset can be very similar to a primary one. The distinction is maybe clearest if you think about more complex features, e.g. formed by aggregating over time windows. A primary dataset would have a column that gives a cleaned version of the original data, but without doing any complex calculations on it, just simple transformations. Say the raw data is the colour of all cars driving past your house over a week. By the time the data is in primary, it will be clean (e.g. correcting "rde" to "red", maybe mapping "crimson" and "red" to the same colour). Between primary and the feature layer, we will have done some less trivial calculations on it, e.g. to find one-hot encoded most common car colour each day.

Is it OK for a primary dataset to consume data from another primary dataset?

In my opinion, yes. This might be necessary if you want to join multiple primary tables together. In general if you are building complex pipelines it will become very difficult if you don't allow this. e.g. in the feature layer I might want to form a dataset containing composite_feature = feature_1 * feature_2 from the two inputs feature_1 and feature_2. There's no way of doing this without having multiple sub-layers within the feature layer.

However, something that is generally worth avoiding is a node that consumes data from many different layers. e.g. a node that takes in one dataset from the feature layer and one from the intermediate layer. This seems a bit strange (why has the latter dataset not passed through the feature layer?).

Is it good practice to build a feature dataset from the INT layer? or should it always pass through Primary?

Building features from the intermediate layer isn't unheard of, but it seems a bit weird. The primary layer is typically an important one which forms the basis for all feature engineering. If your data is in a shape that you can build features then that means it's probably primary layer already. In this case, maybe you don't need an intermediate layer.

The above points might be summarised by the following rules (which should no doubt be broken when required):

1. The input datasets for a node in layer L should all be in the same layer, which can be either L or L-1
2. The output datasets for a node in layer L should all be in the same layer L, which can be either L or L+1

If anyone can offer any further advice or blogs\docs talking about Kedro Data Modelling that would be awesome!

I'm also interested in seeing what others think here! One possibly useful thing to note is that kedro was inspired by cookiecutter data science, and the kedro layer structure is an extended version of what's suggested there. Maybe other projects have taken this directory structure and adapted it in different ways.