data_pipeline | Code for the data processing pipeline | Continuous Deployment library

Default learning rate of Adam is 0.001, which, depending on task, might be too high.

It looks like instead of converging your neural network became divergent (it left the previous ~0.2 loss minima and fell into different region).

Lowering your learning rate at some point (after 50% or 70% percent of training) would probably fix the issue.

Usually people divide the learning rate by 10 (0.0001 in your case) or by half (0.0005 in your case). Try with dividing by half and see if the issue persist, in general you would want to keep your learning rate as high as possible until divergence occurs as is probably the case here.

This is what schedulers are for (gamma specifies learning rate multiplier, might want to change that to 0.5 first).

One can think of lower learning rate phase as fine-tuning already found solution (placing weights in better region of the loss valley) and might require some patience.

The fitted transformers are available in the attributes transformers_ (a list) and named_transformers_ (a dict-like with keys the names you provided). So, for example,

Azure still is not compatible with 3.9. See also at https://github.com/numpy/numpy/issues/17482

If you are trying to preprocess your category features you need to use OneHotEncoder or OrdinalEncoder as per comments.

Here is an example of how to do that:

Edit : pyspark does not support a vector as a target label hence only string encoding works.

The problematic code is -