scoring | Crawl scoring server , scripts , and pages

Q : " What is going wrong? "

A :
There is not a single thing that we can say that it "goes wrong", the code-execution eco-system is so multi-layered, that it is not as trivial as we might wish to enjoy & there are several (different, some hidden) places, where configurations decide, how many CPU-cores will actually bear the overall processing-load.

Situation is also version-dependent & configuration-specific ( both Scikit, Numpy, Scipy have mutual dependencies & underlying dependencies on respective compilation options for numerical packages used )

Given a documented feature of interpretation of negative numbers in top-level n_jobs parameter in RandomizedSearchCV(...) methods, submit the very same task, yet configured so that it has got explicit amount of permitted (top-level) n_jobs = CPU_cores_allowed_to_load and observe, when & how many cores do actually get loaded during the whole flow of processing.

Results:
if and only if that very number of "permitted" CPU-cores was loaded, the top-level call did correctly "propagate" the parameter settings to each & every method or procedure used alongside the flow of processing

In case your observation proves the settings were not "obeyed", we can only review the whole scope of all source-code verticals to decide, who is to be blamed for such dis-obedience of not keeping the work compliant with the top-level set ceiling for the n_jobs. While O/S tools for CPU-core affinity mappings may give us some chances to "externally" restrict the number of such cores used, some other adverse effects ( the add-on management costs being the least performance-punishing ones ) will arise - thermal-management introduced CPU-core "hopping", being the disallowed by affinity maps, will on contemporary processors cause a more and more reduced clock-frequency (as cores get indeed hot in numerically intensive processing), thus prolonging the overall task processing times, as there are "cooler" (thus faster) CPU-cores in the system (those, that were prevented from being used by the affinity-mapping), yet these are very the same CPU-cores, that the affinity-mappings disallowed from being used for temporally placing our task processing (while the hot ones, from which the flow of the processing was reallocated due to reached thermal-ceilings, got some time to cold down and re-gain the chances to run at not decreased CPU-clock-rates)

Top-level call might have set an n_jobs-parameter, yet any lower-level component might have "obeyed" that one value ( without knowing, how many other, concurrently working peers did the same - as in joblib.Parallel() and similar constructors do, not mentioning the other, inherently deployed, GIL-evading multithreading libraries - as that happen to lack any mutual coordination so as to keep the top-level set n_jobs-ceiling )

The point of refit is that the model will be refitted using the best parameter set found before and the entire dataset. To find the best parameters, cross-validation is used which means that the dataset is always split into a training and a validation set, i.e. not the entire dataset is used for training here.

When you define multiple metrics, you have to tell scikit-learn how it should determine what best means for you. For convenience, you can just specify any of your scorers to be used as the decider so to say. In that case, the parameter set that maximizes this metric will be used for refitting.

If you want something more sophisticated, like taking the parameter set that returned the highest mean of all scorers, you have to pass a function to refit that given all the created metrics returns the index of the corresponding best parameter set. This parameter set will then be used to refit the model.

Those metrics will be passed as a dictionary of strings as keys and NumPy arrays as values. Those NumPy arrays have as many entries as parameter sets that have been evaluated. You find a lot of things in there. What is probably the most relevant is mean_test_*scorer-name*. Those arrays contain for each tested parameter set the mean scorer-name-scorer computed across the cv splits.

In code, to get the index of the parameter set, that returns the highest mean across all scorers, you can do the following

I tried with scikeras but I got errors because it doesn't accept not-numerical inputs (in our case the input is in str format). So I came back to the standard keras wrapper.

The focal point here is that the model is not built correctly. The TextVectorization must be put inside the Sequential model like shown in the official documentation.

So the build_model function becomes:

apply() can run a function for each row of a dataframe. If you make a simple function to score the way you want, apply can do the rest:

It is simple enough to verify that "encode_a_d" step in the pipe with SklearnTransformerWrapper produces NaNs during cross-validation:

Finally after lot of head banging, I have been able to consistently repro this bug in another Azure ML Workspace.

I tried deploying the same sample in a brand new Azure ML workspace created and it went smoothly.

At this point I remembered that I had upgraded the Storage Account of my previous AML Workspace to DataLake Gen2.

So I did the same upgrade in this new workspace’s storage account. After the upgrade, when I try to deploy the same endpoint, I get the same DriverFileNotFoundError!

It seems Azure ML does not support Storage Account with DataLake Gen2 capabilities although the support page says otherwise. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#supported-data-storage-service-types.

At this point my only option is to recreate a new workspace and deploy my code there. Hope Azure team fixes this soon.

1. To shorten the training process by simply stopping the training for loop after a certain number like so.

You end up with the error with precision because some of your penalization is too strong for this model, if you check the results, you get 0 for f1 score when C = 0.001 and C = 0.01

As the Pipeline object doesn't support .deploy method, how to deploy this pipeline?

Pipeline does not have a .deploy() method, no

Use pipeline.upsert(role_arn='...') to create/update the pipeline definition to SageMaker, then call pipeline.start() . Docs here

While inference/scoring, When we receive a raw data(single row for each source), how to trigger the pipeline?

There are actually two types of pipelines in SageMaker. Model Building Pipelines (which you have in your question), and Serial Inference Pipelines, which are used for Inference. AWS definitely should have called the former "workflows"

You can use a model building pipeline to setup a serial inference pipeline

To do pre-processing in a serial inference pipeline, you want to train an encoder/estimator (such as SKLearn) and save its model. Then train a learning algorithm, and save its model, then create a PipelineModel using both models