datascience | assignments discussed in data science course | Machine Learning library

Typical solution is to build external partitioned table on top of hdfs directory:

You should use df.loc[indexes] to select rows by their indexes. If you want to select rows by their integer location you should use df.iloc[indexes].

In addition to that, you can read this page on Indexing and Selecting data with pandas.

Minimizing error is a huge, complex problem. As such, a lot of very clever people have thought up a lot of cool solutions. Here are a few:

(out of all of them, I think bayesian optimization with sklearn might be a good choice for your use case, though I've never used it)

(also, delete the last "s" in the image url to see the full size)

• genetic algorithms: formats your problem like chromosomes in a genome and "breeds" an optimal solution (a personal favorite of mine)

• simulated anealing: formats your problem like hot metal being annealed, which attempts to move to a stable state while losing heat

• random search: better than it sounds. randomly tests a verity of input variables.

• Grid Search: Simple to implement, but often less effective than methods which employ true randomness (duplicate exploration along particular axis of interest. This strategy often wastes computational resources)

A lot of these come up in hyperparameter optimization for ML models.

• Gradient Descent: uses the gradient calculated in a differentiable function to step toward local minima

• DeepAR: uses Bayesian optimization, combined with random search, to reduce loss in hyperparameter tuning. While I believe this is only available on AWS, It looks like sklearn has an implementation of Bayesian optimization

• scipy.optimize.minimize: I know you're already using this, but there are 15 different algorithms that can be used by changing the method flag.

while error minimization is simple conceptually, in practice complex error topologies in high dimensional spaces can be very difficult to traverse efficiently. It harkens to local and global extrema, the explore/exploit problem, and our mathematical understanding of what computational complexity even is. Often, a good error reduction is accomplished through a combination of thorough understanding of the problem, and experimentation with multiple algorithms and hyperparameters. In ML, this is often referred to as hyperparameter tuning, and is a sort of "meta" error reduction step, if you will.

note: feel free to recommend more optimization methods, I'll add them to the list.

You need only two axes. ax2 has to be the twin axis of ax1. You can plot as many plots as you want on each axis. See the following code where sin and cos are plotted on the left y-axis whereas cubic and quadratic are plotted on the right y-axis:

For us it ended up being an empty .seq file that was written by one of our ETL tools. Removing that invalid file resolved the issue for us.

We need ifelse instead of if/else as if/else is not vectorized

This finally solved it:

You can consider following example: