levenberg-marquardt | Curve fitting method in JavaScript

I'm working on some code that I'm writing which uses the [GNU Scientific Library (GSL)][1]'s Nonlinear least-squares algorithm for curve fitting.

I have been successful in obtaining a working code that estimate the right parameters from the fitting analysis using a C++ wrapper from https://github.com/Eleobert/gsl-curve-fit/blob/master/example.cpp.

Now, I would like to fix some of the parameters of the function to be fit. And I would like to modify the function in such a way that I can already input the value of the parameter to be fixed.

Any idea on how to do? I'm showing here the full code.

This is the code for performing nonlinear least-squares fitting:

I am using lmfit for solving a non-linear optimization problem. It works fine to the point where I am trying to implement a measurement error as the standard deviation of the dependent variable y (sigma_y). My problem is, that I cannot interpret the Information criteria properly. When implementing the return (model - y)/(sigma_y) they just raise from really low to very high values.

i.e. [left: return (model - y) -weighting-> right: return (model - y)/(sigma_y)]:

• chi-square 0.00159805 -> 47.3184972
• reduced chi-square 1.7756e-04 -> 5.25761080 expectation value is 1 || SO discussion
• Akaike info crit -93.2055413 -> 20.0490661 the more negative, the better
• Bayesian info crit -92.4097507 -> 20.8448566 the more negative, the better

My guess is, that this is somehow connected to bad usage of lmfit (wrong calculation of Information criteria, bad error scaling) or to a general lack of understanding these criteria (to me reduced chi-square of 5.258 (under-estimated) or 1.776e-4 (over-estimated) sounds like a really poor fit, but the plot of residuals etc. looks okay for me...)

Here is my example code that reproduces the problem:

I'm using Eigen's Levenberg-Marquardt implementation and wondering how to set some boundaries on the parameters which should be optimized.

As I'm migrating some GNU octave programs to Eigen I expected that there might be some boundaries which can be easily provided as parameters to the module.

The layout of my implemenation is nearly the same as in this example. I'm not providing the df() implemenatation but rather use Eigen::NumericalDiff in order to approximate it.

So how do I enforce some boundaries on the parameters which are supplied to minimize()? I thought about setting the errors(fvec) in the operator() to some high values when leaving my expected ranges, but in some small tests this resulted in strange results.

My code is very simple for one layer of 20 neurons:

I have implemented a 3D gaussian fit using scipy.optimize.leastsq and now I would like to tweak the arguments ftol and xtol to optimize the performances. However, I don't understand the "units" of these two parameters in order to make a proper choice. Is it possible to calculate these two parameters from the results? That would give me an understanding of how to choose them. My data is numpy arrays of np.uint8. I tried to read the FORTRAN source code of MINIPACK but my FORTRAN knowledge is zero. I also read checked the Levenberg-Marquardt algorithm, but I could not really get a number that was below the ftol for example.

Here is a minimal example of what I do:

After a lot of research and experimentation I've learned a few things which have led me to re-frame the question. Rather than trying to find an "exponential regression", I'm really trying to optimize a non-linear error function with bounded input and potentially unbounded output.

So long as a function is linear, there exists a way to directly compute the optimal parameters to minimize the squared error terms (by taking the derivative of the function, locating the point where the derivative equals zero, then using this local minima as your solution). Many times when people say "exponential regression" they're referring to an equation of the form a * e^(b*x). Ths reason, described below, is that by taking the natural log of both sides this maps perfectly onto a linear equation and so can be directly computed in a single step using the same method.

However in my case the equation a * b^x does not map onto a linear equation, and so there is no direct solution. Instead a solution must be determined iteratively.

There are a few non-linear curve fitting algorithms out there. Notably Levenberg-Marquardt. I found a handful of implementations of this algorithm:

• C++: https://www.gnu.org/software/gsl/doc/html/nls.html
• Python: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
• JavaScript: https://github.com/mljs/levenberg-marquardt

Unfortunately I tried all three of these implementations and the curve fitting was just atrocious. I have some sample data with 11,000 points for which I know the optimal parameters are a = 0.08 and b = 1.19, however these algorithms often returned bizarre results like a = 117, b = 0.000001 or a = 0, b = 3243224

Next I tried verifying my understanding of the problem by using Excel. An error function can be written defined as sum((y - y')^2) where y' is the estimated value given your parameters and an input x. The problem then falls to minimizing this error function. I opened up my data (CSV), added a column for the "computed" values, added another column for the squared error terms, then finally used Solver to optimize the sum of the error terms. This worked beautifully! I got back a = 0.0796, b = 1.1897. Plotting two lines on the same graph (original data and estimated data) showed a really good fit.

I tried doing the same using OpenOffice at first, however the solver built into OpenOffice was just as bad as the Levenberg-Marquardt experiments I did, and repeatedly gave worthless solutions. Even when I set initial values it would "optimize" the problem and come up with something far worse than it started.

Having proven my concept in Excel I then tried using optimization-js. I tried both their genetic optimization and powell optimization (because I don't have a gradient function) and in both cases it produced awful results.

I did find a question regarding how Excel's Solver works which linked to an ugly PDF. I haven't taken the time to read the PDF yet, but it may provide hints for solving the problem manually. I also found a Python example that reportedly implements Generalized Gradient Descent (the same algorithm as Excel), so if I can make sense of it and rewrite it to accept a generic function as input then I may be able to use that.

How, preferably in JavaScript (though other languages are acceptable so long as they can be run on AWS Lambda), can I optimize the parameters to the following function to minimize its output?

Some weeks ago I started coding the Levenberg-Marquardt algorithm from scratch in Matlab. I'm interested in the polynomial fitting of the data but I haven't been able to achieve the level of accuracy I would like. I'm using a fifth order polynomial after I tried other polynomials and it seemed to be the best option. The algorithm always converges to the same function minimization no matter what improvements I try to implement. So far, I have unsuccessfully added the following features:

• Geodesic acceleration term as a second order correction
• Delayed gratification for updating the damping parameter
• Gain factor to get closer to the Gauss-Newton direction or the steepest descent direction depending on the iteration.
• Central differences and forward differences for the finite difference method

I don't have experience in nonlinear least squares, so I don't know if there is a way to minimize the residual even more or if there isn't more room for improvement with this method. I attach below an image of the behavior of the polynomial for the last iterations. If I run the code for more iterations, the curve ends up not changing from iteration to iteration. As it is observed, there is a good fit from time = 0 to time = 12. But I'm not able to fix the behavior of the function from time = 12 to time = 20. Any help will be very appreciated.

I would like to minimize x and y in function f using least square (Levenberg-Marquardt). In Python I can use lmfit like follows