SmartImage | Reverse image search tool ( SauceNao ImgOps trace.moe | Search Engine library

The code looks fine at first glance. You should first check that your code is capable of converging to a target at all , for example by feeding a target image that is either generated by your algorithm with a random genome (or a very simple image that it should be easily recreated by your algorithm).

You are using the SAD (sum of absolute differences) metric between pixels to calculate fitness. You can try using SSD (sum of squared differences) like you are doing in the histogram difference method but between pixels or blocks, that will heavily penalize large differences so the remaining images won't be too different from the target. You can try using a more perceptual image space like HSV so the images will be closer visually even if they are farther in RGB space.

I think comparing the histogram of the entire image may be too lax, as there are many different images that will result in the same histogram. Comparing individual pixels may be too strict, the image needs to be aligned very precisely to get low differences, so everything gets low fitness values unless you are very lucky so the convergence will be too slow. I would recommend that you compare the histogram between overlapping blocks, and don't use all the 256 levels, use only about 16 levels or so (or use some kind of overlapping).

Read about Histogram of oriented gradients (HOG) and other similar techniques to get ideas to improve your fitness function. I took an online course about object recognition in images, Coursera - Deteccion de Objetos by the University of Barcelona but it's in Spanish. I'm pretty sure you can find similar study materials in English.

Edit: before trying something more complex a good idea would be doing the SAD or SSD on the average of each overlapping block (which would have a similar effect to strongly blurring the reference and generated images and then comparing the pixels, but faster). The fitness function should be resilient against small changes. An image that it's shifted by a few pixels or that is very similar after discarding the low-level detail should have much better fitness than a very different image and I think blurring will have that effect.