kandi background
Explore Kits

scipy | SciPy library main repository

 by   scipy Python Version: v1.8.0 License: BSD-3-Clause

 by   scipy Python Version: v1.8.0 License: BSD-3-Clause

Download this library from

kandi X-RAY | scipy Summary

scipy is a Python library. scipy has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub.
SciPy library main repository
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • scipy has a highly active ecosystem.
  • It has 9300 star(s) with 4133 fork(s). There are 332 watchers for this library.
  • There were 5 major release(s) in the last 12 months.
  • There are 1469 open issues and 6486 have been closed. On average issues are closed in 460 days. There are 321 open pull requests and 0 closed requests.
  • It has a positive sentiment in the developer community.
  • The latest version of scipy is v1.8.0
scipy Support
Best in #Python
Average in #Python
scipy Support
Best in #Python
Average in #Python

quality kandi Quality

  • scipy has 0 bugs and 0 code smells.
scipy Quality
Best in #Python
Average in #Python
scipy Quality
Best in #Python
Average in #Python

securitySecurity

  • scipy has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • scipy code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.
scipy Security
Best in #Python
Average in #Python
scipy Security
Best in #Python
Average in #Python

license License

  • scipy is licensed under the BSD-3-Clause License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
scipy License
Best in #Python
Average in #Python
scipy License
Best in #Python
Average in #Python

buildReuse

  • scipy releases are available to install and integrate.
  • Build file is available. You can build the component from source.
  • scipy saves you 191325 person hours of effort in developing the same functionality from scratch.
  • It has 216613 lines of code, 17977 functions and 957 files.
  • It has high code complexity. Code complexity directly impacts maintainability of the code.
scipy Reuse
Best in #Python
Average in #Python
scipy Reuse
Best in #Python
Average in #Python
Top functions reviewed by kandi - BETA

kandi has reviewed scipy and discovered the below as its top functions. This is intended to give you an instant insight into scipy implemented functionality, and help decide if they suit your requirements.

  • Calculate least squares .
    • Linear Grammar algorithm .
      • r Solve an IVP .
        • r Solve a problem .
          • Compute the lsqr of A and b .
            • Integrate quadratic integrand .
              • Test a permutation test .
                • r Solve the linear operator .
                  • r Solve a binary quadratic problem .
                    • Solve linear problem .

                      Get all kandi verified functions for this library.

                      Get all kandi verified functions for this library.

                      scipy Key Features

                      SciPy library main repository

                      Padding scipy affine_transform output to show non-overlapping regions of transformed images

                      copy iconCopydownload iconDownload
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      from scipy.ndimage import rotate, shift
                      
                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      
                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      
                      np.pad(src, number, mode='constant')
                      
                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      
                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      
                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      
                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      
                      def centralized(a, width, height):
                          '''
                          Image centralized to the given width and height
                          by padding with zeros (black)
                          '''
                          assert width >= a.shape[0] and height >= a.shape[1]
                          ap = np.zeros((width, height) + a.shape[2:], a.dtype)
                          ccx = (width - a.shape[0])//2
                          ccy = (height - a.shape[1])//2
                          ap[ccx:ccx+a.shape[0], ccy:ccy+a.shape[1], ...] = a
                          return ap
                      def image_pair(im, width, height, displacement=(0,0), angle=0):
                          '''
                          this build an a pair of images as numpy arrays
                          from the input image.
                          Both images will be padded with zeros (black)
                          and roughly centralized.
                          and will have the specified shape
                          
                          make sure that the width and height chosen are enough 
                          to fit the rotated image
                          '''
                          a = np.array(im)
                          a1 = centralized(a, width, height)
                          a2 = centralized(ndimage.rotate(a, angle), width, height)
                          a2 = np.roll(a2, displacement, axis=(0,1))
                          return a1, a2
                      
                      def random_transform():
                          angle = np.random.rand() * 360
                          displacement = np.random.randint(-100, 100, 2)
                          return displacement, angle
                      
                      a1, a2 = image_pair(im, 512, 512, *random_transform())
                      plt.subplot(121)
                      plt.imshow(a1)
                      plt.subplot(122)
                      plt.imshow(a2)
                      
                      
                      def compute_correlation(a1, a2):
                          A1 = np.fft.rfftn(a1, axes=(0,1))
                          A2 = np.fft.rfftn(a2, axes=(0,1))
                          C = np.fft.irfftn(np.sum(A1 * np.conj(A2), axis=2))
                          return C
                      
                      displacement, _ = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle=0)
                      C = compute_correlation(a1, a2)
                      np.unravel_index(np.argmax(C), C.shape), displacement
                      a3 = np.roll(a2, np.unravel_index(np.argmax(C), C.shape), axis=(0,1))
                      assert np.all(a3 == a1)
                      
                      def get_aligned(a1, a2, angle):
                          a1_rotated = ndimage.rotate(a1, angle, reshape=False)
                          C = compute_correlation(a2, a1_rotated)
                          found_displacement = np.unravel_index(np.argmax(C), C.shape)
                          a1_aligned = np.roll(a1_rotated, found_displacement, axis=(0,1))
                          return a1_aligned
                      
                      displacement, angle = random_transform()
                      a1, a2 = image_pair(im, 521, 512, displacement, angle)
                      C_max = []
                      C_argmax = []
                      angle_guesses = np.arange(0, 360, 5)
                      for angle_guess in angle_guesses:
                          a1_rotated = ndimage.rotate(a1, angle_guess, reshape=False)
                          C = compute_correlation(a1_rotated, a2)
                          i = np.argmax(C)
                          v = C.reshape(-1)[i]
                          C_max.append(v)
                          C_argmax.append(i)
                      
                      plt.plot(angle_guesses, C_max);
                      
                      a1_aligned = get_aligned(a1, a2, angle_guesses[np.argmax(C_max)])
                      plt.subplot(121)
                      plt.imshow(a2)
                      plt.subplot(122)
                      plt.imshow(a1_aligned)
                      
                      def affine_test(angle=0, translate=(0, 0), shape=(200, 100), buffered_shape=(300, 200), nblob=50):
                          # Maxiumum translation allowed is half difference between shape and buffered_shape
                      
                          np.random.seed(42)
                      
                          # Generate a buffered_shape-sized base image
                          base = np.zeros(buffered_shape, dtype=np.float32)
                          random_locs = np.random.choice(np.arange(2, buffered_shape[0] - 2), nblob * 2, replace=False)
                          i = random_locs[:nblob]
                          j = random_locs[nblob:]
                          for k, (_i, _j) in enumerate(zip(i, j)):
                              base[_i - 2 : _i + 2, _j - 2 : _j + 2] = k + 10
                      
                          # Impose a rotation and translation on source
                          src = rotate(base, angle, reshape=False, order=1, mode="constant")
                          bsc = (np.array(buffered_shape) / 2).astype(int)
                          sc = (np.array(shape) / 2).astype(int)
                          src = src[
                              bsc[0] - sc[0] + translate[0] : bsc[0] + sc[0] + translate[0],
                              bsc[1] - sc[1] + translate[1] : bsc[1] + sc[1] + translate[1],
                          ]
                          # Cut-out destination from the centre of the base image
                          dst = base[bsc[0] - sc[0] : bsc[0] + sc[0], bsc[1] - sc[1] : bsc[1] + sc[1]]
                      
                          src_y, src_x = src.shape
                      
                          def get_matrix_offset(centre, angle, scale):
                              """Follows OpenCV.getRotationMatrix2D"""
                              angle_rad = angle * np.pi / 180
                              alpha = np.round(scale * np.cos(angle_rad), 8)
                              beta = np.round(scale * np.sin(angle_rad), 8)
                              return (
                                  np.array([[alpha, beta], [-beta, alpha]]),
                                  np.array(
                                      [
                                          (1 - alpha) * centre[0] - beta * centre[1],
                                          beta * centre[0] + (1 - alpha) * centre[1],
                                      ]
                                  ),
                              )
                      
                          matrix, offset = get_matrix_offset(np.array([((src_y - 1) / 2) - translate[0], ((src_x - 1) / 2) - translate[
                          1]]), angle, 1)
                      
                          offset += np.array(translate)
                      
                          M = np.column_stack((matrix, offset))
                          M = np.vstack((M, [0, 0, 1]))
                          iM = np.linalg.inv(M)
                          imatrix = iM[:2, :2]
                          ioffset = iM[:2, 2]
                      
                          # Determine the outer bounds of the new image
                          lin_pts = np.array([[0, src_y-1, src_y-1, 0], [0, 0, src_x-1, src_x-1]])
                          transf_lin_pts = np.dot(matrix, lin_pts) + offset.reshape(2, 1) # - np.array(translate).reshape(2, 1) # both?
                      
                          # Find min and max bounds of the transformed image
                          min_x = np.floor(np.min(transf_lin_pts[1])).astype(int)
                          min_y = np.floor(np.min(transf_lin_pts[0])).astype(int)
                          max_x = np.ceil(np.max(transf_lin_pts[1])).astype(int)
                          max_y = np.ceil(np.max(transf_lin_pts[0])).astype(int)
                      
                          # Add translation to the transformation matrix to shift to positive values
                          anchor_x, anchor_y = 0, 0
                          if min_x < 0:
                              anchor_x = -min_x
                          if min_y < 0:
                              anchor_y = -min_y
                      
                          dot_anchor = np.dot(imatrix, [anchor_y, anchor_x])
                          shifted_offset = ioffset - dot_anchor
                      
                          # Create padded destination image
                          dst_y, dst_x = dst.shape[:2]
                          pad_widths = [anchor_y, max(max_y, dst_y) - dst_y, anchor_x, max(max_x, dst_x) - dst_x]
                          dst_padded = np.pad(
                              dst,
                              ((pad_widths[0], pad_widths[1]), (pad_widths[2], pad_widths[3])),
                              "constant",
                              constant_values=-10,
                          )
                      
                          dst_pad_y, dst_pad_x = dst_padded.shape
                          # Create the aligned and padded source image
                          source_aligned = affine_transform(
                              src,
                              imatrix,
                              offset=shifted_offset,
                              output_shape=(dst_pad_y, dst_pad_x),
                              order=3,
                              mode="constant",
                              cval=-10,
                          )
                      
                      affine_test(angle=-25, translate=(10, -40))
                      
                      def affine_test(angle=0, translate=(0, 0), shape=(200, 100), buffered_shape=(300, 200), nblob=50):
                          # Maxiumum translation allowed is half difference between shape and buffered_shape
                      
                          np.random.seed(42)
                      
                          # Generate a buffered_shape-sized base image
                          base = np.zeros(buffered_shape, dtype=np.float32)
                          random_locs = np.random.choice(np.arange(2, buffered_shape[0] - 2), nblob * 2, replace=False)
                          i = random_locs[:nblob]
                          j = random_locs[nblob:]
                          for k, (_i, _j) in enumerate(zip(i, j)):
                              base[_i - 2 : _i + 2, _j - 2 : _j + 2] = k + 10
                      
                          # Impose a rotation and translation on source
                          src = rotate(base, angle, reshape=False, order=1, mode="constant")
                          bsc = (np.array(buffered_shape) / 2).astype(int)
                          sc = (np.array(shape) / 2).astype(int)
                          src = src[
                              bsc[0] - sc[0] + translate[0] : bsc[0] + sc[0] + translate[0],
                              bsc[1] - sc[1] + translate[1] : bsc[1] + sc[1] + translate[1],
                          ]
                          # Cut-out destination from the centre of the base image
                          dst = base[bsc[0] - sc[0] : bsc[0] + sc[0], bsc[1] - sc[1] : bsc[1] + sc[1]]
                      
                          src_y, src_x = src.shape
                      
                          def get_matrix_offset(centre, angle, scale):
                              """Follows OpenCV.getRotationMatrix2D"""
                              angle_rad = angle * np.pi / 180
                              alpha = np.round(scale * np.cos(angle_rad), 8)
                              beta = np.round(scale * np.sin(angle_rad), 8)
                              return (
                                  np.array([[alpha, beta], [-beta, alpha]]),
                                  np.array(
                                      [
                                          (1 - alpha) * centre[0] - beta * centre[1],
                                          beta * centre[0] + (1 - alpha) * centre[1],
                                      ]
                                  ),
                              )
                      
                          matrix, offset = get_matrix_offset(np.array([((src_y - 1) / 2) - translate[0], ((src_x - 1) / 2) - translate[
                          1]]), angle, 1)
                      
                          offset += np.array(translate)
                      
                          M = np.column_stack((matrix, offset))
                          M = np.vstack((M, [0, 0, 1]))
                          iM = np.linalg.inv(M)
                          imatrix = iM[:2, :2]
                          ioffset = iM[:2, 2]
                      
                          # Determine the outer bounds of the new image
                          lin_pts = np.array([[0, src_y-1, src_y-1, 0], [0, 0, src_x-1, src_x-1]])
                          transf_lin_pts = np.dot(matrix, lin_pts) + offset.reshape(2, 1) # - np.array(translate).reshape(2, 1) # both?
                      
                          # Find min and max bounds of the transformed image
                          min_x = np.floor(np.min(transf_lin_pts[1])).astype(int)
                          min_y = np.floor(np.min(transf_lin_pts[0])).astype(int)
                          max_x = np.ceil(np.max(transf_lin_pts[1])).astype(int)
                          max_y = np.ceil(np.max(transf_lin_pts[0])).astype(int)
                      
                          # Add translation to the transformation matrix to shift to positive values
                          anchor_x, anchor_y = 0, 0
                          if min_x < 0:
                              anchor_x = -min_x
                          if min_y < 0:
                              anchor_y = -min_y
                      
                          dot_anchor = np.dot(imatrix, [anchor_y, anchor_x])
                          shifted_offset = ioffset - dot_anchor
                      
                          # Create padded destination image
                          dst_y, dst_x = dst.shape[:2]
                          pad_widths = [anchor_y, max(max_y, dst_y) - dst_y, anchor_x, max(max_x, dst_x) - dst_x]
                          dst_padded = np.pad(
                              dst,
                              ((pad_widths[0], pad_widths[1]), (pad_widths[2], pad_widths[3])),
                              "constant",
                              constant_values=-10,
                          )
                      
                          dst_pad_y, dst_pad_x = dst_padded.shape
                          # Create the aligned and padded source image
                          source_aligned = affine_transform(
                              src,
                              imatrix,
                              offset=shifted_offset,
                              output_shape=(dst_pad_y, dst_pad_x),
                              order=3,
                              mode="constant",
                              cval=-10,
                          )
                      
                      affine_test(angle=-25, translate=(10, -40))
                      

                      Cannot import name '_centered' from 'scipy.signal.signaltools'

                      copy iconCopydownload iconDownload
                          from statsmodels.tsa.seasonal import seasonal_decompose
                        File "/usr/local/lib/python3.8/site-packages/statsmodels/tsa/seasonal.py", line 12, in <module>
                          from statsmodels.tsa.filters.filtertools import convolution_filter
                        File "/usr/local/lib/python3.8/site-packages/statsmodels/tsa/filters/filtertools.py", line 18, in <module>
                          from scipy.signal.signaltools import _centered as trim_centered
                      ImportError: cannot import name '_centered' from 'scipy.signal.signaltools' (/usr/local/lib/python3.8/site-packages/scipy/signal/signaltools.py)
                      
                      import  scipy.signal.signaltools
                      
                      def _centered(arr, newsize):
                          # Return the center newsize portion of the array.
                          newsize = np.asarray(newsize)
                          currsize = np.array(arr.shape)
                          startind = (currsize - newsize) // 2
                          endind = startind + newsize
                          myslice = [slice(startind[k], endind[k]) for k in range(len(endind))]
                          return arr[tuple(myslice)]
                      
                      scipy.signal.signaltools._centered = _centered
                      

                      Installing scipy and scikit-learn on apple m1

                      copy iconCopydownload iconDownload
                      # SciPy:
                      python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
                      pyenv rehash
                      python -m pip install --no-cache --no-binary :all: --no-use-pep517 scipy"==1.7.1"
                      
                      # Scikit-Learn
                      python -m pip install --no-use-pep517 scikit-learn"==0.24.2"
                      
                      brew install openblas openssl@1.1 pkg-config pyenv pyenv-virtualenv
                      python -m pip install numpy==1.19.5
                      
                      # SciPy:
                      python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
                      pyenv rehash
                      python -m pip install --no-cache --no-binary :all: --no-use-pep517 scipy"==1.7.1"
                      
                      # Scikit-Learn
                      python -m pip install --no-use-pep517 scikit-learn"==0.24.2"
                      
                      brew install openblas openssl@1.1 pkg-config pyenv pyenv-virtualenv
                      python -m pip install numpy==1.19.5
                      
                      
                      
                          >> /opt/homebrew/bin/brew install openblas
                          >> export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
                          >> export CFLAGS="-falign-functions=8 ${CFLAGS}"
                          >> git clone https://github.com/scipy/scipy.git
                          >> cd scipy
                          >> git submodule update --init
                          >> /opt/homebrew/bin/pip3 install .
                          >> /opt/homebrew/bin/pip3 install scikit-learn
                      
                      
                      RUN OPENBLAS="/opt/homebrew/opt/openblas" CFLAGS="-falign-functions=8 ${CFLAGS}" pip3 install scipy
                      RUN OPENBLAS="/opt/homebrew/opt/openblas" CFLAGS="-falign-functions=8 ${CFLAGS}" pip3 install scikit-learn
                      
                      brew install openblas
                      export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
                      export CFLAGS="-falign-functions=8 ${CFLAGS}"
                      # ^ no need to add to .zshrc, just doing this once.
                      pip install scikit-learn # ==0.24.1 if you want
                      
                      Building wheels for collected packages: scikit-learn
                        Building wheel for scikit-learn (pyproject.toml) ... done
                        Created wheel for scikit-learn: filename=scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl size=6364030 sha256=0b0cc9a21af775e0c8077ee71698ff62da05ab62efc914c5c15cd4bf97867b31
                      Successfully built scikit-learn
                      Installing collected packages: scipy, scikit-learn
                      Successfully installed scikit-learn-1.0.1 scipy-1.7.3
                      
                      Collecting click>=7.0
                        Downloading click-8.0.3-py3-none-any.whl
                      
                      Collecting grpcio>=1.28.1
                        Downloading grpcio-1.42.0.tar.gz (21.3 MB)
                           |████████████████████████████████| 21.3 MB 12.7 MB/s
                        Preparing metadata (setup.py) ... done
                      
                      ## later in the process it installs using setuptools 
                      Running setup.py install for grpcio ... done
                      
                      brew install openblas
                      export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
                      export CFLAGS="-falign-functions=8 ${CFLAGS}"
                      # ^ no need to add to .zshrc, just doing this once.
                      pip install scikit-learn # ==0.24.1 if you want
                      
                      Building wheels for collected packages: scikit-learn
                        Building wheel for scikit-learn (pyproject.toml) ... done
                        Created wheel for scikit-learn: filename=scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl size=6364030 sha256=0b0cc9a21af775e0c8077ee71698ff62da05ab62efc914c5c15cd4bf97867b31
                      Successfully built scikit-learn
                      Installing collected packages: scipy, scikit-learn
                      Successfully installed scikit-learn-1.0.1 scipy-1.7.3
                      
                      Collecting click>=7.0
                        Downloading click-8.0.3-py3-none-any.whl
                      
                      Collecting grpcio>=1.28.1
                        Downloading grpcio-1.42.0.tar.gz (21.3 MB)
                           |████████████████████████████████| 21.3 MB 12.7 MB/s
                        Preparing metadata (setup.py) ... done
                      
                      ## later in the process it installs using setuptools 
                      Running setup.py install for grpcio ... done
                      
                      brew install openblas
                      export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
                      export CFLAGS="-falign-functions=8 ${CFLAGS}"
                      # ^ no need to add to .zshrc, just doing this once.
                      pip install scikit-learn # ==0.24.1 if you want
                      
                      Building wheels for collected packages: scikit-learn
                        Building wheel for scikit-learn (pyproject.toml) ... done
                        Created wheel for scikit-learn: filename=scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl size=6364030 sha256=0b0cc9a21af775e0c8077ee71698ff62da05ab62efc914c5c15cd4bf97867b31
                      Successfully built scikit-learn
                      Installing collected packages: scipy, scikit-learn
                      Successfully installed scikit-learn-1.0.1 scipy-1.7.3
                      
                      Collecting click>=7.0
                        Downloading click-8.0.3-py3-none-any.whl
                      
                      Collecting grpcio>=1.28.1
                        Downloading grpcio-1.42.0.tar.gz (21.3 MB)
                           |████████████████████████████████| 21.3 MB 12.7 MB/s
                        Preparing metadata (setup.py) ... done
                      
                      ## later in the process it installs using setuptools 
                      Running setup.py install for grpcio ... done
                      
                      brew install openblas
                      export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
                      export CFLAGS="-falign-functions=8 ${CFLAGS}"
                      # ^ no need to add to .zshrc, just doing this once.
                      pip install scikit-learn # ==0.24.1 if you want
                      
                      Building wheels for collected packages: scikit-learn
                        Building wheel for scikit-learn (pyproject.toml) ... done
                        Created wheel for scikit-learn: filename=scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl size=6364030 sha256=0b0cc9a21af775e0c8077ee71698ff62da05ab62efc914c5c15cd4bf97867b31
                      Successfully built scikit-learn
                      Installing collected packages: scipy, scikit-learn
                      Successfully installed scikit-learn-1.0.1 scipy-1.7.3
                      
                      Collecting click>=7.0
                        Downloading click-8.0.3-py3-none-any.whl
                      
                      Collecting grpcio>=1.28.1
                        Downloading grpcio-1.42.0.tar.gz (21.3 MB)
                           |████████████████████████████████| 21.3 MB 12.7 MB/s
                        Preparing metadata (setup.py) ... done
                      
                      ## later in the process it installs using setuptools 
                      Running setup.py install for grpcio ... done
                      
                      Python 3.9.10 (main, Jan 15 2022, 11:40:53) 
                      [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
                      pip 22.0.3
                      
                      brew install openblas gfortran
                      OPENBLAS="$(brew --prefix openblas)" pip install numpy==1.19.3
                      OPENBLAS="$(brew --prefix openblas)" pip install scipy==1.7.2
                      
                      pip install cython
                      brew install libomp
                      export CC=/usr/bin/clang
                      export CXX=/usr/bin/clang++
                      export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
                      export CFLAGS="$CFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export CXXFLAGS="$CXXFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export LDFLAGS="$LDFLAGS -L/opt/homebrew/Cellar/libomp/13.0.1/lib -lomp"
                      export DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/libomp/13.0.1/lib
                      pip install scikit-learn==0.21.3
                      
                      Python 3.9.10 (main, Jan 15 2022, 11:40:53) 
                      [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
                      pip 22.0.3
                      
                      brew install openblas gfortran
                      OPENBLAS="$(brew --prefix openblas)" pip install numpy==1.19.3
                      OPENBLAS="$(brew --prefix openblas)" pip install scipy==1.7.2
                      
                      pip install cython
                      brew install libomp
                      export CC=/usr/bin/clang
                      export CXX=/usr/bin/clang++
                      export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
                      export CFLAGS="$CFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export CXXFLAGS="$CXXFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export LDFLAGS="$LDFLAGS -L/opt/homebrew/Cellar/libomp/13.0.1/lib -lomp"
                      export DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/libomp/13.0.1/lib
                      pip install scikit-learn==0.21.3
                      
                      Python 3.9.10 (main, Jan 15 2022, 11:40:53) 
                      [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
                      pip 22.0.3
                      
                      brew install openblas gfortran
                      OPENBLAS="$(brew --prefix openblas)" pip install numpy==1.19.3
                      OPENBLAS="$(brew --prefix openblas)" pip install scipy==1.7.2
                      
                      pip install cython
                      brew install libomp
                      export CC=/usr/bin/clang
                      export CXX=/usr/bin/clang++
                      export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
                      export CFLAGS="$CFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export CXXFLAGS="$CXXFLAGS -I/opt/homebrew/Cellar/libomp/13.0.1/include"
                      export LDFLAGS="$LDFLAGS -L/opt/homebrew/Cellar/libomp/13.0.1/lib -lomp"
                      export DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/libomp/13.0.1/lib
                      pip install scikit-learn==0.21.3
                      
                      conda install --channel=conda-forge scikit-learn
                      

                      How could I speed up my written python code: spheres contact detection (collision) using spatial searching

                      copy iconCopydownload iconDownload
                      from pyflann import FLANN
                      
                      p = np.loadtxt("pos_large.csv", delimiter=",")
                      flann = FLANN()
                      flann.build_index(pts=p)
                      idx, dist = flann.nn_index(qpts=p, num_neighbors=50)
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = [np.empty([1, 2], dtype=np.int64)]
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              # Find the nearest point including the current one and
                              # then remove the current point from the output.
                              # The distances can be computed directly without a new query.
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max), dtype=np.int64)
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              if particle_idx > 0:
                                  ends_ind.append(ends_ind_mod_temp)
                              else:
                                  ends_ind[0][:] = ends_ind_mod_temp[0, 0], ends_ind_mod_temp[0, 1]
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)                                # <--- relatively high time consumer
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1], float64)')
                      def compute(poss, all_neighbours, all_neighbours_sizes, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind_lst = [np.empty((1, 2), dtype=np.int64)]
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = all_neighbours[an_offset:an_offset+cur_len]
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              # Compute the distances
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2-x1)**2 + (y2-y1)**2 + (z2-z1)**2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                      
                              if particle_idx > 0:
                                  ends_ind_lst.append(ends_ind_mod_temp)
                              else:
                                  tmp = ends_ind_lst[0]
                                  tmp[:] = ends_ind_mod_temp[0, :]
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      def ends_gap(poss, dia_max):
                          kdtree = cKDTree(poss)
                          tmp = kdtree.query_ball_point(poss, r=dia_max, workers=-1)
                          all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours_sizes = np.array([len(e) for e in tmp], dtype=np.int64)
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes, dia_max)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      ends_gap(poss, dia_max)
                      
                      Initial code with Scipy:             259 s
                      Initial default code with Numpy:     112 s
                      Optimized algorithm:                   1.37 s
                      Final optimized code:                  0.22 s
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = [np.empty([1, 2], dtype=np.int64)]
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              # Find the nearest point including the current one and
                              # then remove the current point from the output.
                              # The distances can be computed directly without a new query.
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max), dtype=np.int64)
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              if particle_idx > 0:
                                  ends_ind.append(ends_ind_mod_temp)
                              else:
                                  ends_ind[0][:] = ends_ind_mod_temp[0, 0], ends_ind_mod_temp[0, 1]
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)                                # <--- relatively high time consumer
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1], float64)')
                      def compute(poss, all_neighbours, all_neighbours_sizes, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind_lst = [np.empty((1, 2), dtype=np.int64)]
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = all_neighbours[an_offset:an_offset+cur_len]
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              # Compute the distances
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2-x1)**2 + (y2-y1)**2 + (z2-z1)**2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                      
                              if particle_idx > 0:
                                  ends_ind_lst.append(ends_ind_mod_temp)
                              else:
                                  tmp = ends_ind_lst[0]
                                  tmp[:] = ends_ind_mod_temp[0, :]
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      def ends_gap(poss, dia_max):
                          kdtree = cKDTree(poss)
                          tmp = kdtree.query_ball_point(poss, r=dia_max, workers=-1)
                          all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours_sizes = np.array([len(e) for e in tmp], dtype=np.int64)
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes, dia_max)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      ends_gap(poss, dia_max)
                      
                      Initial code with Scipy:             259 s
                      Initial default code with Numpy:     112 s
                      Optimized algorithm:                   1.37 s
                      Final optimized code:                  0.22 s
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = [np.empty([1, 2], dtype=np.int64)]
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              # Find the nearest point including the current one and
                              # then remove the current point from the output.
                              # The distances can be computed directly without a new query.
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max), dtype=np.int64)
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              if particle_idx > 0:
                                  ends_ind.append(ends_ind_mod_temp)
                              else:
                                  ends_ind[0][:] = ends_ind_mod_temp[0, 0], ends_ind_mod_temp[0, 1]
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)                                # <--- relatively high time consumer
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1], float64)')
                      def compute(poss, all_neighbours, all_neighbours_sizes, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind_lst = [np.empty((1, 2), dtype=np.int64)]
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = all_neighbours[an_offset:an_offset+cur_len]
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              # Compute the distances
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2-x1)**2 + (y2-y1)**2 + (z2-z1)**2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                      
                              if particle_idx > 0:
                                  ends_ind_lst.append(ends_ind_mod_temp)
                              else:
                                  tmp = ends_ind_lst[0]
                                  tmp[:] = ends_ind_mod_temp[0, :]
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      def ends_gap(poss, dia_max):
                          kdtree = cKDTree(poss)
                          tmp = kdtree.query_ball_point(poss, r=dia_max, workers=-1)
                          all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours_sizes = np.array([len(e) for e in tmp], dtype=np.int64)
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes, dia_max)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      ends_gap(poss, dia_max)
                      
                      Initial code with Scipy:             259 s
                      Initial default code with Numpy:     112 s
                      Optimized algorithm:                   1.37 s
                      Final optimized code:                  0.22 s
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')
                      
                          # tmp = balltree.query_radius(poss, r=dia_max)
                          # all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours = balltree.query_radius(poss, r=dia_max)
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)
                      
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = []                                                                       # <------- this line is modified
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max, return_sorted=True), dtype=np.int64)       # <------- this line is modified
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              ends_ind.append(ends_ind_mod_temp)                                              # <------- this line is modified
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1])')
                      def compute(poss, all_neighbours, all_neighbours_sizes):
                          particle_corsp_overlaps = []
                          ends_ind_lst = []                                                                   # <------- this line is modified
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = np.sort(all_neighbours[an_offset:an_offset+cur_len])              # <------- this line is modified
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2 + (z2 - z1) ** 2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                              ends_ind_lst.append(ends_ind_mod_temp)                                          # <------- this line is modified
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')                                       # <------- new code
                          all_neighbours = balltree.query_radius(poss, r=dia_max)                             # <------- new code and modified
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)   # <------- this line is modified
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)                     # <------- this line is modified
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')
                      
                          # tmp = balltree.query_radius(poss, r=dia_max)
                          # all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours = balltree.query_radius(poss, r=dia_max)
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)
                      
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = []                                                                       # <------- this line is modified
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max, return_sorted=True), dtype=np.int64)       # <------- this line is modified
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              ends_ind.append(ends_ind_mod_temp)                                              # <------- this line is modified
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1])')
                      def compute(poss, all_neighbours, all_neighbours_sizes):
                          particle_corsp_overlaps = []
                          ends_ind_lst = []                                                                   # <------- this line is modified
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = np.sort(all_neighbours[an_offset:an_offset+cur_len])              # <------- this line is modified
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2 + (z2 - z1) ** 2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                              ends_ind_lst.append(ends_ind_mod_temp)                                          # <------- this line is modified
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')                                       # <------- new code
                          all_neighbours = balltree.query_radius(poss, r=dia_max)                             # <------- new code and modified
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)   # <------- this line is modified
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)                     # <------- this line is modified
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')
                      
                          # tmp = balltree.query_radius(poss, r=dia_max)
                          # all_neighbours = np.concatenate(tmp, dtype=np.int64)
                          all_neighbours = balltree.query_radius(poss, r=dia_max)
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)
                      
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, dia_max):
                          particle_corsp_overlaps = []
                          ends_ind = []                                                                       # <------- this line is modified
                      
                          kdtree = cKDTree(poss)
                      
                          for particle_idx in range(len(poss)):
                              cur_point = poss[particle_idx]
                              nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max, return_sorted=True), dtype=np.int64)       # <------- this line is modified
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                      
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where([contact_check <= 0])[1]
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
                              ends_ind.append(ends_ind_mod_temp)                                              # <------- this line is modified
                      
                          ends_ind_org = np.concatenate(ends_ind)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      @nb.jit('(float64[:, ::1], int64[::1], int64[::1])')
                      def compute(poss, all_neighbours, all_neighbours_sizes):
                          particle_corsp_overlaps = []
                          ends_ind_lst = []                                                                   # <------- this line is modified
                          an_offset = 0
                      
                          for particle_idx in range(len(poss)):
                              cur_len = all_neighbours_sizes[particle_idx]
                              nears_i_ind = np.sort(all_neighbours[an_offset:an_offset+cur_len])              # <------- this line is modified
                              an_offset += cur_len
                              assert len(nears_i_ind) > 0
                      
                              if len(nears_i_ind) <= 1:
                                  continue
                      
                              nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
                              dist_i = np.empty(len(nears_i_ind), dtype=np.float64)
                      
                              x1, y1, z1 = poss[particle_idx]
                              for i in range(len(nears_i_ind)):
                                  x2, y2, z2 = poss[nears_i_ind[i]]
                                  dist_i[i] = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2 + (z2 - z1) ** 2)
                      
                              contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
                              connected = contact_check[contact_check <= 0]
                              particle_corsp_overlaps.append(connected)
                      
                              contacts_ind = np.where(contact_check <= 0)
                              contacts_sec_ind = nears_i_ind[contacts_ind]
                              sphere_olps_ind = np.sort(contacts_sec_ind)
                      
                              ends_ind_mod_temp = np.empty((len(sphere_olps_ind), 2), dtype=np.int64)
                              for i in range(len(sphere_olps_ind)):
                                  ends_ind_mod_temp[i, 0] = particle_idx
                                  ends_ind_mod_temp[i, 1] = sphere_olps_ind[i]
                              ends_ind_lst.append(ends_ind_mod_temp)                                          # <------- this line is modified
                      
                          return particle_corsp_overlaps, ends_ind_lst
                      
                      
                      def ends_gap(poss, dia_max):
                          balltree = BallTree(poss, metric='euclidean')                                       # <------- new code
                          all_neighbours = balltree.query_radius(poss, r=dia_max)                             # <------- new code and modified
                          all_neighbours_sizes = np.array([len(e) for e in all_neighbours], dtype=np.int64)   # <------- this line is modified
                          all_neighbours = np.concatenate(all_neighbours, dtype=np.int64)                     # <------- this line is modified
                          particle_corsp_overlaps, ends_ind_lst = compute(poss, all_neighbours, all_neighbours_sizes)
                          ends_ind_org = np.concatenate(ends_ind_lst)
                          ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)
                          gap = np.concatenate(particle_corsp_overlaps)[ends_ind_idx]
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      import numpy as np
                      from scipy import spatial
                      from timeit import default_timer
                      
                      
                      def load_data():
                          centers = np.loadtxt("pos_large.csv", delimiter=",")
                          radii = np.loadtxt("radii_large.csv")
                          assert radii.shape + (3,) == centers.shape
                          return centers, radii
                      
                      
                      def count_contacts(centers, radii):
                          scaled_centers = centers / np.sqrt(centers.shape[1])
                          max_radius = radii.max()
                          tree = spatial.cKDTree(np.c_[scaled_centers, max_radius - radii])
                          count = 0
                          for i, x in enumerate(np.c_[scaled_centers, radii - max_radius]):
                              for j in tree.query_ball_point(x, r=2 * max_radius, p=1):
                                  d = centers[i] - centers[j]
                                  r = radii[i] + radii[j]
                                  if i < j and np.inner(d, d) <= r * r:
                                      count += 1
                          return count
                      
                      
                      def main():
                          centers, radii = load_data()
                          start = default_timer()
                          print(count_contacts(centers, radii))
                          end = default_timer()
                          print(end - start)
                      
                      
                      if __name__ == "__main__":
                          main()
                      
                      import numpy as np
                      import numba as nb
                      from sklearn.neighbors import BallTree
                      from joblib import Parallel, delayed
                      
                      def flatten_neighbours(arr):
                          sizes = np.fromiter(map(len, arr), count=len(arr), dtype=np.int64)
                          values = np.concatenate(arr, dtype=np.int64)
                          return sizes, values
                      
                      @delayed
                      def find_neighbours(searched_pts, ref_pts, max_dist):
                          balltree = BallTree(ref_pts, leaf_size=16, metric='euclidean')
                          res = balltree.query_radius(searched_pts, r=max_dist)
                          return flatten_neighbours(res)
                      
                      def vstack_neighbours(top_infos, bottom_infos):
                          top_sizes, top_values = top_infos
                          bottom_sizes, bottom_values = bottom_infos
                          return np.concatenate([top_sizes, bottom_sizes]), np.concatenate([top_values, bottom_values])
                      
                      @nb.njit('(Tuple([int64[::1],int64[::1]]), Tuple([int64[::1],int64[::1]]), int64)')
                      def hstack_neighbours(left_infos, right_infos, offset):
                          left_sizes, left_values = left_infos
                          right_sizes, right_values = right_infos
                          n = left_sizes.size
                          out_sizes = np.empty(n, dtype=np.int64)
                          out_values = np.empty(left_values.size + right_values.size, dtype=np.int64)
                          left_cur, right_cur, out_cur = 0, 0, 0
                          right_values += offset
                          for i in range(n):
                              left, right = left_sizes[i], right_sizes[i]
                              full = left + right
                              out_values[out_cur:out_cur+left] = left_values[left_cur:left_cur+left]
                              out_values[out_cur+left:out_cur+full] = right_values[right_cur:right_cur+right]
                              out_sizes[i] = full
                              left_cur += left
                              right_cur += right
                              out_cur += full
                          return out_sizes, out_values
                      
                      @nb.njit('(int64[::1], int64[::1], int64[::1], int64[::1])')
                      def reorder_neighbours(in_sizes, in_values, index, reverse_index):
                          n = reverse_index.size
                          out_sizes = np.empty_like(in_sizes)
                          out_values = np.empty_like(in_values)
                          in_offsets = np.empty_like(in_sizes)
                          s, cur = 0, 0
                      
                          for i in range(n):
                              in_offsets[i] = s
                              s += in_sizes[i]
                      
                          for i in range(n):
                              in_ind = reverse_index[i]
                              size = in_sizes[in_ind]
                              in_offset = in_offsets[in_ind]
                              out_sizes[i] = size
                              for j in range(size):
                                  out_values[cur+j] = index[in_values[in_offset+j]]
                              cur += size
                      
                          return out_sizes, out_values
                      
                      @nb.njit
                      def small_inplace_sort(arr):
                          if len(arr) < 80:
                              # Basic insertion sort
                              i = 1
                              while i < len(arr):
                                  x = arr[i]
                                  j = i - 1
                                  while j >= 0 and arr[j] > x:
                                      arr[j+1] = arr[j]
                                      j = j - 1
                                  arr[j+1] = x
                                  i += 1
                          else:
                              arr.sort()
                      
                      @nb.jit('(float64[:, ::1], float64[::1], int64[::1], int64[::1])')
                      def compute(poss, radii, neighbours_sizes, neighbours_values):
                          n, m = neighbours_sizes.size, np.max(neighbours_sizes)
                      
                          # Big buffers allocated with the maximum size.
                          # Thank to virtual memory, it does not take more memory can actually needed.
                          particle_corsp_overlaps = np.empty(neighbours_values.size, dtype=np.float64)
                          ends_ind_org = np.empty((neighbours_values.size, 2), dtype=np.float64)
                      
                          in_offset = 0
                          out_offset = 0
                      
                          buff1 = np.empty(m, dtype=np.int64)
                          buff2 = np.empty(m, dtype=np.float64)
                          buff3 = np.empty(m, dtype=np.float64)
                      
                          for particle_idx in range(n):
                              size = neighbours_sizes[particle_idx]
                              cur = 0
                      
                              for i in range(size):
                                  value = neighbours_values[in_offset+i]
                                  if value != particle_idx:
                                      buff1[cur] = value
                                      cur += 1
                      
                              nears_i_ind = buff1[0:cur]
                              small_inplace_sort(nears_i_ind)  # Note: bottleneck of this function
                              in_offset += size
                      
                              if len(nears_i_ind) == 0:
                                  continue
                      
                              x1, y1, z1 = poss[particle_idx]
                              cur = 0
                      
                              for i in range(len(nears_i_ind)):
                                  index = nears_i_ind[i]
                                  x2, y2, z2 = poss[index]
                                  dist = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2 + (z2 - z1) ** 2)
                                  contact_check = dist - (radii[index] + radii[particle_idx])
                                  if contact_check <= 0.0:
                                      buff2[cur] = contact_check
                                      buff3[cur] = index
                                      cur += 1
                      
                              particle_corsp_overlaps[out_offset:out_offset+cur] = buff2[0:cur]
                      
                              contacts_sec_ind = buff3[0:cur]
                              small_inplace_sort(contacts_sec_ind)
                              sphere_olps_ind = contacts_sec_ind
                      
                              for i in range(cur):
                                  ends_ind_org[out_offset+i, 0] = particle_idx
                                  ends_ind_org[out_offset+i, 1] = sphere_olps_ind[i]
                      
                              out_offset += cur
                      
                          # Truncate the views to their real size
                          particle_corsp_overlaps = particle_corsp_overlaps[:out_offset]
                          ends_ind_org = ends_ind_org[:out_offset]
                      
                          assert len(ends_ind_org) % 2 == 0
                          size = len(ends_ind_org)//2
                          ends_ind = np.empty((size,2), dtype=np.int64)
                          ends_ind_idx = np.empty(size, dtype=np.int64)
                          gap = np.empty(size, dtype=np.float64)
                          cur = 0
                      
                          # Find efficiently duplicates (replace np.unique+np.sort)
                          for i in range(len(ends_ind_org)):
                              left, right = ends_ind_org[i]
                              if left < right:
                                  ends_ind[cur, 0] = left
                                  ends_ind[cur, 1] = right
                                  ends_ind_idx[cur] = i
                                  gap[cur] = particle_corsp_overlaps[i]
                                  cur += 1
                      
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, radii):
                          assert poss.size >= 1
                      
                          # Sort the balls
                          index = np.argsort(radii)
                          reverse_index = np.empty(index.size, np.int64)
                          reverse_index[index] = np.arange(index.size, dtype=np.int64)
                          sorted_poss = poss[index]
                          sorted_radii = radii[index]
                      
                          # Split them in two groups: the small and the big ones
                          split_ind = len(radii) * 3 // 4
                          small_poss, big_poss = np.split(sorted_poss, [split_ind])
                          small_radii, big_radii = np.split(sorted_radii, [split_ind])
                          max_small_radii = sorted_radii[max(split_ind, 0)]
                          max_big_radii = sorted_radii[-1]
                      
                          # Find the neighbours in parallel
                          result = Parallel(n_jobs=4, backend='threading')([
                              find_neighbours(small_poss, small_poss, small_radii+max_small_radii),
                              find_neighbours(small_poss, big_poss,   small_radii+max_big_radii  ),
                              find_neighbours(big_poss,   small_poss, big_radii+max_small_radii  ),
                              find_neighbours(big_poss,   big_poss,   big_radii+max_big_radii    )
                          ])
                          small_small_neighbours = result[0]
                          small_big_neighbours = result[1]
                          big_small_neighbours = result[2]
                          big_big_neighbours = result[3]
                      
                          # Merge the (segmented) arrays in a big one
                          neighbours_sizes, neighbours_values = vstack_neighbours(
                              hstack_neighbours(small_small_neighbours, small_big_neighbours, split_ind),
                              hstack_neighbours(big_small_neighbours, big_big_neighbours, split_ind)
                          )
                      
                          # Reverse the indices.
                          # Note that the results in `neighbours_values` associated to 
                          # `neighbours_sizes[i]` are subsets of `query_radius([poss[i]], r=dia_max)`
                          # on a `BallTree(poss)`.
                          res = reorder_neighbours(neighbours_sizes, neighbours_values, index, reverse_index)
                          neighbours_sizes, neighbours_values = res
                      
                          # Finally compute the neighbours with a method similar to the 
                          # previous one, but using a much faster optimized code.
                          return compute(poss, radii, neighbours_sizes, neighbours_values)
                      
                      result = ends_gap(poss, radii)
                      
                      Small dataset:
                       - Reference optimized Numba code:    256 ms
                       - This highly-optimized Numba code:   82 ms
                      
                      Big dataset:
                       - Reference optimized Numba code:    42.7 s  (take about 7~8 GiB of RAM)
                       - This highly-optimized Numba code:   4.2 s  (take about  1  GiB of RAM)
                      
                      import numpy as np
                      import numba as nb
                      from sklearn.neighbors import BallTree
                      from joblib import Parallel, delayed
                      
                      def flatten_neighbours(arr):
                          sizes = np.fromiter(map(len, arr), count=len(arr), dtype=np.int64)
                          values = np.concatenate(arr, dtype=np.int64)
                          return sizes, values
                      
                      @delayed
                      def find_neighbours(searched_pts, ref_pts, max_dist):
                          balltree = BallTree(ref_pts, leaf_size=16, metric='euclidean')
                          res = balltree.query_radius(searched_pts, r=max_dist)
                          return flatten_neighbours(res)
                      
                      def vstack_neighbours(top_infos, bottom_infos):
                          top_sizes, top_values = top_infos
                          bottom_sizes, bottom_values = bottom_infos
                          return np.concatenate([top_sizes, bottom_sizes]), np.concatenate([top_values, bottom_values])
                      
                      @nb.njit('(Tuple([int64[::1],int64[::1]]), Tuple([int64[::1],int64[::1]]), int64)')
                      def hstack_neighbours(left_infos, right_infos, offset):
                          left_sizes, left_values = left_infos
                          right_sizes, right_values = right_infos
                          n = left_sizes.size
                          out_sizes = np.empty(n, dtype=np.int64)
                          out_values = np.empty(left_values.size + right_values.size, dtype=np.int64)
                          left_cur, right_cur, out_cur = 0, 0, 0
                          right_values += offset
                          for i in range(n):
                              left, right = left_sizes[i], right_sizes[i]
                              full = left + right
                              out_values[out_cur:out_cur+left] = left_values[left_cur:left_cur+left]
                              out_values[out_cur+left:out_cur+full] = right_values[right_cur:right_cur+right]
                              out_sizes[i] = full
                              left_cur += left
                              right_cur += right
                              out_cur += full
                          return out_sizes, out_values
                      
                      @nb.njit('(int64[::1], int64[::1], int64[::1], int64[::1])')
                      def reorder_neighbours(in_sizes, in_values, index, reverse_index):
                          n = reverse_index.size
                          out_sizes = np.empty_like(in_sizes)
                          out_values = np.empty_like(in_values)
                          in_offsets = np.empty_like(in_sizes)
                          s, cur = 0, 0
                      
                          for i in range(n):
                              in_offsets[i] = s
                              s += in_sizes[i]
                      
                          for i in range(n):
                              in_ind = reverse_index[i]
                              size = in_sizes[in_ind]
                              in_offset = in_offsets[in_ind]
                              out_sizes[i] = size
                              for j in range(size):
                                  out_values[cur+j] = index[in_values[in_offset+j]]
                              cur += size
                      
                          return out_sizes, out_values
                      
                      @nb.njit
                      def small_inplace_sort(arr):
                          if len(arr) < 80:
                              # Basic insertion sort
                              i = 1
                              while i < len(arr):
                                  x = arr[i]
                                  j = i - 1
                                  while j >= 0 and arr[j] > x:
                                      arr[j+1] = arr[j]
                                      j = j - 1
                                  arr[j+1] = x
                                  i += 1
                          else:
                              arr.sort()
                      
                      @nb.jit('(float64[:, ::1], float64[::1], int64[::1], int64[::1])')
                      def compute(poss, radii, neighbours_sizes, neighbours_values):
                          n, m = neighbours_sizes.size, np.max(neighbours_sizes)
                      
                          # Big buffers allocated with the maximum size.
                          # Thank to virtual memory, it does not take more memory can actually needed.
                          particle_corsp_overlaps = np.empty(neighbours_values.size, dtype=np.float64)
                          ends_ind_org = np.empty((neighbours_values.size, 2), dtype=np.float64)
                      
                          in_offset = 0
                          out_offset = 0
                      
                          buff1 = np.empty(m, dtype=np.int64)
                          buff2 = np.empty(m, dtype=np.float64)
                          buff3 = np.empty(m, dtype=np.float64)
                      
                          for particle_idx in range(n):
                              size = neighbours_sizes[particle_idx]
                              cur = 0
                      
                              for i in range(size):
                                  value = neighbours_values[in_offset+i]
                                  if value != particle_idx:
                                      buff1[cur] = value
                                      cur += 1
                      
                              nears_i_ind = buff1[0:cur]
                              small_inplace_sort(nears_i_ind)  # Note: bottleneck of this function
                              in_offset += size
                      
                              if len(nears_i_ind) == 0:
                                  continue
                      
                              x1, y1, z1 = poss[particle_idx]
                              cur = 0
                      
                              for i in range(len(nears_i_ind)):
                                  index = nears_i_ind[i]
                                  x2, y2, z2 = poss[index]
                                  dist = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2 + (z2 - z1) ** 2)
                                  contact_check = dist - (radii[index] + radii[particle_idx])
                                  if contact_check <= 0.0:
                                      buff2[cur] = contact_check
                                      buff3[cur] = index
                                      cur += 1
                      
                              particle_corsp_overlaps[out_offset:out_offset+cur] = buff2[0:cur]
                      
                              contacts_sec_ind = buff3[0:cur]
                              small_inplace_sort(contacts_sec_ind)
                              sphere_olps_ind = contacts_sec_ind
                      
                              for i in range(cur):
                                  ends_ind_org[out_offset+i, 0] = particle_idx
                                  ends_ind_org[out_offset+i, 1] = sphere_olps_ind[i]
                      
                              out_offset += cur
                      
                          # Truncate the views to their real size
                          particle_corsp_overlaps = particle_corsp_overlaps[:out_offset]
                          ends_ind_org = ends_ind_org[:out_offset]
                      
                          assert len(ends_ind_org) % 2 == 0
                          size = len(ends_ind_org)//2
                          ends_ind = np.empty((size,2), dtype=np.int64)
                          ends_ind_idx = np.empty(size, dtype=np.int64)
                          gap = np.empty(size, dtype=np.float64)
                          cur = 0
                      
                          # Find efficiently duplicates (replace np.unique+np.sort)
                          for i in range(len(ends_ind_org)):
                              left, right = ends_ind_org[i]
                              if left < right:
                                  ends_ind[cur, 0] = left
                                  ends_ind[cur, 1] = right
                                  ends_ind_idx[cur] = i
                                  gap[cur] = particle_corsp_overlaps[i]
                                  cur += 1
                      
                          return gap, ends_ind, ends_ind_idx, ends_ind_org
                      
                      def ends_gap(poss, radii):
                          assert poss.size >= 1
                      
                          # Sort the balls
                          index = np.argsort(radii)
                          reverse_index = np.empty(index.size, np.int64)
                          reverse_index[index] = np.arange(index.size, dtype=np.int64)
                          sorted_poss = poss[index]
                          sorted_radii = radii[index]
                      
                          # Split them in two groups: the small and the big ones
                          split_ind = len(radii) * 3 // 4
                          small_poss, big_poss = np.split(sorted_poss, [split_ind])
                          small_radii, big_radii = np.split(sorted_radii, [split_ind])
                          max_small_radii = sorted_radii[max(split_ind, 0)]
                          max_big_radii = sorted_radii[-1]
                      
                          # Find the neighbours in parallel
                          result = Parallel(n_jobs=4, backend='threading')([
                              find_neighbours(small_poss, small_poss, small_radii+max_small_radii),
                              find_neighbours(small_poss, big_poss,   small_radii+max_big_radii  ),
                              find_neighbours(big_poss,   small_poss, big_radii+max_small_radii  ),
                              find_neighbours(big_poss,   big_poss,   big_radii+max_big_radii    )
                          ])
                          small_small_neighbours = result[0]
                          small_big_neighbours = result[1]
                          big_small_neighbours = result[2]
                          big_big_neighbours = result[3]
                      
                          # Merge the (segmented) arrays in a big one
                          neighbours_sizes, neighbours_values = vstack_neighbours(
                              hstack_neighbours(small_small_neighbours, small_big_neighbours, split_ind),
                              hstack_neighbours(big_small_neighbours, big_big_neighbours, split_ind)
                          )
                      
                          # Reverse the indices.
                          # Note that the results in `neighbours_values` associated to 
                          # `neighbours_sizes[i]` are subsets of `query_radius([poss[i]], r=dia_max)`
                          # on a `BallTree(poss)`.
                          res = reorder_neighbours(neighbours_sizes, neighbours_values, index, reverse_index)
                          neighbours_sizes, neighbours_values = res
                      
                          # Finally compute the neighbours with a method similar to the 
                          # previous one, but using a much faster optimized code.
                          return compute(poss, radii, neighbours_sizes, neighbours_values)
                      
                      result = ends_gap(poss, radii)
                      
                      Small dataset:
                       - Reference optimized Numba code:    256 ms
                       - This highly-optimized Numba code:   82 ms
                      
                      Big dataset:
                       - Reference optimized Numba code:    42.7 s  (take about 7~8 GiB of RAM)
                       - This highly-optimized Numba code:   4.2 s  (take about  1  GiB of RAM)
                      

                      Colab: (0) UNIMPLEMENTED: DNN library is not found

                      copy iconCopydownload iconDownload
                      !pip install tensorflow==2.7.0
                      
                      'tensorflow==2.7.0',
                      'tf-models-official==2.7.0',
                      'tensorflow_io==0.23.1',
                      

                      Cannot find conda info. Please verify your conda installation on EMR

                      copy iconCopydownload iconDownload
                      wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
                          && /bin/bash ~/miniconda.sh -b -p $HOME/conda
                      
                      echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc
                      
                      
                      conda config --set always_yes yes --set changeps1 no
                      conda config -f --add channels conda-forge
                      
                      
                      conda create -n zoo python=3.7 # "zoo" is conda environment name
                      conda init bash
                      source activate zoo
                      conda install python 3.7.0 -c conda-forge orca 
                      sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv
                      
                      “spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
                      "spark.pyspark.virtualenv.enabled": "true",
                      "spark.pyspark.virtualenv.type":"native",
                      "spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
                      "zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
                      "zeppelin.python": "/home/hadoop/conda/bin/python"
                      
                      wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
                          && /bin/bash ~/miniconda.sh -b -p $HOME/conda
                      
                      echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc
                      
                      
                      conda config --set always_yes yes --set changeps1 no
                      conda config -f --add channels conda-forge
                      
                      
                      conda create -n zoo python=3.7 # "zoo" is conda environment name
                      conda init bash
                      source activate zoo
                      conda install python 3.7.0 -c conda-forge orca 
                      sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv
                      
                      “spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
                      "spark.pyspark.virtualenv.enabled": "true",
                      "spark.pyspark.virtualenv.type":"native",
                      "spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
                      "zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
                      "zeppelin.python": "/home/hadoop/conda/bin/python"
                      

                      Edge weight in networkx

                      copy iconCopydownload iconDownload
                      df["weight"] = 1.0
                      df = df.groupby([<id_columns>]).agg({"wight": sum}).reset_index()
                      
                      G = nx.from_pandas_edgelist(source='source_column', target='target_column', edge_attr="weight")
                      
                      df["weight"] = 1.0
                      df = df.groupby([<id_columns>]).agg({"wight": sum}).reset_index()
                      
                      G = nx.from_pandas_edgelist(source='source_column', target='target_column', edge_attr="weight")
                      

                      ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects

                      copy iconCopydownload iconDownload
                      apt-get install sox ffmpeg libcairo2 libcairo2-dev
                      apt-get install texlive-full
                      pip3 install manimlib  # or pip install manimlib
                      
                      pip3 install manimce  # or pip install manimce
                      
                      apt-get install sox ffmpeg libcairo2 libcairo2-dev
                      apt-get install texlive-full
                      pip3 install manimlib  # or pip install manimlib
                      
                      pip3 install manimce  # or pip install manimce
                      

                      How to find peaks of FFT graph using Python?

                      copy iconCopydownload iconDownload
                      plt.plot(a)
                      plt.plot(peaks, a[peaks])
                      
                      import pandas as pd
                      import matplotlib.pyplot as plt
                      from scipy.fft import fft, fftfreq
                      from scipy.signal import find_peaks
                      
                      # First: Let's generate a dummy dataframe with X,Y
                      # The signal consists in 3 cosine signals with noise added. We terminate by creating
                      # a pandas dataframe.
                      
                      import numpy as np
                      X=np.arange(start=0,stop=20,step=0.01) # 20 seconds long signal sampled every 0.01[s]
                      
                      # Signal components given by [frequency, phase shift, Amplitude]
                      GeneratedSignal=np.array([[5.50, 1.60, 1.0], [10.2, 0.25, 0.5], [18.3, 0.70, 0.2]])
                      
                      Y=np.zeros(len(X))
                      # Let's add the components one by one
                      for P in GeneratedSignal:
                          Y+=np.cos(2*np.pi*P[0]*X-P[1])*P[2] 
                      
                      # Let's add some gaussian random noise (mu=0, sigma=noise):
                      noise=0.5
                      Y+=np.random.randn(len(X))*noise
                      
                      # Let's build the dataframe:
                      dummy_data=pd.DataFrame({'X':X,'Y':Y})
                      print('Dummy dataframe: ')
                      print(dummy_data.head())
                      
                      # Figure-1: The dummy data
                      
                      plt.plot(X,Y)
                      plt.title('Dummy data')
                      plt.xlabel('time [s]')
                      plt.ylabel('Amplitude')
                      plt.show()
                      
                      # ----------------------------------------------------
                      # Processing:
                      
                      headers = ["X","Y"]
                      
                      #original_data = pd.read_csv("testdata.csv",names=headers)
                      # Let's take our dummy data:
                      
                      original_data = dummy_data
                      
                      x = np.array(original_data["X"])
                      y = np.array(original_data["Y"])
                      
                      
                      # Assuming the time step is constant:
                      # (otherwise you'll need to resample the data at a constant rate).
                      dt=x[1]-x[0]  # time step of the data
                      
                      # The fourier transform of y:
                      yf=fft(y, norm='forward')  
                      # Note: see  help(fft) --> norm. I chose 'forward' because it gives the amplitudes we put in.
                      # Otherwise, by default, yf will be scaled by a factor of n: the number of points
                      
                      
                      # The frequency scale
                      n = x.size   # The number of points in the data
                      freq = fftfreq(n, d=dt)
                      
                      # Let's find the peaks with height_threshold >=0.05
                      # Note: We use the magnitude (i.e the absolute value) of the Fourier transform
                      
                      height_threshold=0.05 # We need a threshold. 
                      
                      
                      # peaks_index contains the indices in x that correspond to peaks:
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold)
                      
                      # Notes: 
                      # 1) peaks_index does not contain the frequency values but indices
                      # 2) In this case, properties will contain only one property: 'peak_heights'
                      #    for each element in peaks_index (See help(find_peaks) )
                      
                      # Let's first output the result to the terminal window:
                      print('Positions and magnitude of frequency peaks:')
                      [print("%4.4f    \t %3.4f" %(freq[peaks_index[i]], properties['peak_heights'][i])) for i in range(len(peaks_index))]
                      
                      
                      # Figure-2: The frequencies
                      
                      plt.plot(freq, np.abs(yf),'-', freq[peaks_index],properties['peak_heights'],'x')
                      plt.xlabel("Frequency")
                      plt.ylabel("Amplitude")
                      plt.show()
                      
                      Dummy dataframe: 
                            X         Y
                      0  0.00  0.611829
                      1  0.01  0.723775
                      2  0.02  0.768813
                      3  0.03  0.798328
                      
                      Positions and magnitude of frequency peaks:
                      5.5000       0.4980
                      10.2000      0.2575
                      18.3000      0.0999
                      -18.3000     0.0999
                      -10.2000     0.2575
                      -5.5000      0.4980
                      
                      X=np.arange(start=0,stop=1,step=0.01) # 1 seconds long signal sampled every 0.01[s]
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold, width=0)
                      
                      print(properties)
                      
                      import pandas as pd
                      import matplotlib.pyplot as plt
                      from scipy.fft import fft, fftfreq
                      from scipy.signal import find_peaks
                      
                      # First: Let's generate a dummy dataframe with X,Y
                      # The signal consists in 3 cosine signals with noise added. We terminate by creating
                      # a pandas dataframe.
                      
                      import numpy as np
                      X=np.arange(start=0,stop=20,step=0.01) # 20 seconds long signal sampled every 0.01[s]
                      
                      # Signal components given by [frequency, phase shift, Amplitude]
                      GeneratedSignal=np.array([[5.50, 1.60, 1.0], [10.2, 0.25, 0.5], [18.3, 0.70, 0.2]])
                      
                      Y=np.zeros(len(X))
                      # Let's add the components one by one
                      for P in GeneratedSignal:
                          Y+=np.cos(2*np.pi*P[0]*X-P[1])*P[2] 
                      
                      # Let's add some gaussian random noise (mu=0, sigma=noise):
                      noise=0.5
                      Y+=np.random.randn(len(X))*noise
                      
                      # Let's build the dataframe:
                      dummy_data=pd.DataFrame({'X':X,'Y':Y})
                      print('Dummy dataframe: ')
                      print(dummy_data.head())
                      
                      # Figure-1: The dummy data
                      
                      plt.plot(X,Y)
                      plt.title('Dummy data')
                      plt.xlabel('time [s]')
                      plt.ylabel('Amplitude')
                      plt.show()
                      
                      # ----------------------------------------------------
                      # Processing:
                      
                      headers = ["X","Y"]
                      
                      #original_data = pd.read_csv("testdata.csv",names=headers)
                      # Let's take our dummy data:
                      
                      original_data = dummy_data
                      
                      x = np.array(original_data["X"])
                      y = np.array(original_data["Y"])
                      
                      
                      # Assuming the time step is constant:
                      # (otherwise you'll need to resample the data at a constant rate).
                      dt=x[1]-x[0]  # time step of the data
                      
                      # The fourier transform of y:
                      yf=fft(y, norm='forward')  
                      # Note: see  help(fft) --> norm. I chose 'forward' because it gives the amplitudes we put in.
                      # Otherwise, by default, yf will be scaled by a factor of n: the number of points
                      
                      
                      # The frequency scale
                      n = x.size   # The number of points in the data
                      freq = fftfreq(n, d=dt)
                      
                      # Let's find the peaks with height_threshold >=0.05
                      # Note: We use the magnitude (i.e the absolute value) of the Fourier transform
                      
                      height_threshold=0.05 # We need a threshold. 
                      
                      
                      # peaks_index contains the indices in x that correspond to peaks:
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold)
                      
                      # Notes: 
                      # 1) peaks_index does not contain the frequency values but indices
                      # 2) In this case, properties will contain only one property: 'peak_heights'
                      #    for each element in peaks_index (See help(find_peaks) )
                      
                      # Let's first output the result to the terminal window:
                      print('Positions and magnitude of frequency peaks:')
                      [print("%4.4f    \t %3.4f" %(freq[peaks_index[i]], properties['peak_heights'][i])) for i in range(len(peaks_index))]
                      
                      
                      # Figure-2: The frequencies
                      
                      plt.plot(freq, np.abs(yf),'-', freq[peaks_index],properties['peak_heights'],'x')
                      plt.xlabel("Frequency")
                      plt.ylabel("Amplitude")
                      plt.show()
                      
                      Dummy dataframe: 
                            X         Y
                      0  0.00  0.611829
                      1  0.01  0.723775
                      2  0.02  0.768813
                      3  0.03  0.798328
                      
                      Positions and magnitude of frequency peaks:
                      5.5000       0.4980
                      10.2000      0.2575
                      18.3000      0.0999
                      -18.3000     0.0999
                      -10.2000     0.2575
                      -5.5000      0.4980
                      
                      X=np.arange(start=0,stop=1,step=0.01) # 1 seconds long signal sampled every 0.01[s]
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold, width=0)
                      
                      print(properties)
                      
                      import pandas as pd
                      import matplotlib.pyplot as plt
                      from scipy.fft import fft, fftfreq
                      from scipy.signal import find_peaks
                      
                      # First: Let's generate a dummy dataframe with X,Y
                      # The signal consists in 3 cosine signals with noise added. We terminate by creating
                      # a pandas dataframe.
                      
                      import numpy as np
                      X=np.arange(start=0,stop=20,step=0.01) # 20 seconds long signal sampled every 0.01[s]
                      
                      # Signal components given by [frequency, phase shift, Amplitude]
                      GeneratedSignal=np.array([[5.50, 1.60, 1.0], [10.2, 0.25, 0.5], [18.3, 0.70, 0.2]])
                      
                      Y=np.zeros(len(X))
                      # Let's add the components one by one
                      for P in GeneratedSignal:
                          Y+=np.cos(2*np.pi*P[0]*X-P[1])*P[2] 
                      
                      # Let's add some gaussian random noise (mu=0, sigma=noise):
                      noise=0.5
                      Y+=np.random.randn(len(X))*noise
                      
                      # Let's build the dataframe:
                      dummy_data=pd.DataFrame({'X':X,'Y':Y})
                      print('Dummy dataframe: ')
                      print(dummy_data.head())
                      
                      # Figure-1: The dummy data
                      
                      plt.plot(X,Y)
                      plt.title('Dummy data')
                      plt.xlabel('time [s]')
                      plt.ylabel('Amplitude')
                      plt.show()
                      
                      # ----------------------------------------------------
                      # Processing:
                      
                      headers = ["X","Y"]
                      
                      #original_data = pd.read_csv("testdata.csv",names=headers)
                      # Let's take our dummy data:
                      
                      original_data = dummy_data
                      
                      x = np.array(original_data["X"])
                      y = np.array(original_data["Y"])
                      
                      
                      # Assuming the time step is constant:
                      # (otherwise you'll need to resample the data at a constant rate).
                      dt=x[1]-x[0]  # time step of the data
                      
                      # The fourier transform of y:
                      yf=fft(y, norm='forward')  
                      # Note: see  help(fft) --> norm. I chose 'forward' because it gives the amplitudes we put in.
                      # Otherwise, by default, yf will be scaled by a factor of n: the number of points
                      
                      
                      # The frequency scale
                      n = x.size   # The number of points in the data
                      freq = fftfreq(n, d=dt)
                      
                      # Let's find the peaks with height_threshold >=0.05
                      # Note: We use the magnitude (i.e the absolute value) of the Fourier transform
                      
                      height_threshold=0.05 # We need a threshold. 
                      
                      
                      # peaks_index contains the indices in x that correspond to peaks:
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold)
                      
                      # Notes: 
                      # 1) peaks_index does not contain the frequency values but indices
                      # 2) In this case, properties will contain only one property: 'peak_heights'
                      #    for each element in peaks_index (See help(find_peaks) )
                      
                      # Let's first output the result to the terminal window:
                      print('Positions and magnitude of frequency peaks:')
                      [print("%4.4f    \t %3.4f" %(freq[peaks_index[i]], properties['peak_heights'][i])) for i in range(len(peaks_index))]
                      
                      
                      # Figure-2: The frequencies
                      
                      plt.plot(freq, np.abs(yf),'-', freq[peaks_index],properties['peak_heights'],'x')
                      plt.xlabel("Frequency")
                      plt.ylabel("Amplitude")
                      plt.show()
                      
                      Dummy dataframe: 
                            X         Y
                      0  0.00  0.611829
                      1  0.01  0.723775
                      2  0.02  0.768813
                      3  0.03  0.798328
                      
                      Positions and magnitude of frequency peaks:
                      5.5000       0.4980
                      10.2000      0.2575
                      18.3000      0.0999
                      -18.3000     0.0999
                      -10.2000     0.2575
                      -5.5000      0.4980
                      
                      X=np.arange(start=0,stop=1,step=0.01) # 1 seconds long signal sampled every 0.01[s]
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold, width=0)
                      
                      print(properties)
                      
                      import pandas as pd
                      import matplotlib.pyplot as plt
                      from scipy.fft import fft, fftfreq
                      from scipy.signal import find_peaks
                      
                      # First: Let's generate a dummy dataframe with X,Y
                      # The signal consists in 3 cosine signals with noise added. We terminate by creating
                      # a pandas dataframe.
                      
                      import numpy as np
                      X=np.arange(start=0,stop=20,step=0.01) # 20 seconds long signal sampled every 0.01[s]
                      
                      # Signal components given by [frequency, phase shift, Amplitude]
                      GeneratedSignal=np.array([[5.50, 1.60, 1.0], [10.2, 0.25, 0.5], [18.3, 0.70, 0.2]])
                      
                      Y=np.zeros(len(X))
                      # Let's add the components one by one
                      for P in GeneratedSignal:
                          Y+=np.cos(2*np.pi*P[0]*X-P[1])*P[2] 
                      
                      # Let's add some gaussian random noise (mu=0, sigma=noise):
                      noise=0.5
                      Y+=np.random.randn(len(X))*noise
                      
                      # Let's build the dataframe:
                      dummy_data=pd.DataFrame({'X':X,'Y':Y})
                      print('Dummy dataframe: ')
                      print(dummy_data.head())
                      
                      # Figure-1: The dummy data
                      
                      plt.plot(X,Y)
                      plt.title('Dummy data')
                      plt.xlabel('time [s]')
                      plt.ylabel('Amplitude')
                      plt.show()
                      
                      # ----------------------------------------------------
                      # Processing:
                      
                      headers = ["X","Y"]
                      
                      #original_data = pd.read_csv("testdata.csv",names=headers)
                      # Let's take our dummy data:
                      
                      original_data = dummy_data
                      
                      x = np.array(original_data["X"])
                      y = np.array(original_data["Y"])
                      
                      
                      # Assuming the time step is constant:
                      # (otherwise you'll need to resample the data at a constant rate).
                      dt=x[1]-x[0]  # time step of the data
                      
                      # The fourier transform of y:
                      yf=fft(y, norm='forward')  
                      # Note: see  help(fft) --> norm. I chose 'forward' because it gives the amplitudes we put in.
                      # Otherwise, by default, yf will be scaled by a factor of n: the number of points
                      
                      
                      # The frequency scale
                      n = x.size   # The number of points in the data
                      freq = fftfreq(n, d=dt)
                      
                      # Let's find the peaks with height_threshold >=0.05
                      # Note: We use the magnitude (i.e the absolute value) of the Fourier transform
                      
                      height_threshold=0.05 # We need a threshold. 
                      
                      
                      # peaks_index contains the indices in x that correspond to peaks:
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold)
                      
                      # Notes: 
                      # 1) peaks_index does not contain the frequency values but indices
                      # 2) In this case, properties will contain only one property: 'peak_heights'
                      #    for each element in peaks_index (See help(find_peaks) )
                      
                      # Let's first output the result to the terminal window:
                      print('Positions and magnitude of frequency peaks:')
                      [print("%4.4f    \t %3.4f" %(freq[peaks_index[i]], properties['peak_heights'][i])) for i in range(len(peaks_index))]
                      
                      
                      # Figure-2: The frequencies
                      
                      plt.plot(freq, np.abs(yf),'-', freq[peaks_index],properties['peak_heights'],'x')
                      plt.xlabel("Frequency")
                      plt.ylabel("Amplitude")
                      plt.show()
                      
                      Dummy dataframe: 
                            X         Y
                      0  0.00  0.611829
                      1  0.01  0.723775
                      2  0.02  0.768813
                      3  0.03  0.798328
                      
                      Positions and magnitude of frequency peaks:
                      5.5000       0.4980
                      10.2000      0.2575
                      18.3000      0.0999
                      -18.3000     0.0999
                      -10.2000     0.2575
                      -5.5000      0.4980
                      
                      X=np.arange(start=0,stop=1,step=0.01) # 1 seconds long signal sampled every 0.01[s]
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold, width=0)
                      
                      print(properties)
                      
                      import pandas as pd
                      import matplotlib.pyplot as plt
                      from scipy.fft import fft, fftfreq
                      from scipy.signal import find_peaks
                      
                      # First: Let's generate a dummy dataframe with X,Y
                      # The signal consists in 3 cosine signals with noise added. We terminate by creating
                      # a pandas dataframe.
                      
                      import numpy as np
                      X=np.arange(start=0,stop=20,step=0.01) # 20 seconds long signal sampled every 0.01[s]
                      
                      # Signal components given by [frequency, phase shift, Amplitude]
                      GeneratedSignal=np.array([[5.50, 1.60, 1.0], [10.2, 0.25, 0.5], [18.3, 0.70, 0.2]])
                      
                      Y=np.zeros(len(X))
                      # Let's add the components one by one
                      for P in GeneratedSignal:
                          Y+=np.cos(2*np.pi*P[0]*X-P[1])*P[2] 
                      
                      # Let's add some gaussian random noise (mu=0, sigma=noise):
                      noise=0.5
                      Y+=np.random.randn(len(X))*noise
                      
                      # Let's build the dataframe:
                      dummy_data=pd.DataFrame({'X':X,'Y':Y})
                      print('Dummy dataframe: ')
                      print(dummy_data.head())
                      
                      # Figure-1: The dummy data
                      
                      plt.plot(X,Y)
                      plt.title('Dummy data')
                      plt.xlabel('time [s]')
                      plt.ylabel('Amplitude')
                      plt.show()
                      
                      # ----------------------------------------------------
                      # Processing:
                      
                      headers = ["X","Y"]
                      
                      #original_data = pd.read_csv("testdata.csv",names=headers)
                      # Let's take our dummy data:
                      
                      original_data = dummy_data
                      
                      x = np.array(original_data["X"])
                      y = np.array(original_data["Y"])
                      
                      
                      # Assuming the time step is constant:
                      # (otherwise you'll need to resample the data at a constant rate).
                      dt=x[1]-x[0]  # time step of the data
                      
                      # The fourier transform of y:
                      yf=fft(y, norm='forward')  
                      # Note: see  help(fft) --> norm. I chose 'forward' because it gives the amplitudes we put in.
                      # Otherwise, by default, yf will be scaled by a factor of n: the number of points
                      
                      
                      # The frequency scale
                      n = x.size   # The number of points in the data
                      freq = fftfreq(n, d=dt)
                      
                      # Let's find the peaks with height_threshold >=0.05
                      # Note: We use the magnitude (i.e the absolute value) of the Fourier transform
                      
                      height_threshold=0.05 # We need a threshold. 
                      
                      
                      # peaks_index contains the indices in x that correspond to peaks:
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold)
                      
                      # Notes: 
                      # 1) peaks_index does not contain the frequency values but indices
                      # 2) In this case, properties will contain only one property: 'peak_heights'
                      #    for each element in peaks_index (See help(find_peaks) )
                      
                      # Let's first output the result to the terminal window:
                      print('Positions and magnitude of frequency peaks:')
                      [print("%4.4f    \t %3.4f" %(freq[peaks_index[i]], properties['peak_heights'][i])) for i in range(len(peaks_index))]
                      
                      
                      # Figure-2: The frequencies
                      
                      plt.plot(freq, np.abs(yf),'-', freq[peaks_index],properties['peak_heights'],'x')
                      plt.xlabel("Frequency")
                      plt.ylabel("Amplitude")
                      plt.show()
                      
                      Dummy dataframe: 
                            X         Y
                      0  0.00  0.611829
                      1  0.01  0.723775
                      2  0.02  0.768813
                      3  0.03  0.798328
                      
                      Positions and magnitude of frequency peaks:
                      5.5000       0.4980
                      10.2000      0.2575
                      18.3000      0.0999
                      -18.3000     0.0999
                      -10.2000     0.2575
                      -5.5000      0.4980
                      
                      X=np.arange(start=0,stop=1,step=0.01) # 1 seconds long signal sampled every 0.01[s]
                      
                      peaks_index, properties = find_peaks(np.abs(yf), height=height_threshold, width=0)
                      
                      print(properties)
                      

                      After conda update, python kernel crashes when matplotlib is used

                      copy iconCopydownload iconDownload
                        2021-10-31 10:47:22  (rev 3)
                           bokeh  {2.3.3 (defaults/win-64) -> 2.4.1 (defaults/win-64)}
                           click  {8.0.1 (defaults/noarch) -> 8.0.3 (defaults/noarch)}
                           filelock  {3.0.12 (defaults/noarch) -> 3.3.1 (defaults/noarch)}
                           freetype  {2.10.4 (defaults/win-64) -> 2.11.0 (defaults/win-64)}
                           imagecodecs  {2021.6.8 (defaults/win-64) -> 2021.8.26 (defaults/win-64)}
                           joblib  {1.0.1 (defaults/noarch) -> 1.1.0 (defaults/noarch)}
                           lerc  {2.2.1 (defaults/win-64) -> 3.0 (defaults/win-64)}
                           more-itertools  {8.8.0 (defaults/noarch) -> 8.10.0 (defaults/noarch)}
                           pyopenssl  {20.0.1 (defaults/noarch) -> 21.0.0 (defaults/noarch)}
                           scikit-learn  {0.24.2 (defaults/win-64) -> 1.0.1 (defaults/win-64)}
                           statsmodels  {0.12.2 (defaults/win-64) -> 0.13.0 (defaults/win-64)}
                           sympy  {1.8 (defaults/win-64) -> 1.9 (defaults/win-64)}
                           tqdm  {4.62.2 (defaults/noarch) -> 4.62.3 (defaults/noarch)}
                           xlwings  {0.24.7 (defaults/win-64) -> 0.24.9 (defaults/win-64)}
                      
                      conda update -c anaconda numpy
                      
                      conda upgrade -c conda-forge matplotlib
                      

                      Community Discussions

                      Trending Discussions on scipy
                      • Padding scipy affine_transform output to show non-overlapping regions of transformed images
                      • Cannot import name '_centered' from 'scipy.signal.signaltools'
                      • Installing scipy and scikit-learn on apple m1
                      • Why should I use normalised units in numerical integration?
                      • How could I speed up my written python code: spheres contact detection (collision) using spatial searching
                      • Colab: (0) UNIMPLEMENTED: DNN library is not found
                      • Cannot find conda info. Please verify your conda installation on EMR
                      • Edge weight in networkx
                      • ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
                      • Is it possible to use a collection of hyperspectral 1x1 pixels in a CNN model purposed for more conventional datasets (CIFAR-10/MNIST)?
                      Trending Discussions on scipy

                      QUESTION

                      Padding scipy affine_transform output to show non-overlapping regions of transformed images

                      Asked 2022-Mar-28 at 11:54

                      I have source (src) image(s) I wish to align to a destination (dst) image using an Affine Transformation whilst retaining the full extent of both images during alignment (even the non-overlapping areas).

                      I am already able to calculate the Affine Transformation rotation and offset matrix, which I feed to scipy.ndimage.interpolate.affine_transform to recover the dst-aligned src image.

                      The problem is that, when the images are not fuly overlapping, the resultant image is cropped to only the common footprint of the two images. What I need is the full extent of both images, placed on the same pixel coordinate system. This question is almost a duplicate of this one - and the excellent answer and repository there provides this functionality for OpenCV transformations. I unfortunately need this for scipy's implementation.

                      Much too late, after repeatedly hitting a brick wall trying to translate the above question's answer to scipy, I came across this issue and subsequently followed to this question. The latter question did give some insight into the wonderful world of scipy's affine transformation, but I have as yet been unable to crack my particular needs.

                      The transformations from src to dst can have translations and rotation. I can get translations only working (an example is shown below) and I can get rotations only working (largely hacking around the below and taking inspiration from the use of the reshape argument in scipy.ndimage.interpolation.rotate). However, I am getting thoroughly lost combining the two. I have tried to calculate what should be the correct offset (see this question's answers again), but I can't get it working in all scenarios.

                      Translation-only working example of padded affine transformation, which follows largely this repo, explained in this answer:

                      from scipy.ndimage import rotate, affine_transform
                      import numpy as np
                      import matplotlib.pyplot as plt
                      
                      nblob = 50
                      shape = (200, 100)
                      buffered_shape = (300, 200)  # buffer for rotation and translation
                      
                      
                      def affine_test(angle=0, translate=(0, 0)):
                          np.random.seed(42)
                          # Maxiumum translation allowed is half difference between shape and buffered_shape
                      
                          # Generate a buffered_shape-sized base image with random blobs
                          base = np.zeros(buffered_shape, dtype=np.float32)
                          random_locs = np.random.choice(np.arange(2, buffered_shape[0] - 2), nblob * 2, replace=False)
                          i = random_locs[:nblob]
                          j = random_locs[nblob:]
                          for k, (_i, _j) in enumerate(zip(i, j)):
                              # Use different values, just to make it easier to distinguish blobs
                              base[_i - 2 : _i + 2, _j - 2 : _j + 2] = k + 10
                      
                          # Impose a rotation and translation on source
                          src = rotate(base, angle, reshape=False, order=1, mode="constant")
                          bsc = (np.array(buffered_shape) / 2).astype(int)
                          sc = (np.array(shape) / 2).astype(int)
                          src = src[
                              bsc[0] - sc[0] + translate[0] : bsc[0] + sc[0] + translate[0],
                              bsc[1] - sc[1] + translate[1] : bsc[1] + sc[1] + translate[1],
                          ]
                          # Cut-out destination from the centre of the base image
                          dst = base[bsc[0] - sc[0] : bsc[0] + sc[0], bsc[1] - sc[1] : bsc[1] + sc[1]]
                      
                          src_y, src_x = src.shape
                      
                          def get_matrix_offset(centre, angle, scale):
                              """Follows OpenCV.getRotationMatrix2D"""
                              angle = angle * np.pi / 180
                              alpha = scale * np.cos(angle)
                              beta = scale * np.sin(angle)
                              return (
                                  np.array([[alpha, beta], [-beta, alpha]]),
                                  np.array(
                                      [
                                          (1 - alpha) * centre[0] - beta * centre[1],
                                          beta * centre[0] + (1 - alpha) * centre[1],
                                      ]
                                  ),
                              )
                          # Obtain the rotation matrix and offset that describes the transformation
                          # between src and dst
                          matrix, offset = get_matrix_offset(np.array([src_y / 2, src_x / 2]), angle, 1)
                          offset = offset - translate
                      
                          # Determine the outer bounds of the new image
                          lin_pts = np.array([[0, src_x, src_x, 0], [0, 0, src_y, src_y]])
                          transf_lin_pts = np.dot(matrix.T, lin_pts) - offset[::-1].reshape(2, 1)
                      
                          # Find min and max bounds of the transformed image
                          min_x = np.floor(np.min(transf_lin_pts[0])).astype(int)
                          min_y = np.floor(np.min(transf_lin_pts[1])).astype(int)
                          max_x = np.ceil(np.max(transf_lin_pts[0])).astype(int)
                          max_y = np.ceil(np.max(transf_lin_pts[1])).astype(int)
                      
                          # Add translation to the transformation matrix to shift to positive values
                          anchor_x, anchor_y = 0, 0
                          if min_x < 0:
                              anchor_x = -min_x
                          if min_y < 0:
                              anchor_y = -min_y
                          shifted_offset = offset - np.dot(matrix, [anchor_y, anchor_x])
                      
                          # Create padded destination image
                          dst_h, dst_w = dst.shape[:2]
                          pad_widths = [anchor_y, max(max_y, dst_h) - dst_h, anchor_x, max(max_x, dst_w) - dst_w]
                          dst_padded = np.pad(
                              dst,
                              ((pad_widths[0], pad_widths[1]), (pad_widths[2], pad_widths[3])),
                              "constant",
                              constant_values=-1,
                          )
                          dst_pad_h, dst_pad_w = dst_padded.shape
                      
                          # Create the aligned and padded source image
                          source_aligned = affine_transform(
                              src,
                              matrix.T,
                              offset=shifted_offset,
                              output_shape=(dst_pad_h, dst_pad_w),
                              order=3,
                              mode="constant",
                              cval=-1,
                          )
                      
                          # Plot the images
                          fig, axes = plt.subplots(1, 4, figsize=(10, 5), sharex=True, sharey=True)
                          axes[0].imshow(src, cmap="viridis", vmin=-1, vmax=nblob)
                          axes[0].set_title("Source")
                          axes[1].imshow(dst, cmap="viridis", vmin=-1, vmax=nblob)
                          axes[1].set_title("Dest")
                          axes[2].imshow(source_aligned, cmap="viridis", vmin=-1, vmax=nblob)
                          axes[2].set_title("Source aligned to Dest padded")
                          axes[3].imshow(dst_padded, cmap="viridis", vmin=-1, vmax=nblob)
                          axes[3].set_title("Dest padded")
                          plt.show()
                      

                      e.g.:

                      affine_test(0, (-20, 40))
                      

                      gives:

                      enter image description here

                      With a zoom in showing the aligned in the padded images:

                      enter image description here

                      I require the full extent of the src and dst images aligned on the same pixel coordinates, with both rotations and translations.

                      Any help is greatly appreciated!

                      ANSWER

                      Answered 2022-Mar-22 at 16:44

                      If you have two images that are similar (or the same) and you want to align them, you can do it using both functions rotate and shift :

                      from scipy.ndimage import rotate, shift
                      

                      You need to find first the difference of angle between the two images angle_to_rotate, having that you apply a rotation to src:

                      angle_to_rotate = 25
                      rotated_src = rotate(src, angle_to_rotate , reshape=True, order=1, mode="constant")
                      

                      With reshape=True you avoid losing information from your original src matrix, and it pads the result so the image could be translated around the 0,0 indexes. You can calculate this translation as it is (x*cos(angle),y*sin(angle) where x and y are the dimensions of the image, but it probably won't matter.

                      Now you will need to translate the image to the source, for doing that you can use the shift function:

                      rot_translated_src = shift(rotated_src , [distance_x, distance_y])
                      

                      In this case there is no reshape (because otherwise you wouldn't have any real translation) so if the image was not previously padded some information will be lost.

                      But you can do some padding with

                      np.pad(src, number, mode='constant')
                      

                      To calculate distance_x and distance_y you will need to find a point that serves you as a reference between the rotated_src and the destination, then just calculate the distance in the x and y axis.

                      Summary

                      1. Make some padding in src, and dst
                      2. Find the angular distance between them.
                      3. Rotate src with scipy.ndimage.rotate using reshape=True
                      4. Find the horizontal and vertical distance distance_x, distance_y between the rotated image and dst
                      5. Translate your 'rotated_src' with scipy.ndimage.shift

                      Code

                      from scipy.ndimage import rotate, shift
                      import matplotlib.pyplot as plt
                      import numpy as np
                      

                      First we make the destination image:

                      # make and plot dest
                      dst = np.ones([40,20])
                      dst = np.pad(dst,10)
                      dst[17,[14,24]]=4
                      dst[27,14:25]=4
                      dst[26,[14,25]]=4
                      rotated_dst = rotate(dst, 20, order=1)
                      
                      plt.imshow(dst) # plot it
                      plt.imshow(rotated_dst)
                      plt.show()
                      

                      We make the Source image:

                      # make_src image and plot it
                      src = np.zeros([40,20])
                      src = np.pad(src,10)
                      src[0:20,0:20]=1
                      src[7,[4,14]]=4
                      src[17,4:15]=4
                      src[16,[4,15]]=4
                      plt.imshow(src)
                      plt.show()
                      

                      Then we align the src to the destination:

                      rotated_src = rotate(src, 20, order=1) # find the angle 20, reshape true is by default
                      plt.imshow(rotated_src)
                      plt.show()
                      distance_y = 8 # find this distances from rotated_src and dst
                      distance_x = 12 # use any visual reference or even the corners
                      translated_src = shift(rotated_src, [distance_y,distance_x])
                      plt.imshow(translated_src)
                      plt.show()
                      

                      pd: If you find problems to find the angle and the distances in a programmatic way, please leave a comment providing a bit more of insight of what can be used as a reference that could be for example the frame of the image or some image features / data)

                      Source https://stackoverflow.com/questions/71516584

                      Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                      Vulnerabilities

                      No vulnerabilities reported

                      Install scipy

                      You can download it from GitHub.
                      You can use scipy like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

                      Support

                      For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

                      DOWNLOAD this Library from

                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      Share this Page

                      share link
                      Consider Popular Python Libraries
                      Try Top Libraries by scipy
                      Compare Python Libraries with Highest Support
                      Compare Python Libraries with Highest Quality
                      Compare Python Libraries with Highest Security
                      Compare Python Libraries with Permissive License
                      Compare Python Libraries with Highest Reuse
                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      • © 2022 Open Weaver Inc.