numpy | fundamental package for scientific computing | Data Manipulation library

by numpy Python Version: 1.24.1 License: BSD-3-Clause

kandi X-RAY | numpy Summary

numpy is a Python library typically used in Utilities, Data Manipulation, Numpy applications. numpy has build file available, it has a Permissive License and it has medium support. However numpy has 152 bugs and it has 5 vulnerabilities. You can install using 'pip install numpy' or download it from GitHub, PyPI.

NumPy is the fundamental package for scientific computing with Python.

Support

Quality

Security

License

Reuse

kandi-support Support

numpy has a medium active ecosystem.
It has 22522 star(s) with 7688 fork(s). There are 574 watchers for this library.
There were 8 major release(s) in the last 6 months.
There are 1958 open issues and 9227 have been closed. On average issues are closed in 194 days. There are 201 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of numpy is 1.24.1

numpy Support

Best in #Data Manipulation

Average in #Data Manipulation

numpy Support

Best in #Data Manipulation

Average in #Data Manipulation

quality kandi Quality

numpy has 152 bugs (5 blocker, 0 critical, 140 major, 7 minor) and 1954 code smells.

numpy Quality

Best in #Data Manipulation

Average in #Data Manipulation

numpy Quality

Best in #Data Manipulation

Average in #Data Manipulation

security Security

numpy has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
numpy code analysis shows 5 unresolved vulnerabilities (5 blocker, 0 critical, 0 major, 0 minor).
There are 26 security hotspots that need review.

numpy Security

Best in #Data Manipulation

Average in #Data Manipulation

numpy Security

Best in #Data Manipulation

Average in #Data Manipulation

license License

numpy is licensed under the BSD-3-Clause License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

numpy License

Best in #Data Manipulation

Average in #Data Manipulation

numpy License

Best in #Data Manipulation

Average in #Data Manipulation

build Reuse

numpy releases are available to install and integrate.
Deployable package is available in PyPI.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
numpy saves you 126151 person hours of effort in developing the same functionality from scratch.
It has 132876 lines of code, 10561 functions and 454 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

numpy Reuse

Best in #Data Manipulation

Average in #Data Manipulation

numpy Reuse

Best in #Data Manipulation

Average in #Data Manipulation

Top functions reviewed by kandi - BETA

kandi has reviewed numpy and discovered the below as its top functions. This is intended to give you an instant insight into numpy implemented functionality, and help decide if they suit your requirements.

Create a configuration object .

Create a row from a text file .

Analyze the group .

Einsum operator .

Analyze block .

Pad an array with a given padding .

Compute the gradient of a function .

Calculate the percentile of an array .

Computes the einsum path .

Read data from a file .

Get all kandi verified functions for this library.

numpy Key Features

Website: https://www.numpy.org

Documentation: https://numpy.org/doc

Mailing list: https://mail.python.org/mailman/listinfo/numpy-discussion

Source code: https://github.com/numpy/numpy

Contributing: https://www.numpy.org/devdocs/dev/index.html

Bug reports: https://github.com/numpy/numpy/issues

Report a security vulnerability: https://tidelift.com/docs/security

a powerful N-dimensional array object

sophisticated (broadcasting) functions

tools for integrating C/C++ and Fortran code

useful linear algebra, Fourier transform, and random number capabilities

numpy Examples and Code Snippets

See all related Code Snippets

Community Discussions

Trending Discussions on numpy

Why is `np.sum(range(N))` very slow?
Installing scipy and scikit-learn on apple m1
How could I speed up my written python code: spheres contact detection (collision) using spatial searching
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)
TypeError: load() missing 1 required positional argument: 'Loader' in Google Colab
How do I calculate square root in Python?
Using a pip requirements file in a conda yml file throws AttributeError: 'FileNotFoundError'
Problem with memory allocation in Julia code
Efficient summation in Python
NumPy 1.21.2 may not yet support Python 3.10

Trending Discussions on numpy

QUESTION

Why is `np.sum(range(N))` very slow?

Asked 2022-Mar-29 at 14:31

I saw a video about speed of loops in python, where it was explained that doing sum(range(N)) is much faster than manually looping through range and adding the variables together, since the former runs in C due to built-in functions being used, while in the latter the summation is done in (slow) python. I was curious what happens when adding numpy to the mix. As I expected np.sum(np.arange(N)) is the fastest, but sum(np.arange(N)) and np.sum(range(N)) are even slower than doing the naive for loop.

Why is this?

Here's the script I used to test, some comments about the supposed cause of slowing done where I know (taken mostly from the video) and the results I got on my machine (python 3.10.0, numpy 1.21.2):

updated script:

import numpy as np
from timeit import timeit

N = 10_000_000
repetition = 10

def sum0(N = N):
    s = 0
    i = 0
    while i < N: # condition is checked in python
        s += i
        i += 1 # both additions are done in python
    return s

def sum1(N = N):
    s = 0
    for i in range(N): # increment in C
        s += i # addition in python
    return s

def sum2(N = N):
    return sum(range(N)) # everything in C

def sum3(N = N):
    return sum(list(range(N)))

def sum4(N = N):
    return np.sum(range(N)) # very slow np.array conversion

def sum5(N = N):
    # much faster np.array conversion
    return np.sum(np.fromiter(range(N),dtype = int))

def sum5v2_(N = N):
    # much faster np.array conversion
    return np.sum(np.fromiter(range(N),dtype = np.int_))

def sum6(N = N):
    # possibly slow conversion to Py_long from np.int
    return sum(np.arange(N))

def sum7(N = N):
    # list returns a list of np.int-s
    return sum(list(np.arange(N)))

def sum7v2(N = N):
    # tolist conversion to python int seems faster than the implicit conversion
    # in sum(list()) (tolist returns a list of python int-s)
    return sum(np.arange(N).tolist())

def sum8(N = N):
    return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)

def sum9(N = N):
    return np.arange(N).sum() # remove dispatch overhead

def array_basic(N = N):
    return np.array(range(N))

def array_dtype(N = N):
    return np.array(range(N),dtype = np.int_)

def array_iter(N = N):
    # np.sum's source code mentions to use fromiter to convert from generators
    return np.fromiter(range(N),dtype = np.int_)

print(f"while loop:         {timeit(sum0, number = repetition)}")
print(f"for loop:           {timeit(sum1, number = repetition)}")
print(f"sum_range:          {timeit(sum2, number = repetition)}")
print(f"sum_rangelist:      {timeit(sum3, number = repetition)}")
print(f"npsum_range:        {timeit(sum4, number = repetition)}")
print(f"npsum_iterrange:    {timeit(sum5, number = repetition)}")
print(f"npsum_iterrangev2:  {timeit(sum5, number = repetition)}")
print(f"sum_arange:         {timeit(sum6, number = repetition)}")
print(f"sum_list_arange:    {timeit(sum7, number = repetition)}")
print(f"sum_arange_tolist:  {timeit(sum7v2, number = repetition)}")
print(f"npsum_arange:       {timeit(sum8, number = repetition)}")
print(f"nparangenpsum:      {timeit(sum9, number = repetition)}")
print(f"array_basic:        {timeit(array_basic, number = repetition)}")
print(f"array_dtype:        {timeit(array_dtype, number = repetition)}")
print(f"array_iter:         {timeit(array_iter,  number = repetition)}")

print(f"npsumarangeREP:     {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
print(f"npsumarangeREP:     {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")

# Example output:
#
# while loop:         11.493371912998555
# for loop:           7.385945574002108
# sum_range:          2.4605720699983067
# sum_rangelist:      4.509678105998319
# npsum_range:        11.85120212900074
# npsum_iterrange:    4.464334709002287
# npsum_iterrangev2:  4.498494338993623
# sum_arange:         9.537815956995473
# sum_list_arange:    13.290120724996086
# sum_arange_tolist:  5.231948580003518
# npsum_arange:       0.241889145996538
# nparangenpsum:      0.21876695199898677
# array_basic:        11.736577274998126
# array_dtype:        8.71628468400013
# array_iter:         4.303306431000237
# npsumarangeREP:     21.240833958996518
# npsumarangeREP:     16.690092379001726

ANSWER

Answered 2021-Oct-16 at 17:42

From the cpython source code for sum sum initially seems to attempt a fast path that assumes all inputs are the same type. If that fails it will just iterate:

/* Fast addition by keeping temporary sums in C instead of new Python objects.
   Assumes all inputs are the same type.  If the assumption fails, default
   to the more general routine.
*/

I'm not entirely certain what is happening under the hood, but it is likely the repeated creation/conversion of C types to Python objects that is causing these slow-downs. It's worth noting that both sum and range are implemented in C.

This next bit is not really an answer to the question, but I wondered if we could speed up sum for python ranges as range is quite a smart object.

To do this I've used functools.singledispatch to override the built-in sum function specifically for the range type; then implemented a small function to calculate the sum of an arithmetic progression.

from functools import singledispatch

def sum_range(range_, /, start=0):
    """Overloaded `sum` for range, compute arithmetic sum"""
    n = len(range_)
    if not n:
        return start
    return int(start + (n * (range_[0] + range_[-1]) / 2))

sum = singledispatch(sum)
sum.register(range, sum_range)

def test():
    """
    >>> sum(range(0, 100))
    4950
    >>> sum(range(0, 10, 2))
    20
    >>> sum(range(0, 9, 2))
    20
    >>> sum(range(0, -10, -1))
    -45
    >>> sum(range(-10, 10))
    -10
    >>> sum(range(-1, -100, -2))
    -2500
    >>> sum(range(0, 10, 100))
    0
    >>> sum(range(0, 0))
    0
    >>> sum(range(0, 100), 50)
    5000
    >>> sum(range(0, 0), 10)
    10
    """

if __name__ == "__main__":
    import doctest
    doctest.testmod()

I'm not sure if this is complete, but it's definitely faster than looping.

Source https://stackoverflow.com/questions/69584027

QUESTION

How could I speed up my written python code: spheres contact detection (collision) using spatial searching

Asked 2022-Mar-13 at 15:43

I am working on a spatial search case for spheres in which I want to find connected spheres. For this aim, I searched around each sphere for spheres that centers are in a (maximum sphere diameter) distance from the searching sphere’s center. At first, I tried to use scipy related methods to do so, but scipy method takes longer times comparing to equivalent numpy method. For scipy, I have determined the number of K-nearest spheres firstly and then find them by cKDTree.query, which lead to more time consumption. However, it is slower than numpy method even by omitting the first step with a constant value (it is not good to omit the first step in this case). It is contrary to my expectations about scipy spatial searching speed. So, I tried to use some list-loops instead some numpy lines for speeding up using numba prange. Numba run the code a little faster, but I believe that this code can be optimized for better performances, perhaps by vectorization, using other alternative numpy modules or using numba in another way. I have used iteration on all spheres due to prevent probable memory leaks and …, where number of spheres are high.

import numpy as np
import numba as nb
from scipy.spatial import cKDTree, distance

# ---------------------------- input data ----------------------------
""" For testing by prepared files:
radii = np.load('a.npy')     # shape: (n-spheres, )     must be loaded by np.load('a.npy') or np.loadtxt('radii_large.csv')
poss = np.load('b.npy')      # shape: (n-spheres, 3)    must be loaded by np.load('b.npy') or np.loadtxt('pos_large.csv', delimiter=',')
"""

rnd = np.random.RandomState(70)
data_volume = 200000

radii = rnd.uniform(0.0005, 0.122, data_volume)
dia_max = 2 * radii.max()

x = rnd.uniform(-1.02, 1.02, (data_volume, 1))
y = rnd.uniform(-3.52, 3.52, (data_volume, 1))
z = rnd.uniform(-1.02, -0.575, (data_volume, 1))
poss = np.hstack((x, y, z))
# --------------------------------------------------------------------

# @nb.jit('float64[:,::1](float64[:,::1], float64[::1])', forceobj=True, parallel=True)
def ends_gap(poss, dia_max):
    particle_corsp_overlaps = np.array([], dtype=np.float64)
    ends_ind = np.empty([1, 2], dtype=np.int64)
    """ using list looping """
    # particle_corsp_overlaps = []
    # ends_ind = []

    # for particle_idx in nb.prange(len(poss)):  # by list looping
    for particle_idx in range(len(poss)):
        unshared_idx = np.delete(np.arange(len(poss)), particle_idx)                                                    # <--- relatively high time consumer
        poss_without = poss[unshared_idx]

        """ # SCIPY method ---------------------------------------------------------------------------------------------
        nears_i_ind = cKDTree(poss_without).query_ball_point(poss[particle_idx], r=dia_max)         # <--- high time consumer
        if len(nears_i_ind) > 0:
            dist_i, dist_i_ind = cKDTree(poss_without[nears_i_ind]).query(poss[particle_idx], k=len(nears_i_ind))       # <--- high time consumer
            if not isinstance(dist_i, float):
                dist_i[dist_i_ind] = dist_i.copy()
        """  # NUMPY method --------------------------------------------------------------------------------------------
        lx_limit_idx = poss_without[:, 0] <= poss[particle_idx][0] + dia_max
        ux_limit_idx = poss_without[:, 0] >= poss[particle_idx][0] - dia_max
        ly_limit_idx = poss_without[:, 1] <= poss[particle_idx][1] + dia_max
        uy_limit_idx = poss_without[:, 1] >= poss[particle_idx][1] - dia_max
        lz_limit_idx = poss_without[:, 2] <= poss[particle_idx][2] + dia_max
        uz_limit_idx = poss_without[:, 2] >= poss[particle_idx][2] - dia_max

        nears_i_ind = np.where(lx_limit_idx & ux_limit_idx & ly_limit_idx & uy_limit_idx & lz_limit_idx & uz_limit_idx)[0]
        if len(nears_i_ind) > 0:
            dist_i = distance.cdist(poss_without[nears_i_ind], poss[particle_idx][None, :]).squeeze()                   # <--- relatively high time consumer
        # """  # -------------------------------------------------------------------------------------------------------
            contact_check = dist_i - (radii[unshared_idx][nears_i_ind] + radii[particle_idx])
            connected = contact_check[contact_check <= 0]

            particle_corsp_overlaps = np.concatenate((particle_corsp_overlaps, connected))
            """ using list looping """
            # if len(connected) > 0:
            #    for value_ in connected:
            #        particle_corsp_overlaps.append(value_)

            contacts_ind = np.where([contact_check <= 0])[1]
            contacts_sec_ind = np.array(nears_i_ind)[contacts_ind]
            sphere_olps_ind = np.where((poss[:, None] == poss_without[contacts_sec_ind][None, :]).all(axis=2))[0]       # <--- high time consumer

            ends_ind_mod_temp = np.array([np.repeat(particle_idx, len(sphere_olps_ind)), sphere_olps_ind], dtype=np.int64).T
            if particle_idx > 0:
                ends_ind = np.concatenate((ends_ind, ends_ind_mod_temp))
            else:
                ends_ind[0, 0], ends_ind[0, 1] = ends_ind_mod_temp[0, 0], ends_ind_mod_temp[0, 1]
            """ using list looping """
            # for contacted_idx in sphere_olps_ind:
            #    ends_ind.append([particle_idx, contacted_idx])

    # ends_ind_org = np.array(ends_ind)  # using lists
    ends_ind_org = ends_ind
    ends_ind, ends_ind_idx = np.unique(np.sort(ends_ind_org), axis=0, return_index=True)                                # <--- relatively high time consumer
    gap = np.array(particle_corsp_overlaps)[ends_ind_idx]
    return gap, ends_ind, ends_ind_idx, ends_ind_org

In one of my tests on 23000 spheres, scipy, numpy, and numba-aided methods finished the loop in about 400, 200, and 180 seconds correspondingly using Colab TPU; for 500.000 spheres it take 3.5 hours. These execution times are not satisfying at all for my project, where number of spheres may be up to 1.000.000 in a medium data volume. I will call this code many times in my main code and seeking for ways that could perform this code in milliseconds (as much as fastest that it could). Is it possible?? I would be appreciated if anyone would speed up the code as it is needed.

Notes:

This code must be executable with python 3.7+, on CPU and GPU.
This code must be applicable for data size, at least, 300.000 spheres.
All numpy, scipy, and … equivalent modules instead of my written modules, which make my code faster significantly, will be upvoted.

I would be appreciated for any recommendations or explanations about:

Which method could be faster in this subject?
Why scipy is not faster than other methods in this case and where it could be helpful relating to this subject?
Choosing between iterator methods and matrix form methods is a confusing matter for me. Iterating methods use less memory and could be used and tuned up by numba and … but, I think, are not useful and comparable with matrix methods (which depends on memory limits) like numpy and … for huge sphere numbers. For this case, perhaps I could omit the iteration by numpy, but I guess strongly that it cannot be handled due to huge matrix size operations and memory leaks.

Prepared sample test data:

Poss data: 23000, 500000
Radii data: 23000, 500000
Line by line speed test logs: for two test cases scipy method and numpy time consumption.

ANSWER

Answered 2022-Feb-14 at 10:23

Have you tried FLANN?

This code doesn't solve your problem completely. It simply finds the nearest 50 neighbors to each point in your 500000 point dataset:

from pyflann import FLANN

p = np.loadtxt("pos_large.csv", delimiter=",")
flann = FLANN()
flann.build_index(pts=p)
idx, dist = flann.nn_index(qpts=p, num_neighbors=50)

The last line takes less than a second in my laptop without any tuning or parallelization.

Source https://stackoverflow.com/questions/71104627

QUESTION

Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)

Asked 2022-Mar-05 at 07:13

version pip 21.2.4 python 3.6

The command:

pip install -r  requirments.txt

The content of my requirements.txt:

mongoengine==0.19.1
numpy==1.16.2
pylint
pandas==1.1.5
fawkes

The command is failing with this error

ERROR: Command errored out with exit status 1:
     command: /Users/*/Desktop/ml/*/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"'; __file__='"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-pip-egg-info-97994d6e
         cwd: /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/
    Complete output (1 lines):
    error in mongoengine setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding https://*/pypi/packages/mongoengine-0.19.1.tar.gz#md5=68e613009f6466239158821a102ac084 (from https://*/pypi/simple/mongoengine/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement mongoengine==0.19.1 (from versions: 0.15.0, 0.19.1)
ERROR: No matching distribution found for mongoengine==0.19.1

ANSWER

Answered 2021-Nov-19 at 13:30

It looks like setuptools>=58 breaks support for use_2to3:

setuptools changelog for v58

So you should update setuptools to setuptools<58 or avoid using packages with use_2to3 in the setup parameters.

I was having the same problem, pip==19.3.1

Source https://stackoverflow.com/questions/69100275

QUESTION

TypeError: load() missing 1 required positional argument: 'Loader' in Google Colab

Asked 2022-Mar-04 at 11:01

I am trying to do a regular import in Google Colab.
This import worked up until now.
If I try:

import plotly.express as px

import pingouin as pg

I get an error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-86e89bd44552> in <module>()
----> 1 import plotly.express as px

9 frames
/usr/local/lib/python3.7/dist-packages/plotly/express/__init__.py in <module>()
     13     )
     14 
---> 15 from ._imshow import imshow
     16 from ._chart_types import (  # noqa: F401
     17     scatter,

/usr/local/lib/python3.7/dist-packages/plotly/express/_imshow.py in <module>()
      9 
     10 try:
---> 11     import xarray
     12 
     13     xarray_imported = True

/usr/local/lib/python3.7/dist-packages/xarray/__init__.py in <module>()
      1 import pkg_resources
      2 
----> 3 from . import testing, tutorial, ufuncs
      4 from .backends.api import (
      5     load_dataarray,

/usr/local/lib/python3.7/dist-packages/xarray/tutorial.py in <module>()
     11 import numpy as np
     12 
---> 13 from .backends.api import open_dataset as _open_dataset
     14 from .backends.rasterio_ import open_rasterio as _open_rasterio
     15 from .core.dataarray import DataArray

/usr/local/lib/python3.7/dist-packages/xarray/backends/__init__.py in <module>()
      4 formats. They should not be used directly, but rather through Dataset objects.
      5 
----> 6 from .cfgrib_ import CfGribDataStore
      7 from .common import AbstractDataStore, BackendArray, BackendEntrypoint
      8 from .file_manager import CachingFileManager, DummyFileManager, FileManager

/usr/local/lib/python3.7/dist-packages/xarray/backends/cfgrib_.py in <module>()
     14     _normalize_path,
     15 )
---> 16 from .locks import SerializableLock, ensure_lock
     17 from .store import StoreBackendEntrypoint
     18 

/usr/local/lib/python3.7/dist-packages/xarray/backends/locks.py in <module>()
     11 
     12 try:
---> 13     from dask.distributed import Lock as DistributedLock
     14 except ImportError:
     15     DistributedLock = None

/usr/local/lib/python3.7/dist-packages/dask/distributed.py in <module>()
      1 # flake8: noqa
      2 try:
----> 3     from distributed import *
      4 except ImportError:
      5     msg = (

/usr/local/lib/python3.7/dist-packages/distributed/__init__.py in <module>()
      1 from __future__ import print_function, division, absolute_import
      2 
----> 3 from . import config
      4 from dask.config import config
      5 from .actor import Actor, ActorFuture

/usr/local/lib/python3.7/dist-packages/distributed/config.py in <module>()
     18 
     19 with open(fn) as f:
---> 20     defaults = yaml.load(f)
     21 
     22 dask.config.update_defaults(defaults)

TypeError: load() missing 1 required positional argument: 'Loader'

I think it might be a problem with Google Colab or some basic utility package that has been updated, but I can not find a way to solve it.

ANSWER

Answered 2021-Oct-15 at 21:11

Found the problem.
I was installing pandas_profiling, and this package updated pyyaml to version 6.0 which is not compatible with the current way Google Colab imports packages.
So just reverting back to pyyaml version 5.4.1 solved the problem.

For more information check versions of pyyaml here.
See this issue and formal answers in GitHub

##################################################################
For reverting back to pyyaml version 5.4.1 in your code, add the next line at the end of your packages installations:

!pip install pyyaml==5.4.1

It is important to put it at the end of the installation, some of the installations will change the pyyaml version.

Source https://stackoverflow.com/questions/69564817

QUESTION

How do I calculate square root in Python?

Asked 2022-Feb-17 at 03:40

I need to calculate the square root of some numbers, for example √9 = 3 and √2 = 1.4142. How can I do it in Python?

The inputs will probably be all positive integers, and relatively small (say less than a billion), but just in case they're not, is there anything that might break?

Related

_{Note: This is an attempt at a canonical question after a discussion on Meta about an existing question with the same title.}

ANSWER

Answered 2022-Feb-04 at 19:44

Option 1: math.sqrt()

The math module from the standard library has a sqrt function to calculate the square root of a number. It takes any type that can be converted to float (which includes int) as an argument and returns a float.

>>> import math
>>> math.sqrt(9)
3.0

Option 2: Fractional exponent

The power operator (**) or the built-in pow() function can also be used to calculate a square root. Mathematically speaking, the square root of a equals a to the power of 1/2.

The power operator requires numeric types and matches the conversion rules for binary arithmetic operators, so in this case it will return either a float or a complex number.

>>> 9 ** (1/2)
3.0
>>> 9 ** .5  # Same thing
3.0
>>> 2 ** .5
1.4142135623730951

(Note: in Python 2, 1/2 is truncated to 0, so you have to force floating point arithmetic with 1.0/2 or similar. See Why does Python give the "wrong" answer for square root?)

This method can be generalized to nth root, though fractions that can't be exactly represented as a float (like 1/3 or any denominator that's not a power of 2) may cause some inaccuracy:

>>> 8 ** (1/3)
2.0
>>> 125 ** (1/3)
4.999999999999999

Edge cases Negative and complex

Exponentiation works with negative numbers and complex numbers, though the results have some slight inaccuracy:

>>> (-25) ** .5  # Should be 5j
(3.061616997868383e-16+5j)
>>> 8j ** .5  # Should be 2+2j
(2.0000000000000004+2j)

Note the parentheses on -25! Otherwise it's parsed as -(25**.5) because exponentiation is more tightly binding than unary negation.

Meanwhile, math is only built for floats, so for x<0, math.sqrt() will raise ValueError: math domain error and for complex x, it'll raise TypeError: can't convert complex to float. Instead, you can use cmath.sqrt(), which is more more accurate than exponentiation (and will likely be faster too):

>>> import cmath
>>> cmath.sqrt(-25)
5j
>>> cmath.sqrt(8j)
(2+2j)

Precision

Both options involve an implicit conversion to float, so floating point precision is a factor. For example:

>>> n = 10**30
>>> square = n**2
>>> x = square**.5
>>> x == n
False
>>> x - n  # how far off are they?
0.0
>>> int(x) - n  # how far off is the float from the int?
19884624838656

Very large numbers might not even fit in a float and you'll get OverflowError: int too large to convert to float. See Python sqrt limit for very large numbers?

Other types

Let's look at Decimal for example:

Exponentiation fails unless the exponent is also Decimal:

>>> decimal.Decimal('9') ** .5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for ** or pow(): 'decimal.Decimal' and 'float'
>>> decimal.Decimal('9') ** decimal.Decimal('.5')
Decimal('3.000000000000000000000000000')

Meanwhile, math and cmath will silently convert their arguments to float and complex respectively, which could mean loss of precision.

decimal also has its own .sqrt(). See also calculating n-th roots using Python 3's decimal module

Source https://stackoverflow.com/questions/70793490

QUESTION

Problem with memory allocation in Julia code

Asked 2022-Jan-19 at 09:34

I used a function in Python/Numpy to solve a problem in combinatorial game theory.

import numpy as np
from time import time

def problem(c):
    start = time()
    N = np.array([0, 0])
    U = np.arange(c)
    
    for _ in U:
        bits = np.bitwise_xor(N[:-1], N[-2::-1])
        N = np.append(N, np.setdiff1d(U, bits).min())

    return len(*np.where(N==0)), time()-start 

problem(10000)

Then I wrote it in Julia because I thought it'd be faster due to Julia using just-in-time compilation.

function problem(c)
    N = [0]
    U = Vector(0:c)
    
    for _ in U
        elems = N[1:length(N)-1]
        bits = elems .⊻ reverse(elems)
        push!(N, minimum(setdiff(U, bits))) 
    end
    
    return sum(N .== 0)
end

@time problem(10000)

But the second version was much slower. For c = 10000, the Python version takes 2.5 sec. on an Core i5 processor and the Julia version takes 4.5 sec. Since Numpy operations are implemented in C, I'm wondering if Python is indeed faster or if I'm writing a function with wasted time complexity.

The implementation in Julia allocates a lot of memory. How to reduce the number of allocations to improve its performance?

ANSWER

Answered 2022-Jan-19 at 09:34

The original code can be re-written in the following way:

function problem2(c)
    N = zeros(Int, c+2)
    notseen = falses(c+1)

    for lN in 1:c+1
        notseen .= true
        @inbounds for i in 1:lN-1
            b = N[i] ⊻ N[lN-i]
            b <= c && (notseen[b+1] = false)
        end
        idx = findfirst(notseen)
        isnothing(idx) || (N[lN+1] = idx-1)
    end
    return count(==(0), N)
end

First check if the functions produce the same results:

julia> problem(10000), problem2(10000)
(1475, 1475)

(I have also checked that the generated N vector is identical)

Now let us benchmark both functions:

julia> using BenchmarkTools

julia> @btime problem(10000)
  4.938 s (163884 allocations: 3.25 GiB)
1475

julia> @btime problem2(10000)
  76.275 ms (4 allocations: 79.59 KiB)
1475

So it turns out to be over 60x faster.

What I do to improve the performance is avoiding allocations. In Julia it is easy and efficient. If any part of the code is not clear please comment. Note that I concentrated on showing how to improve the performance of Julia code (and not trying to just replicate the Python code, since - as it was commented under the original post - doing language performance comparisons is very tricky). I think it is better to concentrate in this discussion on how to make Julia code fast.

EDIT

Indeed changing to Vector{Bool} and removing the condition on b and c relation (which mathematically holds for these values of c) gives a better speed:

julia> function problem3(c)
           N = zeros(Int, c+2)
           notseen = Vector{Bool}(undef, c+1)

           for lN in 1:c+1
               notseen .= true
               @inbounds for i in 1:lN-1
                   b = N[i] ⊻ N[lN-i]
                   notseen[b+1] = false
               end
               idx = findfirst(notseen)
               isnothing(idx) || (N[lN+1] = idx-1)
           end
           return count(==(0), N)
       end
problem3 (generic function with 1 method)

julia> @btime problem3(10000)
  20.714 ms (3 allocations: 88.17 KiB)
1475

Source https://stackoverflow.com/questions/70766215

QUESTION

Efficient summation in Python

Asked 2022-Jan-16 at 12:49

I am trying to efficiently compute a summation of a summation in Python:

$\sum_{x=0}^{n}\left(x^2\sum_{y=0}^x y\right )$

WolframAlpha is able to compute it too a high n value: sum of sum.

I have two approaches: a for loop method and an np.sum method. I thought the np.sum approach would be faster. However, they are the same until a large n, after which the np.sum has overflow errors and gives the wrong result.

I am trying to find the fastest way to compute this sum.

import numpy as np
import time

def summation(start,end,func):
    sum=0
    for i in range(start,end+1):
        sum+=func(i)
    return sum

def x(y):
    return y

def x2(y):
    return y**2

def mysum(y):
    return x2(y)*summation(0, y, x)

n=100

# method #1
start=time.time()
summation(0,n,mysum)
print('Slow method:',time.time()-start)

# method #2
start=time.time()
w=np.arange(0,n+1)
(w**2*np.cumsum(w)).sum()
print('Fast method:',time.time()-start)

ANSWER

Answered 2022-Jan-16 at 12:49

(fastest methods, 3 and 4, are at the end)

In a fast NumPy method you need to specify dtype=np.object so that NumPy does not convert Python int to its own dtypes (np.int64 or others). It will now give you correct results (checked it up to N=100000).

# method #2
start=time.time()
w=np.arange(0, n+1, dtype=np.object)
result2 = (w**2*np.cumsum(w)).sum()
print('Fast method:', time.time()-start)

Your fast solution is significantly faster than the slow one. Yes, for large N's, but already at N=100 it is like 8 times faster:

start=time.time()
for i in range(100):
    result1 = summation(0, n, mysum)
print('Slow method:', time.time()-start)

# method #2
start=time.time()
for i in range(100):
    w=np.arange(0, n+1, dtype=np.object)
    result2 = (w**2*np.cumsum(w)).sum()
print('Fast method:', time.time()-start)

Slow method: 0.06906533241271973
Fast method: 0.008007287979125977

EDIT: Even faster method (by KellyBundy, the Pumpkin) is by using pure python. Turns out NumPy has no advantage here, because it has no vectorized code for np.objects.

# method #3
import itertools
start=time.time()
for i in range(100):
    result3 = sum(x*x * ysum for x, ysum in enumerate(itertools.accumulate(range(n+1))))
print('Faster, pure python:', (time.time()-start))

Faster, pure python: 0.0009944438934326172

EDIT2: Forss noticed that numpy fast method can be optimized by using x*x instead of x**2. For N > 200 it is faster than pure Python method. For N < 200 it is slower than pure Python method (the exact value of boundary may depend on machine, on mine it was 200, its best to check it yourself):

# method #4
start=time.time()
for i in range(100):
    w = np.arange(0, n+1, dtype=np.object)
    result2 = (w*w*np.cumsum(w)).sum()
print('Fast method x*x:', time.time()-start)

Source https://stackoverflow.com/questions/69864793

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install numpy

You can install using 'pip install numpy' or download it from GitHub, PyPI.
You can use numpy like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

The NumPy project welcomes your expertise and enthusiasm!. Small improvements or fixes are always appreciated; issues labeled as "good first issue" may be a good starting point. If you are considering larger contributions to the source code, please contact us through the mailing list first.

Find more information at: