parfor | A package to mimic the use of parfor as done in Matlab

 by   wimpomp Python Version: 2024.5.1 License: GPL-3.0

kandi X-RAY | parfor Summary

kandi X-RAY | parfor Summary

parfor is a Python library. parfor has no bugs, it has no vulnerabilities, it has build file available, it has a Strong Copyleft License and it has low support. You can install using 'pip install parfor' or download it from GitHub, PyPI.

Used to parallelize for-loops using parfor in Matlab? This package allows you to do the same in python. Take any normal serial but parallelizable for-loop and execute it in parallel using easy syntax. Don't worry about the technical details of using the multiprocessing module, race conditions, queues, parfor handles all that. Tested on linux on python 2.7 and 3.8 and on Windows and OSX on python 3.8.
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              parfor has a low active ecosystem.
              It has 7 star(s) with 0 fork(s). There are 2 watchers for this library.
              There were 3 major release(s) in the last 6 months.
              There are 0 open issues and 2 have been closed. On average issues are closed in 19 days. There are no pull requests.
              It has a neutral sentiment in the developer community.
              The latest version of parfor is 2024.5.1

            kandi-Quality Quality

              parfor has 0 bugs and 0 code smells.

            kandi-Security Security

              parfor has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              parfor code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              parfor is licensed under the GPL-3.0 License. This license is Strong Copyleft.
              Strong Copyleft licenses enforce sharing, and you can use them when creating open source projects.

            kandi-Reuse Reuse

              parfor releases are not available. You will need to build from source code and install.
              Deployable package is available in PyPI.
              Build file is available. You can build the component from source.
              Installation instructions, examples and code snippets are available.
              It has 480 lines of code, 57 functions and 2 files.
              It has medium code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi has reviewed parfor and discovered the below as its top functions. This is intended to give you an instant insight into parfor implemented functionality, and help decide if they suit your requirements.
            • Print error in main thread
            • Close the queue
            • Empty queue
            • Decorator to create a function from a function
            • Map a function over an iterable
            • Get the latest task from the queue
            • Get a task from the queue
            • Called when a task is done
            • Update the qbar
            • Called when an error occurs
            • Close queue
            • Update the cache
            • Remove items from the queue
            Get all kandi verified functions for this library.

            parfor Key Features

            No Key Features are available at this moment for parfor.

            parfor Examples and Code Snippets

            No Code Snippets are available at this moment for parfor.

            Community Discussions

            QUESTION

            Numba : parallel flag in @jit does not work in my code with numpy 2D arrays
            Asked 2021-Dec-13 at 19:44

            First, I asked this question Numba : Why guvectorize is so slow?. Trying many things as I am discovering numba, this more basic question arose

            Why this code works (parallel flag set to False):

            ...

            ANSWER

            Answered 2021-Dec-13 at 19:44

            Not sure wether it is strictly necessary in your project that you allocate the result array outside of the function scope and pass it to the function as argument. Otherwise this code modification runs perfectly fine:

            Source https://stackoverflow.com/questions/70338614

            QUESTION

            matlab random number generation in parfor loop
            Asked 2021-Dec-06 at 07:30

            I want to generate same normal random numbers for loop and parfor loop. Following MATLAB documentation I tried three different method:

            Method 1: Using rng(seed, 'twister')

            ...

            ANSWER

            Answered 2021-Dec-04 at 00:52

            I'm inferring that your concern is that the pseudorandomly generated numbers for your for and parfor loops did not match. This is however expected behavior. You used a MRG32K3A pseudorandom algorithm, which "does not make a guarantee that a fixed seed and a fixed series of calls to mrg32k3a.RandomState methods using the same parameters will always produce the same results." Source.

            So it appears that everything is working correctly.

            Source https://stackoverflow.com/questions/70221929

            QUESTION

            Parallel Computing for video compression in MATLAB
            Asked 2021-Dec-01 at 04:06

            I need some help with parallel programming in MATLAB. To be clear, I have never implemented parallelization techniques in any of my codes before. I have a video compression engine, developed as part of my university project. It is a basic verion of H.264 video compression engine. I have to implement the parallel proceesing techniques available in MATALB to this engine. Basically, I have a function which divides an image frame into a number of blocks (predtermined by the size of the block). I'm trying to partially or fully parallelize this block of the code. I have used "parfor" when there was no dependency between the blocks, and this worked out well. I have uploaded this implementation. Now I'm trying to parallalize a case were there are dependencies between blocks.

            ...

            ANSWER

            Answered 2021-Dec-01 at 04:06

            You can use a parfor inside a non parallel for, something like this:

            Source https://stackoverflow.com/questions/70177270

            QUESTION

            MATLAB Parfor is running much slower on a 18-core PC than a 10-core PC
            Asked 2021-Aug-27 at 15:53

            I am currently running a compute-intensive financial trading test; the test has 503 independent loops. I have two PCs in my office, one has 10-core Intel i9-10900k @ 3.7GHz with 128GB DDR4; the other one has 18-core Intel i9-7980XE @ 3.4GHz with 128GB DDR4 as well.

            In order to speed up the whole processing, the first 10-core PC runs loop 1-to-189 in parfor; the second 18-core PC runs loop 190-to-503 in parfor.

            After one-day running, I found the 10-core PC finished 25 loops, the 18-core PC only finished 4 loops. I found very curious, can anyone know the reason of this problem?

            P.S. 10-core PC running MATLAB 2020b with latest updates with MOSEK 9.2.35 18-core PC running MATLAB 2021a with latest updates with MOSEK 9.2.36

            I also checked the taskmgr.exe, unlike the 10-core PC, I found in 18-core PC, 10 matlab tasks are in one group, the other 10 tasks are seperated....

            ...

            ANSWER

            Answered 2021-Aug-27 at 15:53

            Many thanks to Michal Adamaszek, When using the MOSEK (interior-point or integer programming) inside parfor loop, it is better to turn off the multi-threads in MOSEK, otherwise, by default, MOSEK will using all cores.

            Source https://stackoverflow.com/questions/68931802

            QUESTION

            Matlab parfor uses fewer cores than the allocated number of cores
            Asked 2021-Jun-28 at 07:00

            I'm running a parallel Matlab job on a single node of a remote cluster. Each node of the cluster has 2 processors with 24 cores each, for a total of 48 cores per node. The job contains some sequential code followed by a single parfor loop. I run it using a slurm bash script.

            The bash script test.sh is:

            ...

            ANSWER

            Answered 2021-Jun-28 at 07:00

            I bet your parallel pool is timing-out in between your parfor loops. It then gets auto-created with size 12, as that is the default preference for "preferred number of workers in a parallel pool" (doc). (Personally, I don't much care for that preference, and always set the value to 99999 and let other things control the size of the pool, but in your case you might not be able to if your SLURM workers don't share a MATLAB preferences directory (prefdir) with your client).

            I suggest you create your pool of size 48 with an IdleTimeout of Inf, like this:

            Source https://stackoverflow.com/questions/68146950

            QUESTION

            Array of structures with parfor in Matlab
            Asked 2021-Jun-21 at 21:04

            I am using Matlab to work on my project, and I want to use "parfor" as a part of my code that returns an array of structures in each its iteration, But when I run my code I got some errors. I just tried to bring an example of my problem in the simplest way. Will be appreciated any helps! Here is the example:

            ...

            ANSWER

            Answered 2021-Jun-21 at 21:04

            Find the solution, just need to define a cell array and then finally merge them! Follow can be an answer:

            Source https://stackoverflow.com/questions/68074506

            QUESTION

            Vectorized hashing/ranking of integer combinations of fixed size via operations on 32-bit integers in MATLAB
            Asked 2021-May-14 at 06:09

            I have huge dynamically created tables/matrices in MATLAB of varying first dimension, whose rows represent (sorted) combinations of integers in the range 1-50 of order 6.

            I would like to assign to each combination a unique value (hash, ranking), so that I can check if the same combinations appear in different tables. Different combinations are not allowed to have same value assigned, i.e. no collisions. I have to make a lot of such comparisons between a lot of such tables. So, for performance reasons, I would like to accomplish this by vectorization of uint32 operations to make it suitable for GPU acceleration in MATLAB.

            Things I have thought of so far:

            1. Lexicographic ranking: no idea how to vectorize the standard fast recursive algorithms well, and the only option seems to be to parfor it through the rows, which is slower than other options. IIRC, the direct explicit formula, though vectorizable, requires computation of binomials, which in turn requires log Gamma function in order to avoid huge factorials + double type to avoid collision if I am not mistaken, i.e. is slower because it's 'very numerical'.
            2. Cantor pairing function: one can successively apply Cantor's pairing, which is nice because it's a polynomial expression, but it produces huge numbers well beyond uint32 and is definitely slower than other options.
            3. Base 51 (no pun intended) integers: sends a combination/row vector (x_1,...,x_6) to x_1 + x_2 * 51 + ... + x_6 * 51^5. This is the fastest I currently have. It's easily vectorizable, but unfortunately still requires uint64 or double for rank-6 combinations of 50 elements, which is slower than uint32 or single type operations would be.

            So, I guess, I am looking for a 'clever' injective function on these combinations that computes within the uint32 range and is also well vectorizable (in MATLAB).

            Any help would be much appreciated!

            EDIT: Here is a routine that benchmarks both ranking and searching in uint32, single, and double. I have used MATLAB's gputimeit to produce accurate results.

            ...

            ANSWER

            Answered 2021-May-10 at 12:41

            You've almost got enough bits for your last idea, so you just need to squeeze a few bits out due to the ordering to get it over the bar. Since the whole sequence is sorted, every pair is also ordered. So use a 50-by-50 look-up table to map the sorted (1st,2nd), (3rd,4th), (5th,6th) pairs into numbers from 0-1274.

            Or if you don't want a table, there are fairly simple explicit functions for mapping a pair (i,j) with j>=i to a linear index. Look up upper- or lower-triangular matrix indexing for details on those. (It'll be something along the lines of n*(n+1)/2 - (n-i)*(n-i-1)/2 + j with some +/-1's thrown in depending on base-0 or base-1 indexing, and n=50 in your case, but I'm sure I'll get it wrong writing it off-the-cuff.)

            Anyway, once you've got three numbers 0-1274, the base-1275 idea will fit in uint32.

            Source https://stackoverflow.com/questions/67455774

            QUESTION

            How to solve local minimum issue when using fminsearch in Matlab?
            Asked 2021-May-13 at 16:38

            I am using 'fminsearch' in Matlab to solve the value function iteration problem.

            ...

            ANSWER

            Answered 2021-May-13 at 06:20

            Multiple startpoints is a well known practice for local optimization algorithms. Perhaps you can try a different algorithm like simumated annealing.

            Another way is to use global optimization algorithms like DIRECT (Dividing Rectangles) or Baysian Optimization if you do not know the gradient of the function

            Source https://stackoverflow.com/questions/67514624

            QUESTION

            How do I pass a template function to a thread within the same .cpp file?
            Asked 2021-May-09 at 16:23

            I have an assignment to implement a parallel version of the longest common subsequence algorithm (just calculating the LCS length). The program must use threads in order to complete the task as quickly as possible (at least, faster than a sequential implementation). Ideally, it should also utilize TLS in the threads. We had a similar assignment that implemented a template within a .hpp file and I want to use the same template, but it does not seem like I can use a .hpp file in this assignment. My problem lies in passing the template function to my threads. Below is the code:

            ...

            ANSWER

            Answered 2021-May-09 at 16:23

            You are right, you cannot pass a function template as if it was a function. They are different things, just like a cookie cutter is not a cookie.

            You have two main problems in your code.

            First, since ParFor::parfor is a template, you can only take a member function pointer to it if you provide template parameters that match what your lambdas use for TLS, so change it to this (for example):

            Source https://stackoverflow.com/questions/67443418

            QUESTION

            Printing what number Pari/GP is working on for a parallel computation
            Asked 2021-Feb-25 at 13:15

            I am using Pari/GP to test (pseudo)primality of a sequence of numbers f(n), where f(n) is some function. I have parallelized the code as

            ...

            ANSWER

            Answered 2021-Feb-25 at 13:03

            I solved (in a very dirty way probably) like this

            Source https://stackoverflow.com/questions/66368778

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install parfor

            You can install using 'pip install parfor' or download it from GitHub, PyPI.
            You can use parfor like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

            Support

            For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
            Install
          • PyPI

            pip install parfor

          • CLONE
          • HTTPS

            https://github.com/wimpomp/parfor.git

          • CLI

            gh repo clone wimpomp/parfor

          • sshUrl

            git@github.com:wimpomp/parfor.git

          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link