ringbuffer | Ring buffer that allows for high-throughput data transfer

 by   bslatkin Python Version: Current License: Apache-2.0

kandi X-RAY | ringbuffer Summary

kandi X-RAY | ringbuffer Summary

ringbuffer is a Python library. ringbuffer has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub.

A multiple writer and reader ring buffer that allows for high-throughput data transfer between multiproccessing Python processes. The goal is to make it easy to construct bandwidth-heavy pipelines (e.g., for video stream processing) that can utilize multiple cores with Python tools like OpenCV, Scikit Learn, and TensorFlow. The RingBuffer data structure's performance is primarily bound by the behavior of the Lock class, which is a Kernel semaphore under the covers. The lock is used to coordinate a readers/writer lock, meaning that lock contention dominates as the number of writes per second increases. Memory performance isn't an issue because all data is transferred through mmap'ed buffers. For examples of how it all fits together, look at example_numpy.py and example_ctypes.py. On an old MacBook, the ring buffer can easily do 2 gigabytes per second of transfer when using large slot sizes (~100MB), a relatively low number of writes per second (~24), and a single reader/writer pair. As you increase the number of writes per second the performance degrades proportionately due to the lock. However, increasing the number of readers has a minimal effect (when the slot sizes are ~10MB) because multiple readers can hold the read lock at the same time.
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              ringbuffer has a low active ecosystem.
              It has 102 star(s) with 20 fork(s). There are 8 watchers for this library.
              OutlinedDot
              It had no major release in the last 6 months.
              There are 4 open issues and 1 have been closed. There are 1 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of ringbuffer is current.

            kandi-Quality Quality

              ringbuffer has 0 bugs and 0 code smells.

            kandi-Security Security

              ringbuffer has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              ringbuffer code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              ringbuffer is licensed under the Apache-2.0 License. This license is Permissive.
              Permissive licenses have the least restrictions, and you can use them in most projects.

            kandi-Reuse Reuse

              ringbuffer releases are not available. You will need to build from source code and install.
              Build file is available. You can build the component from source.
              Installation instructions are not available. Examples and code snippets are available.
              ringbuffer saves you 474 person hours of effort in developing the same functionality from scratch.
              It has 1118 lines of code, 154 functions and 5 files.
              It has medium code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi has reviewed ringbuffer and discovered the below as its top functions. This is intended to give you an instant insight into ringbuffer implemented functionality, and help decide if they suit your requirements.
            • Generate a random writer
            • Checks if the given position has a write conflict
            • Mark the writer as done
            • Context manager for writing write operations
            • Acquire the writer lock
            • Release the writer lock
            • Returns the current position
            • Try to write data
            • Try to read data
            • Acquire the reader lock
            • Release the reader lock
            • Try to read data from the block
            • Context manager for reading
            • Read records from ring
            • Perform a blocking read
            • Wait for a write
            • Creates a new reader
            • Force synchronization of readers
            • Initialize a new writer
            Get all kandi verified functions for this library.

            ringbuffer Key Features

            No Key Features are available at this moment for ringbuffer.

            ringbuffer Examples and Code Snippets

            stream type:
            npmdot img1Lines of Code : 27dot img1no licencesLicense : No License
            copy iconCopy
            /* Create a ring buffer that stores the last 100 records. */
            var bunyan = require('bunyan');
            var ringbuffer = new bunyan.RingBuffer({ limit: 100 });
            var log = bunyan.createLogger({
                name: 'foo',
                streams: [
                    {
                        level: 'info',
              

            Community Discussions

            QUESTION

            The function pointer is changing its own address
            Asked 2022-Feb-17 at 07:53

            I am first time using function pointers and ran into a weird problem. I am writing a code for STM32G4xx. The main idea is to transmit and receive data through LPUART. I have implemented simple FSM to handle TX and RX. LPUART configured in DMA interrupt mode. I have typedef the function pointer and declared the three function pointer variables (ISR handles) in main.h file as follow:

            ...

            ANSWER

            Answered 2022-Feb-17 at 07:53

            As per @Lundin's suggestion, I have put a watchpoint on lpuart_dma_rx_tc_isr_clback function pointer variable. It exposed the out of index bug in my code. The bug is inside while loop in main.c.

            Source https://stackoverflow.com/questions/71139143

            QUESTION

            Are memory orders for each atomic correct in this lock-free SPSC ring buffer queue?
            Asked 2022-Jan-06 at 12:33

            I have a ring buffer that looks like:

            ...

            ANSWER

            Answered 2021-Dec-31 at 12:49

            Previous answers may help as background:
            c++, std::atomic, what is std::memory_order and how to use them?
            https://bartoszmilewski.com/2008/12/01/c-atomics-and-memory-ordering/

            Firstly the system you describe is known as a Single Producer - Single Consumer queue. You can always look at the boost version of this container to compare. I often will examine boost code, even when I work in situations where boost is not allowed. This is because examining and understanding a stable solution will give you insights into problems you may encounter (why did they do it that way? Oh, I see it - etc). Given your design, and having written many similar containers I will say that your design has to be careful about distinguishing empty from full. If you use a classic {begin,end} pair, you hit the problem that due to wrapping

            {begin, begin+size} == {begin, begin} == empty

            Okay, so back synchronisation issue.

            Given that the order only effects reording, the use of release in Publish seems a textbook use of the flag. Nothing will read the value until the size of the container is incremented, so you don't care if the orders of writes of the value itself happen in a random order, you only care that the value must be fully written before the count is increased. So I would concur, you are correctly using the flag in the Publish function.
            I did question whether the "release" was required in the Consume, but if you are moving out of the queue, and those moves are side-effecting, it may be required. I would say that if you are after raw speed, then it may be worth making a second version, that is specialised for trivial objects, that uses relaxed order for incrementing the head.

            You might also consider inplace new/delete as you push/pop. Whilst most moves will leave an object in an empty state, the standard only requires that it is left in a valid state after a move. explicitly deleting the object after the move may save you from obscure bugs later.

            You could argue that the two atomic loads in consume could be memory_order_consume. This relaxes the constraints to say "I don't care what order they are loaded, as long as they are both loaded by the time they are used". Although I doubt in practice it produces any gain. I am also nervous about this suggestion because when I look at the boost version it is remarkably close to what you have. https://www.boost.org/doc/libs/1_66_0/boost/lockfree/spsc_queue.hpp

            Source https://stackoverflow.com/questions/70512371

            QUESTION

            C++ destroys itself by detecting compile errors in the standard library
            Asked 2021-Dec-19 at 12:18

            I can't build the project in the Debug configuration due to the fact that the compiler detects strange compile errors in the standard library. The project can be built in the Release configuration, due to the absence of _DEBUG, but I need to debug the project somehow. How can it be fixed?

            Example ():

            ...

            ANSWER

            Answered 2021-Dec-19 at 12:18

            Indeed, the problem was in the stb_vorbis.c file, whose #define's conflict with the standard library. I discovered this by isolating from stb and then going to this file.

            I was inspired by the solution here. It was suitable to me to rearrange #include directives in such a way that stb_vorbis.c was at the end of the list. Now it is possible to build the entire project completely.

            FileManager.cpp:

            Source https://stackoverflow.com/questions/70408811

            QUESTION

            How to remove items from list in other class?
            Asked 2021-Oct-28 at 17:56

            I have a problem here, I need to do method pop but I can only use functions from SimpleList, any ideas?

            I can only modify class RingBuffer so I can't use remove(), pop() etc.

            ...

            ANSWER

            Answered 2021-Oct-28 at 17:56

            I think you figured it out, but just for you to compare:

            Source https://stackoverflow.com/questions/69757637

            QUESTION

            Ringbuffer for a microcontroller
            Asked 2021-Oct-27 at 17:25

            Here is a scenario I'm facing right now, I have an interrupt(thread) UART that is reading to a ringbuffer from the values that I get from the serial port, and also writing the values from the serial port to the ring buffer. I have a main loop that access that ringbuffer for reading the values from it, while writing an AT Command, and also writing to the ring buffer those AT Commands. Do I need the ringbuffer to be lock free or surround the shared data with a semaphore or a mutex ? I don't have an OS for getting a mutex or semaphore working. I have read alot about the subject and it seems I need a lock free ringbuffer. On ARM I would use a compare and swap instruction. The ringbuffer is implemented as an array so I wouldn't run into ABA problem

            Declaration of buffers:

            ...

            ANSWER

            Answered 2021-Oct-27 at 17:25

            You need synchronization between interrupt handlers and the "main-thread", so yes a sort of mutex is needed. The usual method without OS, is to just disable interrupts before "entering the critical section" and re-enable afterwards. That ensures the critical section is "atomic" (i.e. no interrupts fire in the meantime).

            On ARM I would use a compare and swap instruction

            What compiler are you using?

            GCC and Clang has intrinsics for atomic compare-and-swap.

            See e.g.

            https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html

            https://llvm.org/docs/Atomics.html

            C11 supports it via I believe: https://en.cppreference.com/w/c/atomic/atomic_compare_exchange

            Source https://stackoverflow.com/questions/69464202

            QUESTION

            Is array in java is threadsafe in case : one thread change value, one thread read value?
            Asked 2021-Sep-25 at 10:37

            I write a simple ringbuffer and in method test1() I use one thread is poll() and one thread is offer(). I test many time but it is always true. Can you explain for me?

            ...

            ANSWER

            Answered 2021-Sep-25 at 07:19

            There is no happens before edge between a write to an array at some position and a read from the same position. So if you don't have any ordering guarantee in place, your code is suffering from a data race.

            If you also allow for concurrent offers and concurrent polls, then you also have race conditions on your hands.

            It has been quite some time I played with ringbuffers. But normally you make use of a tail and head sequence (e.g. a long). If you make the ringbuffer a power of 2, you can do a cheap mod on the conversion of the sequences to indices. And the head and tail sequence could be relatively expensive volatiles (I really would start with that) and later on you could play with relaxed memory order modes. The head and tail will give you the appropriate happens before edges so don't need to do anything special to the array. With this approach you can also get rid of the 'size'; you can calculate the size as the difference between tail and thehead; the problem with size is that it will cause contention between a thread read/writing to the ringbuffer. Also you need to properly pad the the head/tail fields to prevent false sharing.

            Source https://stackoverflow.com/questions/69323845

            QUESTION

            Tomcat in Eclipse does not start servlet anymore: migrating Jersey 2.32 -> 3.0.1
            Asked 2021-Sep-17 at 06:48

            I know there are many old questions regarding this topic: 1, 2, 3

            But I am using a different approach in for my pretty straight forward servlet.

            Setup
            1. Tomcat 9
            2. Maven for clean builds and managing dependencies
            3. Jersey
            4. Eclipse 2021-09 (4.21.0)
            I am not using a web.xml, like this:

            4.7.2.1. Descriptor-less deployment

            There are multiple deployment options in the Servlet 3.0 container for a JAX-RS application defined by implementing a custom Application subclass. For simple deployments, no web.xml is necessary at all. Instead, an @ApplicationPath annotation can be used to annotate the custom Application subclass and define the base application URI for all JAX-RS resources configured in the application:

            Example there

            ...

            ANSWER

            Answered 2021-Sep-16 at 14:50

            Jersey 3.0 is an implementation of Jakarta RESTful Web Services 3.0, which is part of Jakarta EE 9.

            This is the first version, which uses the new jakarta.* package namespace instead of the old namespace javax.*. Therefore Jersey 3.0 works only with other Jakarta EE 9 technologies and is not compatible with Jakarta EE 8 specifications.

            If you wish to use Jersey 3.0, you need to upgrade to Tomcat 10.

            Source https://stackoverflow.com/questions/69209718

            QUESTION

            Writing audio stream to circular buffer but Segmentation error reading value
            Asked 2021-Mar-07 at 17:09

            I very new concepts of audio processing and have a basic knowledge of C++, my end goal is to create a means to measure harmonic distortion on line-in audio; my thoughts are to write audio-in stream to a circular buffer, then read from the circular buffer to an FFT function; from the FFT data I can workout audio distortion.

            So far I'm using Portaudio to write streamed audio to the circular buffer from it's streaming input callback; I've created a Thread that checks if there's any data in the circular buffer, if there is data, read it into a temporary buffer. As a means of a simple test, I'm merely echo'ing the number of elements to read from the circular buffer, this works but after a second or so, produces an incorrect value for 'read elements available' (18446744073709542400) followed by a Segmentation error; sample output and my code below:

            ...

            ANSWER

            Answered 2021-Mar-07 at 17:09

            RingBuffer::readAvailable() is returning a small negative number as a size_t. Since size_t is an unsigned type and because you're using %lu in the printf, it's being displayed as though it's a huge unsigned long. (Somehow your output has extra digits.)

            It's possible RingBuffer has a bug. It seems to assume that that tail index will always be less then or equal to the head index. I'm not an expert in the memory ordering constraints used in lockless code, but it looks to me like it could be a synchronization problem with when it reads the head and tail to compute the amount available.

            Source https://stackoverflow.com/questions/66515015

            QUESTION

            AvSetMmThreadCharacteristicsW for UWP
            Asked 2021-Feb-19 at 14:31

            I'm working on a WASAPI UWP audio application with cpp/winrt which needs to take audio from an input and send it to an output after being processed.

            I want to set my audio thread characteristics with AvSetMmThreadCharacteristicsW(L"Pro Audio", &taskIndex), but I just noticed this function (and most of avrt.h) is limited to WINAPI_PARTITION_DESKTOP and WINAPI_PARTITION_GAMES.

            I think I need this because when my code is integrated into my UWP app, the audio input is full of discontinuity, and I have no issue in my test code which uses the avrt API.

            Is there another way to configure my thread for audio processing?

            Edit: here is my test program https://github.com/loics2/test-wasapi. The interesting part happens in the AudioStream class. I can't share my UWP app, but I can copy as is these classes into a Windows Runtime Component.

            Edit 2: here's the audio thread code :

            ...

            ANSWER

            Answered 2021-Feb-19 at 14:31

            @IInspectable was right, there's something wrong with my code : the audio processing is done by a library which then calls callbacks with some results.

            In my callback, I try to raise a winrt::event, but it sometimes takes more than 50ms. When it happens, it blocks the audio thread, and creates discontinuity...

            Source https://stackoverflow.com/questions/66211356

            QUESTION

            Where is the pointer pointing?
            Asked 2020-Dec-25 at 08:46

            This is the defination for my class.

            ...

            ANSWER

            Answered 2020-Dec-25 at 08:39

            rings.get():

            This is getting raw pointer from rings which gives you RingBuffer *

            rings_->ring1_.get():

            I suppose you mean rings->ring1_.get(), which first dereferences rings and then get raw pointer from member ring1_. It is the same with rings.get()->ring1_.get(). The final result is a RingBufferLock *

            Source https://stackoverflow.com/questions/65446498

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install ringbuffer

            You can download it from GitHub.
            You can use ringbuffer like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

            Support

            For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
            CLONE
          • HTTPS

            https://github.com/bslatkin/ringbuffer.git

          • CLI

            gh repo clone bslatkin/ringbuffer

          • sshUrl

            git@github.com:bslatkin/ringbuffer.git

          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link