Documentr | Generate Sexy Documentation From Markdown
kandi X-RAY | Documentr Summary
kandi X-RAY | Documentr Summary
Generate Sexy Documentation From Markdown
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of Documentr
Documentr Key Features
Documentr Examples and Code Snippets
Community Discussions
Trending Discussions on Documentr
QUESTION
I want to use apache arrow because it enables execution engines to take advantage of the latest SIMD (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing. (https://arrow.apache.org/).
From documentration (https://arrow.apache.org/docs/memory_layout.html), I understand that memory allocation make sure about 64 byte alignment.
In order to verify this 64 bytes alignment, I use the __array_interface__ data member of a numpy array that points to the data-area storing the array contents and compute a modulo 64 on it. If the result is 0 then the memory address is aligned on at least 64 Bytes.
When I execute the code bellow, on my system (Fedora) it seems to work (the result of modulo 64 is zero) but when I execute the same code on my colleague's system (Fedora too) it does not work: the result of modulo 64 is not zero. So the memory is not aligned on 64 bytes.
Please find my code here:
...ANSWER
Answered 2018-Feb-17 at 16:12The memory in Arrow is 64 byte aligned but in your example code, the conversion to Pandas/NumPy makes a copy of the data as a nested array of lists is differently represented in Arrow and in NumPy. In Arrow this is done using one buffer that holds the data of all lists while there is another buffer that holds the offsets for each list in that Array. As NumPy has no native list type, it is represented as a NumPy array that contains other NumPy arrays as elements. These are represented in the first NumPy array as Python objects.
Thus using the NumPy functions you see the memory as allocated by NumPy, not by Arrow. Thus if your memory address is on a 64 byte boundary, it is only by chance.
In the next version (0.9) of pyarrow
there will be a buffers
property to access the underlying memory addresses. You should then be able to directly check if the Arrow memory is allocated on a 64 byte aligned address (it always should be).
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install Documentr
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page