memchr | Optimized string search routines for Rust | DevOps library

stderr thread 'main' panicked at 'Error: pyo3 requires a nightly or dev version of Rust.'

You have to install a nightly or dev version of Rust (do you have rust installed at all ?)

EDIT You can install rust nightly on macOS with following commands in the shell:

Unclear exactly what you want to do BUT:

Not really a bug, but the old Valgrind version probably doesn't know about sse string operations.

As an optimization, str* and mem* functions can do 8 byte reads in the knowledge that memory will always be allocated in 8 byte increments. So this may read beyond the end of the string but it will never read unallocated memory.

Reqwest lists OpenSSL as a requirement on Linux due to it using native-tls, which depends on openssl. You need to install the pkg-config and libssl-dev packages:

This declaration

The function

I assume that memchr may always look at maxlen bytes no matter the contents of those bytes.

That assumption is wrong. From POSIX:

Implementations shall behave as if they read the memory byte by byte from the beginning of the bytes pointed to by s and stop at the first occurrence of c (if it is found in the initial n bytes).

The problem was originated because I redefined __asm to be replaced by nothing (e.g. #define __asm) since I didn't want to touch the assembly code yet. Remember that I said I am porting ARM to x86, so I thought that easiest way to get rid of the compile errors was to remove all those __asm instructions, but not considering the effects of doing such a thing.

In other words, when I included the string.h header, the header itself uses assembly call as the error messaged pointed out:

No, it's not a branch, that's the whole point of cmovcc.

It's an ALU select that has a data dependency on both inputs, not a control dependency. (With a memory source, it unconditionally loads the memory source, unlike ARM predicated load instructions that are truly NOPed. So you can't use it with maybe-bad pointers for branchless bounds or NULL checks. That's maybe the clearest illustration that it's definitely not a branch.)

But anyway, it's not predicted or speculated in any way; as far as the CPU scheduler is concerned it's just like an adc instruction: 2 integer inputs + FLAGS, and 1 integer output. (Only difference from adc/sbb is that it doesn't write FLAGS. And of course runs on an execution unit with different internals).

Whether that's good or bad entirely depends on the use-case. See also gcc optimization flag -O3 makes code slower than -O2 for much more about cmov upside / downside

Note that repne scasb is not fast. "Fast Strings" only works for rep stos / movs.

repne scasb runs about 1 count per clock cycle on modern CPUs, i.e. typically about 16x worse than a simple SSE2 pcmpeqb/pmovmskb/test+jnz loop. And with clever optimization you can go even faster, up to 2 vectors per clock saturating the load ports.

(e.g. see glibc's memchr for ORing pcmpeqb results for a whole cache line together to feed one pmovmskb, IIRC. Then go back and sort out where the actual hit was.)

repne scasb also has startup overhead, but microcode branching is different from regular branching: it's not branch-predicted on Intel CPUs. So this can't mispredict, but is total garbage for performance with anything but very small buffers.

SSE2 is baseline for x86-64 and efficient unaligned loads + pmovmskb make it a no-brainer for memchr where you can check for length >= 16 to avoid crossing into an unmapped page.

Fast strlen:

• Why is this code 6.5x slower with optimizations enabled? shows a simple not-unrolled strlen for 16-byte-aligned inputs using SSE2.
• Why does glibc's strlen need to be so complicated to run quickly? links to some more stuff about hand-optimized asm strlen functions in glibc. (And how to make a bithack strlen in GNU C avoid strict-aliasing UB.)
• https://codereview.stackexchange.com/a/213558 scalar bithack strlen, including the same 4-byte-at-a-time bithack that the glibc question was about. Better than byte-at-a-time but pointless with SSE2 (which x86-64 guarantees). However, @CodyGray's tutorial-style answer may be a useful for beginners. Note that it doesn't take into account Is it safe to read past the end of a buffer within the same page on x86 and x64?