rayon | Rayon : A data parallelism library for Rust | Reactive Programming library

The closure's signature is dictated by the function signature of ThreadPool::install:

I was under the impression that only something that is sent via the channels must implement Send + Sync and I'm probably wrong here.

You are slightly wrong:

• A type is Send if its values can be sent across threads. Many types are Send, String is, for example.
• A type is Sync if references to its values can be accessed from multiple threads without incurring any data-race. Perhaps surprisingly this means that String is Sync -- by virtue of being immutable when shared -- and in general T is Sync for any &T that is Send.

Note that those rules do not care how values are sent or shared across threads, only that they are.

This is important here because the closure you use to start a thread is itself sent across threads: it is created on the "launcher" thread, and executed on the "launched" thread.

As a result, this closure must be Send, and this means that anything it captures must in turn be Send.

Why doesn't Rc implement Send?

Because its reference count is non-atomic.

That is, if you have:

• Thread A: one Rc.
• Thread B: one Rc (same pointee).

And then Thread A drops its handle at the same time Thread B creates a clone, you'd expect the count to be 2 (still) but due to non-atomic accesses it could be only 1 despite 2 handles still existing:

• Thread A reads count (2).
• Thread B reads count (2).
• Thread B writes incremented count (3).
• Thread A writes decremented count (1).

And then, the next time B drops a handle, the item is destructed and the memory released and any further access via the last handle will blow up in your face.

I've tried to synchronize the access (though it looks like useless as there is no shared variable in threads), but no luck:

You can't wrap a type to make it Send, it doesn't help because it doesn't change the fundamental properties.

The above race condition on Rc could happen even with a Rc wrapped in Arc<...>>.

And therefore !Send is contagious and "infects" any containing type.

Migrating to Arc is highly undesired due to performance issue and lot's of related code to be migrated to Arc too.

Arc itself has relatively little performance overhead, so it seems unlikely it would matter unless you keep cloning those Checker, which you could probably improve on -- passing references, instead of clones.

The higher overhead here will come from Mutex (or RwLock) if Checker is not Sync. As I mentioned, any immutable value is trivially Sync, so if you can refactor the internal state of Checker to be Sync, then you can avoid the Mutex entirely and just have Checker contain Arc.

If you have mutable state at the moment, consider extracting it, going towards:

Why indeed... Digging into it, this is due to the way rayon determines if a Range implements IntoParallelIterator.

Rust tries hard to prevent you from doing precisely what you want to do: retain access to the whole collection while modifying it. If you're unwilling to adjust the layout of your data to accommodate the borrow checker, you can use interior mutability to make Child::calculate_distances take &self rather than &mut self. Then your problem goes away because it's perfectly fine to hand out multiple shared references to self.children.

Ideally you'd use a RefCell because you don't access the same Child from multiple threads. But Rust doesn't allow that because, based on the signatures of the functions involved, you could do so, which would be a data race. Declaring distances: RefCell> makes Child no longer Sync, removing access to Vec::par_iter().

What you can do is use a Mutex. Although it feels initially wasteful, have in mind that each call to Child::calculate_distances() receives a different Child, so the mutex will always be uncontended and therefore cheap to lock (not involve a system call). And you'd lock it only once per Child::calculate_distances(), not on every access to the array. The code would look like this (playground):

remove this if statement:

Thanks @Jmb for the discussion, I got rid of the threads and I spawn a tokio task as follows:

create a vector to hold or the futures so we could wait for them:

I can't seem to find any parallel permutation implementations, so your best bet is probably to use ParallelBridge to convert the iterator into a parallel one. Note that, unless your processing is intensive, you shouldn't use this and just use regular iterator methods instead because there's an added synchronization cost.

I assume the order doesn't matter, therefore you don't need to have an order of your output data.

You could use a mpsc::channel to transfer your data from the for_each closure to your database api, e.g.