deduplicated | Check duplicated files

For every unique combination of the first two indexes, I want all of the rows (and the index name) of their third index transformed into a json string column.

For example

I want my dataframe to return unique rows based on two logical conditions (OR not AND).

But when I ran this, df %>% group_by(sex) %>% distinct(state, education) %>% summarise(n=n()) I got deduplicated rows based on the two conditions joined by AND not OR.

Is there a way to get something like this df %>% group_by(sex) %>% distinct(state | education) %>% summarise(n=n()) so that the deduplicated rows will be joined by OR not AND?

Thank you.

I have a need to do some processing on many thousands of strings (each string being an element in a list, imported from records in a SQL table).

Each string comprises a number of phrases delimited by a consistent delimiter. I need to 1) eliminate duplicate phrases in the string; 2) sort the remaining phrases and return the deduplicated, sorted phrases as a delimited string.

This is what I've conjured:

When I upgrade IoTDB to version 0.12.3, I could not query for more than 1000 timeseries anymore. Even I modified the configuration max_deduplicated_path_num, I could still get the error: Too many paths in one query! Currently allowed max deduplicated path number is 1000. Please use slimit or adjust max_deduplicated_path_num in iotdb-engine.properties.

I have X sources that contain info about assets (hostname, IPs, MACs, os, etc.) in our environment. The sources contain anywhere from 1500 to 150k entries (at least the ones I use now). My script is supposed to query each of them, gather that data, deduplicate it by merging info about the same assets from different sources, and return unified list of all entries. My current implementation does work, but it's slow for bigger datasets. I'm curious if there is better way to accomplish what I'm trying to do.

Universal problem
Deduplication of data by merging similar entries with the caveat that merging two assets might change whether the resulting asset will be similar to the third asset that was similar to the first two before merging.
Example:
~ similarity, + merging
(before) A ~ B ~ C
(after) (A+B) ~ C or (A+B) !~ C

I tried looking for people having the same issue, I only found What is an elegant way to remove duplicate mutable objects in a list in Python?, but it didn't include merging of data which is crucial in my case.

Simplified for ease of reading and understanding with unneeded parts removed - general functionality is intact.

I've got a certain spider which inherits from SitemapSpider. As expected, the first request on startup is to sitemap.xml of my website. However, for it to work correctly I need to add a header to all the requests, including the initial ones which fetch the sitemap. I do so with DownloaderMiddleware, like this:

If using SeekToCurrentErrorHandler with stateful retry, such that the message is polled from the broker for each retry, there is a risk that for a long retry period that a consumer group rebalance could cause the partition to be re-assigned to another consumer. Hence the stateful retry period/attempts would be reset, as the new consumer has no knowledge of the state of the retry.

Taking an example, if a retry max period was 24 hours, but consumer group re-balancing was happening on average every 12 hours, the retry could never complete, and the message (and those behind it) would eventually expire from the topic once they exceeded the retention period. (Assuming the cause of the retryable exception was not resolved in this time). The message would not end up on the DLT after 24 hours as expected, as retries would not be exhausted due to the reset.

I assume that even if a consumer is retrying by re-polling messages, there is no guarantee that following a re-balance that this consumer would retain assignment to this partition. Or is it the case that we can be confident that so long as this consumer instance is alive that it would typically retain assignment to the partition it is polling?

Are there best practises/guidelines on use of stateful retry to cater for this?

Stateless retry means any total retry time that exceeds the poll timeout would cause rebalancing and duplicate message delivery. To avoid that then the retry period must be very limited. Or is the guideline to allow this, ensure messages are deduplicated by the consumer, so that the duplicate messages are acceptable and long running stateless retries can be configured?

Is the only safe and stable option for enabling a retry period of something like several hours (e.g. to cater for a service being unavailable for this period) to use retry topics?

Thanks, Rob.

I am working on outputting total deduplicated counts from a pre-aggregated frame as follows.

I currently have a data frame that displays like so. It's the initial structure and the point that I have gotten to by filtering out unneeded columns.

We can see from the example above that 101 is found in both Grape and Flower. I would like to arrange the format so that the distinct string values from the "Source" column become their own sources, as from there I can perform a groupBy for a specific arrangement of yes's and no's as so.

I agree that creating this manually via the above example is a good fit, but I am working with +100m rows and need something more susinct.

What I've extracted so far is a list of distinct Source values and arranged them into a list:

We have the NPE at the line of coe mentioned at the topic when calling for