hyperloglog | Beta bias correction and TailCut space reduction

I'm using the HyperLogLog (hll) field to represent unique users, using the Django django-pg-hll package. What I'd like to do is get a cumulative total of unique users over a specific time period, but I'm having trouble doing this.

Given a model like:

Context: we have telemetry system for our service and would like to track retention, how many users use various features, etc.

There are two options to deal with user identifiable information and be GDPR compliant:

1. Support deleting user information based on request
2. Keep data for less than 30 days

Option #1 is hard to implement (for telemetry system). Option #2 doesn't allow answering questions such as "what is 6-month retention for feature X?".

One idea how to get answers for above question is to calculate HyperLogLog blobs per feature every week/day and store them separately forever. This will allow moving forward to merge/dcount/calculate retention based on these blobs.

Assuming that any user identifiable information is gone after 30 days (after user account gets deleted), will HyperLogLog blobs still allow to track users or not (i.e. to answer whether a particular user used feature X two years ago)?

If it allows then it is not compliant (doesn't mean that it is compliant if it doesn't allow).

I am trying to use redis in my nodejs project. I see that to build redis you need make command and gcc. I have instaled cygwin on my windows machine and then installed both make and gcc.

I downloaded redis from here https://redis.io/download and as per the instructions -

To calculate quantiles, I use approxQuantile method accessible from the stat() function in any Dataset or Dataframe of Spark. The way it approximate them is explained by in this post.

I want to implement URL filtering for the distributed crawling system on top of Redis database (e.g. don't visit the same URL twice, so I need somehow to keep tracking all of them with the minimal memory fingerprint, there is no need to store full URLs, just check if some particular URL has been visited or not). Bloom filters sounds right in this case, and I saw a native module for Redis implementing the Bloom filters. But it also has the built-in HyperLogLog data structure, so I'm wondering which one is a better choice in my scenario.

I want to implement a special counter: all increment operations time out after a fixed period of time (say 30 days).

An example:

• Day 0: counter = 0. TTL = 30 days
• Day 1: increment counter (+1)
• Day 2: increment counter (+1)
• Day 3: value of counter == 2
• Day 31: value of counter == 1
• Day 32: value of counter == 0

A naïve implementation is to maintain a set of timestamps, where each timestamp equals the time of an increment. The value of the counter equals the size of the set after subtracting all timestamps that have timed out.

This naïve counter has O(n) space (size of the set), has O(n) lookup and O(1) inserts. The values are exact.

Trade speed and memory for accuracy.

I want a counter with O(1) lookup and insert, O(1) space. The accuracy < exact.

Alternatively, I would accept O(log n) space and lookup.

The counter representation should be suited for storage in a database field, i.e., I should be able to update and poll the counter rapidly without too much (de)serialization overhead.

I'm essentially looking for a counter that resembles a HyperLogLog counter, but for a different type of approximate count: decaying increments vs. number of distinct elements

How could I implement such a counter?

I need to keep a 28 day history for some dashboard data. Essentially I have an event/action that is recorded through our BI system. I want to count the number of events and the distinct users who do that event for the past 1 day, 7 days and 28 days. I also use grouping sets (cube) to get the fully segmented data by country/browser/platform etc.

The old way was to do this keeping a 28 day history per user, for all segments. So if a user accessed the site from mobile and desktop every day for all 28 days they would have 54 rows in the DB. This ends up being a large table and is time consuming even to calculate approx_distinct and not distinct. But the issue is that I also wish to calculate approx_percentiles.

So I started investigating the user of HyperLogLog https://prestodb.io/docs/current/functions/hyperloglog.html
This works great, its much more efficient storing the sketches daily rather than the entire list of unique users per day. As I am using approx_distinct the values are close enough and it works.

I then noticed a similar function for medians. Qdigest. https://prestodb.io/docs/current/functions/qdigest.html Unfortunately the documentation is not nearly as good on this page as it is on previous pages, so it took me a while to figure it out. This works great for calculating daily medians. But it does not work if I want to calculate the median actions per user over the longer time period. The examples in HyperLogLog demonstrate how to calculate approx_distinct users over a time period but the Qdigest docs do not give such an example.

The results that I get when I try something to the HLL example for date ranges with Qdigest I get results similar to 1 day results.

I got a List of integers and I want to count the number of times each integer appears in the list.

For example: [0,5,0,1,3,3,1,1,1] gives (0 -> 2), (1 -> 4), (3 -> 2), (5 -> 1). I only need the count, not the value (the goal is to have an histogram of the counts).

A common approach would be to group by value then count the cardinality of each set. In SQL: SELECT count(*) FROM myTable GROUPBY theColumnContainingIntegers.

Is there a faster way to do this? A heuristic or a probabilistic approach is fine since I am computing a large data set and sacrifying precision for speed is fine.

Something similar to HyperLogLog algorithm (used to count the number of distinct elements in a data set) would be great, but I did not find anything like this...

I am learning HyperLogLogs examples with redis-cli

The redis-cli examples show how you can use HyperLogLog commands to record and count unique user visits to a website.
The command PFADD adds one or many strings to a HyperLogLog. PFADD returns 1 if the cardinality was changed and 0 if it remains the same:

Nonetheless, It report error when I follow the instructions: