bloom-filter | in-memory bloom filter in ruby

I've been working on my website on and off for a couple years, learned a ton about JavaScript, CSS, HTML, Bootstrap, Jekyll, Travis-CI, and Github Pages in the process. (Long list is a major factor in why it's taken so long.)

I've discovered that if I push to (or have Travis deploy to) the gh-pages branch of a repo, it actually becomes a subdomain of my website. Examples: here, here, here.

This is pretty awesome, but those sub-pages end up feeling like they're not a part of the same website, because they provide no way to get back to the main page. I'd like them to include my navbar.

Is there an elegant way to do this?

I have a strange issue while querying from Presto (AWS EMR). I was using Presto 0.194 and everything was ok, after I upgraded to 0.224, I cannot run my queries. I'm using LDAP authentication for presto and also file base authorization for Hive using a authorization.json file. I'm using the same json file which was working fine in the old version. Any help would highly appreciated.

Error: Query 20191005_104119_00006_3snge failed: Access Denied: View owner 'username' cannot create view that selects from ...

config.propertis:

How to resolve an error that I'm getting trying to assign values to memory allocated by glMapBufferRange This is a problematic function, due to not having casts in python. I am trying to assign values to this memory, though am experiencing the error stated in the title. I have also tried to create a np.array with a dimension of the size material but to no avail. Perhaps just not doing it correctly.

Update and SUCCESS! With the amazing help of Rabbid76 and the fixes first to allocate and be able to assign to glMapBufferRange memory, and then to the sbmloader.py the program renders successfully. Thank you.

Final results also found on my github PythonOpenGLSuperBible7Glut

support files: hdrbloom_support.zip

expected output rendering is what the actual results are rendering:

source code:

I'm considering redis for my next project (in-memory, fast) but now I have the problem of figuring out how and if at all it could actually achieve my goal. The goal is to store "large" (millions) amount of fixed-length bit strings and then searching over the database with a input (query) bit string. Search means to return everything which fulfills below condition:

query & value = query

eg. if all bits set in the query are also set in the value return that key eg. bloom-filter albeit in my domain of work it isn't usually called like that.

I found the module RedisBloom but I already have my bloom filter (bit strings) available from external program and would simply like to use RedisBloom for storage of them and searching (exists command). therefore in my case the "Add" command should take the input as is and not hash it again.

Is that possible? And if not other suggestions?

We have a table with 100M rows in google cloud datastore. What is the most efficient way to look up the existence of a large number of keys (500K-1M)?

For context, a use case could be that we have a big content datastore (think of all webpages in a domain). This datastore contains pre-crawled content and metadata for each document. Each document, however, could be liked by many users. Now when we have a new user and he/she says he/she likes document {a1, a2, ..., an}, we want to tell if all these document ak {k in 1 to n} are already crawled. That's the reason we want to do the lookup mentioned above. If there is a subset of documents that we don't have yet, we would start to crawl them immediately. Yes, the ultimate goal is to retrieve all these document content and use them to build the user profile.

My current thought is to issue a bunch of batch lookup requests. Each lookup request can contain up to 1K of keys [1]. However to get the existence of every key in a set of 1M, I still need to issue 1000 requests.

An alternative is to use a customized middle layer to provide a quick look up (for example, can use bloom filter or something similar) to save the time between multiple requests. Assuming we never delete keys, every time we insert a key, we add it through the middle layer. The bloom-filter keeps track of what keys we have (with a tolerable false positive rate). Since this is a custom layer, we could provide a micro-service without a limit. Say we could respond to a request asking for the existence of 1M keys. However, this definitely increases our design/implementation complexity.

Is there any more efficient ways to do that? Maybe a better design? Thanks!

[1] https://cloud.google.com/datastore/docs/concepts/limits

I'm trying to replicate a web design trick known as "gooey effect" (see it live here). It's a technique applying SVG filters on moving ellipses in order to get a blob-like motion. The process is rather simple:

• apply a gaussian blur
• increase the contrast of the alpha channel only

The combination of the two creates a blob effect

The last step (increasing the alpha channel contrast) is usually done through a "color matrix filter".

A color matrix is composed of 5 columns (RGBA + offset) and 4 rows.

The values in the first four columns are multiplied with the source red, green, blue, and alpha values respectively. The fifth column value is added (offset).

In CSS, increasing the alpha channel contrast is as simple as calling a SVG filter and specifying the contrast value (here 18):

In Processing though, it seems to be a bit more complicated. I believe (I may be wrong) the only way to apply a color matrix filter is to create one in a shader. After a few tries I came up with these (very basic) vertex and fragment shaders for color rendering:

colorvert.glsl

Question

In Clojure, it is possible to check if a variable represents a function, using ifn?.

What I'm asking myself is, if there is a way to check if a var is a function which has certain post (or pre) conditions.

Bonus: is it possible to construct the post-condition to be checked parametrically?

Example

Let's say that foo represents a function (i.e. (ifn? foo) returns true). How can I check that foo has a certain post condition, e.g. that it returns numbers between 0 and 10. In other words, how can I check that foo has the following post-condition?

{:post (and (number? %) (<= 0 %) (<= % 10))

Bonus: is it possible to check for N, instead of 10, where N is a parameter I can choose?

{:post (and (number? %) (<= 0 %) (<= % 10))

Background

I'm implementing Bloom-filter data structure, and it would be really nice if I could validate the hash-functions. (I.e., that they are functions returning only numbers between 0 and the number of bits in the Bloom-filter.)

Remarks

I'm already aware of some workarounds, e.g. like wrapping the hash-functions in other functions that contain the post-conditions, or taking their modulo of the number of bits. While I appreciate any further workarounds for the above use case, please, include an explicit answer to the main question (i.e. "is it possible to check if a function has a certain post-condition?") too. "No, it is not possible" is also an answer I can accept of course.

I have been trying to implement my own (simple) Bloom Filter but am stuck on hashing, I understand the concept of hashing the item multiple times and populating the bit array with the indices.

However, I am seeing a ton of collisions in my hashing, I am using 1 hash algorithm (I have tried FNV, murmurhash and now farmhash) with various seeds (based on current nanoseconds).

I must be doing something wrong, I am calculating the k functions by following the information here and setting the same amount of seeds.

Any help would be great, thanks.