fuzzy | handy little library for writing expressive | Testing library

Here's a simple approach:

1. Obtain binary image. Load image, convert to grayscale, Gaussian blur, then Otsu's threshold.

2. Merge small contours into a single large contour. We dilate using cv2.dilate to merge circles into a single contour.

3. Find external contours. Finally we find external contours with the external cv2.RETR_EXTERNAL flag and cv2.drawContours()

Visualization of the image pipeline

Input image

Threshold for binary image

Dilate

Detected contours in green

Code

There is no MUST but a SHOULD relation for comparisons being consistent with the commonly understood relations. Python expressively does not enforce this and float is an inbuilt type with different behaviour due to float("nan").

Expressions: Value comparisons

[…]
User-defined classes that customize their comparison behavior should follow some consistency rules, if possible:

• […]
• Comparison should be symmetric. In other words, the following expressions should have the same result:
  • x == y and y == x
  • x != y and y != x
  • x < y and y > x
  • x <= y and y >= x
• Comparison should be transitive. The following (non-exhaustive) examples illustrate that:
  • x > y and y > z implies x > z
  • x < y and y <= z implies x < z

Python does not enforce these consistency rules. In fact, the not-a-number values are an example for not following these rules.

Still, keep in mind that exceptions are incredibly rare and subject to being ignored: most people would treat float as having total order, for example. Using uncommon comparison relations can seriously increase maintenance effort.

Canonical ways to model "fuzzy matching" via operators are as subset, subsequence or containment using unsymmetric operators.

• The set and frozenset support >, >= and so on to indicate that one set encompases all values of another.

Maybe agrep is what the question is asking for.

You could try this:

The following works with the posted data examples but it uses two joins and is probably ineffective for larger data sets.

Not just XML preview, the Jetpack Compose preview is also very blurry on my Macbook. I tried the old trick Help > Edit custom properties and add:

hidpi=false

But this did not work (maybe it will work on Windows, you can try). Currently I have to revert back to Android Studio 4.2.2 .

You need to add "compat" like this

Consider below approach

There are a couple of things you can do to improve your computation:

Improve parallelism

As Nithish mentioned in the comments, you don't have enough partitions in your input data frames to make use of all your CPU cores. You're not using all your CPU capability and this will slow you down.

To increase your parallelism, repartition dfc to at least your number of cores:

dfc = dfc.repartition(dfc.sql_ctx.sparkContext.defaultParallelism)

You need to do this because your crossJoin is run as a BroadcastNestedLoopJoin which doesn't reshuffle your large input dataframe.

Separate your computation stages

A Spark dataframe/RDD is conceptually just a directed action graph (DAG) of operations to run on your input data but it does not hold data. One consequence of this behavior is that, by default, you'll rerun your computations as many times as you reuse your dataframe.

In your fuzzy_match_approve function, you run 2 separate filters on your df, this means you rerun the whole cross-join operations twice. You really don't want this !

One easy way to avoid this is to use cache() on your fuzzy_match result which should be fairly small given your inputs and matching criteria.