fuzzywuzzy | Fuzzy String Matching in Python | Search Engine library

by seatgeek Python Version: 0.18.0 License: GPL-2.0

kandi X-RAY | fuzzywuzzy Summary

fuzzywuzzy is a Python library typically used in Database, Search Engine applications. fuzzywuzzy has no bugs, it has no vulnerabilities, it has build file available, it has a Strong Copyleft License and it has high support. You can install using 'pip install fuzzywuzzy' or download it from GitHub, PyPI.

Fuzzy String Matching in Python

Support

Quality

Security

License

Reuse

kandi-support Support

fuzzywuzzy has a highly active ecosystem.
It has 8822 star(s) with 886 fork(s). There are 271 watchers for this library.
It had no major release in the last 12 months.
There are 81 open issues and 103 have been closed. On average issues are closed in 124 days. There are 26 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of fuzzywuzzy is 0.18.0

fuzzywuzzy Support

Best in #Search Engine

Average in #Search Engine

fuzzywuzzy Support

Best in #Search Engine

Average in #Search Engine

quality kandi Quality

fuzzywuzzy has 0 bugs and 0 code smells.

fuzzywuzzy Quality

Best in #Search Engine

Average in #Search Engine

fuzzywuzzy Quality

Best in #Search Engine

Average in #Search Engine

security Security

fuzzywuzzy has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
fuzzywuzzy code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

fuzzywuzzy Security

Best in #Search Engine

Average in #Search Engine

fuzzywuzzy Security

Best in #Search Engine

Average in #Search Engine

license License

fuzzywuzzy is licensed under the GPL-2.0 License. This license is Strong Copyleft.
Strong Copyleft licenses enforce sharing, and you can use them when creating open source projects.

fuzzywuzzy License

Best in #Search Engine

Average in #Search Engine

fuzzywuzzy License

Best in #Search Engine

Average in #Search Engine

build Reuse

fuzzywuzzy releases are available to install and integrate.
Deployable package is available in PyPI.
Build file is available. You can build the component from source.

fuzzywuzzy Reuse

Best in #Search Engine

Average in #Search Engine

fuzzywuzzy Reuse

Best in #Search Engine

Average in #Search Engine

Top functions reviewed by kandi - BETA

kandi has reviewed fuzzywuzzy and discovered the below as its top functions. This is intended to give you an instant insight into fuzzywuzzy implemented functionality, and help decide if they suit your requirements.

UWRatio between two strings

Get opcodes
Get all matching blocks
Return the similarity between two strings

Uratio ratio

Return the similarity between two sequences
Return the ratio between two strings
Compare two strings

Removes duplicates from a list

Extract elements from a query
Extract a single item from a query

Print the result from a timeit

Extract the best matches from the query

Extracts the best match from choices

Generate a quick ratio

Get all kandi verified functions for this library.

fuzzywuzzy Key Features

Fuzzy String Matching in Python

fuzzywuzzy Examples and Code Snippets

See all related Code Snippets

Community Discussions

Trending Discussions on fuzzywuzzy

check if the string equals the first letters of a list of words
The airflow scheduler stops working after updating pypi packages on google cloud composer 2.0.1
Pipreqs: SyntaxError: invalid non-printable character U+FEFF
Fuzzy matching for groups in pandas
How to set a column value by fuzzy string matching with another dataframe?
Setting a Threshold for fuzzywuzzy process.extractOne
How to replace using for() with all() in a pandas dataframe?
how token sort ratio works?
Optimize the traversal of a column of a dataframe
fuzzywuzzy returning single characters, not strings

Trending Discussions on fuzzywuzzy

QUESTION

check if the string equals the first letters of a list of words

Asked 2022-Mar-28 at 14:59

I am confused about a simple task

the user will give me a string and my program will check if this string equals the first letters of a list of words ( like this example)

>>> html_attr = ["onerror","onload"]
>>> example_task(html_attr,"on")
["onerror","onload"]
>>> example_task(html_attr,"one")
["onerror"]

should I use fuzzywuzzy here or what ?

thanks

ANSWER

Answered 2022-Mar-28 at 14:59

No need for some weird libraries, Python has a nice builtin str function called startswith that does just that.

def example_task(words, beginning):
    return [w for w in words if w.startswith(beginning)]

Fuzzywuzzy would come in handy if you don't want an exact match.

Source https://stackoverflow.com/questions/71649455

QUESTION

Fuzzy matching for groups in pandas

Asked 2022-Mar-21 at 07:59

I have the following dataset:

import pandas as pd
from fuzzywuzzy import fuzz 

adf = pd.DataFrame({'my_addresses':['234 Rue Morgue','234 R. Morgue','234 La rue morgue, 234','312 La rue moulin','24 Rue Marbeau','24 La Rue Marbeau', '28 Boulevard Suchet','28 Blvd Suchet'],'my_group':['A1','A1','A1','A1','B2','B2','B2','B2']})
adf

    my_addresses            my_group
0   234 Rue Morgue          A1
1   234 R. Morgue           A1
2   234 La rue morgue, 234  A1
3   312 La rue moulin       A1
4   24 Rue Marbeau          B2
5   24 La Rue Marbeau       B2
6   28 Boulevard Suchet     B2
7   28 Blvd Suchet          B2

And I want to apply a custom function. This function creates subgroups given a pandas object, using fuzzy match.

def match_groups(grp_obj, threshold = 67):
    groups = []
    for i in range(len(grp_obj.to_list())):
        groups.append(i+1)
    for i, val_i in enumerate(grp_obj.to_list()):
        for j, val_j in enumerate(grp_obj.to_list()):
            if j>i:
                ratio = fuzz.ratio(val_i, val_j)
                if ratio >= threshold:
                    groups[j] = groups[i]
    return groups

For example, let's calculate subgroups for the group B2:

match_groups(adf[adf['my_group']=='B2'].my_addresses, 67)

It returns this list:

[1, 1, 3, 3]

Ok. Now I want to apply this function to each group. This is the expected result:

    my_addresses            my_group    my_subgroup
0   234 Rue Morgue          A1          1
1   234 R. Morgue           A1          1
2   234 La rue morgue, 234  A1          1
3   312 La rue moulin       A1          4
4   24 Rue Marbeau          B2          1
5   24 La Rue Marbeau       B2          1
6   28 Boulevard Suchet     B2          3
7   28 Blvd Suchet          B2          3

So I tried this:

adf['my_subgroup'] = adf.groupby('my_group')['my_addresses'].apply(match_groups)

But it returns me:


    my_addresses            my_group    my_subgroup
0   234 Rue Morgue          A1          NaN
1   234 R. Morgue           A1          NaN
2   234 La rue morgue, 234  A1          NaN
3   312 La rue moulin       A1          NaN
4   24 Rue Marbeau          B2          NaN
5   24 La Rue Marbeau       B2          NaN
6   28 Boulevard Suchet     B2          NaN
7   28 Blvd Suchet          B2          NaN

I think my problem is that the function returns a list, but I'm not sure how to pass this to the data frame as the column required. Please, any help will be appreciated.

ANSWER

Answered 2022-Mar-21 at 07:59

One way might be to create a parallel DataFrame, then join. Here are a couple of variations on that approach. There may well be a better way.

Here's a slightly modified match_groups function, so that it takes a Series rather than a DataFrame:

def match_groups(addresses, threshold):
    subgroups = [i for i in range(1, len(addresses)+1)]
    for i, val_i in enumerate(addresses):
        for j, val_j in enumerate(addresses):
            if j>i:
                ratio = fuzz.ratio(val_i, val_j)
                if ratio >= threshold:
                    subgroups[j] = subgroups[i]
    return subgroups

This still returns a list.

By the way, computing subgroups this way can give different results depending on the order it sees the input, so be careful.

Using concat and join

We make one DataFrame for each group, computing our subgroups, and preserving the index values from the original DataFrame, adf, then paste them back together with concat:

df_subgroups = pd.concat(
    pd.DataFrame({
        "my_subgroup": match_groups(
            adf.my_addresses[adf.my_group==grp],
            threshold=67)},
        index=adf.index[adf.my_group==grp])
    for grp in adf.my_group.unique()
)

The result looks like this:

   my_subgroup
0            1
1            1
2            1
3            4
4            1
5            1
6            3
7            3

Now, join on the shared index:

adf.join(df_subgroups)

             my_addresses my_group  my_subgroup
0          234 Rue Morgue       A1            1
1           234 R. Morgue       A1            1
2  234 La rue morgue, 234       A1            1
3       312 La rue moulin       A1            4
4          24 Rue Marbeau       B2            1
5       24 La Rue Marbeau       B2            1
6     28 Boulevard Suchet       B2            3
7          28 Blvd Suchet       B2            3

Using groupby, transform, and join

We can do a little better with groupby and transform. Transform takes care of the concat-ing and and index preserving step above.

df_subgroups = adf.groupby('my_group').transform(match_groups, threshold=67)
df_subgroups.columns = ["my_subgroup"]

We get the same result as before - a DataFrame with the original index preserved so we can join. Annoyingly, it names the new column "my_addresses", so we give it the name we want.

   my_subgroup
0            1
1            1
2            1
3            4
4            1
5            1
6            3
7            3

Then, apply the same join as before:

adf.join(df_subgroups)

             my_addresses my_group  my_subgroup
0          234 Rue Morgue       A1            1
1           234 R. Morgue       A1            1
2  234 La rue morgue, 234       A1            1
3       312 La rue moulin       A1            4
4          24 Rue Marbeau       B2            1
5       24 La Rue Marbeau       B2            1
6     28 Boulevard Suchet       B2            3
7          28 Blvd Suchet       B2            3

Source https://stackoverflow.com/questions/71552594

QUESTION

How to set a column value by fuzzy string matching with another dataframe?

Asked 2022-Mar-02 at 14:16

I have referred to this post but cannot get it to run for my particular case. I have two dataframes:

import pandas as pd

df1 = pd.DataFrame(
    {
        "ein": {0: 1001, 1: 1500, 2: 3000},
        "ein_name": {0: "H for Humanity", 1: "Labor Union", 2: "Something something"},
        "lname": {0: "Cooper", 1: "Cruise", 2: "Pitt"},
        "fname": {0: "Bradley", 1: "Thomas", 2: "Brad"},
    }
)

df2 = pd.DataFrame(
    {
        "lname": {0: "Couper", 1: "Cruise", 2: "Pit"},
        "fname": {0: "Brad", 1: "Tom", 2: "Brad"},
        "score": {0: 3, 1: 3.5, 2: 4},
    }
)

Then I do:

from fuzzywuzzy import fuzz
from fuzzywuzzy import process
from itertools import product

N = 60
names = {
    tup: fuzz.ratio(*tup)
    for tup in product(df1["lname"].tolist(), df2["lname"].tolist())
}

s1 = pd.Series(names)
s1 = s1[s1 > N]
s1 = s1[s1.groupby(level=0).idxmax()]

degrees = {
    tup: fuzz.ratio(*tup)
    for tup in product(df1["fname"].tolist(), df2["fname"].tolist())
}

s2 = pd.Series(degrees)
s2 = s2[s2 > N]
s2 = s2[s2.groupby(level=0).idxmax()]

df2["lname"] = df2["lname"].map(s1).fillna(df2["lname"])
df2["fname"] = df2["fname"].map(s2).fillna(df2["fname"])
df = df1.merge(df2, on=["lname", "fname"], how="outer")

The result is not what I expect. Can you help me with editing this code please? Note that I have millions of lines in df1 and millions in df2, so I need some efficiency as well.

Basically, I need to match people from df1 to people in df2. In the above example, I am matching them on last name (lname) and first name (fname). I also have a third one, which I leave out here for parsimony.

The expected result should look like:

ein ein_name    lname   fname   score
0   1001    H for Humanity  Cooper  Bradley 3
1   1500    Labor Union Cruise  Thomas  3.5
2   3000    Something something Pitt    Brad    4

ANSWER

Answered 2021-Dec-26 at 17:50

You could try this:

from functools import cache

import pandas as pd
from fuzzywuzzy import fuzz

# First, define indices and values to check for matches
indices_and_values = [(i, value) for i, value in enumerate(df2["lname"] + df2["fname"])]

# Define helper functions to find matching rows and get corresponding score
@cache
def find_match(x):
    return [i for i, value in indices_and_values if fuzz.ratio(x, value) > 75]

def get_score(x):
    try:
        return df2.loc[x[0], "score"]
    except (KeyError, IndexError):
        return pd.NA

# Add scores to df1:
df1["score"] = (
    (df1["lname"] + df1["fname"])
    .apply(find_match)
    .apply(get_score)
)

And then:

print(df1)

    ein             ein_name   lname    fname  score
0  1001       H for Humanity  Cooper  Bradley    3.0
1  1500          Labor Union  Cruise   Thomas    3.5
2  3000  Something something    Pitt     Brad    4.0

Given the size of your dataframes, I suppose you have namesakes (identical first and last names), hence the use of @cache decorator from Python standard library in order to try speeding things up (but you can do without it).

Source https://stackoverflow.com/questions/70472121

QUESTION

How to replace using for() with all() in a pandas dataframe?

Asked 2022-Feb-21 at 13:36

I have a university activity that makes the following dataframe available:

      import pandas as pd
      from fuzzywuzzy import fuzz
      from fuzzywuzzy import process

      df1 = pd.DataFrame({'id': [0,1,2,3],
                          'name_city': ['RIO DE JANEIRO', 'SAO PAULO', 
                                        'ITU', 'CURITIBA'],
                          'city':['RIO JANEIRO', 'SAOO PAULO', 
                                  'FLORIANOPOLIS', 'BELO HORIZONTE']})

      print(df1)

            id  name_city             city
            0   RIO DE JANEIRO     RIO JANEIRO
            1   SAO PAULO          SAOO PAULO
            2   ITU                FLORIANOPOLIS
            3   CURITIBA           BELO HORIZONTE

I need to assess what is the similarity between the names of cities. So I'm using the fuzzy library and I made the following code:

      for i in range(0, len(df1)):
         df1['value_fuzzy'].iloc[i] = fuzz.ratio(df1['name_city'].iloc[i], df1['city'].iloc[i]

      This code works perfectly, the output is as desired:

      print(df1)

           id   name_city           city               value_fuzzy
           0    RIO DE JANEIRO     RIO JANEIRO             88
           1    SAO PAULO          SAOO PAULO              95
           2    ITU                FLORIANOPOLIS           12
           3    CURITIBA           BELO HORIZONTE          27

However, I would like to replace the for. So I tried to use all() which I found this example: Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

        df1['value_fuzzy'] = fuzz.ratio(df1['name_city'].all(), df1['city'].all())

But this code with all() returns wrong output:

       id   name_city        city          value_fuzzy
       0    RIO DE JANEIRO  RIO JANEIRO     27
       1    SAO PAULO       SAOO PAULO      27
       2    ITU             FLORIANOPOLIS   27
       3    CURITIBA        BELO HORIZONTE  27

Is there another way to generate the desired output without using for()?

ANSWER

Answered 2022-Feb-21 at 12:43

You can't use fuzz.ratio this way directly, the function is not vectorial. You need to pass it to apply:

df1['value_fuzzy'] = df1.apply(lambda r: fuzz.ratio(r['name_city'], r['city']), axis=1)

NB. Note that whatever you do, unless the function is specifically rewritten to handle vectorial input, that won't improve efficiency

output:

   id       name_city            city  value_fuzzy
0   0  RIO DE JANEIRO     RIO JANEIRO           88
1   1       SAO PAULO      SAOO PAULO           95
2   2             ITU   FLORIANOPOLIS           12
3   3        CURITIBA  BELO HORIZONTE           27

Source https://stackoverflow.com/questions/71206431

QUESTION

how token sort ratio works?

Asked 2022-Feb-17 at 05:13

Can someone explain me how this function of the library fuzzywuzzy in Python works? I know how the Levenshtein distance works but I don't understand how the ratio is computed.

b = fuzz.token_sort_ratio('controlled', 'comparative')

The result is 38

ANSWER

Answered 2022-Feb-17 at 05:13

Levenshtein distance

As you probably already know the Levenshtein distance is the minimum amount of insertions / deletions / substitutions to convert one sequence into another sequence. It can be normalized as dist / max_dist, where max_dist is the maximum distance possible given the two sequence lengths. In the case of the Levenshtein distance this results in the normalization dist / max(len(s1), len(s2)). In addition a normalized similarity can be calculated by inverting this: 1 - normalized distance.

>>> from rapidfuzz.distance import Levenshtein
>>> Levenshtein.distance('controlled', 'comparative')
8
>>> Levenshtein.similarity('controlled', 'comparative')
3
>>> Levenshtein.normalized_distance('controlled', 'comparative')
0.7272727272727273
>>> Levenshtein.normalized_similarity('controlled', 'comparative')
0.2727272727272727

Indel distance

The Indel distance is the minimum amount of insertions / deletions to convert one sequence into another sequence. So it behaves similar to the Levenshtein distance, but does not allow substitutions. Since any substitution can be replaced by a insertion + deletion this can be calculated e.g. by modifying the cost of substitutions in the Levenshtein distance to 2. The Indel distance can be normalized in a similar way to the Levenshtein distance, but uses a different max_dist: dist / (len(s1) + len(s2)).

>>> from rapidfuzz.distance import Indel
>>> Indel.distance('controlled', 'comparative')
13
>>> Indel.similarity('controlled', 'comparative')
8
>>> Indel.normalized_distance('controlled', 'comparative')
0.6190476190476191
>>> Indel.normalized_similarity('controlled', 'comparative')
0.38095238095238093

ratio

The ratio in fuzzywuzzy/thefuzz/rapidfuzz is the normalized indel similarity scaled to 100.

>>> from rapidfuzz.distance import Indel
>>> from rapidfuzz import fuzz
>>> Indel.normalized_similarity('controlled', 'comparative') * 100
38.095238095238095
>>> fuzz.ratio('controlled', 'comparative')
38.095238095238095

The only difference in fuzzywuzzy/thefuzz is, that results are rounded:

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio('controlled', 'comparative')
38

token_sort_ratio

token_sort_ratio is a variant of ratio, which sorts the words in both sequences before comparing them:

s1 = " ".join(sorted(s1.split()))
s2 = " ".join(sorted(s2.split()))
fuzz.ratio(s1, s2)

In your example token_sort_ratio will have the same result as ratio, since both sequences are already sorted.

Source https://stackoverflow.com/questions/71146287

QUESTION

fuzzywuzzy returning single characters, not strings

Asked 2022-Jan-28 at 02:42

I'm not sure where I'm going wrong here and why my data is returning wrong. Writing this code to use fuzzywuzzy to clean bad input road names against a list of correct names, replacing the incorrect with the closest match.

It's returning all lines of data2 back. I'm looking for it to return the same, or replaced lines of data1 back to me.

My Minimal, Reproducible Example:

import pandas as pd
import os
import csv
import usaddress
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

data1 =('3176 DETRIT ROAD')
data2 =('DETROIT RD')

try:
    data1 = usaddress.tag(data1)
except usaddress.RepeatedLabelError:
    pass

roaddnum2 = data1[0].get('AddressNumber', '')
roadir2 = data1[0].get('StreetNamePreDirectional', '')
roadname2 = data1[0].get('StreetName', '')
roaddsg2 = data1[0].get('StreetNamePostType', '')

street2 = (roadir2, roadname2, roaddsg2)
street2 = " ".join(street2)
street2 = street2.strip()
data2 = data2.split(',')

if street2 not in data2:
    street2 = process.extract(street2, data2)
    print(street2[0])

My full code

import pandas as pd
import os
import csv
import usaddress
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

def convert_tolist(string):
    li = list(string.split(" "))
    return li

with open(r"Cass_Howard - Copy.csv") as csv_file,\
        open("Final_Test_Clean.csv", "w") as f,\
        open(r"TEST_no_dups12.csv") as ul:
    csv_reader = csv.reader(csv_file, delimiter=',')
    next(csv_reader)
    csv_reader = csv.reader(f, delimiter=',')
    file_1 = csv_file
    file_2 = ul

    for data1, data2 in zip(file_1, file_2):
        data1 = data1.split(',')
        data1 = data1[18]
        data1 = data1.upper()
        data2 = data2.strip()
        data2 = data2.split(',')
        data2 = ''.join(data2)
        try:
            data1 = usaddress.tag(data1)
        except usaddress.RepeatedLabelError:
            pass

        roaddnum2 = data1[0].get('AddressNumber', '')
        roadir2 = data1[0].get('StreetNamePreDirectional', '')
        roadname2 = data1[0].get('StreetName', '')
        roaddsg2 = data1[0].get('StreetNamePostType', '')

        street2 = (roadir2, roadname2, roaddsg2)
        street2 = " ".join(street2)
        street2 = street2.strip()
        data1 = list(data1)
        convert_tolist(data2)


        if street2 not in data2:
            street2 = process.extract(street2, data2)
            print(street2)

street2 query data(around 950 lines)

DETROIT ROAD
DETROIT ROAD
MANNIX ST
MANNIX ST

data2 choices data(around 200 lines)

ACRES
ADERSON RD
AIRPORT RD
ALGONQUIN

ANSWER

Answered 2022-Jan-25 at 18:21

Okay, I'm not certain I've fully understood your issue, but modifying your reprex, I have produced the following solution.

import usaddress
from fuzzywuzzy import process

data1 = "3176 DETRIT ROAD"
choices = ["DETROIT RD"]

try:
    data1 = usaddress.tag(data1)
except usaddress.RepeatedLabelError:
    pass

parts = [
    data1[0].get("StreetNamePreDirectional"),
    data1[0].get("StreetName"),
    data1[0].get("StreetNamePostType"),
]

street = " ".join([x for x in parts if x])

if street not in choices:
    street = " ".join(
        [data1[0].get("AddressNumber"), process.extract(street, choices)[0][0]]
    )

print(street)

This yields:

3176 DETROIT RD

So basically, it has replaces the street name bits with the best match from the choices list using the process.extract() function.

And here it is as a callable function:

import usaddress
from fuzzywuzzy import process


def best_street(addr: str, choices: list[str]) -> str:
    try:
        usaddr = usaddress.tag(addr)
    except usaddress.RepeatedLabelError:
        pass
    street_parts = [
        usaddr[0].get("StreetNamePreDirectional"),
        usaddr[0].get("StreetName"),
        usaddr[0].get("StreetNamePostType"),
    ]
    street = " ".join([x for x in street_parts if x])
    return " ".join(
        [usaddr[0].get("AddressNumber"), process.extract(street, choices)[0][0]]
    )


if __name__ == "__main__":
    choices = ["AIRPORT RD", "DETROIT RD"]
    print(best_street("123 Detrt", choices))
    print(best_street("9876 AIRPUMP DR", choices))

yielding:

123 DETROIT RD
9876 AIRPORT RD

Source https://stackoverflow.com/questions/70851051

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install fuzzywuzzy

You can install using 'pip install fuzzywuzzy' or download it from GitHub, PyPI.
You can use fuzzywuzzy like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: