pybktree | Python BK-tree data structure | Dataset library

 by   benhoyt Python Version: v1.1 License: MIT

kandi X-RAY | pybktree Summary

kandi X-RAY | pybktree Summary

pybktree is a Python library typically used in Artificial Intelligence, Dataset, Example Codes applications. pybktree has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub.

Python BK-tree data structure to allow fast querying of "close" matches
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              pybktree has a low active ecosystem.
              It has 118 star(s) with 16 fork(s). There are 4 watchers for this library.
              OutlinedDot
              It had no major release in the last 12 months.
              There are 0 open issues and 5 have been closed. On average issues are closed in 3 days. There are no pull requests.
              It has a neutral sentiment in the developer community.
              The latest version of pybktree is v1.1

            kandi-Quality Quality

              pybktree has 0 bugs and 0 code smells.

            kandi-Security Security

              pybktree has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              pybktree code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              pybktree is licensed under the MIT License. This license is Permissive.
              Permissive licenses have the least restrictions, and you can use them in most projects.

            kandi-Reuse Reuse

              pybktree releases are available to install and integrate.
              Build file is available. You can build the component from source.
              It has 109 lines of code, 6 functions and 2 files.
              It has high code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi has reviewed pybktree and discovered the below as its top functions. This is intended to give you an instant insight into pybktree implemented functionality, and help decide if they suit your requirements.
            • Returns a list of n items starting from the tree .
            • Adds an item to the tree .
            • Initialize the tree .
            • Return an iterator over the tree .
            • Return a pretty representation of the tree .
            • Compute hamming distance between x and y .
            Get all kandi verified functions for this library.

            pybktree Key Features

            No Key Features are available at this moment for pybktree.

            pybktree Examples and Code Snippets

            No Code Snippets are available at this moment for pybktree.

            Community Discussions

            QUESTION

            How to implement a fast fuzzy-search engine using BK-tree when the corpus has 10 billion unique DNA sequences?
            Asked 2021-May-05 at 05:52

            I am trying to use the BK-tree data structure in python to store a corpus with ~10 billion entries (1e10) in order to implement a fast fuzzy search engine.

            Once I add over ~10 million (1e7) values to a single BK-tree, I start to see a significant degradation in the performance of querying.

            I was thinking to store the corpus into a forest of a thousand BK-trees and to query them in parallel.

            Does this idea sound feasible? Should I create and query 1,000 BK-trees simultaneously? What else can I do in order to use BK-tree for this corpus.

            I use pybktree.py and my queries are intended to find all entries within an edit distance d.

            Is there some architecture or database which will allow me to store those trees?

            Note: I don’t run out of memory, rather the tree begins to be inefficient (presumably each node has too many children).

            ...

            ANSWER

            Answered 2021-Jan-18 at 12:18
            Few thoughts

            BK-trees
            Kudos to Ben Hoyt and his link to the issue which I will draw from. That being said, the first observation from the mentioned issue is that the BK tree isn't exactly logarithmic. From what you told us your usual d is ~6, which is 3/10 of your string length. Unfortunately, that means that if we look at the tables from the issue you will get the complexity of somewhere between O(N^0.8) to O(N). In the optimistic case of the exponent being 0.8(it will likely be slightly worse) you get an improvement factor of ~100 on your 10B entries. So if you have a reasonably fast implementation of BK-trees it can still be worth it to use them or use them as a basis for a further optimization.

            The downside of this is that even if you use 1000 trees in parallel, you will only get the improvement from the parallelization as the perfomance of the trees depends on the d rather than on the amount of the nodes within the tree. However even if you run all the 1000 trees at once with a massive machine, we are at the ~10M nodes/tree which you reported as slow. Still, computation wise, this seems doable.

            A brute force approach
            If you don't mind paying a little I would look into something like Google cloud big query if that doesn't clash with some kind of data confidentiality. They will brute force the solution for you - for a fee. The current rate is $5/TB of a query. Your dataset is ~10B rows * 20chars. Taking one byte per char, one query would take 200GB so ~1$ per query if you went the lazy way.
            However, since the charge is per byte of a data in a column and not per complexity of a question, you could improve on this by storing your strings as bits - 2bits per a letter, this would save you 75% of the expenses.
            Improving further, you can write your query in such a way that it will ask for a dozen strings at once. You might need to be a bit careful to use a batch of similar strings for the purpose of the query to avoid clogging of the result with too many one-offs though.

            Brute forcing of the BK-trees
            Since if you go with the route above, you will have to pay depending on the volume, the ~100-fold decrease in the computations needed becomes ~100-fold decrease in price which might be useful, especially if you have a lot of queries to run.
            However you would need to figure out a way to store this tree in a several layers of databases to query recursively as the Bigquery pricing depends on the volume of the data in the queried table.
            Building a smart batch engine for recursive processing of the queries to minimize the costs could be fun optimization excercise.

            A choice of language
            One more thing. While I think that Python is a good language for fast prototyping, analysis and thinking about code in general you are past that stage. You are currently looking for a way to do a specific, well defined and well thought operation as fast as possible. Python is not a great language for this as this example shows. While I used all the tricks I could think of in Python, the Java and C solutions were still several times faster. (Not to mention the rust one that beat us all - but he beat us by algorithm as well so it's hard to compare.) So if you go from python to a faster language, you might gain another factor or ten or maybe even more of a performance gain. This could be another fun optimization exercise.
            Note: I am being rather conservative with the estimate as the fuzzywuzzy already offers to use a C library in the background so I'm not too sure about how much of the work still depends on the python. My experience in similar cases is that the performance gain can be factor of 100 from pure python(or worse, pure R) to a compiled language.

            Source https://stackoverflow.com/questions/65588433

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install pybktree

            You can download it from GitHub.
            You can use pybktree like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

            Support

            For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
            CLONE
          • HTTPS

            https://github.com/benhoyt/pybktree.git

          • CLI

            gh repo clone benhoyt/pybktree

          • sshUrl

            git@github.com:benhoyt/pybktree.git

          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link