gensim | Topic Modelling for Humans | Topic Modeling library

 by   RaRe-Technologies Python Version: 4.3.2 License: LGPL-2.1

kandi X-RAY | gensim Summary

kandi X-RAY | gensim Summary

gensim is a Python library typically used in Institutions, Learning, Education, Artificial Intelligence, Topic Modeling applications. gensim has no bugs, it has no vulnerabilities, it has build file available, it has a Weak Copyleft License and it has high support. You can install using 'pip install gensim' or download it from GitHub, PyPI.

Topic Modelling for Humans
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              gensim has a highly active ecosystem.
              It has 14417 star(s) with 4368 fork(s). There are 432 watchers for this library.
              There were 1 major release(s) in the last 12 months.
              There are 369 open issues and 1436 have been closed. On average issues are closed in 113 days. There are 31 open pull requests and 0 closed requests.
              It has a positive sentiment in the developer community.
              The latest version of gensim is 4.3.2

            kandi-Quality Quality

              gensim has 0 bugs and 0 code smells.

            kandi-Security Security

              gensim has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              gensim code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              gensim is licensed under the LGPL-2.1 License. This license is Weak Copyleft.
              Weak Copyleft licenses have some restrictions, but you can use them in commercial projects.

            kandi-Reuse Reuse

              gensim releases are available to install and integrate.
              Deployable package is available in PyPI.
              Build file is available. You can build the component from source.
              Installation instructions, examples and code snippets are available.
              It has 61066 lines of code, 2260 functions and 199 files.
              It has high code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi has reviewed gensim and discovered the below as its top functions. This is intended to give you an instant insight into gensim implemented functionality, and help decide if they suit your requirements.
            • Update the model with a given corpus
            • Perform the inference on the given document
            • Compute the phinorm
            • Evaluate the model
            • Updates the Lda model with a given corpus
            • Evaluate a single step
            • Add metrics to the plot
            • Set the model
            • Fit LDAPE algorithm
            • Merge two projections
            • Write a corpus to a file
            • Estimate the probability of a boolean sliding window
            • Extract articles and positions from file
            • Load a model
            • Add new documents to the LsiModel
            • Return unit vector
            • Updates the model with the given corpus
            • Update the LDA
            • Add a model to the model
            • Train the model
            • Evaluate the word analogies in the model
            • Evaluate a list of words
            • Compute the difference between two topics
            • Construct a sparse term similarity matrix
            • Compute the inner product between two matrices
            • Compute the distance between two documents
            Get all kandi verified functions for this library.

            gensim Key Features

            No Key Features are available at this moment for gensim.

            gensim Examples and Code Snippets

            How to get the dimensions of a word2vec object in python?
            Pythondot img1Lines of Code : 13dot img1License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            import gensim
            gensim.__version__
            # 3.6.0
            
            from gensim.test.utils import common_texts
            from gensim.models import Word2Vec
            
            model = Word2Vec(sentences=common_texts, window=5, min_count=1, workers=4) # do not specify size, leave the default 10
            How to get average pairwise cosine similarity per group in Pandas
            Pythondot img2Lines of Code : 31dot img2License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            # get average if more than 1 word is included in the "text" column
            def document_vector(items):
                # remove out-of-vocabulary words
                doc = [word for word in items.split() if word in model_glove]
                if doc:
                    doc_vector = model_gl
            'Doc2Vec' object has no attribute 'outputs', while saving doc2vec for tensorflow serving
            Pythondot img3Lines of Code : 5dot img3License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            filename = 'my_doc2vec_model'
            initial_model.save(filename)
            
            reloaded_model = Doc2Vec.load(filename)
            
            Using gensim most_similar function on a subset of total vocab
            Pythondot img4Lines of Code : 5dot img4License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            finite_set = set(['word_d', 'word_e', 'word_f'])  # set for efficient 'in'
            all_candidates = wv_from_bin.most_similar(positive=["word_a", "word_b"],
                                                      topn=len(vw_from_bin))
            filtered_results = [word_s
            No such file or directory: 'GoogleNews-vectors-negative300.bin'
            Pythondot img5Lines of Code : 4dot img5License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            serv/GoogleNews-vectors-negative300.bin
            
            /Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/GoogleNews-vectors-negative300.bin
            
            How to store the Phrase trigrams gensim model after training
            Pythondot img6Lines of Code : 4dot img6License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            trigram_transformer.save(TRIPHRASER_PATH)
            
            reloads_trigram_transformer = Phrases.load(TRIPHRASER_PATH)
            
            Plotly - Highlight data point and nearest three points on hover
            Pythondot img7Lines of Code : 107dot img7License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            import gensim
            import numpy as np
            import pandas as pd
            from sklearn.manifold import TSNE
            import plotly.express as px
            import plotly.graph_objects as go
            
            import json
            
            import dash
            from dash import dcc, html, Input, Output
            
            external_stylesheets 
            Problem with creating dictionary with gensim for LDA
            Pythondot img8Lines of Code : 6dot img8License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            from gensim import corpora
            corpus = [
                ['door', 'cat', 'mom'],
            ]
            dictionary = corpora.Dictionary(corpus)
            
            gensim/ Training a LDA Model: 'int' object is not subscriptable
            Pythondot img9Lines of Code : 2dot img9License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            corpus2 = [dct.doc2bow(filtered_sentence),]
            
            copy iconCopy
            error: can't find Rust compiler
            
            RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
            ENV PATH="/root/.cargo/bin:${PATH}"
            

            Community Discussions

            QUESTION

            How to get average pairwise cosine similarity per group in Pandas
            Asked 2022-Mar-29 at 20:51

            I have a sample dataframe as below

            ...

            ANSWER

            Answered 2022-Mar-29 at 18:47

            Remove the .vocab here in model_glove.vocab, this is not supported in the current version of gensim any more: Edit: also needs split() to iterate over words and not characters here.

            Source https://stackoverflow.com/questions/71666450

            QUESTION

            KeyedVectors\' object has no attribute \'wv for gensim 4.1.2
            Asked 2022-Mar-20 at 19:43

            i have migrated from gensim 3.8.3 to 4.1.2 and i am using this

            claim = [token for token in claim_text if token in w2v_model.wv.vocab]

            reference = [token for token in ref_text if token in w2v_model.wv.vocab]

            i am not sure how to replace w2v_model.wv.vocab to newer attribute and i am getting this error

            KeyedVectors' object has no attribute 'wv' can anyone please help.

            ...

            ANSWER

            Answered 2022-Mar-20 at 19:43

            You only use the .wv property to fetch the KeyedVectors object from another more complete algorithmic model, like a full Word2Vec model (which contains a KeyedVectors in its .wv attribute).

            If you're already working with just-the-vectors, there's no need to request the word-vectors subcomponent. Whatever you were going to do, you just do to the KeyedVectors directly.

            However, you're also using the .vocab attribute, which has been replaced. See the migration FAQ for more details:

            https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4#4-vocab-dict-became-key_to_index-for-looking-up-a-keys-integer-index-or-get_vecattr-and-set_vecattr-for-other-per-key-attributes

            (Mainly: instead of doing an in w2v_model.wv.vocab, you may only need to do in kv_model or in kv_model.key_to_index.)

            Source https://stackoverflow.com/questions/71544767

            QUESTION

            Gensim phrases model vocabulary length does not correspond to amount of iteratively added documents
            Asked 2022-Mar-14 at 19:50

            I iteratively apply the...

            ...

            ANSWER

            Answered 2022-Mar-14 at 19:50

            By default, to avoid using an unbounded amount of RAM, the Gensim Phrases class uses a default parameter max_vocab_size=40000000, per the source code & docs at:

            https://radimrehurek.com/gensim/models/phrases.html#gensim.models.phrases.Phrases

            Unfortunately, the mechanism behind this cap is very crude & non-intuitive. Whenever the tally of all known keys in they survey-dict (which includes both unigrams & bigrams) hits this threshold (default 40,000,000), a prune operation is performed that discards all token counts (unigrams & bigrams) at low-frequencies until the total unique-keys is under the threshold. And, it sets the low-frequency floor for future prunes to be at least as high as was necessary for this prune.

            For example, the 1st time this is hit, it might need to discard all the 1-count tokens. And due to the typical Zipfian distribution of word-frequencies, that step along might not just get the total count of known tokens slightly under the threshold, but massively under the threshold. And, any subsequent prune will start by eliminated at least everything with fewer than 2 occurrences.

            This results in the sawtooth counts you're seeing. When the model can't fit in max_vocab_size, it overshrinks. It may do this many times in the course of processing a very-large corpus. As a result, final counts of lower-frequency words/bigrams can also be serious undercounts - depending somewhat arbitrarily on whether a key's counts survived the various prune-thresholds. (That's also influenced by where in the corpus a token appears. A token that only appears in the corpus after the last prune will still have a precise count, even if it only appears once! Although rare tokens that appeared any number of times could be severely undercounted, if they were always below the cutoff at each prior prune.)

            The best solution would be to use a precise count that uses/correlates some spillover storage on-disk, to only prune (if at all) at the very end, ensuring only the truly-least-frequent keys are discarded. Unfortunately, Gensim's never implemented that option.

            The next-best, for many cases, could be to use a memory-efficient approximate counting algorithm, that vaguely maintains the right magnitudes of counts for a much-larger number of keys. There's been a litte work in Gensim on this in the past, but not yet integrated with the Phrases functionality.

            That leaves you with the only practical workaround in the short term: change the max_vocab_size parameter to be larger.

            You could try setting it to math.inf (might risk lower performance due to int-vs-float comparisons) or sys.maxsize – essentially turning off the pruning entirely, to see if your survey can complete without exhausting your RAM. But, you might run out of memory anyway.

            You could also try a larger-but-not-essentially-infinite cap – whatever fits in your RAM – so that far less pruning is done. But you'll still see the non-intuitive decreases in total counts, sometimes, if in fact the threshold is ever enforced. Per the docs, a very rough (perhaps outdated) estimate is that the default max_vocab_size=40000000 consumes about 3.6GB at peak saturation. So if you've got a 64GB machine, you could possibly try a max_vocab_size thats 10-14x larger than the default, etc.

            Source https://stackoverflow.com/questions/71457117

            QUESTION

            Can I use a different corpus for fasttext build_vocab than train in Gensim Fasttext?
            Asked 2022-Mar-07 at 22:50

            I am curious to know if there are any implications of using a different source while calling the build_vocab and train of Gensim FastText model. Will this impact the contextual representation of the word embedding?

            My intention for doing this is that there is a specific set of words I am interested to get the vector representation for and when calling model.wv.most_similar. I only want words defined in this vocab list to get returned rather than all possible words in the training corpus. I would use the result of this to decide if I want to group those words to be relevant to each other based on similarity threshold.

            Following is the code snippet that I am using, appreciate your thoughts if there are any concerns or implication with this approach.

            • vocab.txt contains a list of unique words of interest
            • corpus.txt contains full conversation text (i.e. chat messages) where each line represents a paragraph/sentence per chat

            A follow up question to this is what values should I set for total_examples & total_words during training in this case?

            ...

            ANSWER

            Answered 2022-Mar-07 at 22:50

            Incase someone has similar question, I'll paste the reply I got when asking this question in the Gensim Disussion Group for reference:

            You can try it, but I wouldn't expect it to work well for most purposes.

            The build_vocab() call establishes the known vocabulary of the model, & caches some stats about the corpus.

            If you then supply another corpus – & especially one with more words – then:

            • You'll want your train() parameters to reflect the actual size of your training corpus. You'll want to provide a true total_examples and total_words count that are accurate for the training-corpus.
            • Every word in the training corpus that's not in the know vocabulary is ignored completely, as if it wasn't even there. So you might as well filter your corpus down to just the words-of-interest first, then use that same filtered corpus for both steps. Will the example texts still make sense? Will that be enough data to train meaningful, generalizable word-vectors for just the words-of-interest, alongside other words-of-interest, without the full texts? (You could look at your pref-filtered corpus to get a sense of that.) I'm not sure - it could depend on how severely trimming to just the words-of-interest changed the corpus. In particular, to train high-dimensional dense vectors – as with vector_size=300 – you need a lot of varied data. Such pre-trimming might thin the corpus so much as to make the word-vectors for the words-of-interest far less useful.

            You could certainly try it both ways – pre-filtered to just your words-of-interest, or with the full original corpus – and see which works better on downstream evaluations.

            More generally, if the concern is training time with the full corpus, there are likely other ways to get an adequate model in an acceptable amount of time.

            If using corpus_file mode, you can increase workers to equal the local CPU core count for a nearly-linear speedup from number of cores. (In traditional corpus_iterable mode, max throughput is usually somewhere in the 6-12 workers threads, as long as you ahve that many cores.)

            min_count=1 is usually a bad idea for these algorithms: they tend to train faster, in less memory, leaving better vectors for the remaining words when you discard the lowest-frequency words, as the default min_count=5 does. (It's possible FastText can eke a little bit of benefit out of lower-frequency words via their contribution to character-n-gram-training, but I'd only ever lower the default min_count if I could confirm it was actually improving relevant results.

            If your corpus is so large that training time is a concern, often a more-aggressive (smaller) sample parameter value not only speeds training (by dropping many redundant high-frequency words), but ofthen improves final word-vector quality for downstream purposes as well (by letting the rarer words have relatively more influence on the model in the absense of the downsampled words).

            And again if the corpus is so large that training time is a concern, than epochs=100 is likely overkill. I believe the GoogleNews vectors were trained using only 3 passes – over a gigantic corpus. A sufficiently large & varied corpus, with plenty of examples of all words all throughout, could potentially train in 1 pass – because each word-vector can then get more total training-updates than many epochs with a small corpus. (In general larger epochs values are more often used when the corpus is thin, to eke out something – not on a corpus so large you're considering non-standard shortcuts to speed the steps.)

            -- Gordon

            Source https://stackoverflow.com/questions/71289683

            QUESTION

            Unpickle instance from Jupyter Notebook in Flask App
            Asked 2022-Feb-28 at 18:03

            I have created a class for word2vec vectorisation which is working fine. But when I create a model pickle file and use that pickle file in a Flask App, I am getting an error like:

            AttributeError: module '__main__' has no attribute 'GensimWord2VecVectorizer'

            I am creating the model on Google Colab.

            Code in Jupyter Notebook:

            ...

            ANSWER

            Answered 2022-Feb-24 at 11:48

            Import GensimWord2VecVectorizer in your Flask Web app python file.

            Source https://stackoverflow.com/questions/71231611

            QUESTION

            Word2Vec returning vectors for individual character and not words
            Asked 2022-Feb-12 at 13:11

            For the following list:

            ...

            ANSWER

            Answered 2022-Feb-12 at 13:11

            Word2Vec expects a list of lists as input, where the corpus (main list) is composed of individual documents. The individual documents are composed of individual words (tokens). Word2Vec iterates over all documents and all tokens. In your example you have passed a single list to Word2Vec, therefore Word2Vec interprets each word as an individual document and iterates over each word character which is interpreted as a token. Therefore you have built a vocabulary of characters not words. To build a vocabulary of words you can pass a nested list to Word2Vec as in the example below.

            Source https://stackoverflow.com/questions/71091209

            QUESTION

            No such file or directory: 'GoogleNews-vectors-negative300.bin'
            Asked 2022-Feb-04 at 06:08

            I have this code :

            ...

            ANSWER

            Answered 2022-Feb-04 at 06:08

            The 'current working directory' that the Python process will consider active, and thus will use as the expected location for your plain relative filename GoogleNews-vectors-negative300.bin, will depend on how you launched Flask.

            You could print out the directory to be sure – see some ways at How do you properly determine the current script directory? – but I suspect it may just be the /Users/Ile-Maurice/Desktop/Flask/flaskapp/ directory.

            If so, you could relatively-reference your file with the path relative to the above directory...

            Source https://stackoverflow.com/questions/70973660

            QUESTION

            How to store the Phrase trigrams gensim model after training
            Asked 2022-Feb-04 at 01:01

            I would like to know can I store the gensim Phrase model after training it on the sentences

            ...

            ANSWER

            Answered 2022-Feb-03 at 18:40

            Convert list or that partular format into an numpy array and save it as a .npy file easy to save and easy to read, using this by numpy gives you advantage of loading it in almost every platform like google colab, replit ..... refer to this link for more details on saving a npy file numpy.save()

            Using pickle is also a good option but things get a bit tricky at points when difference in encoding standards and such problems arise.

            Source https://stackoverflow.com/questions/70976566

            QUESTION

            Plotly - Highlight data point and nearest three points on hover
            Asked 2022-Feb-02 at 04:15

            I have made a scatter plot of the word2vec model using plotly.
            I want functionality of highlighting the specific data point on hover along with the top 3 nearest vectors to that. It would be of great help if anyone can guide me with this or suggest any other option

            model
            csv

            Code:

            ...

            ANSWER

            Answered 2022-Feb-02 at 04:15

            In plotly-python, I don't think there's an easy way of retrieving the location of the cursor. You can attempt to use go.FigureWidget to highlight a trace as described in this answer, but i think you're going to be limited with with plotly-python and i'm not sure if highlighting the closest n points will be possible.

            However, I believe that you can accomplish what you want in plotly-dash since callbacks are supported - meaning you would be able to retrieve location of your cursor and then calculate the n closest data points to your cursor and highlight the data points as needed.

            Below is an example of such a solution. If you haven't seen it before, it looks complicated, but what is happening is that I am taking the point where you clicked as an input. plotly is plotly.js under the hood so it comes us in the form of a dictionary (and not some kind of plotly-python object). Then I calculate the closest three data points to the clicked input point by comparing the coordinates of every other point in the dataframe, add the information from the three closest points as traces to the input with the color teal (or any color of your choosing), and send this modified input back as the output, and update the figure.

            I am using click instead of hover because hover would cause the highlighted points to flicker too much as you drag your mouse through the points.

            Also the dash app doesn't work perfectly as I believe there is some issue when you double click on points (you can see me click once in the gif below before getting it to start working), but this basic framework is hopefully close enough to what you want. Cheers!

            Source https://stackoverflow.com/questions/70944316

            QUESTION

            gensim w2k - additional file
            Asked 2022-Feb-01 at 14:52

            I trained w2v on rather big (> 200 million sentences) corpus, and got, in addition to file w2v_model.model, files: w2v_model.model.trainables.syn1neg.npy and w2v.model_model.wv.vectors.npy. Model file was successfully loaded and read all npy files without any exceptions. The obtained model performed OK.

            Now I retrained the model on much bigger corpus (> 1 billion sentences). The same 3 files were automatically saved, as expected.

            When I try to load my new retrained model:

            ...

            ANSWER

            Answered 2022-Jan-24 at 18:39

            If a .save() is creating any files with the word trainables in it, you're using a older version fo Gensim. Any new training should definitely prefer using a current version. As of now (January 2022), that's gensim-4.1.2, released 2021-09.

            If an attempt at a .load() generated that particular error, then there should've been that file, alongside the others you mention, created when the .save() had been done. (In fact, the only way that the main file you named with path_filename should be able to know that other filename is if that other file was written successfully, allowing the main file to complete writing.)

            Are you sure that file wasn't written, but then somehow left behind, perhaps getting deleted or not moving alongside the other few files to some new filesystem path?

            In general, I would suggest:

            • using latest Gensim for any new training
            • always enable Python logging at the INFO level, & watch the logging/console output of training/saving processes closely to see confirmation of expected activity/steps
            • keep all files from a .save() that begin with the same main filename (in your examples above, w2v_US.model) together - & keep in mind that for larger models it may be a larger roster of files than for a small test model

            You will probably have to re-train the model, but you might be able to re-generate a compatible lockf file via steps like the following:

            • save aside all files of any potential use
            • from the exact same configuration as your original .save() – including the same outdated Gensim version, exact same model parameters, & exact same training corpus – repeat all the model-building steps you did before up through the .build_vocab() step. (That is: no extra need to .train().) This will create an untrained dummy model that should exactly match the vocabulary 'shape' of your broken model.
            • use .save() to save that dummy model again - watching the logs/output for errors. There should be, alongside the other files, a file with a name like dummy.model.trainables.vectors_lockf.npy. If so, you might be able to copy that away, rename it to tbe the file expected by the original model whose load failed, then leave it alongside that original model - and the .load() might then succeed, or fail in a different way.

            (If there were other problems/corruption at the time of the original model creation, this might not work. In particular, I wonder if when you talk about retraining the model, you didn't start with a fresh Word2Vec instance, but somehow expanded the older one, which might've added other problems/complications. In that case, a full retraining, ideally in the latest Gensim, would be necessary, and also a better basis for going forward.)

            Source https://stackoverflow.com/questions/70693372

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install gensim

            This software depends on [NumPy and Scipy], two Python packages for scientific computing. You must have them installed prior to installing gensim. It is also recommended you install a fast BLAS library before installing NumPy. This is optional, but using an optimized BLAS such as MKL, [ATLAS] or [OpenBLAS] is known to improve performance by as much as an order of magnitude. On OSX, NumPy picks up its vecLib BLAS automatically, so you don’t need to do anything special.

            Support

            [QuickStart][Tutorials][Official API Documentation] [QuickStart]: https://radimrehurek.com/gensim/auto_examples/core/run_core_concepts.html [Tutorials]: https://radimrehurek.com/gensim/auto_examples/ [Official Documentation and Walkthrough]: http://radimrehurek.com/gensim/ [Official API Documentation]: http://radimrehurek.com/gensim/apiref.html
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
            Install
          • PyPI

            pip install gensim

          • CLONE
          • HTTPS

            https://github.com/RaRe-Technologies/gensim.git

          • CLI

            gh repo clone RaRe-Technologies/gensim

          • sshUrl

            git@github.com:RaRe-Technologies/gensim.git

          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Consider Popular Topic Modeling Libraries

            gensim

            by RaRe-Technologies

            Familia

            by baidu

            BERTopic

            by MaartenGr

            Top2Vec

            by ddangelov

            lda

            by lda-project

            Try Top Libraries by RaRe-Technologies

            smart_open

            by RaRe-TechnologiesPython

            sqlitedict

            by RaRe-TechnologiesPython

            bounter

            by RaRe-TechnologiesPython

            gensim-data

            by RaRe-TechnologiesPython

            movie-plots-by-genre

            by RaRe-TechnologiesJupyter Notebook