kandi background
Explore Kits

bert-extractive-summarizer | Easy to use extractive text summarization with BERT | Natural Language Processing library

 by   dmmiller612 Python Version: 0.10.1 License: MIT

 by   dmmiller612 Python Version: 0.10.1 License: MIT

Download this library from

kandi X-RAY | bert-extractive-summarizer Summary

bert-extractive-summarizer is a Python library typically used in Artificial Intelligence, Natural Language Processing, Deep Learning, Pytorch, Bert applications. bert-extractive-summarizer has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install bert-extractive-summarizer' or download it from GitHub, PyPI.
This repo is the generalization of the lecture-summarizer repo. This tool utilizes the HuggingFace Pytorch transformers library to run extractive summarizations. This works by first embedding the sentences, then running a clustering algorithm, finding the sentences that are closest to the cluster's centroids. This library also uses coreference techniques, utilizing the https://github.com/huggingface/neuralcoref library to resolve words in summaries that need more context. The greedyness of the neuralcoref library can be tweaked in the CoreferenceHandler class. As of the most recent version of bert-extractive-summarizer, by default, CUDA is used if a gpu is available.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • bert-extractive-summarizer has a medium active ecosystem.
  • It has 851 star(s) with 235 fork(s). There are 21 watchers for this library.
  • There were 3 major release(s) in the last 12 months.
  • There are 27 open issues and 63 have been closed. On average issues are closed in 321 days. There are 3 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of bert-extractive-summarizer is 0.10.1
This Library - Support
Best in #Natural Language Processing
Average in #Natural Language Processing
This Library - Support
Best in #Natural Language Processing
Average in #Natural Language Processing

quality kandi Quality

  • bert-extractive-summarizer has 0 bugs and 0 code smells.
This Library - Quality
Best in #Natural Language Processing
Average in #Natural Language Processing
This Library - Quality
Best in #Natural Language Processing
Average in #Natural Language Processing

securitySecurity

  • bert-extractive-summarizer has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • bert-extractive-summarizer code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.
This Library - Security
Best in #Natural Language Processing
Average in #Natural Language Processing
This Library - Security
Best in #Natural Language Processing
Average in #Natural Language Processing

license License

  • bert-extractive-summarizer is licensed under the MIT License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
This Library - License
Best in #Natural Language Processing
Average in #Natural Language Processing
This Library - License
Best in #Natural Language Processing
Average in #Natural Language Processing

buildReuse

  • bert-extractive-summarizer releases are available to install and integrate.
  • Deployable package is available in PyPI.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
This Library - Reuse
Best in #Natural Language Processing
Average in #Natural Language Processing
This Library - Reuse
Best in #Natural Language Processing
Average in #Natural Language Processing
Top functions reviewed by kandi - BETA

kandi has reviewed bert-extractive-summarizer and discovered the below as its top functions. This is intended to give you an instant insight into bert-extractive-summarizer implemented functionality, and help decide if they suit your requirements.

  • Run clustering .
  • Initializer .
  • Extract embeddings from text .
  • Run the embeddings algorithm .
  • Calculate the optimal cluster clustering .
  • Find the closest point to centroids .
  • Return a list of sentences from a document .
  • Creates a matrix representation of embeddings .
  • Convert raw text to paragraphs .
  • Convert raw text to paragraphs .

bert-extractive-summarizer Key Features

Easy to use extractive text summarization with BERT

Install

copy iconCopydownload iconDownload
pip install bert-extractive-summarizer

Simple Example

copy iconCopydownload iconDownload
from summarizer import Summarizer

body = 'Text body that you want to summarize with BERT'
body2 = 'Something else you want to summarize with BERT'
model = Summarizer()
model(body)
model(body2)

Use SBert

copy iconCopydownload iconDownload
pip install -U sentence-transformers

Retrieve Embeddings

copy iconCopydownload iconDownload
from summarizer import Summarizer
body = 'Text body that you want to summarize with BERT'
model = Summarizer()
result = model.run_embeddings(body, ratio=0.2)  # Specified with ratio. 
result = model.run_embeddings(body, num_sentences=3)  # Will return (3, N) embedding numpy matrix.
result = model.run_embeddings(body, num_sentences=3, aggregate='mean')  # Will return Mean aggregate over embeddings. 

Use Coreference

copy iconCopydownload iconDownload
pip install spacy
pip install transformers # > 4.0.0
pip install neuralcoref

python -m spacy download en_core_web_md

Custom Model Example

copy iconCopydownload iconDownload
from transformers import *

# Load model, model config and tokenizer via Transformers
custom_config = AutoConfig.from_pretrained('allenai/scibert_scivocab_uncased')
custom_config.output_hidden_states=True
custom_tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')
custom_model = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased', config=custom_config)

from summarizer import Summarizer

body = 'Text body that you want to summarize with BERT'
body2 = 'Something else you want to summarize with BERT'
model = Summarizer(custom_model=custom_model, custom_tokenizer=custom_tokenizer)
model(body)
model(body2)

Large Example

copy iconCopydownload iconDownload
from summarizer import Summarizer

body = '''
The Chrysler Building, the famous art deco New York skyscraper, will be sold for a small fraction of its previous sales price.
The deal, first reported by The Real Deal, was for $150 million, according to a source familiar with the deal.
Mubadala, an Abu Dhabi investment fund, purchased 90% of the building for $800 million in 2008.
Real estate firm Tishman Speyer had owned the other 10%.
The buyer is RFR Holding, a New York real estate company.
Officials with Tishman and RFR did not immediately respond to a request for comments.
It's unclear when the deal will close.
The building sold fairly quickly after being publicly placed on the market only two months ago.
The sale was handled by CBRE Group.
The incentive to sell the building at such a huge loss was due to the soaring rent the owners pay to Cooper Union, a New York college, for the land under the building.
The rent is rising from $7.75 million last year to $32.5 million this year to $41 million in 2028.
Meantime, rents in the building itself are not rising nearly that fast.
While the building is an iconic landmark in the New York skyline, it is competing against newer office towers with large floor-to-ceiling windows and all the modern amenities.
Still the building is among the best known in the city, even to people who have never been to New York.
It is famous for its triangle-shaped, vaulted windows worked into the stylized crown, along with its distinctive eagle gargoyles near the top.
It has been featured prominently in many films, including Men in Black 3, Spider-Man, Armageddon, Two Weeks Notice and Independence Day.
The previous sale took place just before the 2008 financial meltdown led to a plunge in real estate prices.
Still there have been a number of high profile skyscrapers purchased for top dollar in recent years, including the Waldorf Astoria hotel, which Chinese firm Anbang Insurance purchased in 2016 for nearly $2 billion, and the Willis Tower in Chicago, which was formerly known as Sears Tower, once the world's tallest.
Blackstone Group (BX) bought it for $1.3 billion 2015.
The Chrysler Building was the headquarters of the American automaker until 1953, but it was named for and owned by Chrysler chief Walter Chrysler, not the company itself.
Walter Chrysler had set out to build the tallest building in the world, a competition at that time with another Manhattan skyscraper under construction at 40 Wall Street at the south end of Manhattan. He kept secret the plans for the spire that would grace the top of the building, building it inside the structure and out of view of the public until 40 Wall Street was complete.
Once the competitor could rise no higher, the spire of the Chrysler building was raised into view, giving it the title.
'''

model = Summarizer()
result = model(body, min_length=60)
full = ''.join(result)
print(full)
"""
The Chrysler Building, the famous art deco New York skyscraper, will be sold for a small fraction of its previous sales price. 
The building sold fairly quickly after being publicly placed on the market only two months ago.
The incentive to sell the building at such a huge loss was due to the soaring rent the owners pay to Cooper Union, a New York college, for the land under the building.'
Still the building is among the best known in the city, even to people who have never been to New York.
"""

Calculating Elbow

copy iconCopydownload iconDownload
from summarizer import Summarizer

body = 'Your Text here.'
model = Summarizer()
res = model.calculate_elbow(body, k_max=10)
print(res)

Summarizer Options

copy iconCopydownload iconDownload
model = Summarizer(
    model: This gets used by the hugging face bert library to load the model, you can supply a custom trained model here
    custom_model: If you have a pre-trained model, you can add the model class here.
    custom_tokenizer:  If you have a custom tokenizer, you can add the tokenizer here.
    hidden: Needs to be negative, but allows you to pick which layer you want the embeddings to come from.
    reduce_option: It can be 'mean', 'median', or 'max'. This reduces the embedding layer for pooling.
    sentence_handler: The handler to process sentences. If want to use coreference, instantiate and pass CoreferenceHandler instance
)

model(
    body: str # The string body that you want to summarize
    ratio: float # The ratio of sentences that you want for the final summary
    min_length: int # Parameter to specify to remove sentences that are less than 40 characters
    max_length: int # Parameter to specify to remove sentences greater than the max length,
    num_sentences: Number of sentences to use. Overrides ratio if supplied.
)

Running the Service

copy iconCopydownload iconDownload
make docker-service-build
make docker-service-run

Community Discussions

Trending Discussions on Natural Language Processing
  • number of matches for keywords in specified categories
  • Apple's Natural Language API returns unexpected results
  • Tokenize text but keep compund hyphenated words together
  • Create new boolean fields based on specific bigrams appearing in a tokenized pandas dataframe
  • ModuleNotFoundError: No module named 'milvus'
  • Which model/technique to use for specific sentence extraction?
  • Assigning True/False if a token is present in a data-frame
  • How to calculate perplexity of a sentence using huggingface masked language models?
  • Mapping values from a dictionary's list to a string in Python
  • What are differences between AutoModelForSequenceClassification vs AutoModel
Trending Discussions on Natural Language Processing

QUESTION

number of matches for keywords in specified categories

Asked 2022-Apr-14 at 13:32

For a large scale text analysis problem, I have a data frame containing words that fall into different categories, and a data frame containing a column with strings and (empty) counting columns for each category. I now want to take each individual string, check which of the defined words appear, and count them within the appropriate category.

As a simplified example, given the two data frames below, i want to count how many of each animal type appear in the text cell.

df_texts <- tibble(
  text=c("the ape and the fox", "the tortoise and the hare", "the owl and the the 
  grasshopper"),
  mammals=NA,
  reptiles=NA,
  birds=NA,
  insects=NA
)

df_animals <- tibble(animals=c("ape", "fox", "tortoise", "hare", "owl", "grasshopper"),
           type=c("mammal", "mammal", "reptile", "mammal", "bird", "insect"))

So my desired result would be:

df_result <- tibble(
  text=c("the ape and the fox", "the tortoise and the hare", "the owl and the the 
  grasshopper"),
  mammals=c(2,1,0),
  reptiles=c(0,1,0),
  birds=c(0,0,1),
  insects=c(0,0,1)
)

Is there a straightforward way to achieve this keyword-matching-and-counting that would be applicable to a much larger dataset?

Thanks in advance!

ANSWER

Answered 2022-Apr-14 at 13:32

Here's a way do to it in the tidyverse. First look at whether strings in df_texts$text contain animals, then count them and sum by text and type.

library(tidyverse)

cbind(df_texts[, 1], sapply(df_animals$animals, grepl, df_texts$text)) %>% 
  pivot_longer(-text, names_to = "animals") %>% 
  left_join(df_animals) %>% 
  group_by(text, type) %>% 
  summarise(sum = sum(value)) %>% 
  pivot_wider(id_cols = text, names_from = type, values_from = sum)

  text                                   bird insect mammal reptile
  <chr>                                 <int>  <int>  <int>   <int>
1 "the ape and the fox"                     0      0      2       0
2 "the owl and the the \n  grasshopper"     1      0      0       0
3 "the tortoise and the hare"               0      0      1       1

To account for the several occurrences per text:

cbind(df_texts[, 1], t(sapply(df_texts$text, str_count, df_animals$animals, USE.NAMES = F))) %>% 
  setNames(c("text", df_animals$animals)) %>% 
  pivot_longer(-text, names_to = "animals") %>% 
  left_join(df_animals) %>% 
  group_by(text, type) %>% 
  summarise(sum = sum(value)) %>% 
  pivot_wider(id_cols = text, names_from = type, values_from = sum)

Source https://stackoverflow.com/questions/71871613

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install bert-extractive-summarizer

You can install using 'pip install bert-extractive-summarizer' or download it from GitHub, PyPI.
You can use bert-extractive-summarizer like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Share this Page

share link
Reuse Pre-built Kits with bert-extractive-summarizer
Consider Popular Natural Language Processing Libraries
Compare Natural Language Processing Libraries with Highest Support
Compare Natural Language Processing Libraries with Highest Quality
Compare Natural Language Processing Libraries with Highest Security
Compare Natural Language Processing Libraries with Permissive License
Compare Natural Language Processing Libraries with Highest Reuse
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.