kandi background

sentence-transformers | Multilingual Sentence & Image Embeddings with BERT | Natural Language Processing library

by UKPLab Python Version: v2.0.0 License: Apache-2.0

by UKPLab Python Version: v2.0.0 License: Apache-2.0

Download this library from

kandi X-RAY | sentence-transformers Summary

sentence-transformers is a Python library typically used in Artificial Intelligence, Natural Language Processing, Deep Learning, Pytorch, Tensorflow, Bert, Transformer applications. sentence-transformers has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install sentence-transformers' or download it from GitHub, PyPI.

This framework provides an easy method to compute dense vector representations for sentences, paragraphs, and images. The models are based on transformer networks like BERT / RoBERTa / XLM-RoBERTa etc. and achieve state-of-the-art performance in various task. Text is embedding in vector space such that similar text is close and can efficiently be found using cosine similarity. We provide an increasing number of state-of-the-art pretrained models for more than 100 languages, fine-tuned for various use-cases. Further, this framework allows an easy fine-tuning of custom embeddings models, to achieve maximal performance on your specific task. For the full documentation, see www.SBERT.net.

Support

Support

Quality

Quality

Security

Security

License

License

Reuse

Reuse

kandi-support Support

sentence-transformers has a medium active ecosystem.
It has 5944 star(s) with 1145 fork(s). There are 104 watchers for this library.
It had no major release in the last 12 months.
There are 503 open issues and 539 have been closed. On average issues are closed in 37 days. There are 11 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of sentence-transformers is v2.0.0

This Library - Support

Best in #Natural Language Processing

Average in #Natural Language Processing

This Library - Support

Best in #Natural Language Processing

Average in #Natural Language Processing

quality kandi Quality

sentence-transformers has 0 bugs and 0 code smells.

This Library - Quality

Best in #Natural Language Processing

Average in #Natural Language Processing

This Library - Quality

Best in #Natural Language Processing

Average in #Natural Language Processing

security Security

sentence-transformers has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
sentence-transformers code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

This Library - Security

Best in #Natural Language Processing

Average in #Natural Language Processing

This Library - Security

Best in #Natural Language Processing

Average in #Natural Language Processing

license License

sentence-transformers is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

This Library - License

Best in #Natural Language Processing

Average in #Natural Language Processing

This Library - License

Best in #Natural Language Processing

Average in #Natural Language Processing

build Reuse

sentence-transformers releases are available to install and integrate.
Deployable package is available in PyPI.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.
sentence-transformers saves you 3939 person hours of effort in developing the same functionality from scratch.
It has 12029 lines of code, 425 functions and 182 files.
It has high code complexity. Code complexity directly impacts maintainability of the code.

This Library - Reuse

Best in #Natural Language Processing

Average in #Natural Language Processing

This Library - Reuse

Best in #Natural Language Processing

Average in #Natural Language Processing

Top functions reviewed by kandi - BETA

kandi has reviewed sentence-transformers and discovered the below as its top functions. This is intended to give you an instant insight into sentence-transformers implemented functionality, and help decide if they suit your requirements.

Fit the AdamW model .
Compute metrics for each query .
Performs community detection .
Predict from sentences .
Perform a semantic search using the semantic search algorithm .
Download a repository from a repository .
Performs morphological mining .
Compute scores for the given model .
Forward mini batch of mini batches .
Create a TREC dataset .

sentence-transformers Key Features

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)

Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (EMNLP 2020)

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks (NAACL 2021)

The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes (arXiv 2020)

TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning (arXiv 2021)

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv 2021)

sentence-transformers Examples and Code Snippets

See all related Code Snippets

Installation

Copy

Download

pip install -U sentence-transformers

Getting Started

Copy

Download

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

Citing & Authors

Copy

Download

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

The last dimension of the inputs to a Dense layer should be defined. Found None. Full input shape received: <unknown>

Copy

Download

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings

    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        input_ids.set_shape(tf.TensorShape((None, None)))
        token_type_ids.set_shape(tf.TensorShape((None, None)))
        attention_mask.set_shape(tf.TensorShape((None, None)))

        model_output = self.process(input_ids, token_type_ids, attention_mask)
        model_output[0].set_shape(tf.TensorShape((None, None, 384)))
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings

    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
outputs = tf.keras.layers.Dense(32)(outputs)
model = tf.keras.Model(inputs, outputs)
print(model(tf.constant(['some text', 'more text'])))
print(model.summary())

tf.Tensor(
[[-0.06719425 -0.02954631 -0.05811356 -0.1456391  -0.13001677  0.00145465
   0.0401044   0.05949172 -0.02589339  0.07255618 -0.00958113  0.01159782
   0.02508018  0.03075579 -0.01910635 -0.03231853  0.00875124  0.01143366
  -0.04365401 -0.02090197  0.07030752 -0.02872834  0.10535908  0.05691438
  -0.017165   -0.02044982  0.02580127 -0.04564123 -0.0631128  -0.00303708
   0.00133517  0.01613527]
 [-0.11922387  0.02304137 -0.02670465 -0.13117084 -0.11492493  0.03961402
   0.08129141 -0.05999354  0.0039564   0.02892766  0.00493046  0.00440936
  -0.07966737  0.11354238  0.03141225  0.00048972  0.04658606 -0.03658888
  -0.05292419 -0.04639702  0.08445395  0.00522146  0.04359548  0.0290177
  -0.02171512 -0.03399373 -0.00418095 -0.04019783 -0.04733383 -0.03972956
   0.01890458 -0.03927581]], shape=(2, 32), dtype=float32)
Model: "model_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_18 (InputLayer)       [(None, 1)]               0         
                                                                 
 s_bert_17 (SBert)           (None, 384)               22713216  
                                                                 
 dense_78 (Dense)            (None, 32)                12320     
                                                                 
=================================================================
Total params: 22,725,536
Trainable params: 22,725,536
Non-trainable params: 0
_________________________________________________________________
None

-----------------------

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings

    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        input_ids.set_shape(tf.TensorShape((None, None)))
        token_type_ids.set_shape(tf.TensorShape((None, None)))
        attention_mask.set_shape(tf.TensorShape((None, None)))

        model_output = self.process(input_ids, token_type_ids, attention_mask)
        model_output[0].set_shape(tf.TensorShape((None, None, 384)))
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings

    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
outputs = tf.keras.layers.Dense(32)(outputs)
model = tf.keras.Model(inputs, outputs)
print(model(tf.constant(['some text', 'more text'])))
print(model.summary())

tf.Tensor(
[[-0.06719425 -0.02954631 -0.05811356 -0.1456391  -0.13001677  0.00145465
   0.0401044   0.05949172 -0.02589339  0.07255618 -0.00958113  0.01159782
   0.02508018  0.03075579 -0.01910635 -0.03231853  0.00875124  0.01143366
  -0.04365401 -0.02090197  0.07030752 -0.02872834  0.10535908  0.05691438
  -0.017165   -0.02044982  0.02580127 -0.04564123 -0.0631128  -0.00303708
   0.00133517  0.01613527]
 [-0.11922387  0.02304137 -0.02670465 -0.13117084 -0.11492493  0.03961402
   0.08129141 -0.05999354  0.0039564   0.02892766  0.00493046  0.00440936
  -0.07966737  0.11354238  0.03141225  0.00048972  0.04658606 -0.03658888
  -0.05292419 -0.04639702  0.08445395  0.00522146  0.04359548  0.0290177
  -0.02171512 -0.03399373 -0.00418095 -0.04019783 -0.04733383 -0.03972956
   0.01890458 -0.03927581]], shape=(2, 32), dtype=float32)
Model: "model_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_18 (InputLayer)       [(None, 1)]               0         
                                                                 
 s_bert_17 (SBert)           (None, 384)               22713216  
                                                                 
 dense_78 (Dense)            (None, 32)                12320     
                                                                 
=================================================================
Total params: 22,725,536
Trainable params: 22,725,536
Non-trainable params: 0
_________________________________________________________________
None

Use `sentence-transformers` inside of a keras model

Copy

Download

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x.decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0]
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.model({'input_ids': input_ids, 'token_type_ids': token_type_ids, 'attention_mask': attention_mask})
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
sbert(['some text', 'more text'])

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.process(input_ids, token_type_ids, attention_mask)
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
model = tf.keras.Model(inputs, outputs)
model(tf.constant(['some text', 'more text']))

TensorShape([2, 384]).shape

-----------------------

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x.decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0]
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.model({'input_ids': input_ids, 'token_type_ids': token_type_ids, 'attention_mask': attention_mask})
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
sbert(['some text', 'more text'])

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.process(input_ids, token_type_ids, attention_mask)
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
model = tf.keras.Model(inputs, outputs)
model(tf.constant(['some text', 'more text']))

TensorShape([2, 384]).shape

-----------------------

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x.decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0]
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.model({'input_ids': input_ids, 'token_type_ids': token_type_ids, 'attention_mask': attention_mask})
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
sbert(['some text', 'more text'])

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.process(input_ids, token_type_ids, attention_mask)
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
model = tf.keras.Model(inputs, outputs)
model(tf.constant(['some text', 'more text']))

TensorShape([2, 384]).shape

Using sentence transformers with limited access to internet

Copy

Download

['1_Pooling', 'config_sentence_transformers.json', 'tokenizer.json', 'tokenizer_config.json', 'modules.json', 'sentence_bert_config.json', 'pytorch_model.bin', 'special_tokens_map.json', 'config.json', 'train_script.py', 'data_config.json', 'README.md', '.gitattributes', 'vocab.txt']

Using Sentence-Bert with other features in scikit-learn

Copy

Download

X_train = pd.DataFrame({
    'tweet':['foo', 'foo', 'bar'],
    'feature1':[1, 1, 0],
    'feature2':[1, 0, 1],
})
y_train = [1, 1, 0]

model = SentenceTransformer('mrm8488/bert-tiny-finetuned-squadv2') # model named is changed for time and computation gians :)
embedder = FunctionTransformer(lambda item:model.encode(item, convert_to_tensor=True, show_progress_bar=False).detach().cpu().numpy())

preprocessor = ColumnTransformer(
    transformers=[('embedder', embedder, 'tweet')],
    remainder='passthrough'
    )
X_train = preprocessor.fit_transform(X_train) # X_train.shape => (len(df), your_transformer_model_hidden_dim + your_features_count)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

-----------------------

X_train = pd.DataFrame({
    'tweet':['foo', 'foo', 'bar'],
    'feature1':[1, 1, 0],
    'feature2':[1, 0, 1],
})
y_train = [1, 1, 0]

model = SentenceTransformer('mrm8488/bert-tiny-finetuned-squadv2') # model named is changed for time and computation gians :)
embedder = FunctionTransformer(lambda item:model.encode(item, convert_to_tensor=True, show_progress_bar=False).detach().cpu().numpy())

preprocessor = ColumnTransformer(
    transformers=[('embedder', embedder, 'tweet')],
    remainder='passthrough'
    )
X_train = preprocessor.fit_transform(X_train) # X_train.shape => (len(df), your_transformer_model_hidden_dim + your_features_count)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

-----------------------

X_train = pd.DataFrame({
    'tweet':['foo', 'foo', 'bar'],
    'feature1':[1, 1, 0],
    'feature2':[1, 0, 1],
})
y_train = [1, 1, 0]

model = SentenceTransformer('mrm8488/bert-tiny-finetuned-squadv2') # model named is changed for time and computation gians :)
embedder = FunctionTransformer(lambda item:model.encode(item, convert_to_tensor=True, show_progress_bar=False).detach().cpu().numpy())

preprocessor = ColumnTransformer(
    transformers=[('embedder', embedder, 'tweet')],
    remainder='passthrough'
    )
X_train = preprocessor.fit_transform(X_train) # X_train.shape => (len(df), your_transformer_model_hidden_dim + your_features_count)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

-----------------------

X_train = pd.DataFrame({
    'tweet':['foo', 'foo', 'bar'],
    'feature1':[1, 1, 0],
    'feature2':[1, 0, 1],
})
y_train = [1, 1, 0]

model = SentenceTransformer('mrm8488/bert-tiny-finetuned-squadv2') # model named is changed for time and computation gians :)
embedder = FunctionTransformer(lambda item:model.encode(item, convert_to_tensor=True, show_progress_bar=False).detach().cpu().numpy())

preprocessor = ColumnTransformer(
    transformers=[('embedder', embedder, 'tweet')],
    remainder='passthrough'
    )
X_train = preprocessor.fit_transform(X_train) # X_train.shape => (len(df), your_transformer_model_hidden_dim + your_features_count)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

String comparison with BERT seems to ignore "not" in sentence

Copy

Download

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("roberta-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("roberta-large-mnli")

def test_entailment(text1, text2):
    batch = tokenizer(text1, text2, return_tensors='pt').to(model.device)
    with torch.no_grad():
        proba = torch.softmax(model(**batch).logits, -1)
    return proba.cpu().numpy()[0, model.config.label2id['ENTAILMENT']]

def test_equivalence(text1, text2):
    return test_entailment(text1, text2) * test_entailment(text2, text1)

print(test_equivalence("I'm a good person", "I'm not a good person"))  # 2.0751484e-07
print(test_equivalence("I'm a good person", "You are a good person"))  # 0.49342492
print(test_equivalence("I'm a good person", "I'm not a bad person"))   # 0.94236994

python packages not being installed on the virtual environment using ubuntu

Copy

Download

pip3 install package_name --user

AttributeError: Caught AttributeError in DataLoader worker process 0. - fine tuning pre-trained transformer model

Copy

Download

import pandas as pd
# initialise data of lists.
data = {'input':[
          "Alpro, Cioccolato bevanda a base di soia 1 ltr", #Alpro, Chocolate soy drink 1 ltr
          "Milka  cioccolato al latte 100 g", #Milka milk chocolate 100 g
          "Danone, HiPRO 25g Proteine gusto cioccolato 330 ml", #Danone, HiPRO 25g Protein chocolate flavor 330 ml
         ]
        }
 
# Creates pandas DataFrame.
x_sample = pd.DataFrame(data)
print(x_sample['input'])

# load model
from sentence_transformers import SentenceTransformer, SentencesDataset, InputExample, losses, evaluation
from torch.utils.data import DataLoader

embedder = SentenceTransformer('sentence-transformers/paraphrase-xlm-r-multilingual-v1') # or any other pretrained model
print("embedder loaded...")

# define your train dataset, the dataloader, and the train loss
# train_dataset = SentencesDataset(x_sample["input"].tolist(), embedder)
# train_dataloader = DataLoader(train_dataset, shuffle=False, batch_size=4, num_workers=1)
# train_loss = losses.CosineSimilarityLoss(embedder)

# dummy evaluator to make the api work
sentences1 = ['latte al cioccolato', 'latte al cioccolato','latte al cioccolato']
sentences2 = ['Alpro, Cioccolato bevanda a base di soia 1 ltr', 'Danone, HiPRO 25g Proteine gusto cioccolato 330 ml','Milka  cioccolato al latte 100 g']
scores = [0.99,0.95,0.4]
evaluator = evaluation.EmbeddingSimilarityEvaluator(sentences1, sentences2, scores)

examples = []
for s1,s2,l in zip(sentences1, sentences2, scores):
  examples.append(InputExample(texts=[s1, s2], label=l))
train_dataloader = DataLoader(examples, shuffle=False, batch_size=4, num_workers=1)
train_loss = losses.CosineSimilarityLoss(embedder)
# tune the model
embedder.fit(train_objectives=[(train_dataloader, train_loss)], 
    epochs=5, 
    warmup_steps=500, 
    evaluator=evaluator, 
    evaluation_steps=1,
    output_path='fine_tuned_bert',
    save_best_model= True,
    show_progress_bar= True
    )

Heroku: Compiled Slug Size is too large Python

Copy

Download

heroku run bash -a <appname>

du -ha --max-depth 1 /app | sort -hr

-----------------------

heroku run bash -a <appname>

du -ha --max-depth 1 /app | sort -hr

Download pre-trained BERT model locally

Copy

Download

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
model = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
tokenizer.save_pretrained('./local_directory/')
model.save_pretrained('./local_directory/')

Sentence embedding using T5

Copy

Download

model.encoder(input_ids=s, attention_mask=attn, return_dict=True)
pooled_sentence = output.last_hidden_state # shape is [batch_size, seq_len, hidden_size]
# pooled_sentence will represent the embeddings for each word in the sentence
# you need to sum/average the pooled_sentence
pooled_sentence = torch.mean(pooled_sentence, dim=1)

See all related Code Snippets

Community Discussions

Trending Discussions on sentence-transformers

The last dimension of the inputs to a Dense layer should be defined. Found None. Full input shape received: <unknown>
Use `sentence-transformers` inside of a keras model
Using sentence transformers with limited access to internet
Using Sentence-Bert with other features in scikit-learn
String comparison with BERT seems to ignore "not" in sentence
Token indices sequence length Issue
python packages not being installed on the virtual environment using ubuntu
AttributeError: Caught AttributeError in DataLoader worker process 0. - fine tuning pre-trained transformer model
How to increase dimension-vector size of BERT sentence-transformers embedding
BERT problem with context/semantic search in italian language

Trending Discussions on sentence-transformers

QUESTION

The last dimension of the inputs to a Dense layer should be defined. Found None. Full input shape received: <unknown>

Asked 2022-Mar-10 at 08:57

I am having trouble when switching a model from some local dummy data to using a TF dataset.

Sorry for the long model code, I have tried to shorten it as much as possible.

The following works fine:

import tensorflow as tf
import tensorflow_recommenders as tfrs
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
        def __call(i, t, a):
            model_output = self.model(
                {'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()}
            )
            return model_output[0]
        return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.process(input_ids, token_type_ids, attention_mask)
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings


sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input(shape=(1,), dtype=tf.string)
outputs = sbert(inputs)
model = tf.keras.Model(inputs, outputs)
model(tf.constant(['some text', 'more text']))

The call to the model outputs tensors - yipee :)

Now I want to use this layer inside of a larger two tower model:

class Encoder(tf.keras.Model):
    def __init__(self):
        super().__init__()
        
        self.text_embedding = self._build_text_embedding()
    
    def _build_text_embedding(self):
        sbert = SBert(tokenizer, model)
        inputs = tf.keras.layers.Input(shape=(1,), dtype=tf.string)
        outputs = sbert(inputs)
        return tf.keras.Model(inputs, outputs)
    
    def call(self, inputs):
        return self.text_embedding(inputs)

    
class RecModel(tfrs.models.Model):
    def __init__(self):
        super().__init__()
        
        self.query_model = tf.keras.Sequential([
            Encoder(),
            tf.keras.layers.Dense(32)
        ])
        
        self.candidate_model = tf.keras.Sequential([
            Encoder(),
            tf.keras.layers.Dense(32)
        ])
    
        self.retrieval_task = tfrs.tasks.Retrieval(
            metrics=tfrs.metrics.FactorizedTopK(
                candidates=tf.data.Dataset.from_tensor_slices(
                    data['text']
                ).batch(1).map(self.candidate_model),
            ),
            batch_metrics=[
                tf.keras.metrics.TopKCategoricalAccuracy(k=5)
            ]
        )

    def call(self, features):
        query_embeddings = self.query_model(features['query'])
        candidate_embeddings = self.candidate_model(features['text'])
        return (
            query_embeddings,
            candidate_embeddings,
        )   

    def compute_loss(self, features, training=False):
        query_embeddings, candidate_embeddings = self(features)
        retrieval_loss = self.retrieval_task(query_embeddings, candidate_embeddings)
        return retrieval_loss

Create a small dummy dataset:

data = {
    'query': ['blue', 'cat', 'football'],
    'text': ['a nice colour', 'a type of animal', 'a sport']
}

ds = tf.data.Dataset.from_tensor_slices(data).batch(1)

Try to compile:

model = RecModel()
model.compile(optimizer=tf.keras.optimizers.Adagrad())

And we hit the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-df4cc46e0307> in <module>
----> 1 model = RecModel()
      2 model.compile(optimizer=tf.keras.optimizers.Adagrad())

<ipython-input-8-a774041744b9> in __init__(self)
     33                 candidates=tf.data.Dataset.from_tensor_slices(
     34                     data['text']
---> 35                 ).batch(1).map(self.candidate_model),
     36             ),
     37             batch_metrics=[

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in map(self, map_func, num_parallel_calls, deterministic, name)
   2014         warnings.warn("The `deterministic` argument has no effect unless the "
   2015                       "`num_parallel_calls` argument is specified.")
-> 2016       return MapDataset(self, map_func, preserve_cardinality=True, name=name)
   2017     else:
   2018       return ParallelMapDataset(

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, input_dataset, map_func, use_inter_op_parallelism, preserve_cardinality, use_legacy_function, name)
   5193         self._transformation_name(),
   5194         dataset=input_dataset,
-> 5195         use_legacy_function=use_legacy_function)
   5196     self._metadata = dataset_metadata_pb2.Metadata()
   5197     if name:

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/data/ops/structured_function.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
    269         fn_factory = trace_tf_function(defun_kwargs)
    270 
--> 271     self._function = fn_factory()
    272     # There is no graph to add in eager mode.
    273     add_to_graph &= not context.executing_eagerly()

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/eager/function.py in get_concrete_function(self, *args, **kwargs)
   3069     """
   3070     graph_function = self._get_concrete_function_garbage_collected(
-> 3071         *args, **kwargs)
   3072     graph_function._garbage_collector.release()  # pylint: disable=protected-access
   3073     return graph_function

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   3034       args, kwargs = None, None
   3035     with self._lock:
-> 3036       graph_function, _ = self._maybe_define_function(args, kwargs)
   3037       seen_names = set()
   3038       captured = object_identity.ObjectIdentitySet(

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3290 
   3291           self._function_cache.add_call_context(cache_key.call_context)
-> 3292           graph_function = self._create_graph_function(args, kwargs)
   3293           self._function_cache.add(cache_key, cache_key_deletion_observer,
   3294                                    graph_function)

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3138             arg_names=arg_names,
   3139             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3140             capture_by_value=self._capture_by_value),
   3141         self._function_attributes,
   3142         function_spec=self.function_spec,

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes, acd_record_initial_resource_uses)
   1159         _, original_func = tf_decorator.unwrap(python_func)
   1160 
-> 1161       func_outputs = python_func(*func_args, **func_kwargs)
   1162 
   1163       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/data/ops/structured_function.py in wrapped_fn(*args)
    246           attributes=defun_kwargs)
    247       def wrapped_fn(*args):  # pylint: disable=missing-docstring
--> 248         ret = wrapper_helper(*args)
    249         ret = structure.to_tensor_list(self._output_structure, ret)
    250         return [ops.convert_to_tensor(t) for t in ret]

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/data/ops/structured_function.py in wrapper_helper(*args)
    175       if not _should_unpack(nested_args):
    176         nested_args = (nested_args,)
--> 177       ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
    178       if _should_pack(ret):
    179         ret = tuple(ret)

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    687       try:
    688         with conversion_ctx:
--> 689           return converted_call(f, args, kwargs, options=options)
    690       except Exception as e:  # pylint:disable=broad-except
    691         if hasattr(e, 'ag_error_metadata'):

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in converted_call(f, args, kwargs, caller_fn_scope, options)
    375 
    376   if not options.user_requested and conversion.is_allowlisted(f):
--> 377     return _call_unconverted(f, args, kwargs, options)
    378 
    379   # internal_convert_user_code is for example turned off when issuing a dynamic

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in _call_unconverted(f, args, kwargs, options, update_cache)
    456 
    457   if kwargs is not None:
--> 458     return f(*args, **kwargs)
    459   return f(*args)
    460 

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/keras/layers/core/dense.py in build(self, input_shape)
    137     last_dim = tf.compat.dimension_value(input_shape[-1])
    138     if last_dim is None:
--> 139       raise ValueError('The last dimension of the inputs to a Dense layer '
    140                        'should be defined. Found None. '
    141                        f'Full input shape received: {input_shape}')

ValueError: Exception encountered when calling layer "sequential_5" (type Sequential).

The last dimension of the inputs to a Dense layer should be defined. Found None. Full input shape received: <unknown>

Call arguments received:
  • inputs=tf.Tensor(shape=(None,), dtype=string)
  • training=None
  • mask=None

I am not quite sure where I should set the shape - as using regular tensors and not TF dataset works ok.

ANSWER

Answered 2022-Mar-10 at 08:57

You will have to explicitly set the shapes of the tensors coming from tf.py_functions. Using None will allow variable input lengths. The Bert output dimension (384,) is, however, necessary:

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings

    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        input_ids.set_shape(tf.TensorShape((None, None)))
        token_type_ids.set_shape(tf.TensorShape((None, None)))
        attention_mask.set_shape(tf.TensorShape((None, None)))

        model_output = self.process(input_ids, token_type_ids, attention_mask)
        model_output[0].set_shape(tf.TensorShape((None, None, 384)))
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings

    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
outputs = tf.keras.layers.Dense(32)(outputs)
model = tf.keras.Model(inputs, outputs)
print(model(tf.constant(['some text', 'more text'])))
print(model.summary())

tf.Tensor(
[[-0.06719425 -0.02954631 -0.05811356 -0.1456391  -0.13001677  0.00145465
   0.0401044   0.05949172 -0.02589339  0.07255618 -0.00958113  0.01159782
   0.02508018  0.03075579 -0.01910635 -0.03231853  0.00875124  0.01143366
  -0.04365401 -0.02090197  0.07030752 -0.02872834  0.10535908  0.05691438
  -0.017165   -0.02044982  0.02580127 -0.04564123 -0.0631128  -0.00303708
   0.00133517  0.01613527]
 [-0.11922387  0.02304137 -0.02670465 -0.13117084 -0.11492493  0.03961402
   0.08129141 -0.05999354  0.0039564   0.02892766  0.00493046  0.00440936
  -0.07966737  0.11354238  0.03141225  0.00048972  0.04658606 -0.03658888
  -0.05292419 -0.04639702  0.08445395  0.00522146  0.04359548  0.0290177
  -0.02171512 -0.03399373 -0.00418095 -0.04019783 -0.04733383 -0.03972956
   0.01890458 -0.03927581]], shape=(2, 32), dtype=float32)
Model: "model_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_18 (InputLayer)       [(None, 1)]               0         
                                                                 
 s_bert_17 (SBert)           (None, 384)               22713216  
                                                                 
 dense_78 (Dense)            (None, 32)                12320     
                                                                 
=================================================================
Total params: 22,725,536
Trainable params: 22,725,536
Non-trainable params: 0
_________________________________________________________________
None

Source https://stackoverflow.com/questions/71414627

QUESTION

Use `sentence-transformers` inside of a keras model

Asked 2022-Mar-09 at 17:10

I would like to use a model from sentence-transformers inside of a larger Keras model.

Here is the full example:

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            return self.tokenizer(
                inputs, padding=True, truncation=True, return_tensors='tf'
            )
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int64])
    
    def mean_pooling(model_output, attention_mask):
        token_embeddings = model_output[0]
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), token_embeddings.shape),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings

    def call(self, inputs):
        encoded_input = self.tf_encode(inputs)
        model_output = self.model(encoded_input)
        embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
        return embeddings
    
    
sbert = SBert(tokenizer, model)
sbert(['some text', 'more text'])

I am able to use the model and tokenizer outside of TF / Keras with no problems, the issue seems to happen when we try and build the graph and TF passing a symbolic tensor to the tokenizer, generating an error - this is why I have tried to wrap in tf.py_function but with no success...

The error:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-20-a0c4a906e456> in <module>
     44 
     45 sbert = SBert(tokenizer, model)
---> 46 sbert(['some text', 'more text'])

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

<ipython-input-20-a0c4a906e456> in call(self, inputs)
     36     def call(self, inputs):
     37         tf.print(inputs, output_stream=sys.stdout)
---> 38         encoded_input = self.tf_encode(inputs)
     39         tf.print(encoded_input, output_stream=sys.stdout)
     40         model_output = self.model(encoded_input)

<ipython-input-20-a0c4a906e456> in tf_encode(self, inputs)
     20                 inputs, padding=True, truncation=True, return_tensors='tf'
     21             )
---> 22         return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int64])
     23 
     24     def mean_pooling(model_output, attention_mask):

InvalidArgumentError: Exception encountered when calling layer "s_bert_6" (type SBert).

ValueError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
Traceback (most recent call last):

  File "/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 269, in __call__
    return func(device, token, args)

  File "/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 147, in __call__
    outputs = self._call(device, args)

  File "/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 154, in _call
    ret = self._func(*args)

  File "/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
    return func(*args, **kwargs)

  File "<ipython-input-20-a0c4a906e456>", line 20, in encode
    inputs, padding=True, truncation=True, return_tensors='tf'

  File "/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2378, in __call__
    "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) "

ValueError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).

 [Op:EagerPyFunc]

Call arguments received:
  • inputs=["'some text'", "'more text'"]

ANSWER

Answered 2022-Mar-09 at 17:10

tf.py_function does not seem to work with a dict output that’s why you can try returning three separate tensors. Also, I am decoding the inputs to remove the b in the front of each string:

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel


MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)


class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x.decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0]
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.model({'input_ids': input_ids, 'token_type_ids': token_type_ids, 'attention_mask': attention_mask})
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
sbert(['some text', 'more text'])

If you want to use a Keras model, you will have to do something like this:

import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

MODEL_PATH = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = TFAutoModel.from_pretrained(MODEL_PATH, from_pt=True)

class SBert(tf.keras.layers.Layer):
    def __init__(self, tokenizer, model):
        super(SBert, self).__init__()
        
        self.tokenizer = tokenizer
        self.model = model
        
    def tf_encode(self, inputs):
        def encode(inputs):
            inputs = [x[0].decode("utf-8") for x in inputs.numpy()]
            outputs = self.tokenizer(inputs, padding=True, truncation=True, return_tensors='tf')
            return outputs['input_ids'], outputs['token_type_ids'], outputs['attention_mask']
        return tf.py_function(func=encode, inp=[inputs], Tout=[tf.int32, tf.int32, tf.int32])
    
    def process(self, i, t, a):
      def __call(i, t, a):
        model_output = self.model({'input_ids': i.numpy(), 'token_type_ids': t.numpy(), 'attention_mask': a.numpy()})
        return model_output[0]
      return tf.py_function(func=__call, inp=[i, t, a], Tout=[tf.float32])

    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = tf.squeeze(tf.stack(model_output), axis=0)
        input_mask_expanded = tf.cast(
            tf.broadcast_to(tf.expand_dims(attention_mask, -1), tf.shape(token_embeddings)),
            tf.float32
        )
        a = tf.math.reduce_sum(token_embeddings * input_mask_expanded, axis=1)
        b = tf.clip_by_value(tf.math.reduce_sum(input_mask_expanded, axis=1), 1e-9, tf.float32.max)
        embeddings = a / b
        embeddings, _ = tf.linalg.normalize(embeddings, 2, axis=1)
        return embeddings
    def call(self, inputs):
        input_ids, token_type_ids, attention_mask = self.tf_encode(inputs)
        model_output = self.process(input_ids, token_type_ids, attention_mask)
        embeddings = self.mean_pooling(model_output, attention_mask)
        return embeddings
    
    
sbert = SBert(tokenizer, model)
inputs = tf.keras.layers.Input((1,), dtype=tf.string)
outputs = sbert(inputs)
model = tf.keras.Model(inputs, outputs)
model(tf.constant(['some text', 'more text']))

TensorShape([2, 384]).shape

Source https://stackoverflow.com/questions/71411065

QUESTION

Using sentence transformers with limited access to internet

Asked 2022-Jan-19 at 13:27

I have access to the latest packages but I cannot access internet from my python enviroment.

Package versions that I have are as below

huggingface-hub-0.4.0 sacremoses-0.0.47 tokenizers-0.10.3 transformers-4.15.0
sentence-transformers-2.1.0 sentencepiece-0.1.96 torchvision-0.11.2

print (torch.__version__)
1.10.1+cu102

I went to the location and copied all the files in a folder

os.listdir('multi-qa-mpnet-base-dot-v1_Jan2022/')

['config_sentence_transformers.json',
 'config.json',
 'gitattributes',
 'modules.json',
 'data_config.json',
 'sentence_bert_config.json',
 'README.md',
 'special_tokens_map.json',
 'tokenizer_config.json',
 'train_script.py',
 'vocab.txt',
 'tokenizer.json',
 '1_Pooling',
 '.ipynb_checkpoints',
 '9e1e76b7a067f72e49c7f571cd8e811f7a1567bec49f17e5eaaea899e7bc2c9e']

Then I went to the url and tried to execute the code listed there

But I get below error

model = SentenceTransformer('multi-qa-mpnet-base-dot-v1_Jan2022/')

OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index', 'flax_model.msgpack'] found in directory multi-qa-mpnet-base-dot-v1_Jan2022/ or `from_tf` and `from_flax` set to False.

Where could I get those 4 files ('pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index', 'flax_model.msgpack') or what else do I need to change? These files are not available at the first URL mentioned above

ANSWER

Answered 2022-Jan-19 at 13:27

Based on the things you mentioned, I checked the source code of sentence-transformers on Google Colab. After running the model and getting the files, I check the directory and I saw the pytorch_model.bin there.

And according to sentence-transformers code: Link

the flax_model.msgpack , rust_model.ot, tf_model.h5 are getting ignored when the it is trying to download.

and these are the files that it downloads :

['1_Pooling', 'config_sentence_transformers.json', 'tokenizer.json', 'tokenizer_config.json', 'modules.json', 'sentence_bert_config.json', 'pytorch_model.bin', 'special_tokens_map.json', 'config.json', 'train_script.py', 'data_config.json', 'README.md', '.gitattributes', 'vocab.txt']

The only thing that you have to have to load the model is pytorch_model.bin file. I tested with copying the modules to another directory and it worked. And according to your question, you haven't downloaded this file, so that is the problem.

All in all, you should download the model using its command and then move the files to another directory and initialize the SentenceTransformer class with that dir.

I wish it would be helpful.

Source https://stackoverflow.com/questions/70716702

QUESTION

Using Sentence-Bert with other features in scikit-learn

Asked 2021-Oct-16 at 12:47

I have a dataset, one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. Can I replace the text column with tensors? And, how can I train the model. The code below is how I transform the text into vectors.

model = SentenceTransformer('sentence-transformers/LaBSE')
sentence_embeddings = model.encode(X_train['tweet'], convert_to_tensor=True, show_progress_bar=True)
sentence_embeddings1 = model.encode(X_test['tweet'], convert_to_tensor=True, show_progress_bar=True)

ANSWER

Answered 2021-Oct-16 at 12:47

Let's assume this is your data

X_train = pd.DataFrame({
    'tweet':['foo', 'foo', 'bar'],
    'feature1':[1, 1, 0],
    'feature2':[1, 0, 1],
})
y_train = [1, 1, 0]

and you are willing to use it with sklearn API (cross-validation, pipeline, grid-search, and so on). There is a utility named ColumnTransformer which can map pandas data frames to the desired data using user-defined arbitrary functions! what you have to do is define a function and create an official sklearn.transformer from it.

model = SentenceTransformer('mrm8488/bert-tiny-finetuned-squadv2') # model named is changed for time and computation gians :)
embedder = FunctionTransformer(lambda item:model.encode(item, convert_to_tensor=True, show_progress_bar=False).detach().cpu().numpy())

After that, you would be able to use the transformer like any other transformer and map your text column into semantic space, like:

preprocessor = ColumnTransformer(
    transformers=[('embedder', embedder, 'tweet')],
    remainder='passthrough'
    )
X_train = preprocessor.fit_transform(X_train) # X_train.shape => (len(df), your_transformer_model_hidden_dim + your_features_count)

X_train would be the data you wanted. It's proper to use with sklearn ecosystem.

gnb = GaussianNB()
gnb.fit(X_train, y_train)

output: GaussianNB(priors=None, var_smoothing=1e-09)

caveat: Numerical features and the tweets embeddings should belong to the same SCALE otherwise some would dominate others and degrade the performance

Source https://stackoverflow.com/questions/69585176

QUESTION

python packages not being installed on the virtual environment using ubuntu

Asked 2021-Aug-18 at 18:11

I have a requirements.txt file which holds all information of my python packages I need for my Flask application. Here is what I did:

python3 -m venv venv
source venv/bin/activate
sudo pip install -r requirements.txt

When I tried to check if the packages were installed on the virtual environment using pip list, I do not see the packages. Can someone tell what went wrong?

(venv) ubuntu@ip-xxx-xx-x-xxx:/var/www/html/flaskapp$ ls
__pycache__  application.py  flaskapp.wsgi  requirements.txt  utils.py  venv
(venv) ubuntu@ip-xxx-xx-x-xxx:/var/www/html/flaskapp$ sudo pip install -r requirements.txt
Requirement already satisfied: certifi==2021.5.30 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 1)) (2021.5.30)
Requirement already satisfied: charset-normalizer==2.0.4 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 2)) (2.0.4)
Requirement already satisfied: click==8.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (8.0.1)
Requirement already satisfied: filelock==3.0.12 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 4)) (3.0.12)
Requirement already satisfied: Flask==2.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 5)) (2.0.1)
Requirement already satisfied: huggingface-hub==0.0.12 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 6)) (0.0.12)
Requirement already satisfied: idna==3.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 7)) (3.2)
Requirement already satisfied: itsdangerous==2.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 8)) (2.0.1)
Requirement already satisfied: Jinja2==3.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 9)) (3.0.1)
Requirement already satisfied: joblib==1.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 10)) (1.0.1)
Requirement already satisfied: MarkupSafe==2.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 11)) (2.0.1)
Requirement already satisfied: nltk==3.6.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 12)) (3.6.2)
Requirement already satisfied: numpy==1.21.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 13)) (1.21.1)
Requirement already satisfied: packaging==21.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 14)) (21.0)
Requirement already satisfied: Pillow==8.3.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 15)) (8.3.1)
Requirement already satisfied: pyparsing==2.4.7 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 16)) (2.4.7)
Requirement already satisfied: PyYAML==5.4.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 17)) (5.4.1)
Requirement already satisfied: regex==2021.8.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 18)) (2021.8.3)
Requirement already satisfied: requests==2.26.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 19)) (2.26.0)
Requirement already satisfied: sacremoses==0.0.45 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 20)) (0.0.45)
Requirement already satisfied: scikit-learn==0.24.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 21)) (0.24.2)
Requirement already satisfied: scipy==1.7.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 22)) (1.7.1)
Requirement already satisfied: sentence-transformers==2.0.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 23)) (2.0.0)
Requirement already satisfied: sentencepiece==0.1.96 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 24)) (0.1.96)
Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 25)) (1.16.0)
Requirement already satisfied: textblob==0.15.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 26)) (0.15.3)
Requirement already satisfied: threadpoolctl==2.2.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 27)) (2.2.0)
Requirement already satisfied: tokenizers==0.10.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 28)) (0.10.3)
Requirement already satisfied: torch==1.9.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 29)) (1.9.0)
Requirement already satisfied: torchvision==0.10.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 30)) (0.10.0)
Requirement already satisfied: tqdm==4.62.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 31)) (4.62.0)
Requirement already satisfied: transformers==4.9.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 32)) (4.9.2)
Requirement already satisfied: typing-extensions==3.10.0.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 33)) (3.10.0.0)
Requirement already satisfied: urllib3==1.26.6 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 34)) (1.26.6)
Requirement already satisfied: Werkzeug==2.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 35)) (2.0.1)
(venv) ubuntu@ip-xxx-xx-x-xxx:/var/www/html/flaskapp$ pip list
Package       Version
------------- -------
pip           20.0.2 
pkg-resources 0.0.0  
setuptools    44.0.0 
(venv) ubuntu@ip-xxx-xx-x-xxx:/var/www/html/flaskapp$

ANSWER

Answered 2021-Aug-18 at 18:05

If you want to use python3+ to install the packages try to use pip3 install package_name

And to solve the errno 13 try to add --user at the end

pip3 install package_name --user

let me know if it works

Source https://stackoverflow.com/questions/68837021

QUESTION

AttributeError: Caught AttributeError in DataLoader worker process 0. - fine tuning pre-trained transformer model

Asked 2021-Aug-17 at 13:41

can anyone help me to resolve this error?

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-aaa58b106c77> in <module>()
     25     output_path='fine_tuned_bert',
     26     save_best_model= True,
---> 27     show_progress_bar= True
     28     )

4 frames
/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
    423             # have message field
    424             raise self.exc_type(message=msg)
--> 425         raise self.exc_type(msg)
    426 
    427 

AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/SentenceTransformer.py", line 518, in smart_batching_collate
    num_texts = len(batch[0].texts)
AttributeError: 'str' object has no attribute 'texts'

Code:

import pandas as pd
# initialise data of lists.
data = {'input':[
          "Alpro, Cioccolato bevanda a base di soia 1 ltr", #Alpro, Chocolate soy drink 1 ltr
          "Milka  cioccolato al latte 100 g", #Milka milk chocolate 100 g
          "Danone, HiPRO 25g Proteine gusto cioccolato 330 ml", #Danone, HiPRO 25g Protein chocolate flavor 330 ml
         ]
        }
 
# Creates pandas DataFrame.
x_sample = pd.DataFrame(data)
print(x_sample['input'])

# load model
from sentence_transformers import SentenceTransformer, SentencesDataset, InputExample, losses, evaluation
from torch.utils.data import DataLoader

embedder = SentenceTransformer('sentence-transformers/paraphrase-xlm-r-multilingual-v1') # or any other pretrained model
print("embedder loaded...")

# define your train dataset, the dataloader, and the train loss
train_dataset = SentencesDataset(x_sample["input"].tolist(), embedder)
train_dataloader = DataLoader(train_dataset, shuffle=False, batch_size=4, num_workers=1)
train_loss = losses.CosineSimilarityLoss(embedder)

# dummy evaluator to make the api work
sentences1 = ['latte al cioccolato', 'latte al cioccolato','latte al cioccolato']
sentences2 = ['Alpro, Cioccolato bevanda a base di soia 1 ltr', 'Danone, HiPRO 25g Proteine gusto cioccolato 330 ml','Milka  cioccolato al latte 100 g']
scores = [0.99,0.95,0.4]
evaluator = evaluation.EmbeddingSimilarityEvaluator(sentences1, sentences2, scores)

# tune the model
embedder.fit(train_objectives=[(train_dataloader, train_loss)], 
    epochs=5, 
    warmup_steps=500, 
    evaluator=evaluator, 
    evaluation_steps=1,
    output_path='fine_tuned_bert',
    save_best_model= True,
    show_progress_bar= True
    )

ANSWER

Answered 2021-Aug-17 at 13:41

[Updated] I skimmed several lines of documentation here about how to use the fit() method and I realized there is a simpler solution to do what you desired. The only changes you need to consider are to define proper InputExample for constructing a DataLoader & create a loss!

import pandas as pd
# initialise data of lists.
data = {'input':[
          "Alpro, Cioccolato bevanda a base di soia 1 ltr", #Alpro, Chocolate soy drink 1 ltr
          "Milka  cioccolato al latte 100 g", #Milka milk chocolate 100 g
          "Danone, HiPRO 25g Proteine gusto cioccolato 330 ml", #Danone, HiPRO 25g Protein chocolate flavor 330 ml
         ]
        }
 
# Creates pandas DataFrame.
x_sample = pd.DataFrame(data)
print(x_sample['input'])

# load model
from sentence_transformers import SentenceTransformer, SentencesDataset, InputExample, losses, evaluation
from torch.utils.data import DataLoader

embedder = SentenceTransformer('sentence-transformers/paraphrase-xlm-r-multilingual-v1') # or any other pretrained model
print("embedder loaded...")

# define your train dataset, the dataloader, and the train loss
# train_dataset = SentencesDataset(x_sample["input"].tolist(), embedder)
# train_dataloader = DataLoader(train_dataset, shuffle=False, batch_size=4, num_workers=1)
# train_loss = losses.CosineSimilarityLoss(embedder)

# dummy evaluator to make the api work
sentences1 = ['latte al cioccolato', 'latte al cioccolato','latte al cioccolato']
sentences2 = ['Alpro, Cioccolato bevanda a base di soia 1 ltr', 'Danone, HiPRO 25g Proteine gusto cioccolato 330 ml','Milka  cioccolato al latte 100 g']
scores = [0.99,0.95,0.4]
evaluator = evaluation.EmbeddingSimilarityEvaluator(sentences1, sentences2, scores)

examples = []
for s1,s2,l in zip(sentences1, sentences2, scores):
  examples.append(InputExample(texts=[s1, s2], label=l))
train_dataloader = DataLoader(examples, shuffle=False, batch_size=4, num_workers=1)
train_loss = losses.CosineSimilarityLoss(embedder)
# tune the model
embedder.fit(train_objectives=[(train_dataloader, train_loss)], 
    epochs=5, 
    warmup_steps=500, 
    evaluator=evaluator, 
    evaluation_steps=1,
    output_path='fine_tuned_bert',
    save_best_model= True,
    show_progress_bar= True
    )

Source https://stackoverflow.com/questions/68760136

QUESTION

How to increase dimension-vector size of BERT sentence-transformers embedding

Asked 2021-Aug-15 at 13:35

I am using sentence-transformers for semantic search but sometimes it does not understand the contextual meaning and returns wrong result eg. BERT problem with context/semantic search in italian language

by default the vector side of embedding of the sentence is 78 columns, so how do I increase that dimension so that it can understand the contextual meaning in deep.

code:

# Load the BERT Model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-base-nli-mean-tokens')

# Setup a Corpus
# A corpus is a list with documents split by sentences.

sentences = ['Absence of sanity', 
             'Lack of saneness',
             'A man is eating food.',
             'A man is eating a piece of bread.',
             'The girl is carrying a baby.',
             'A man is riding a horse.',
             'A woman is playing violin.',
             'Two men pushed carts through the woods.',
             'A man is riding a white horse on an enclosed ground.',
             'A monkey is playing drums.',
             'A cheetah is running behind its prey.']

# Each sentence is encoded as a 1-D vector with 78 columns 
sentence_embeddings = model.encode(sentences) ### how to increase vector dimention 

print('Sample BERT embedding vector - length', len(sentence_embeddings[0]))

print('Sample BERT embedding vector - note includes negative values', sentence_embeddings[0])

ANSWER

Answered 2021-Aug-10 at 07:39

Increasing the dimension of a trained model is not possible (without many difficulties and re-training the model). The model you are using was pre-trained with dimension 768, i.e., all weight matrices of the model have a corresponding number of trained parameters. Increasing the dimensionality would mean adding parameters which however need to be learned.

Also, the dimension of the model does not reflect the amount of semantic or context information in the sentence representation. The choice of the model dimension reflects more a trade-off between model capacity, the amount of training data, and reasonable inference speed.

If the model that you are using does not provide representation that is semantically rich enough, you might want to search for better models, such as RoBERTa or T5.

Source https://stackoverflow.com/questions/68686272

QUESTION

BERT problem with context/semantic search in italian language

Asked 2021-Aug-09 at 19:33

I am using BERT model for context search in Italian language but it does not understand the contextual meaning of the sentence and returns wrong result.

in below example code when I compare "milk with chocolate flavour" with two other type of milk and one chocolate so it returns high similarity with chocolate. it should return high similarity with other milks.

can anyone suggest me any improvement on the below code so that it can return semantic results?

Code :

!python -m spacy download it_core_news_lg
!pip install sentence-transformers


import scipy
import numpy as np
from sentence_transformers import models, SentenceTransformer
model = SentenceTransformer('distiluse-base-multilingual-cased') # workes with Arabic, Chinese, Dutch, English, French, German, Italian, Korean, Polish, Portuguese, Russian, Spanish, Turkish

corpus = [
          "Alpro, Cioccolato bevanda a base di soia 1 ltr", #Alpro, Chocolate soy drink 1 ltr(soya milk)
          "Milka  cioccolato al latte 100 g", #Milka milk chocolate 100 g
          "Danone, HiPRO 25g Proteine gusto cioccolato 330 ml", #Danone, HiPRO 25g Protein chocolate flavor 330 ml(milk with chocolate flabor)
         ]
corpus_embeddings = model.encode(corpus)


queries = [
            'latte al cioccolato', #milk with chocolate flavor,
          ]
query_embeddings = model.encode(queries)


# Calculate Cosine similarity of query against each sentence i
closest_n = 10
for query, query_embedding in zip(queries, query_embeddings):
    distances = scipy.spatial.distance.cdist([query_embedding], corpus_embeddings, "cosine")[0]

    results = zip(range(len(distances)), distances)
    results = sorted(results, key=lambda x: x[1])

    print("\n======================\n")
    print("Query:", query)
    print("\nTop 10 most similar sentences in corpus:")

    for idx, distance in results[0:closest_n]:
        print(corpus[idx].strip(), "(Score: %.4f)" % (1-distance))

Output :

======================

Query: latte al cioccolato

Top 10 most similar sentences in corpus:
Milka  cioccolato al latte 100 g (Score: 0.7714)
Alpro, Cioccolato bevanda a base di soia 1 ltr (Score: 0.5586)
Danone, HiPRO 25g Proteine gusto cioccolato 330 ml (Score: 0.4569)

ANSWER

Answered 2021-Aug-09 at 19:33

The problem is not with your code, it is just the insufficient model performance.

There are a few things you can do. First, you can try Universal Sentence Encoder (USE). From my experience their embeddings are a little bit better, at least in English.

Second, you can try a different model, for example sentence-transformers/xlm-r-distilroberta-base-paraphrase-v1. It is based on ROBERTa and might give a better performance.

Now you can combine together embeddings from several models (just by concatenating the representations). In some cases it helps, on expense of much heavier compute.

And finally you can create your own model. It is well known that single language models perform significantly better than multilingual ones. You can follow the guide and train your own Italian model.

Source https://stackoverflow.com/questions/68627093

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install sentence-transformers

We recommend Python 3.6 or higher, PyTorch 1.6.0 or higher and transformers v4.6.0 or higher. The code does not work with Python 2.7.
See Quickstart in our documenation. This example shows you how to use an already trained Sentence Transformer model to embed sentences for another task. First download a pretrained model. Then provide some sentences to the model. And that's it already. We now have a list of numpy arrays with the embeddings.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

DOWNLOAD this Library from

Reuse Trending Solutions

14 best Python Telegram Bot

21 best C# Game Development

Build AI Fake News Detector

Build Text Summarizer in Python

Getting Started with Web3

24 best Python Blockchain

26 best Python Face Recognition

Build Ludo Game in Python

Build Location Based AR App

Build AI Study Assistant Bot

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items

Find more libraries

Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Explore Related Topics

Artificial Intelligence

Natural Language Processing

Share this Page

Reuse Pre-built Kits with sentence-transformers

Build NLP Paraphrase Generator

BuildWithAI Challenge 1

Arqum-Kit

Food Wastage Reducer

Health Monitor

See all related Kits

Reuse Natural Language Processing Kits

Build AI Virtual Assistant

Build NLP Paraphrase Generator

19 best Python NLP

15 best Python Topic Modelling

11 best Java Ethereum

See all related Kits

Reuse Artificial Intelligence Kits

Build Text Summarizer in Python

26 best Python Face Recognition

Build AI Study Assistant Bot

Resolving Gender Bias in AI

Stop words : NLP

See all related Kits

Consider Popular Natural Language Processing Libraries

transformers

by huggingface

funNLP

by fighting41love

bert

by google-research

Python

by geekcomputers

HanLP

by hankcs

See all Natural Language Processing Libraries

Try Top Libraries by UKPLab

emnlp2017-bilstm-cnn-crf

by UKPLab Python

deeplearning4nlp-tutorial

by UKPLab Python

EasyNMT

by UKPLab Python

elmo-bilstm-cnn-crf

by UKPLab Python

emnlp2017-relation-extraction

by UKPLab Python

See all Libraries by this author

Compare Natural Language Processing Libraries with Highest Support

transformers

by huggingface

bert

by google-research

allennlp

by allenai

text

by pytorch

flair

by flairNLP

See all Natural Language Processing Libraries

Compare Natural Language Processing Libraries with Highest Quality

nadia

by mmberg

MinorThird

by TeamCohen

baseqa

by oaqa

SmithHeilmann_fork

by kutschkem

wikientities

by xiaoganghan

See all Natural Language Processing Libraries

Compare Natural Language Processing Libraries with Highest Security

stanbol-freeling

by insideout10

CLAVIN-NERD

by Novetta

nadia

by mmberg

coreNlp

by jconwell

corenlp-examples

by drewfarris

See all Natural Language Processing Libraries

Compare Natural Language Processing Libraries with Permissive License

nadia

by mmberg

coreNlp

by jconwell

LrcParser

by authorfu

allsummarizer

by kariminf

Xponents

by OpenSextant

See all Natural Language Processing Libraries

Compare Natural Language Processing Libraries with Highest Reuse

elasticsearch-analysis-decompound

by jprante

elasticsearch-analysis-turkishstemmer

by skroutz

semantic

by crm416

treetools

by wmaier

esperanto-analyzer

by fidelisrafael

See all Natural Language Processing Libraries

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items

Find more libraries

Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

© 2022 Open Weaver Inc.
Terms
Privacy policy
Contact us

© 2022 Open Weaver Inc.