textclassification | partially refactored version of the ArgMining Repository | Natural Language Processing library
kandi X-RAY | textclassification Summary
kandi X-RAY | textclassification Summary
This is a partially refactored version of the ArgMining Repository and the code for the argument mining tasks that were researched by Matthias Liebeck as part of his PhD thesis.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Reads all scores files and returns a list of scores
- Reads the results of the finished evaluation
- Read job_parameters
- Print the fasttext results
- Filter fasttext results
- Print the table of the subtask tables
- Group subtask results by feature and classifier
- Filter results by lda path
- Filter subtask results by a given feature and classifier
- Get the LSTM model
- Combine meta parameters
- Get the combination of a dictionary
- Process a text document
- Argument parser
- Loads a TensorFlow dataset
- Print fasttext results
- Reduce training set
- Print the best fit results
- Generate an embedding layer
- Get a list of finished jobs
- Create a new classifier
- Compute word2vec embedding
- Get model parameters
- Return a list of finished jobs
- Return the combination of two word embedding features
- Group results by subtask
textclassification Key Features
textclassification Examples and Code Snippets
Community Discussions
Trending Discussions on textclassification
QUESTION
I'm trying to learn how to use some ML stuff for Android. I got the Text Classification demo working and seems to work fine. So then I tried creating my own model.
The code I used to create my own model was this:
...ANSWER
Answered 2021-May-27 at 15:50In your codes you trained a MobileBERT model, but saved to the path of average_word_vec? spec = model_spec.get('mobilebert_classifier') model.export(export_dir='average_word_vec')
One posssiblity is: you use the model of average_word_vec, but add MobileBERT metadata, thus the preprocessing doesn't match.
Could you follow the Model Maker tutorial and try again? https://colab.sandbox.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/model_maker_text_classification.ipynb Make sure change the export path.
QUESTION
this is probably a very basic question, but i struggle to get the math right. I have a list with arrays of different sizes. The shapes look like so:
...ANSWER
Answered 2021-Jan-14 at 10:04You need to specify the padding for each edge of each dimension. The padding size is a fixed difference to the shape, thus you have to adapt it to the "missing" size:
QUESTION
I'm trying to train a network for a textclassification where the texts are labeled with 6 different categories. Each text can have only one label. So far I built the following simple network:
...ANSWER
Answered 2021-Jan-08 at 20:07In your last layer
QUESTION
I have a TensorFlow model that I built (a 1D CNN) that I would now like to implement into .NET.
In order to do so I need to know the Input and Output nodes.
When I uploaded the model on Netron I get a different graph depending on my save method and the only one that looks correct comes from an h5 upload. Here is the model.summary():
If I save the model as an h5 model.save("Mn_pb_model.h5") and load that into the Netron to graph it, everything looks correct:
However, ML.NET will not accept h5 format so it needs to be saved as a pb.
In looking through samples of adopting TensorFlow in ML.NET, this sample shows a TensorFlow model that is saved in a similar format to the SavedModel format - recommended by TensorFlow (and also recommended by ML.NET here "Download an unfrozen [SavedModel format] ..."). However when saving and loading the pb file into Netron I get this:
And zoomed in a little further (on the far right side),
As you can see, it looks nothing like it should.
Additionally the input nodes and output nodes are not correct so it will not work for ML.NET (and I think something is wrong).
I am using the recommended way from TensorFlow to determine the Input / Output nodes:
When I try to obtain a frozen graph and load it into Netron, at first it looks correct, but I don't think that it is:
There are four reasons I do not think this is correct.
- it is very different from the graph when it was uploaded as an h5 (which looks correct to me).
- as you can see from earlier, I am using 1D convolutions throughout and this is showing that it goes to 2D (and remains that way).
- this file size is 128MB whereas the one in the TensorFlow to ML.NET example is only 252KB. Even the Inception model is only 56MB.
- if I load the Inception model in TensorFlow and save it as an h5, it looks the same as from the ML.NET resource, yet when I save it as a frozen graph it looks different. If I take the same model and save it in the recommended
SavedModelformat, it shows up all messed up in Netron. Take any model you want and save it in the recommendedSavedModelformat and you will see for yourself (I've tried it on a lot of different models).
Additionally in looking at the model.summary() of Inception with it's graph, it is similar to its graph in the same way my model.summary() is to the h5 graph.
It seems like there should be an easier way (and a correct way) to save a TensorFlow model so it can be used in ML.NET.
Please show that your suggested solution works: In the answer that you provide, please check that it works (load the pb model [this should also have a Variables folder in order to work for ML.NET] into Netron and show that it is the same as the h5 model, e.g., screenshot it). So that we are all trying the same thing, here is a link to a MNIST ML crash course example. It takes less than 30s to run the program and produces a model called my_model. From here you can save it according to your method and upload it to see the graph on Netron. Here is the h5 model upload:
ANSWER
Answered 2020-Nov-24 at 10:57This answer is made of 3 parts:
- going through other programs
- NOT going through other programs
- Difference between op-level graph and conceptual graph (and why Netron show you different graphs)
1. Going through other programs:
ML.net needs an ONNX model, not a pb file.
There is several ways to convert your model from TensorFlow to an ONNX model you could load in ML.net :
- With WinMLTools tools: https://docs.microsoft.com/en-us/windows/ai/windows-ml/convert-model-winmltools
- With MMdnn: https://github.com/microsoft/MMdnn
- With tf2onnx: https://github.com/onnx/tensorflow-onnx
- If trained with Keras, with keras2onnx: https://github.com/onnx/keras-onnx
This SO post could help you too: Load model with ML.NET saved with keras
And here you will find more informations on the h5 and pb files formats, what they contain, etc.: https://www.tensorflow.org/guide/keras/save_and_serialize#weights_only_saving_in_savedmodel_format
2. But you are asking "TensorFlow -> ML.NET without going through other programs":
2.A An overview of the problem:
First, the pl file format you made using the code you provided from seems, from what you say, to not be the same as the one used in the example you mentionned in comment (https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/text-classification-tf)
Could to try to use the pb file that will be generated via tf.saved_model.save ? Is it working ?
A thought about this microsoft blog post:
From this page we can read:
In ML.NET you can load a frozen TensorFlow model .pb file (also called “frozen graph def” which is essentially a serialized graph_def protocol buffer written to disk)
and:
That TensorFlow .pb model file that you see in the diagram (and the labels.txt codes/Ids) is what you create/train in Azure Cognitive Services Custom Vision then exporte as a frozen TensorFlow model file to be used by ML.NET C# code.
So, this pb file is a type of file generated from Azure Cognitive Services Custom Vision.
Perharps you could try this way too ?
2.B Now, we'll try to provide the solution:
In fact, in TensorFlow 1.x you could save a frozen graph easily, using freeze_graph.
But TensorFlow 2.x does not support freeze_graph and converter_variables_to_constants.
You could read some usefull informations here too: Tensorflow 2.0 : frozen graph support
Some users are wondering how to do in TF 2.x: how to freeze graph in tensorflow 2.0 (https://github.com/tensorflow/tensorflow/issues/27614)
There are some solutions however to create the pb file you could load in ML.net as you want:
https://leimao.github.io/blog/Save-Load-Inference-From-TF2-Frozen-Graph/
How to save Keras model as frozen graph? (already linked in your question though)
Difference between op-level graph and conceptual graph (and why Netron show you different graphs):As @mlneural03 said in a comment to you question, Netron shows a different graph depending on what file format you give:
- If you load a h5 file, Netron wil display the conceptual graph
- If you load a pb file, Netron wil display the op-level graph
What is the difference between a op-level graph and a conceptual graph ?
- In TensorFlow, the nodes of the op-level graph represent the operations ("ops"), like tf.add , tf.matmul , tf.linalg.inv, etc.
- The conceptual graph will show you your your model's structure.
That's completely different things.
"ops" is an abbreviation for "operations". Operations are nodes that perform the computations.
So, that's why you get a very large graph with a lot of nodes when you load the pb fil in Netron: you see all the computation nodes of the graph. but when you load the h5 file in Netron, you "just" see your model's tructure, the design of your model.
In TensorFlow, you can view your graph with TensorBoard:
- By default, TensorBoard displays the op-level graph.
- To view the coneptual graph, in TensorBoard, select the "keras" tag.
There is a Jupyter Notebook that explains very clearly the difference between the op-level graph and the coneptual graph here: https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/graphs.ipynb
You can also read this "issue" on the TensorFlow Github too, related to your question: https://github.com/tensorflow/tensorflow/issues/39699
In a nutshell:In fact there is no problem, just a little misunderstanding (and that's OK, we can't know everything).
You would like to see the same graphs when loading the h5 file and the pb file in Netron, but it has to be unsuccessful, because the files does not contains the same graphs. These graphs are two ways of displaying the same model.
The pb file created with the method we described will be the correct pb file to load whith ML.NET, as described in the Microsoft's tutorial we talked about. SO, if you load you correct pb file as described in these tutorials, you wil load your real/true model.
QUESTION
I am making quiz app for maths formulas. And because of the limitations, I am using "mathview extends webview" I want to show formulas in quiz options. For that I am using webview for each four options in quiz, so I want to make whole webview clickable like a button. How can I do this?
Below is my mathview code and here is my mainactivity
...ANSWER
Answered 2020-Apr-13 at 12:00You can treat MathView as any other view and setOnClickListener on it.
within onCreate() of your activity use:-
QUESTION
I am new to Tensor flow and machine learning. Here I am trying to create a text classification of my own. I am facing below issue. I am getting below error while loading tflite model on Android.
...ANSWER
Answered 2020-Jan-18 at 16:17This link helped me resolve this issue.
Setting converter.experimental_new_converter =True will help
QUESTION
I am getting the following warnings in the pre-launch report from Google Play.
I am at a loss as to how to correct these. Any help or recommendations is appreciated i am having lots of issues here
...ANSWER
Answered 2019-May-13 at 03:42These warnings refers to usage of restricted non-SDK interfaces. https://developer.android.com/distribute/best-practices/develop/restrictions-non-sdk-interfaces
These calls may lead to incorrect behavior or app's crashes. It is recommended to avoid them. All usages belong to blacklist, greylist or whitelist. If you can`t get rid of these usages, check affiliation to list. Only blacklisted calls lead to crashes. Also, just to remind, Android Q (targetSDK=29) has updated blacklist https://developer.android.com/preview/non-sdk-q
QUESTION
I'm developing a Scala feature extracting app using Apache Spark TF-IDF. I need to read in from a directory of text files. I'm trying to convert an RDD to a dataframe but I'm getting the error "value toDF() is not a member of org.apache.spark.rdd.RDD[streamedRDD]". This is what I have right now ...
I have spark-2.2.1 & Scala 2.1.11. Thanks in advance.
Code:
...ANSWER
Answered 2018-Mar-16 at 19:22The problem here is that map function returns a type of Dataset[Row] which you assign to tweetsDF. It should be:
QUESTION
I have a JSON object and I am iterating through it. I am using different values from different levels of it.
But I am not able to create path dynamically to reiterate the object.
...ANSWER
Answered 2019-Feb-19 at 13:27Thank you T.J. Crowder. I modified code as following and it is working now.
QUESTION
def TextClassification(a):
import pandas as pd
df = pd.read_excel('../Desktop/Stage/Classeur1.xlsx')
#Removing NULL element
df=df[pd.notnull(df['Réclamation'])]
df=df[pd.notnull(df['Catégorie'])]
#Removing punctuation
df['Réclamation'] = [''.join(c for c in s if c not in string.punctuation) for s in df['Réclamation']]
df['Catégorie'] = [''.join(c for c in s if c not in string.punctuation) for s in df['Catégorie']]
#Removing é è and all numbers
ch = ['0','1','2','3','4','5','6','7','8','9']
for c in ch:
df['Réclamation'] = [w.replace(c, '') for w in df['Réclamation']]
df['Catégorie'] = [w.replace(c, '') for w in df['Catégorie']]
df['Réclamation']= [w.replace('è', 'e') for w in df['Réclamation']]
df['Réclamation']= [w.replace('é', 'e') for w in df['Réclamation']]
df['Catégorie']= [w.replace('è', 'e') for w in df['Catégorie']]
df['Catégorie']= [w.replace('é', 'e') for w in df['Catégorie']]
#Lower case
df['Réclamation']=df['Réclamation'].apply(lambda x: " ".join(x.lower() for x in x.split()))
df['Catégorie']=df['Catégorie'].apply(lambda x: " ".join(x.lower() for x in x.split()))
df['category_id'] = df['Catégorie'].factorize()[0]
category_id_df = df[['Catégorie', 'category_id']].sort_values('category_id')
category_to_id = dict(category_id_df.values)
id_to_category = dict(category_id_df[['category_id', 'Catégorie']].values)
#library for data preprocessing and maodel building
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.svm import LinearSVC
#split data(train/test)
X_train, X_test, y_train, y_test = train_test_split(df['Réclamation'], df['Catégorie'], random_state = 0)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = LinearSVC().fit(X_train_tfidf, y_train)
return(print(clf.predict(count_vect.transform(a)))
ANSWER
Answered 2019-Feb-10 at 01:11You are missing a )
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install textclassification
You can use textclassification like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
