kandi background
Explore Kits

trident-ml | Trident-ML : A realtime online machine | Machine Learning library

 by   pmerienne Java Version: Current License: No License

 by   pmerienne Java Version: Current License: No License

Download this library from

kandi X-RAY | trident-ml Summary

trident-ml is a Java library typically used in Artificial Intelligence, Machine Learning, Deep Learning, Pytorch applications. trident-ml has no bugs, it has no vulnerabilities, it has build file available and it has low support. You can download it from GitHub, Maven.
Trident-ML is based on Trident, a high-level abstraction for doing realtime computing. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar. It's recommended to read the Storm and Trident documentation.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • trident-ml has a low active ecosystem.
  • It has 385 star(s) with 90 fork(s). There are 44 watchers for this library.
  • It had no major release in the last 12 months.
  • There are 1 open issues and 11 have been closed. On average issues are closed in 63 days. There are 1 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of trident-ml is current.
trident-ml Support
Best in #Machine Learning
Average in #Machine Learning
trident-ml Support
Best in #Machine Learning
Average in #Machine Learning

quality kandi Quality

  • trident-ml has 0 bugs and 0 code smells.
trident-ml Quality
Best in #Machine Learning
Average in #Machine Learning
trident-ml Quality
Best in #Machine Learning
Average in #Machine Learning

securitySecurity

  • trident-ml has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • trident-ml code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.
trident-ml Security
Best in #Machine Learning
Average in #Machine Learning
trident-ml Security
Best in #Machine Learning
Average in #Machine Learning

license License

  • trident-ml does not have a standard license declared.
  • Check the repository for any license declaration and review the terms closely.
  • Without a license, all rights are reserved, and you cannot use the library in your applications.
trident-ml License
Best in #Machine Learning
Average in #Machine Learning
trident-ml License
Best in #Machine Learning
Average in #Machine Learning

buildReuse

  • trident-ml releases are not available. You will need to build from source code and install.
  • Deployable package is available in Maven.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
trident-ml Reuse
Best in #Machine Learning
Average in #Machine Learning
trident-ml Reuse
Best in #Machine Learning
Average in #Machine Learning
Top functions reviewed by kandi - BETA

kandi has reviewed trident-ml and discovered the below as its top functions. This is intended to give you an instant insight into trident-ml implemented functionality, and help decide if they suit your requirements.

  • Initialize the centroids .
    • Retrieves the labels for the given tuple .
      • Tokenizes the text .
        • Compares this object with another instance .
          • Updates the model .
            • Returns the value of the given feature .
              • Preprocess the given tweet .
                • Calculates the Euclidean distance between documents .
                  • Update state and statistics for the stream .
                    • Creates the feature statistics object .

                      Get all kandi verified functions for this library.

                      Get all kandi verified functions for this library.

                      trident-ml Key Features

                      Trident-ML : A realtime online machine learning library

                      Create instances

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      toppology
                        // Emit tuples with 2 random features (named x0 and x1) and an associated boolean label (named label)
                        .newStream("randomFeatures", new RandomFeaturesSpout())
                        
                        // Transform trident tuple to instance
                        .each(new Fields("label", "x0", "x1"), new InstanceCreator<Boolean>(), new Fields("instance"));
                      

                      Supervised classification

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      // Create perceptron state from labeled instances stream
                      TridentState perceptronModel = toppology
                        // Emit tuple with a labeled instance of enhanced NAND features
                        // i.e. : {label=true, features=[1.0 0.0 1.0]} or {label=false, features=[1.0 1.0 1.0]}  
                        .newStream("nandsamples", new NANDSpout())
                      				
                        // Update perceptron
                        .partitionPersist(new MemoryMapState.Factory(), new Fields("instance"), new ClassifierUpdater<Boolean>("perceptron", new PerceptronClassifier()));
                      
                      // Classify instance from a DRPC stream
                      toppology.newDRPCStream("predict", localDRPC)
                        // Transform DRPC ARGS to unlabeled instance
                        .each(new Fields("args"), new DRPCArgsToInstance(), new Fields("instance"))
                      
                        // Classify instance using perceptron state
                        .stateQuery(perceptronModel, new Fields("instance"), new ClassifyQuery<Boolean>("perceptron"), new Fields("prediction"));
                      

                      Unsupervised classification

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      // Training stream
                      TridentState kmeansState = toppology
                        // Emit tuples with a instance containing an integer as label and 3 double features named (x0, x1 and x2)
                        .newStream("samples", new RandomFeaturesForClusteringSpout())
                      
                        // Convert trident tuple to instance
                        .each(new Fields("label", "x0", "x1", "x2"), new InstanceCreator<Integer>(), new Fields("instance"))
                      
                        // Update a 3 classes kmeans
                        .partitionPersist(new MemoryMapState.Factory(), new Fields("instance"), new ClusterUpdater("kmeans", new KMeans(3)));
                      
                      // Cluster stream
                      toppology.newDRPCStream("predict", localDRPC)
                        // Convert DRPC args to instance
                        .each(new Fields("args"), new DRPCArgsToInstance(), new Fields("instance"))
                      
                        // Query kmeans to classify instance
                        .stateQuery(kmeansState, new Fields("instance"), new ClusterQuery("kmeans"), new Fields("prediction"));
                      

                      Stream statistics

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      // Update stream statistics
                      TridentState streamStatisticsState = toppology
                        // emit tuples with random features
                        .newStream("randomFeatures", new RandomFeaturesSpout())
                      
                        // Transform trident tuple to instance
                        .each(new Fields("x0", "x1"), new InstanceCreator(), new Fields("instance"))
                      
                        // Update stream statistics
                        .partitionPersist(new MemoryMapState.Factory(), new Fields("instance"), new StreamStatisticsUpdater("randomFeaturesStream", StreamStatistics.fixed()));
                      
                      // Query stream statistics (with DRPC)
                      toppology.newDRPCStream("queryStats", localDRPC)
                        // Query stream statistics
                        .stateQuery(streamStatisticsState, new StreamStatisticsQuery("randomFeaturesStream"), new Fields("streamStats"));
                      
                      

                      Preprocessing data

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      toppology
                        // Emit tuples with 2 random features (named x0 and x1) and an associated boolean label (named label)
                        .newStream("randomFeatures", new RandomFeaturesSpout())
                      
                        // Convert trident tuple to instance
                        .each(new Fields("label", "x0", "x1"), new InstanceCreator<Boolean>(), new Fields("instance"))
                      	  
                        // Scales features to unit norm
                        .each(new Fields("instance"), new Normalizer(), new Fields("scaledInstance"));
                      

                      Pre-trained classifier

                      copy iconCopydownload iconDownload
                      TridentTopology toppology = new TridentTopology();
                      
                      // Classification stream
                      toppology.newDRPCStream("classify", localDRPC)
                        // Query classifier with text instance
                        .each(new Fields("args"), new TwitterSentimentClassifier(), new Fields("sentiment"));
                      

                      Maven integration :

                      copy iconCopydownload iconDownload
                      <repositories>
                      	<repository>
                      		<id>clojars.org</id>
                      		<url>http://clojars.org/repo</url>
                      	</repository>
                      </repositories>
                      
                      <dependency>
                      	<groupId>com.github.pmerienne</groupId>
                      	<artifactId>trident-ml</artifactId>
                      	<version>0.0.4</version>
                      </dependency>
                      

                      Community Discussions

                      Trending Discussions on Machine Learning
                      • Using RNN Trained Model without pytorch installed
                      • Flux.jl : Customizing optimizer
                      • How can I check a confusion_matrix after fine-tuning with custom datasets?
                      • CUDA OOM - But the numbers don't add upp?
                      • How to compare baseline and GridSearchCV results fair?
                      • Getting Error 524 while running jupyter lab in google cloud platform
                      • TypeError: brain.NeuralNetwork is not a constructor
                      • Ordinal Encoding or One-Hot-Encoding
                      • How to increase dimension-vector size of BERT sentence-transformers embedding
                      • How to identify what features affect predictions result?
                      Trending Discussions on Machine Learning

                      QUESTION

                      Using RNN Trained Model without pytorch installed

                      Asked 2022-Feb-28 at 20:17

                      I have trained an RNN model with pytorch. I need to use the model for prediction in an environment where I'm unable to install pytorch because of some strange dependency issue with glibc. However, I can install numpy and scipy and other libraries. So, I want to use the trained model, with the network definition, without pytorch.

                      I have the weights of the model as I save the model with its state dict and weights in the standard way, but I can also save it using just json/pickle files or similar.

                      I also have the network definition, which depends on pytorch in a number of ways. This is my RNN network definition.

                      import torch
                      import torch.nn as nn
                      import torch.nn.functional as F
                      import torch.optim as optim
                      import random
                      
                      torch.manual_seed(1)
                      random.seed(1)
                      device = torch.device('cpu')
                      
                      class RNN(nn.Module):
                        def __init__(self, input_size, hidden_size, output_size,num_layers, matching_in_out=False, batch_size=1):
                          super(RNN, self).__init__()
                          self.input_size = input_size
                          self.hidden_size = hidden_size
                          self.output_size = output_size
                          self.num_layers = num_layers
                          self.batch_size = batch_size
                          self.matching_in_out = matching_in_out #length of input vector matches the length of output vector 
                          self.lstm = nn.LSTM(input_size, hidden_size,num_layers)
                          self.hidden2out = nn.Linear(hidden_size, output_size)
                          self.hidden = self.init_hidden()
                        def forward(self, feature_list):
                          feature_list=torch.tensor(feature_list)
                          
                          if self.matching_in_out:
                            lstm_out, _ = self.lstm( feature_list.view(len( feature_list), 1, -1))
                            output_space = self.hidden2out(lstm_out.view(len( feature_list), -1))
                            output_scores = torch.sigmoid(output_space) #we'll need to check if we need this sigmoid
                            return output_scores #output_scores
                          else:
                            for i in range(len(feature_list)):
                              cur_ft_tensor=feature_list[i]#.view([1,1,self.input_size])
                              cur_ft_tensor=cur_ft_tensor.view([1,1,self.input_size])
                              lstm_out, self.hidden = self.lstm(cur_ft_tensor, self.hidden)
                              outs=self.hidden2out(lstm_out)
                            return outs
                        def init_hidden(self):
                          #return torch.rand(self.num_layers, self.batch_size, self.hidden_size)
                          return (torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device),
                                  torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device))
                      

                      I am aware of this question, but I'm willing to go as low level as possible. I can work with numpy array instead of tensors, and reshape instead of view, and I don't need a device setting.

                      Based on the class definition above, what I can see here is that I only need the following components from torch to get an output from the forward function:

                      • nn.LSTM
                      • nn.Linear
                      • torch.sigmoid

                      I think I can easily implement the sigmoid function using numpy. However, can I have some implementation for the nn.LSTM and nn.Linear using something not involving pytorch? Also, how will I use the weights from the state dict into the new class?

                      So, the question is, how can I "translate" this RNN definition into a class that doesn't need pytorch, and how to use the state dict weights for it? Alternatively, is there a "light" version of pytorch, that I can use just to run the model and yield a result?

                      EDIT

                      I think it might be useful to include the numpy/scipy equivalent for both nn.LSTM and nn.linear. It would help us compare the numpy output to torch output for the same code, and give us some modular code/functions to use. Specifically, a numpy equivalent for the following would be great:

                      rnn = nn.LSTM(10, 20, 2)
                      input = torch.randn(5, 3, 10)
                      h0 = torch.randn(2, 3, 20)
                      c0 = torch.randn(2, 3, 20)
                      output, (hn, cn) = rnn(input, (h0, c0))
                      

                      and also for linear:

                      m = nn.Linear(20, 30)
                      input = torch.randn(128, 20)
                      output = m(input)
                      

                      ANSWER

                      Answered 2022-Feb-17 at 10:47

                      You should try to export the model using torch.onnx. The page gives you an example that you can start with.

                      An alternative is to use TorchScript, but that requires torch libraries.

                      Both of these can be run without python. You can load torchscript in a C++ application https://pytorch.org/tutorials/advanced/cpp_export.html

                      ONNX is much more portable and you can use in languages such as C#, Java, or Javascript https://onnxruntime.ai/ (even on the browser)

                      A running example

                      Just modifying a little your example to go over the errors I found

                      Notice that via tracing any if/elif/else, for, while will be unrolled

                      import torch
                      import torch.nn as nn
                      import torch.nn.functional as F
                      import torch.optim as optim
                      import random
                      
                      torch.manual_seed(1)
                      random.seed(1)
                      device = torch.device('cpu')
                      
                      class RNN(nn.Module):
                        def __init__(self, input_size, hidden_size, output_size,num_layers, matching_in_out=False, batch_size=1):
                          super(RNN, self).__init__()
                          self.input_size = input_size
                          self.hidden_size = hidden_size
                          self.output_size = output_size
                          self.num_layers = num_layers
                          self.batch_size = batch_size
                          self.matching_in_out = matching_in_out #length of input vector matches the length of output vector 
                          self.lstm = nn.LSTM(input_size, hidden_size,num_layers)
                          self.hidden2out = nn.Linear(hidden_size, output_size)
                        def forward(self, x, h0, c0):
                          lstm_out, (hidden_a, hidden_b) = self.lstm(x, (h0, c0))
                          outs=self.hidden2out(lstm_out)
                          return outs, (hidden_a, hidden_b)
                        def init_hidden(self):
                          #return torch.rand(self.num_layers, self.batch_size, self.hidden_size)
                          return (torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device).detach(),
                                  torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device).detach())
                      
                      # convert the arguments passed during onnx.export call
                      class MWrapper(nn.Module):
                          def __init__(self, model):
                              super(MWrapper, self).__init__()
                              self.model = model;
                          def forward(self, kwargs):
                              return self.model(**kwargs)
                      

                      Run an example

                      rnn = RNN(10, 10, 10, 3)
                      X = torch.randn(3,1,10)
                      h0,c0  = rnn.init_hidden()
                      print(rnn(X, h0, c0)[0])
                      

                      Use the same input to trace the model and export an onnx file

                      
                      torch.onnx.export(MWrapper(rnn), {'x':X,'h0':h0,'c0':c0}, 'rnn.onnx', 
                                        dynamic_axes={'x':{1:'N'},
                                                     'c0':{1: 'N'},
                                                     'h0':{1: 'N'}
                                                     },
                                        input_names=['x', 'h0', 'c0'],
                                        output_names=['y', 'hn', 'cn']
                                       )
                      

                      Notice that you can use symbolic values for the dimensions of some axes of some inputs. Unspecified dimensions will be fixed with the values from the traced inputs. By default LSTM uses dimension 1 as batch.

                      Next we load the ONNX model and pass the same inputs

                      import onnxruntime
                      ort_model = onnxruntime.InferenceSession('rnn.onnx')
                      print(ort_model.run(['y'], {'x':X.numpy(), 'c0':c0.numpy(), 'h0':h0.numpy()}))
                      

                      Source https://stackoverflow.com/questions/71146140

                      Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                      Vulnerabilities

                      No vulnerabilities reported

                      Install trident-ml

                      You can download it from GitHub, Maven.
                      You can use trident-ml like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the trident-ml component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

                      Support

                      Storm allows trident-ml to process batches of tuples in a distributed way (batches will be computed among several nodes). This means that trident-ml can scale horizontally with workload. However Storm prevents state updates to append simultaneously and the model learning is done in a state update. That's why, the learning step can't be distributed. Thankfully this lack of parallelization isn't a real bottle neck because the incremental algorithms are very fast (and simple!). Distributed algorithms will not be implemented in trident-ml, the whole design prevents this. So you can't achieve distributed learning however but you can still partition the streams to pre-process/enrich your data in a distributed manner.

                      DOWNLOAD this Library from

                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      Share this Page

                      share link
                      Consider Popular Machine Learning Libraries
                      Try Top Libraries by pmerienne
                      Compare Machine Learning Libraries with Highest Support
                      Compare Machine Learning Libraries with Highest Quality
                      Compare Machine Learning Libraries with Highest Security
                      Compare Machine Learning Libraries with Permissive License
                      Compare Machine Learning Libraries with Highest Reuse
                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      • © 2022 Open Weaver Inc.