kandi background
Explore Kits

deeplearning4j | Model import deployment framework | Machine Learning library

 by   eclipse Java Version: Current License: Apache-2.0

 by   eclipse Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | deeplearning4j Summary

deeplearning4j is a Java library typically used in Institutions, Learning, Education, Artificial Intelligence, Machine Learning, Deep Learning, Spark applications. deeplearning4j has build file available, it has a Permissive License and it has high support. However deeplearning4j has 1424 bugs and it has 3 vulnerabilities. You can download it from GitHub, Maven.
Deeplearning4J has quite a few dependencies. For this reason we only support usage with a build tool. Add these dependencies to your pom.xml file to use Deeplearning4J with the CPU backend. A full standalone project example is available in the example repository, if you want to start a new Maven project from scratch.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • deeplearning4j has a highly active ecosystem.
  • It has 12434 star(s) with 4926 fork(s). There are 780 watchers for this library.
  • It had no major release in the last 12 months.
  • There are 762 open issues and 4831 have been closed. On average issues are closed in 59 days. There are 3 open pull requests and 0 closed requests.
  • It has a positive sentiment in the developer community.
  • The latest version of deeplearning4j is current.
deeplearning4j Support
Best in #Machine Learning
Average in #Machine Learning
deeplearning4j Support
Best in #Machine Learning
Average in #Machine Learning

quality kandi Quality

  • deeplearning4j has 1424 bugs (107 blocker, 60 critical, 825 major, 432 minor) and 33169 code smells.
deeplearning4j Quality
Best in #Machine Learning
Average in #Machine Learning
deeplearning4j Quality
Best in #Machine Learning
Average in #Machine Learning

securitySecurity

  • deeplearning4j has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • deeplearning4j code analysis shows 3 unresolved vulnerabilities (3 blocker, 0 critical, 0 major, 0 minor).
  • There are 525 security hotspots that need review.
deeplearning4j Security
Best in #Machine Learning
Average in #Machine Learning
deeplearning4j Security
Best in #Machine Learning
Average in #Machine Learning

license License

  • deeplearning4j is licensed under the Apache-2.0 License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
deeplearning4j License
Best in #Machine Learning
Average in #Machine Learning
deeplearning4j License
Best in #Machine Learning
Average in #Machine Learning

buildReuse

  • deeplearning4j releases are not available. You will need to build from source code and install.
  • Deployable package is available in Maven.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
  • deeplearning4j saves you 1371853 person hours of effort in developing the same functionality from scratch.
  • It has 601973 lines of code, 61116 functions and 6123 files.
  • It has medium code complexity. Code complexity directly impacts maintainability of the code.
deeplearning4j Reuse
Best in #Machine Learning
Average in #Machine Learning
deeplearning4j Reuse
Best in #Machine Learning
Average in #Machine Learning
Top functions reviewed by kandi - BETA

kandi has reviewed deeplearning4j and discovered the below as its top functions. This is intended to give you an instant insight into deeplearning4j implemented functionality, and help decide if they suit your requirements.

  • Extract op descriptors .
  • Process a class and add the op names to the result set .
  • Check for workspace namespaces .
  • Generate a tensor with the shape matrices for each dimension
  • Create a grad function based on the input variables .
  • Helper function to perform the activation of the neural network
  • Get the output of all layers inside the layer .
  • Helper method to compute backprop gradients for training .
  • Helper method to get the outputs for a particular output frame
  • Gets the system info .

deeplearning4j Key Features

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.

Using Eclipse Deeplearning4J in your project

copy iconCopydownload iconDownload
<dependencies>
  <dependency>
      <groupId>org.deeplearning4j</groupId>
      <artifactId>deeplearning4j-core</artifactId>
      <version>1.0.0-M1.1</version>
  </dependency>
  <dependency>
      <groupId>org.nd4j</groupId>
      <artifactId>nd4j-native-platform</artifactId>
      <version>1.0.0-M1.1</version>
  </dependency>
</dependencies>

A taste of code

copy iconCopydownload iconDownload
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
                .seed(seed)
                .l2(0.0005)
                .weightInit(WeightInit.XAVIER)
                .updater(new Adam(1e-3))
                .list()
                .layer(new ConvolutionLayer.Builder(5, 5)
                        .stride(1,1)
                        .nOut(20)
                        .activation(Activation.IDENTITY)
                        .build())
                .layer(new SubsamplingLayer.Builder(PoolingType.MAX)
                        .kernelSize(2,2)
                        .stride(2,2)
                        .build())
                .layer(new ConvolutionLayer.Builder(5, 5)
                        .stride(1,1)
                        .nOut(50)
                        .activation(Activation.IDENTITY)
                        .build())
                .layer(new SubsamplingLayer.Builder(PoolingType.MAX)
                        .kernelSize(2,2)
                        .stride(2,2)
                        .build())
                .layer(new DenseLayer.Builder().activation(Activation.RELU)
                        .nOut(500).build())
                .layer(new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                        .nOut(outputNum)
                        .activation(Activation.SOFTMAX)
                        .build())
                .setInputType(InputType.convolutionalFlat(28,28,1))
                .build();

Building from source

copy iconCopydownload iconDownload
./change-cuda-versions.sh x.x
./change-scala-versions.sh 2.xx
./change-spark-versions.sh x
mvn clean install -Dmaven.test.skip -Dlibnd4j.cuda=x.x -Dlibnd4j.compute=xx

Different predictions for the same data

copy iconCopydownload iconDownload
import org.nd4j.linalg.dataset.api.preprocessor.serializer.NormalizerSerializer; 
NormalizerSerializer SUT = NormalizerSerializer.getDefault(); 

SUT.write(normalizer,new File("outputFile.bin")); 

NormalizeStandardize restored = SUT.restore(new File("outputFile.bin");

DL4J-Image become too bright

copy iconCopydownload iconDownload
// We should also specify the relative monitor/camera response for proper gamma correction.
CanvasFrame frame = new CanvasFrame("Some Title", CanvasFrame.getDefaultGamma()/grabber.getGamma());

What is the output of the dl4j lstm neural network?

copy iconCopydownload iconDownload
 [   a    | b |   c   |   d    | e |     f     ] 
0.0          0.3              0.5             1.0

Deeplearning4j - how to use saved model?

copy iconCopydownload iconDownload
int batchSize = 150;
DataSetIterator iterator = new RecordReaderDataSetIterator(recordReader, batchSize);
DataSet allData = iterator.next();

DataNormalization normalizer = new NormalizerStandardize();
normalizer.fit(allData);
normalizer.transform(allData);

INDArray output = model.output(allData.getFeatures());

// Output
System.out.println(output);

BUILD FAILURE while installing DL4J

copy iconCopydownload iconDownload
  git clone https://github.com/eclipse/deeplearning4j-examples
      cd deeplearning4j-examples/dl4j-examples
   mvn clean install -DskipTests
-----------------------
  git clone https://github.com/eclipse/deeplearning4j-examples
      cd deeplearning4j-examples/dl4j-examples
   mvn clean install -DskipTests
-----------------------
  git clone https://github.com/eclipse/deeplearning4j-examples
      cd deeplearning4j-examples/dl4j-examples
   mvn clean install -DskipTests

DeepLearning4J Doc2Vec input structure

copy iconCopydownload iconDownload
    ArrayList<LabelledDocument> labelledDocumentList = new ArrayList<LabelledDocument>();

    for(Document input : documents){
      LabelledDocument doc = new LabelledDocument();
      doc.setContent(input.content);
      doc.addLabel(input.label);
      labelledDocumentList.add(doc);
    }
    
    SimpleLabelAwareIterator iter = new simpleLabelAwareIterator(labelledDocumentList);
    
    TokenizerFactory t = new UimaTokenizerFactory();
    ParagraphVectors vec = new ParagraphVectors.Builder()
                        .minWordFrequency(1)
                        .labels(Arrays.asList("A", "B"))
                        .layerSize(100)
                        .stopWords(new ArrayList<String>())
                        .windowSize(5).iterate(iter).tokenizerFactory(t).build();
    
    vec.fit();
    tools.saveObject(vec, "models/modelName");

Visualizing training progresss in deeplearning4j throws error when starting server

copy iconCopydownload iconDownload
12:39:05.487 [vert.x-eventloop-thread-0] INFO org.deeplearning4j.ui.VertxUIServer - Deeplearning4j UI server started at: http://localhost:9000` 
12:39:05.490 [main] INFO org.deeplearning4j.ui.VertxUIServer - StatsStorage instance attached to UI: InMemoryStatsStorage(uid=bd548909)
12:39:05.803 [Thread-5] INFO org.deeplearning4j.ui.VertxUIServer - Deeplearning4j UI server is auto-stopping after thread (name: main) died.

Neural Network returning NaN as output

copy iconCopydownload iconDownload
 f[j + i * 7] = (rows[j][i] == 0 ? .5f : rows[j][i] == 1 ? 0f : 1f);
rows[j][i] == 1 ? Float.MIN_VALUE : 1f;

rows[j][i] == 1 ?  Float.MIN_NORMAL : Float.MAX_VALUE/2;

rows[j][i] == 1 ? -Float.MAX_VALUE/2 : Float.MAX_VALUE/2;
new NeuralNetConfiguration
  .Builder()
  .weightInit(WeightInit.XAVIER)
  (...)
  .layer(new DenseLayer.Builder().nIn(42).nOut(30).activation(Activation.RELU)
        .weightInit(WeightInit.XAVIER)
        .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer) //this   
        .build())
  
  (...)
-----------------------
 f[j + i * 7] = (rows[j][i] == 0 ? .5f : rows[j][i] == 1 ? 0f : 1f);
rows[j][i] == 1 ? Float.MIN_VALUE : 1f;

rows[j][i] == 1 ?  Float.MIN_NORMAL : Float.MAX_VALUE/2;

rows[j][i] == 1 ? -Float.MAX_VALUE/2 : Float.MAX_VALUE/2;
new NeuralNetConfiguration
  .Builder()
  .weightInit(WeightInit.XAVIER)
  (...)
  .layer(new DenseLayer.Builder().nIn(42).nOut(30).activation(Activation.RELU)
        .weightInit(WeightInit.XAVIER)
        .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer) //this   
        .build())
  
  (...)
-----------------------
 f[j + i * 7] = (rows[j][i] == 0 ? .5f : rows[j][i] == 1 ? 0f : 1f);
rows[j][i] == 1 ? Float.MIN_VALUE : 1f;

rows[j][i] == 1 ?  Float.MIN_NORMAL : Float.MAX_VALUE/2;

rows[j][i] == 1 ? -Float.MAX_VALUE/2 : Float.MAX_VALUE/2;
new NeuralNetConfiguration
  .Builder()
  .weightInit(WeightInit.XAVIER)
  (...)
  .layer(new DenseLayer.Builder().nIn(42).nOut(30).activation(Activation.RELU)
        .weightInit(WeightInit.XAVIER)
        .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer) //this   
        .build())
  
  (...)
-----------------------
float[] f = new float[7 * 6];
Nd4j.create(f);
float[][] f = new float[1][7 * 6];
Nd4j.createFromArray(f);
-----------------------
float[] f = new float[7 * 6];
Nd4j.create(f);
float[][] f = new float[1][7 * 6];
Nd4j.createFromArray(f);

DeepLearning4J Problems with INDArray

copy iconCopydownload iconDownload
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
            .seed(randSeed).optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
            .updater(new Nesterovs(0.1, 0.9)).list()
            .layer(new DenseLayer.Builder().nIn(numRows * numColums).nOut(63).activation(Activation.RELU)
                    .weightInit(WeightInit.XAVIER).build())
            .layer(new DenseLayer.Builder().nIn(63).nOut(63).activation(Activation.RELU)
                    .weightInit(WeightInit.XAVIER).build())
            .layer(new OutputLayer.Builder(LossFunction.NEGATIVELOGLIKELIHOOD).nIn(numRows * numColums).nOut(7)
                    .activation(Activation.SOFTMAX).weightInit(WeightInit.XAVIER).build())
            .setInputType(InputType.feedForward(numRows * numColums))
            .build();

How to split a DataSetIterator into testing and training iterator?

copy iconCopydownload iconDownload
List<DataSet> trainList = new ArrayList<>();
List<DataSet> testList= new ArrayList<>();

while (yourDataSetIterator.hasNext())
{
    DataSet ds = yourDataSetIterator.next();
    SplitTestAndTrain splData = ds.splitTestAndTrain(0.5); //half for each         
    DataSet trainDs = splData.getTrain();
    trainList.add(trainDs);
    DataSet testDs  = splData.getTest();
    testList.add(testDs);
    (...)
}

Iterator<DataSet> trainIterator = trainList.iterator(); 
Iterator<DataSet> testIterator  = testList.iterator(); 

Community Discussions

Trending Discussions on deeplearning4j
  • Could not initialize class org.nd4j.linalg.factory.Nd4j in docker container
  • Different predictions for the same data
  • DL4J-Image become too bright
  • Reduce DeepLearning4j dependency size of exported jar
  • What is the output of the dl4j lstm neural network?
  • Deeplearning4j - how to iterate multiple DataSets for large data?
  • Deeplearning4j - how to use saved model?
  • Maven: NDJ4 Has Backend, Error Says It Doesn't
  • Ask About The &quot;default&quot; Size of Vocabulary in Word2Vec in Deeplearning4j Library
  • BUILD FAILURE while installing DL4J
Trending Discussions on deeplearning4j

QUESTION

Could not initialize class org.nd4j.linalg.factory.Nd4j in docker container

Asked 2022-Feb-09 at 11:28

I am trying to import a KERAS file in a docker container with a program that has the following java code sample:

String weights = getFile(_modelFile).getAbsolutePath();
String modelConfiguration = getFile(_configFile).getAbsolutePath();
_model = KerasModelImport.importKerasSequentialModelAndWeights(modelConfiguration, weights);

I am using Docker with WSL2 and the docker container has the following OS:
dsasad

If I run the program in WSL2 it works fine, but if I run it in the docker container I get the following error:

2022-01-11 09:34:36,124 [ListenerHTTP-46 ] WARN xpto.jaf.protocols.json.JSONServlet JAF_G1000 [] - xpto.jaf.JafError: Failed to invoke operation method - java.lang.NoClassDefFoundError - java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j at xpto.jaf.services.operations.MethodOperationManager$JavaMethodOperation.execute(MethodOperationManager.java:851) at xpto.jaf.services.operations.WrappedOperation.execute(WrappedOperation.java:141) at xpto.jaf.services.sessions.SessionBase.execute(SessionBase.java:263) at xpto.jaf.protocols.json.JSONServlet.processOperation(JSONServlet.java:217) at xpto.jaf.protocols.json.JSONServlet.doPost(JSONServlet.java:161) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j at org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.readDataSet(Hdf5Archive.java:295) at org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.readDataSet(Hdf5Archive.java:109) at org.deeplearning4j.nn.modelimport.keras.utils.KerasModelUtils.importWeights(KerasModelUtils.java:284) at org.deeplearning4j.nn.modelimport.keras.KerasSequentialModel.(KerasSequentialModel.java:151) at org.deeplearning4j.nn.modelimport.keras.KerasSequentialModel.(KerasSequentialModel.java:57) at org.deeplearning4j.nn.modelimport.keras.utils.KerasModelBuilder.buildSequential(KerasModelBuilder.java:326) at org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasSequentialModelAndWeights(KerasModelImport.java:296) at xpto.ml.model.ModelH5.getModel(ModelH5.java:327) at xpto.ml.model.ModelH5.validateH5Files(ModelH5.java:238) at xpto.ml.operations.H5Operations.validateH5Model(H5Operations.java:53) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at xpto.jaf.services.operations.MethodOperationManager$JavaMethodOperation.execute(MethodOperationManager.java:841) ... 36 common frames omitted

Nd4j is a Dl4j dependency and it is present in both executions which makes me think the problem could be the Docker itself or the OS on the docker container.

Hope to have explained my problem well. Thanks in advance.

EDIT: Both versions of Dl4j and Nd4j are 1.0.0-M1.1

ANSWER

Answered 2022-Jan-11 at 11:59

This still doesn't show a cause. NoClassDeffFoundErrors are usually related to a clashing dependencies. You could have different versions of dl4j/nd4j on your classpath but I doubt it. Most of the time this is the side effect of a native dependency crash somehow.

Of note here:

I wouldn't recommend running the keras converter (or any model import process) in line. I would recommend converting the models separately. This is mainly for performance reasons

Whatever your problem is there are usually a few:

  1. glibc version with hdf5. Keras import uses hdf5 underneath the covers which means c code.

  2. Nd4j native dependency crash: this is also usually glibc related. We load nd4j in to memory to create and set native arrays (which means more java calling in to c++) that then can trigger a crash depending on what OS you're running on

  3. Another hdf5 error: this could be an invalid model or some hdf5 version error.

In any case, we would need more information before we can help you. Whatever you're reporting here isn't enough. Could you mention your docker container OS and what version of dl4j/nd4j is bundled here?

Edit: I see it's oracle linux 7 which is effectively RHEL/Centos. If you're using docker I would recommend a newer image maybe.

Beyond that if it is an nd4j related crash (still not verifiable from your stack trace) if you are using the latest version you might be seeing a crash due to glibc version.

If so there was a recent update to the nd4j classifiers you can find here: https://repo1.maven.org/maven2/org/nd4j/nd4j-native/1.0.0-M1.1/

Older glibcs need to use linux-x86_64-compat as a migration path

Source https://stackoverflow.com/questions/70666185

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install deeplearning4j

You can download it from GitHub, Maven.
You can use deeplearning4j like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the deeplearning4j component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For support for the project, please go over to https://community.konduit.ai/.

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Share this Page

share link
Compare Machine Learning Libraries with Highest Support
Compare Machine Learning Libraries with Highest Quality
Compare Machine Learning Libraries with Highest Security
Compare Machine Learning Libraries with Permissive License
Compare Machine Learning Libraries with Highest Reuse
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.