neural-network | For educational purposes | Machine Learning library

I have managed to solve the error related to the mlUpdate task, the issue was that I was referencing the .mlmodel instead of the compiled version, which is .mlmodelc . When building the iOS app from Xcode this file is automatically generated.

I now get the following error:

Technically the input will be 1D, but that doesn't matter.

The internal architecture of your neural network will take care of recognizing the different words. You could for example have a convolution with a stride equal to the embedding size.

You can flatten a 2D input to become 1D and it will work fine. This is the way you'd normally do it with word embeddings.

If I understand your question properly, you want to get output feature maps of each layer of a model. Normally, as we mentioned in the comment box, a model with one (or multiple) inputs and one (or multiple) outputs. But in order to inspect the activation feature maps of inside layers, we can adopt some strategies. Some possible scenarios: (1). Want to get output feature maps of each layer in run-time or training time. (2). Want to get output feature maps of each layer in the inference time or after training. And as you quoted:

I am now running an image through the trained network and want to get the network output (feature maps etc.) at every stage.

That goes to number 2, get the feature maps in inference time. Below is a simple possible workaround to do this. First, we build a model, and then after training, we will modify the trained model to get feature maps of each layer within it (technically creating the same model with some modification).

RTX 3060 cards are based on the Ampere architecture for which compatible CUDA version start with 11.x.

Your issue can be resolved once you upgrade tensorflow version to 2.4.0, CUDA to 11.0 and cuDNN to 8.0.

For more details you can refer here.

The NaN you see is due to underflow, you need to use BigDecimal class instead of double for higher precision. Refer these for better understanding bigdecimal class java sample use , BigDecimal API Reference

This is a borderline question; you should still be able to extract this understanding from the basic literature ... eventually.

Your insight is exactly correct: you are measuring execution time per epoch, rather than total Time-to-Train (TTT). You have also carried the generic "smaller batches" advice ad absurdum: a batch size of 1 is almost guaranteed to be sub-optimal.

The mechanics are very simple at a macro level.

With a batch size of 60k (the entire training set), you run all 60k images through the model, average their results, and then do one back-propagation for that average result. This tends to lose the learning you can get from focusing on little-seen features.

With a batch size of 1, you run each image individually through the model, average the one result (a very simple operation :-) ), and do a back propagation. This tends to over-emphasize individual effects, especially retaining superstitious effects from each single image. It also gives too much weight to the initial assumptions of the first few images.

The most obvious effect of the tiny batch size is that you're doing 60k back-props instead of 1, so each epoch takes much longer.

Either of these approaches is an extreme case, usually absurd in application.

You need to experiment to find the "sweet spot" that gives you the fastest convergence to acceptable (near-optimal) accuracy. There are a few considerations in choosing your experimental design:

• Memory size: you want to be able to ingest the entire batch into memory at once. This allows your model to pipeline reading and processing. If you exceed available memory, you will lose a lot of time to swapping. If you under-use the memory, you leave some potential performance untapped.
• Processors: if you're on a multi-processor chip, you want to keep them all busy. If you care to assign processors through your OS controls, you'll also want to play with how many to assign to model computation, and how many to assign to I/O and system use. For instance, in one project I did, our group found that our 32 cores were best used with 28 allocated to computation, 4 reserved for I/O and other system functions.
• Scaling: some characteristics work best in powers of 2. You may find that a batch size that is 2^n or 3 * 2^n for some n, works best, simply because of block sizes and other system allocations.

The experimental design that has worked best for me over the years is to start with a power of 2 that is roughly the square root of the training set size. For you, there's an obvious starting guess of 256. Thus, you'd run experiments at perhaps 64, 128, 256, 512, and 1024. See which ones give you the fastest convergence.

Then do one step of refinement, using that factor of 3. For instance, if you find that the best performance comes at 128, also try 96 and 192.

You will likely see very little difference between your "sweet spot" and the adjacent batch sizes; this is the nature of most complex information systems.

Exactly what it says on the tin, the list of members in the class declaration:

You set label_mode='categorical' then this is a multi-class classification and you need to use softmax activation in your last dense layer. Because softmax force the outputs sum to be equal to 1. You can kinda interpret them as probabilities. With sigmoid it will not be possible to find the dominant class. It can assign any values without restriction.

My model's last layer: Dense(5, activation = 'softmax')

My model's loss: loss=tf.keras.losses.CategoricalCrossentropy(), same as yours. Labels are one-hot-encoded in this case.

Explanation: I used a 5 class classification for demo purposes, but it follows the same logic.

This one was a bit tricky and requires slice (see this answer for more info about slice, but it should be intuitive). Also this answer for slice trick. Please see comments for explanation: