maml | Agnostic Meta-Learning for Fast Adaptation | Machine Learning library

TLDR: Use mdl.train() since that uses batch statistics (but inference will not be deterministic anymore). You probably won't want to use mdl.eval() in meta-learning.

BN intended behaviour:

• Importantly, during inference (eval/testing) running_mean, running_std is used - that was calculated from training(because they want a deterministic output and to use estimates of the population statistics).
• During training the batch statistics is used but a population statistic is estimated with running averages. I assume the reason batch_stats is used during training is to introduce noise that regularizes training (noise robustness)
• in meta-learning I think using batch statistics is the best during testing (and not calculate the running means) since we are supposed to be seeing new /tasksdistribution anyway. Price we pay is loss of determinism. Could be interesting just out of curiosity what the accuracy is using population stats estimated from meta-trian.

This is likely why I don't see divergence in my testing with the mdl.train().

So just make sure you use mdl.train() (since that uses batch statistics https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d) but that either the new running stats that cheat aren't saved or used later.

You're better off using PowerShell's language parser, System.Management.Automation.Language.Parser, rather than a regex-based solution.^[1]

I'm assuming you're always interested in the outer here-string, not one that happens to be nested inside another.

Assuming a file file.ps1 with the following verbatim content:

Decided to answer my question. So I ended up using the torch lightning's manual optimization so that I can customize the optimization step. This would make both approaches using the same framework, and I think is better than maintaining 2 separate repos.

It is not a tensorflow bug :) You are updating the Variables of your model with basic Tensors, so in the second iteration, when you call .gradient(support_loss, model.trainable_variables) your model actually doesn't have any trainable variables anymore. Modify your code like so to use the methods for manipulating Variables:

The only difference is that in second approach you'll have to keep much more stuff in memory - until you call backward you'll have all unrolled parameters fnet.parameters(time=T) (along with intermediate computation tensors) for each of task_num iterations as part of the graph for the aggregated meta_loss. If you call backward on every task then you only need to keep full set of unrolled parameters (and other pieces of the graph) for one task.

So to answer your question's title: because in this case the memory footprint is task_num times bigger.

In a nutshell what you're doing is similar to comparing loopA(N) and loopB(N) in the following code. Here loopA will get as much memory as it can and OOM with sufficiently large N, while loopB will use about same amount of memory for any large N: