wgan-gp | pytorch implementation of Paper `` Improved Training | Machine Learning library
kandi X-RAY | wgan-gp Summary
kandi X-RAY | wgan-gp Summary
A pytorch implementation of Paper "Improved Training of Wasserstein GANs"
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Load MNIST dataset
- Generate an epoch from cifar files
- Unpickle a pickle file
wgan-gp Key Features
wgan-gp Examples and Code Snippets
Lines of Code : 50
License : No License
Copypytorch-WGAN-GP
+---[dir_checkpoint]
| \---[scope]
| \---[name_data]
| +---model_epoch00000.pth
| | ...
| \---model_epoch12345.pth
+---[dir_data]
| \---[name_data]
| +---000000.png
| | ...
| from ydata_synthetic.synthesizers.regular import WGAN_GP
from ydata_synthetic.synthesizers import ModelParameters, TrainParameters
# Define the GAN and training parameters
noise_dim = 32
dim = 128
batch_size = 128
log_step = 100
epochs = 500
learni def get_loss_fn():
def d_loss_fn(real_logits, fake_logits):
return tf.reduce_mean(fake_logits) - tf.reduce_mean(real_logits)
def g_loss_fn(fake_logits):
return -tf.reduce_mean(fake_logits)
return d_loss_fn, g_loss_fn
de Community Discussions
Trending Discussions on wgan-gp
QUESTION
I am trying to implement a WGAN-GP model using tensorflow and keras (for credit card fraud data from kaggle).
I mostly followed the sample code that is provided in keras website and several other sample codes on the internet (but changed them from image to my data), and it is pretty straightforward.
But when I want to update the critic, the gradient of loss w.r.t critic's weights becomes all nan after a few batches. And this causes the critic's weights to become nan and after that the generator's weights become nan,... So everything become nan!
I used tf.debugging.enable_check_numerics and found that the problem arises because a -Inf appears in the gradient after some iterations.
This is directly related to the gradient-penalty term in the loss, because when I remove that the problem goes away.
Please note that the gp itself is not nan, but when I get the gradient of the loss w.r.t critic's weights (c_grads in the code below) it contains -Inf and then somehow becomes all nan.
I checked the math and network architecture for possible mistakes (like probability of gradient vanishing, etc.), and I checked my code for possible bugs for hours and hours. But I'm stuck.
I would very much appreciate it if anyone can find the root of the problem
Note: Bear in mind that the critic's output and loss function is slightly different from the original paper (because I'm trying to make it conditional) but that has nothing to do with the problem because as I said before, the whole problem goes away when I just remove the gradient penalty term
This is my critic:
...ANSWER
Answered 2020-Aug-28 at 00:26So after much more digging into the internet, it turns out that this is because of the numerical instability of tf.norm (and some other functions as well).
In the case of norm function, the problem is that when calculating its gradient, its value appears in the denominator. So d(norm(x))/dx at x = 0 would become 0 / 0 (this is the mysterious division-by-zero I was looking for!)
The problem is that the computational graph sometimes ends up with things like a / a where a = 0 which numerically is undefined but the limit exists. And because of the way tensorflow works (which computes the gradients using the chain rule) it results in nans or +/-Infs.
The best way probably would be for tensorflow to detect these patterns and replace them with their analytically-simplified equivalent. But until they do so, we have another way, and that is using something called tf.custom_gradient to define our custom function with our custom gradient (related issue on their github)
Although in my case there was actually an even simpler solution (although it wasn't simple when I didn't know that the tf.norm was the culprit):
So instead of:
QUESTION
I am working on WGAN and would like to implement WGAN-GP.
In its original paper, WGAN-GP is implemented with a gradient penalty because of the 1-Lipschitiz constraint. But packages out there like Keras can clip the gradient norm at 1 (which by definition is equivalent to 1-Lipschitiz constraint), so why do we bother to penalize the gradient? Why don't we just clip the gradient?
...ANSWER
Answered 2019-Nov-06 at 06:10The reason is that clipping in general is a pretty hard constraint in a mathematical sense, not in a sense of implementation complexity. If you check original WGAN paper, you'll notice that clip procedure inputs model's weights and some hyperparameter c, which controls range for clipping.
If c is small then weights would be severely clipped to a tiny values range. The question is how to determine an appropriate c value. It depends on your model, dataset in a question, training procedure and so on and so forth. So why not to try soft penalizing instead of hard clipping? That's why WGAN-GP paper introduces additional constraint to a loss function that forces gradient's norm to be as much close to 1 as possible, avoiding hard collapsing to a predefined values.
QUESTION
I'm studying gan with keras-gan/wgan-gp example with my own dataset. I save models with
wgan.generator.save('generator.h5')
wgan.critic.save('critic.h5')
and load with
model = load_model('generator.h5')
model = load_model('critic.h5')
But this only works fine at the fist time.When I saved the models again after the second training and run
model = load_model('generator.h5')
model = load_model('critic.h5')
again, the error occur ：
ValueError Traceback (most recent call last) in () ----> 1 model = load_model('generator.h5')
D:\keras\engine\saving.py in load_model(filepath, custom_objects, compile) 262 263 # set weights --> 264 load_weights_from_hdf5_group(f['model_weights'], model.layers) 265 266 if compile:
D:\keras\engine\saving.py in load_weights_from_hdf5_group(f, layers, reshape) 914 original_keras_version, 915 original_backend, --> 916 reshape=reshape) 917 if len(weight_values) != len(symbolic_weights): 918 raise ValueError('Layer #' + str(k) +
D:\keras\engine\saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape) 555 weights = convert_nested_time_distributed(weights) 556 elif layer.class.name in ['Model', 'Sequential']: --> 557 weights = convert_nested_model(weights) 558 559 if original_keras_version == '1':
D:\keras\engine\saving.py in convert_nested_model(weights) 543 weights=weights[:num_weights], 544 original_keras_version=original_keras_version, --> 545 original_backend=original_backend)) 546 weights = weights[num_weights:] 547 return new_weights
D:\keras\engine\saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape) 555 weights = convert_nested_time_distributed(weights) 556 elif layer.class.name in ['Model', 'Sequential']: --> 557 weights = convert_nested_model(weights) 558 559 if original_keras_version == '1':
D:\keras\engine\saving.py in convert_nested_model(weights) 531 weights=weights[:num_weights], 532 original_keras_version=original_keras_version, --> 533 original_backend=original_backend)) 534 weights = weights[num_weights:] 535
D:\keras\engine\saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape) 673 weights[0] = np.reshape(weights[0], layer_weights_shape) 674 elif layer_weights_shape != weights[0].shape: --> 675 weights[0] = np.transpose(weights[0], (3, 2, 0, 1)) 676 if layer.class.name == 'ConvLSTM2D': 677 weights1 = np.transpose(weights1, (3, 2, 0, 1))
c:\users\administrator\appdata\local\programs\python\python35\lib\site-packages\numpy\core\fromnumeric.py in transpose(a, axes) 596 597 """ --> 598 return _wrapfunc(a, 'transpose', axes) 599 600
c:\users\administrator\appdata\local\programs\python\python35\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds) 49 def _wrapfunc(obj, method, *args, **kwds): 50 try: ---> 51 return getattr(obj, method)(*args, **kwds) 52 53 # An AttributeError occurs if the object does not have
ValueError: axes don't match array`
I'm using
Python 3.5.3
Keras 2.2.2
h5py 2.8.0
tensorflow-gpu 1.9.0
keras-contrib 2.0.8
Keras-Applications 1.0.4
Keras-Preprocessing 1.0.2
Any advice and suggestions will be appreciated.
...ANSWER
Answered 2018-Sep-30 at 01:05Try downgrading the keras version to 2.1.5. It solved the problem for me.
QUESTION
This is the loss function of WGAN-GP
...ANSWER
Answered 2018-Nov-21 at 14:07One thing to note is that your gradient penalty calculation is wrong. The following line:
QUESTION
I have implemented a conditional WGAN-GP which works fine for sampling digits from 0-9, but as soon as I want to sample a single digit I get dimensionality issues.
...ANSWER
Answered 2018-Jun-14 at 08:58This line
QUESTION
I'm implementing a WGAN-GP in Keras where I calculate the random weighted average of two tensors.
...ANSWER
Answered 2018-May-23 at 10:00Custom operations that use backend function need to be wrapped around a Layer. If you don't have any trainable weights, as in your case, the simplest approach is to use a Lambda layer:
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install wgan-gp
You can use wgan-gp like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
