gpt-2-simple | Python package to easily retrain OpenAI | Machine Learning library
kandi X-RAY | gpt-2-simple Summary
kandi X-RAY | gpt-2-simple Summary
Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Argument parser
- Download the GPT2 model
- Download file with progress bar
- Finalize gpt2
- Simulate the model
- Attention layer
- Compute the attention mask
- Block block
- Generate a sequence of tokens
- Compute top - k logits
- Compute the top - p logits for the given logits
- Copy a checkpoint to gdrive
- Returns the name of the tar file
- Encodes a dataset into a numpy array
- Copies a checkpoint from gdrive
- Reset a TensorFlow session
- Copies a file from the gdrive
- Copy a file to gdrive
gpt-2-simple Key Features
gpt-2-simple Examples and Code Snippets
Lines of Code : 10
License : No License
Copygit clone https://github.com/minimaxir/gpt-2-simple
cd gpt-2-simple
virtualenv .env
# Wait for it to setup
.env\Scripts\activate # On Windows
source .env\bin\activate # On Unix
pip install -r requirements.txt
# IF USING A GPU TO FINETUNE/GENERATE
p gpt2.generate(sess,
length=200,
temperature=0.7,
prefix='<|startoftext|>',
truncate='<|endoftext|>',
include_prefix=False
)
$ pipenv shell # this take you to the virtual environment
$ python -m ipykernel install --user --name= # change the kernel name as you see fit
$ jupyter lab # run a jupyterlab instance on your localhost
sess = gpt2.start_tf_sess()
gpt2.finetune(sess,
file_name,
model_name=model_name,
checkpoint_dir=checkpoint_dir,
run_name=run_name,
steps=25,
)
Training...
[26 | 7.48] loss=0.49 avg=0.49
enc = gpt_2_simple.src.encoder.get_encoder(checkpoint_path)
N_token = len(enc.encoder)
tokens_decoded = [enc.decode([token]) for token in range(N_token)]
probs = tf.nn.softmax(logits)
gpt_2_simple --sample_every 50 finetune yourtext.txt
gpt_2_simple generate --prefix "Once upon a time" --nsamples 5
mkdir -p checkpoint/
cp -r models/345M checkpoint/run1
Community Discussions
Trending Discussions on gpt-2-simple
QUESTION
I'm using the gpt-2 simple package: https://github.com/minimaxir/gpt-2-simple
I would like to get a probability for all possible next tokens as an output. something like:
[ ['A', 0.25], ['B',0.25], ['C',0.25], ['D',0.25] ]
I've modified the gpt_2_simple python code looks something like this:
...ANSWER
Answered 2020-Feb-03 at 01:531) get list of decoded tokens:
QUESTION
I wish to to fine-tune a GPT-2 implementation on some text data. I then want to use this model to complete a text prompt. I can do the first part easily enough using Max Woolf's gpt-2-simple implementation. And Neil Shepherd's fork of OpenAI allows for GPT-2 to be trained on new data and completes text.
However, my corpus is too small to train on and not get gibberish back. Is there any way I can combine the two functions? Ideally, I'd like to be able to do this via a python interface (as opposed to CLI), as I'd like to use pandas for data cleaning and what-have-you. Thanks.
...ANSWER
Answered 2020-Jan-28 at 09:23Huggingface's Transformers package has a GPT-2 implementation (including pre-trained models) for PyTorch and TensorFlow. You can easily work with them in Python.
Fine-tuning of GPT-2, however, requires a lot of memory and I am not sure is you will be able to do the full backpropagation on that. In that case, you fine-tune just a few highest layers.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install gpt-2-simple
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
