subword-nmt | Unsupervised Word Segmentation for Neural Machine Translation and Text Generation | Translation library

by rsennrich Python Version: 0.3.8 License: MIT

X-Ray Key Features Code Snippets(4)Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | subword-nmt Summary

subword-nmt is a Python library typically used in Utilities, Translation, Deep Learning, Pytorch, Tensorflow, Neural Network applications. subword-nmt has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install subword-nmt' or download it from GitHub, PyPI.

This repository contains preprocessing scripts to segment text into subword units. The primary purpose is to facilitate the reproduction of our experiments on Neural Machine Translation with subword units (see below for reference).

Support

Quality

Security

License

Reuse

Support

subword-nmt has a medium active ecosystem.

It has 2023 star(s) with 451 fork(s). There are 55 watchers for this library.

It had no major release in the last 12 months.

There are 2 open issues and 84 have been closed. On average issues are closed in 69 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of subword-nmt is 0.3.8

Quality

subword-nmt has 0 bugs and 0 code smells.

Security

subword-nmt has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

subword-nmt code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

subword-nmt is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

subword-nmt releases are available to install and integrate.

Deployable package is available in PyPI.

Build file is available. You can build the component from source.

Installation instructions, examples and code snippets are available.

subword-nmt saves you 874 person hours of effort in developing the same functionality from scratch.

It has 2011 lines of code, 89 functions and 17 files.

It has high code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed subword-nmt and discovered the below as its top functions. This is intended to give you an instant insight into subword-nmt implemented functionality, and help decide if they suit your requirements.

Learn a BPE
Prune stats that are less than threshold
Calculate the pair frequencies for each pair
Reads the vocabulary file
Process a file
Segment a sentence
Process a single line
Process bpe file
Learn joint BPE from input files
Learn a bpe file
Reads a vocabulary
Merge two vocabulary
Calculate the correct value of two n - grams
Get the vocab from a training file
Create argument parser
Compute the F1 precision recall
Extract n - grams from a string
Segment character n - grams
Calculate the stats for a vocabulary

Get all kandi verified functions for this library.

subword-nmt Key Features

No Key Features are available at this moment for subword-nmt.

subword-nmt Examples and Code Snippets

StupidNMT: Hard-Coded Gaussian Attention for Neural Machine Translation,Basic Usage,Preprocessing

Python

Lines of Code : 13

License : Permissive (BSD-3-Clause)

Copy

python main.py --dataset wmt_en_de -d data/raw/wmt -p data/preprocessed/wmt -v pass

git clone https://github.com/rsennrich/subword-nmt.git
git clone https://github.com/rsennrich/wmt16-scripts.git
git clone https://github.com/moses-smt/mosesdecoder.g

mtrain,Installation,Environment variables

Python

Lines of Code : 10

License : Weak Copyleft (LGPL-3.0)

Copy

export MOSES_HOME=/path/to/moses
export FASTALIGN_HOME=/path/to/fastalign/bin
export MULTEVAL_HOME=/path/to/multeval

export MOSES_HOME=/path/to/moses
export FASTALIGN_HOME=/path/to/fastalign/bin
export MULTEVAL_HOME=/path/to/multeval
export NEMATUS_

Syntax-Guided Controlled Generation of Paraphrases,Custom Dataset Processing

Python

Lines of Code : 4

License : Permissive (Apache-2.0)

Copy

cd src/evaluation/apps/stanford-corenlp-full-2018-10-05
java -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,ssplit,pos,lemma,ner,parse -parse.model /edu/stanford/nlp/models/srparser/englishSR.ser.gz -status_port  -port  -ti

How can I create and fit vocab.bpe file (GPT and GPT2 OpenAI models) with my own corpus text?

Python

Lines of Code : 2

License : Strong Copyleft (CC BY-SA 4.0)

Copy

python learn_bpe -o ./vocab.bpe -i dataset.txt --symbols 50000

Community Discussions

Trending Discussions on subword-nmt

Windows "No such file or directory"

Using SentencePiece as a command

QUESTION

Windows "No such file or directory"

Asked 2019-Jul-22 at 17:01

I am trying to run a bash script from my Python code. I am calling the script in a subprocess like so:

...

ANSWER

Answered 2019-Jul-22 at 17:01

After lots of debugging, I found the issue. While the paths I listed exist if I ls them in powershell, typing bash in powershell doesn't just open a bash shell, it actually changes the directory structure. I think this may be related to the Windows Subsystem for Linux, but the result is that C: changes to /mnt/c once inside the bash shell. Replacing this in all my paths, I was able to run my scripts.

Source https://stackoverflow.com/questions/57120989

QUESTION

Using SentencePiece as a command

Asked 2019-Mar-21 at 13:13

I need to use Google's SentencePiece from

SentencePiece Github

I have installed it via pip and I would like to run the example command to train a model like

...

ANSWER

Answered 2019-Mar-21 at 13:13

subword-nmt creates a script subword-nmt when installed. Python sentencepiece doesn't install any scripts, it's only a Python wrapper for the C++ library.

To execute spm_* scripts from sentencepiece you certainly have to install C++ version.

Source https://stackoverflow.com/questions/55278519

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install subword-nmt

install via pip (from PyPI):.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: