tools | Various utilities for processing the data

tools is a Python library. tools has no vulnerabilities, it has a Strong Copyleft License and it has low support. However tools has 6 bugs and it build file is not available. You can download it from GitHub.

This repository contains various scripts in Perl and Python that can be used as tools for Universal Dependencies. Reads a CoNLL-U file and verifies that it complies with the UD specification. It must be run with the language code and there must exist corresponding lists of treebank-specific features and dependency relations in order to check that they are valid, too. The script runs under Python 3 and needs the third-party module regex. If you do not have the regex module, install it using pip install --user regex. NOTE: Depending on the configuration of your system, it is possible that both Python 2 and 3 are installed; then you may have to run python3 instead of python, and pip3 instead of pip. You can run python validate.py --help for a list of available options. Reads CoNLL-U files from STDIN and verifies that every sentence has a unique id in the sent_id comment. All files of one treebank (repository) must be supplied at once in order to test treebank-wide id uniqueness. Converts Unicode to the NFC normalized form. Can be applied to any UTF-8-encoded text file, including CoNLL-U. As a result, if there are character combinations that by definition must look the same, the same sequence of bytes will be used to represent the glyph, thus improving accuracy of models (as long as they are applied to normalized data too). Beware: The output may slightly differ depending on your version of Perl because the Unicode standard evolves and newer Perl versions incorporate newer versions of Unicode data. Reads a CoNLL-U file, collects various statistics and prints them. This Perl script should not be confused with conll-stats.py, an old Python 2 program that collects just a few very basic statistics. The Perl script (conllu-stats.pl) is used to generate the stats.xml files in each data repository. The script depends on Perl libraries YAML and JSON::Parse that may not be installed automatically with Perl. If they are not installed on your system, you should be able to install them with the cpan command: cpan YAML and cpan JSON::Parse. Reads a CoNLL-U file, collects statistics of multi-word tokens and prints them. Reads a CoNLL-U file, collects statistics about the enhanced graphs in the DEPS column and prints them. This script uses the modules Graph.pm and Node.pm that lie in the same folder. On UNIX-like systems it should be able to tell Perl where to find the modules even if the script is invoked from a remote folder. If that does not work, use perl -I libfolder script to invoke it. Also note that other third-party modules are needed that are not automatically included in the installation of Perl: Moose, MooseX::SemiAffordanceAccessor, List::MoreUtils. You may need to install these modules using the cpan tool (simply go to commandline and type sudo cpan Moose). Reads a CoNLL-U file, removes empty nodes and adjusts the enhanced graphs so that a path traversing one or more empty nodes is contracted into a single edge: if there was a "conj" edge from node 27 to node 33.1, and a nsubj edge from node 33.1 to node 33, the resulting graph will have an edge from 27 to 33, labeled conj>nsubj. This script uses the modules Graph.pm and Node.pm that lie in the same folder. On UNIX-like systems it should be able to tell Perl where to find the modules even if the script is invoked from a remote folder. If that does not work, use perl -I libfolder script to invoke it. Also note that other third-party modules are needed that are not automatically included in the installation of Perl: Moose, MooseX::SemiAffordanceAccessor, List::MoreUtils. You may need to install these modules using the cpan tool (simply go to commandline and type sudo cpan Moose). Compares two CoNLL-U files and searches for sentences that occur in both (verbose duplicates of token sequences). Some treebanks, especially those where the original text had been acquired from the web, contained duplicate documents that were found at different addresses and downloaded twice. This tool helps to find out whether one of the duplicates fell in the training data and the other in development or test. The output has to be verified manually, as some “duplicates” are repetitions that occur naturally in the language (in particular short sentences such as “Thank you.”). The script can also help to figure out whether training-dev-test data split has been changed between two releases so that a previously training sentence is now in test or vice versa. That is something we want to avoid. Similar to overlap.py but it works with the sentence-level attribute text. It remembers all sentences from STDIN or from input files whose names are given as arguments. The find script prints the duplicate sentences (ordered by length and number of occurrences) to STDOUT. The remove script works as a filter: it prints the CoNLL-U data from the input, except for the second and any subsequent occurrence of the duplicate sentences.