twarc | command line tool ( and Python library

by DocNow Python Version: v2.14.0 License: MIT

X-Ray Key Features Code Snippets(3)Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | twarc Summary

twarc is a Python library. twarc has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. However twarc has 3 bugs. You can download it from GitHub.

twarc is a command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter API. It has separate commands (twarc and twarc2) for working with the older v1.1 API and the newer v2 API and Academic Access (respectively).

Support

Quality

Security

License

Reuse

Support

twarc has a medium active ecosystem.

It has 1306 star(s) with 257 fork(s). There are 34 watchers for this library.

It had no major release in the last 12 months.

There are 42 open issues and 380 have been closed. On average issues are closed in 12 days. There are 5 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of twarc is v2.14.0

Quality

twarc has 3 bugs (2 blocker, 0 critical, 1 major, 0 minor) and 77 code smells.

Security

twarc has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

twarc code analysis shows 0 unresolved vulnerabilities.

There are 16 security hotspots that need review.

License

twarc is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

twarc releases are available to install and integrate.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed twarc and discovered the below as its top functions. This is intended to give you an instant insight into twarc implemented functionality, and help decide if they suit your requirements.

Performs Twitter search .
Return an argument parser .
Perform Twitter search .
Main entry point for twitter .
Wait for a job to finish .
Performs a handshake .
Extract related data from a response .
Generate a timeline .
Decorator for the rate limit .
Compute tweets for a given query .

Get all kandi verified functions for this library.

twarc Key Features

No Key Features are available at this moment for twarc.

twarc Examples and Code Snippets

twarc-csv,Extra Command Line Options

Python

Lines of Code : 43

License : Permissive (MIT)

Copy

twarc2 csv --help

Usage: twarc2 csv [OPTIONS] [INFILE] [OUTFILE]

  Convert tweets to CSV.

Options:
  --input-data-type [tweets|users|counts|compliance]
                                  Input data type - you can turn "tweets",

so-me,Full guide,Basic install

Shell

Lines of Code : 31

License : Permissive (Apache-2.0)

Copy

sudo apt-get install twarc jq wget

pip3 install twarc

git clone https://github.com/netarchivesuite/webarchive-discovery.git
pushd webarchive-discovery/
git checkout solrconfig
cp -r warc-indexer/src/main/solr/solr7/ ../so-me_solr7_config
git checko

twarc-report,Recommended Directory Structure

Python

Lines of Code : 15

License : Permissive (CC0-1.0)

Copy

twarc-report/ # local clone
    projects/
        assets/ # copy of twarc-report/assets/
        projectA/
            data/ # created by harvest.py
                tweets/ # populated with tweet*.json files by harvest.py
            metadata.json

Community Discussions

Trending Discussions on twarc

Write to a file until it reaches a certain size, then start new file

why I could not rehydrate more than 18 tweets out of 24000 tweet ids using TWARC/ hydrator app? Does any one know a better way?

QUESTION

Write to a file until it reaches a certain size, then start new file

Asked 2020-Oct-02 at 21:35

I am using the command line tool twarc to download Twitter data as a csv. I have set up my twarc commands and they successfully execute on the command line without issue. Example command:

twarc dosomething > outputfile.jsonl

While I would like to carry out a collection process over an extended period of time, the output files become a bit too large (10+GB) after running for more than a day.

I would like to run a bash script that executes the twarc command, runs until the output file reaches a certain limit, and then starts a new file.

These questions are related...

...although I've had little luck with the translation.

Could anyone provide some insight on setting up a basic bash script to execute a command, wait until a file grows to X size, and then start again on a new file? Could take it from there...

...

ANSWER

Answered 2020-Oct-02 at 20:16

The tool you're looking for is aptly named split:

Source https://stackoverflow.com/questions/64177692

QUESTION

why I could not rehydrate more than 18 tweets out of 24000 tweet ids using TWARC/ hydrator app? Does any one know a better way?

Asked 2020-Aug-06 at 12:21

I have a question regarding rehydrate of the tweet's text. Any help would be appreciated.

This is the source of my data; which is about corona tweets:

source of data set

I have downloaded a data set from it which is in the photo (named 01-feb-2020)

Then, I filter this data to show me the only tweets from 'GB' which is almost 24000 tweets

I have used twarc to hydrate my tweets' text as below :

first, install twarc using pip

then, type this in the command line: twarc configure

then, inter consumer key and secret key

then, write a command:

...

ANSWER

Answered 2020-Aug-05 at 18:24

The Tweet ID collection method (which was copy-pasting ) was not correct. After writing a proper code to save tweet ID into text file, the problem has been solved.

Also, Andy Piper mentioned the same thing in the comment part which I copy past here.

How are you getting from JSON format downloaded, into a CSV format? I'm wondering whether the Tweet ID values are valid. – Andy Piper 5 hours ago

I've managed to reproduce this now, and I believe that in the process of converting your JSON input to CSV / Excel to a list of Tweet IDs to hydrate, you are probably using JavaScript (?) and the Tweet IDs are losing their accuracy. The clue was when I noticed all of the Tweet IDs ending in 0000 in my Excel column. You'll need to use a more precise method of getting the Tweet IDs into twarc

Source https://stackoverflow.com/questions/63254995

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install twarc

You can download it from GitHub.
You can use twarc like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

New features are welcome and encouraged for twarc. However, to keep the core twarc library and command line tool sustainable we will look at new functionality with the following principles in mind:. For features and approaches that fall outside of this, twarc enables external packages to hook into the twarc2 command line tool via click-plugins. This means that if you want to propose new functionality, you can create your own package without coordinating with core twarc.

Find more information at: