CommonCrawl | distributed system for mining common crawl using SQS | Function As A Service library

Since every few hours a new WARC file is added to the news dataset, a static file list does not make sense. Instead you can get a list of files using the AWS CLI - for any subset by year or month, e.g.

The WARC standard does not define a standard way to indicate the number of WARC records in the WARC file itself. The number of response records in Common Crawl WARC files is usually between 30,000 and 50,000 - note that there are also request and metadata records. The WARC standard recommends 1 GB as target size of WARC files which puts a natural limit to the number of records.

Why not use a WARC parser library (I'd recommend warcio) to do the parsing including gzip decompression?

Alternatively, have a look at gzipstream to read from a stream of gzipped content and decompress the data on the fly.

Does your compiler not at least warn you about the naked conditional ret == Z_STREAM_END;? You want an if there and some braces around the inflateReset() related statements.

There's still a problem in that you are leaving the outer loop if strm.avail_in is zero. That will happen every time, except when reaching the end of member. It can even happen then if you just so happen to exhaust the input buffer to decompress that member. Just make the outer loop a while (true).

Even after fixing all that, you would then discard the remaining available input when you do the read at the top of the outer loop. Only do that read if strm.avail_in is zero.

A simpler approach would be to do the reset in the inner loop. Like this (example in C):

this error is because of the libraries being used by the newsplease. mistake is made when we manually install every library, while installing focus on the versions of packages. version info of every library is given in setup.py file. install exact version given in setup.py file. now there may be problems while executing the setup.py.

so use this command -

I am not sure what the issue is but after reading this post

Decompressing using GZipStream returns only the first line

I changed to SharZipLib (http://www.icsharpcode.net/opensource/sharpziplib/) and it worked

No, it is not thread safe if the native code was compiled with CLD2_DYNAMIC_MODE set, which you could test using the function isDataDynamic().

The native function manipulates the static class variable kScoringtables. If CLD2_DYNAMIC_MODE is defined at compilation, this variable is initialized to a set of null tables (NULL_TABLES) and can later be loaded with dynamic data, or unloaded, potentially by other threads.

It would be possible for the kScoringtables.quadgram_obj to be non-null at the line 1762 null check and then the kScoringtables address altered before it is added to the cross-thread ScoringContext object on line 1777. In this case, the wrong pointer would be passed to ApplyHints on line 1785, potentially causing bad things to happen at line 1606.

This would be a very rare race condition, but possible nonetheless, and is not thread safe for the same reason the standard "lazy getter" is not thread safe.

To make this thread-safe, you would have to either test that isDataDynamic() returns false, or ensure the loadDataFromFile, loadDataFromRawAddress, and unloadData functions could not be called by a different thread while you are executing this method (or at least until you are past line 1777...)

The Scala job takes longer because it has a misconfiguration and, therefore, the Python and Scala jobs had been provided with unequal resources.

There are two mistakes in the code:

Seems like each line is its own json object.

You may get away with the following code which will structure this output into correct json: