common-crawl | playing around with the common crawl dataset | Crawler library

I would like to download a subset of a WAT archive segment from Amazon S3.

Background:

Searching the Common Crawl index at http://index.commoncrawl.org yields results with information about the location of WARC files on AWS S3. For example, searching for url=www.celebuzz.com/2017-01-04/*&output=json yields JSON-formatted results, one of which is

{ "urlkey":"com,celebuzz)/2017-01-04/watch-james-corden-george-michael-tribute", ... "filename":"crawl-data/CC-MAIN-2017-34/segments/1502886104631.25/warc/CC-MAIN-20170818082911-20170818102911-00023.warc.gz", ... "offset":"504411150", "length":"14169", ... }

The filename entry indicates which archive segment contains the WARC file for this particular page. This archive file is huge; but fortunately the entry also contains offset and length fields, which can be used to request the range of bytes containing the relevant subset of the archive segment (see, e.g., lines 22-30 in this gist).

My question:

Given the location of a WARC file segment, I know how to construct the name of the corresponding WAT archive segment (see, e.g., this tutorial). I only need a subset of the WAT file, so I would like to request a range of bytes. But how do I find the corresponding offset and length for the WAT archive segment?

I have checked the API documentation for the Common Crawl index server, and it isn't clear to me that this is even possible. But in case it is, I'm posting this question.

I am attempting to create a database of Digital Object Identifier (DOI) found on the internet.

By manually searching the CommonCrawl Index Server manually I have obtained some promising results.

However I wish to develop a programmatic solution.

This may result in my process only requiring to read the index files and not the underlying WARC data files.

The manual steps I wish to automate are these:-

1). for each CommonCrawl Currently available index collection(s):

2). I search ... "Search a url in this collection: (Wildcards -- Prefix: http://example.com/* Domain: *.example.com) " e.g. link.springer.com/*

3). this returns almost 6MB of json data that contains approx 22K unique DOIs.

How can I browse all available CommonCrawl indexes instead of searching for specific URLs?

From reading the API documentation for CommonCrawl I cannot see how I can browse all the indexes to extract all DOIs for all domains.

UPDATE

I found this example java code https://github.com/Smerity/cc-warc-examples/blob/master/src/org/commoncrawl/examples/S3ReaderTest.java

that shows how to access a common crawl dataset.

However when I run it I receive this exception