browsertrix-crawler | fidelity browser-based crawler

Browsertrix Crawler requires Docker to be installed on the machine running the crawl. Assuming Docker is installed, you can run a crawl and test your archive with the following steps. You don't even need to clone this repo, just choose a directory where you'd like the crawl data to be placed, and then run the following commands. Replace [URL] with the web site you'd like to crawl.
Run docker pull webrecorder/browsertrix-crawler
docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --url [URL] --generateWACZ --text --collection test
The crawl will now run and progress of the crawl will be output to the console. Depending on the size of the site, this may take a bit!
Once the crawl is finished, a WACZ file will be created in crawls/collection/test/test.wacz from the directory you ran the crawl!
You can go to ReplayWeb.page and open the generated WACZ file and browse your newly crawled archive!
To include automated text extraction for full text search, add the --text flag.
To limit the crawl to a maximum number of pages, add --limit P where P is the number of pages that will be crawled.
To run more than one browser worker and crawl in parallel, and --workers N where N is number of browsers to run in parallel. More browsers will require more CPU and network bandwidth, and does not guarantee faster crawling.
To crawl into a new directory, specify a different name for the --collection param, or, if omitted, a new collection directory based on current time will be created.
Browsertrix Crawler uses a browser image which supports amd64 and arm64 (currently oldwebtoday/chrome:91). This means Browsertrix Crawler can be built natively on Apple M1 systems using the default settings. Simply running docker-compose build on an Apple M1 should build a native version that should work for development. On M1 system, the browser used will be Chromium instead of Chrome since there is no Linux build of Chrome for ARM, and this now is handled automatically as part of the build. Note that Chromium is different than Chrome, and for example, some video codecs may not be supported in the ARM / Chromium-based version that would be in the amd64 / Chrome version. For production crawling, it is recommended to run on an amd64 Linux environment.