oodles | Distributed web crawler , indexer and search engine

oodles is a C++ library. oodles has no bugs, it has no vulnerabilities and it has low support. You can download it from GitHub.

oodles is a personal project and currently very much a work-in-progress. it is a distributed web crawler, indexer and search engine. currently only the crawler components are being actively developed. html/ - html parser and document structures utility/ - general utility data structures and algorithms test/ - unit tests and simple test programs bin/ - output of all compiled programs common/ - code common to all oodles components url/ - url parser, normaliser and class and iterator code sched/ - crawler scheduler code prog/ - oodles components program source code net/ - networking code (split by core then protocol). first, copy .config-example to .config and modify it according to your development platform. for linux and osx 10.5 this should be pretty straight-forward. you may wish to modify the values of these flags for cppflags. -doodles_net_buffer_size=16384 (default is 4096 or whatever you set below) -doodles_page_size (default is 4096). the latter is only used if you play with the oodles allocator, which i don't recommend presently. you must have boost >= 1.42 installed as the .config may suggest... all test programs will be under bin/test. test input data is under test/input. --[ testing transfer rate of core networking code. on a unix os such as linux or *bsd (so a mac too) download and install