web_crawler | web crawlers written on python | Crawler library

thanks to @jackFeeting comment, I updated lxml and my code worked just fine. pip3 install --upgrade lxml updated from version 4.4.1 to 4.6.2

There are likely more reasons behind this design choice, but one is almost certainly use of Microservices. It is a popular technique, so demonstrating command of it is a good idea for answering design questions and benefits of it are well described on Wikipedia:

• Modularity: This makes the application easier to understand, develop, test, and become more resilient to architecture erosion.[6] This benefit is often argued in comparison to the complexity of monolithic architectures.[33]
• Scalability: Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.[34]
• Integration of heterogeneous and legacy systems: microservices is considered as a viable mean for modernizing existing monolithic software application.[35][36] There are experience reports of several companies who have successfully replaced (parts of) their existing software by microservices, or are in the process of doing so.[37] The process for Software modernization of legacy applications is done using an incremental approach.[38]
• Distributed development: it parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently.[39] It also allows the architecture of an individual service to emerge through continuous refactoring.[40] Microservice-based architectures facilitate continuous integration, continuous delivery and deployment.[41] [42]

All of those apply in this case. Indeed, well-defined API makes the modules separate, reusable, easy to understand. Most likely each of the 3 modules will have very different execution time and CPU/memory requirements, so scaling them separately makes a lot of sense. Some companies like Amazon mentioned on the page might go much further splitting those modules into microservices based on the team number, so this split into 3 services can very well be chosen based on the assumption of having 3 teams, rather than technical constraints.

The page also describes criticism of the technique.