shutils | Shell utils and cheatsheets for sysadmin work | Awesome List library

... why I get such performance ?

See https://docs.python.org/3/library/threading.html:

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

Your code runs in a non-preemptive framework, yet it never yields control, not until it exits. So another thread won't be scheduled until then. You used some thread machinery, but you may as well have written a two-line sequential function that calls io_job() followed by cpu_job().

What you're looking for is multiprocessing.

Also, if you literally want to copy file trees around with tools like rsync, consider using gmake -jN or GNU parallel (sudo apt install parallel). Here's an example command:

You should join the directory name of the old file path with the new file name instead:

As mentioned by Miraj50 it was indeed a Meta Refresh that took it to the final url. So i extracted the final urls from the meta tag and was able to download all the 45 pdfs. Below is the code for the same -