Pixiv | Get image from pixiv | Computer Vision library

So according to comments the problem lies in unequal batch sizes, due to the final batch being smaller than the specified batch size. I believe this is due to this line:

Looks like an existing bug with tqdm. https://github.com/tqdm/tqdm/issues/766

Option 1:

• Provides tqdm the total size
• On each iteration, update progress. Expect the progress bar to keep moving.
• Works fine.

Option 2:

• Provides tqdm the total size along with a generator function that tracks the progress.
• On each iteration, it should automatically get the update from generator and push the progress bar.
• However, you also call progress.update manually, which should not be the case.
• Instead let the generator do the job.
• But this doesn't work either, and the issue is already reported.

Suggestion on Option 1: To avoid closing streams manually, you can enclose them inside with statement. Same applies to tqdm as well.

A little trial and error got my answer. Thanks goes to https://www.dostips.com/DtTipsStringManipulation.php

@echo off

setlocal EnableExtensions DisableDelayedExpansion

pushd "%~dp0" || exit /B

move /Y "Folder2.jpg" "Folder.jpg"

for %%I in (.) do set "FOLDER=%%~nxI"

for /f "tokens=1 delims= " %%a in ("%FOLDER%") do set FOLDER=%%a

"%ProgramFiles(x86)%\gallery-dl\gallery-dl.exe" -d "U:\11Web\gallery-dl" --download-archive "%~dp0zzzGDB.sqlite3" "https://www.pixiv.net/en/users/%FOLDER%/illustrations"

if not errorlevel 1 if exist "zzzGDB.sqlite3" del "Folder.jpg"

popd

endlocal

pause

There are actually two elements on this page that use that XPath. The webdriver will choose the first element that meets the requirements set by the developer. Unfortunately the element you want is the second element in the DOM. However updating your XPath to be more specific will help:

The differences are:

1. You're using different SSL information: You're browser has a built-in set of certificate authorities. Python uses a set which comes with the OS. They differ & if the site you're accessing uses one know to your browser but not known to python, the python will throw an exception.
2. You're accessing using different User-Agents. Your browser is telling the server it's Chrome or IE or whatever. Python is telling the server it's python. For whatever reason, the server may decide it doesn't like that and return Forbidden.
3. The server may be working harder than you think: while it appears the request is for a simple file, you're really requesting a resource. It may be (though unlikely in this case) that the resource you're requesting results in multiple interactions between the server and your browser -- cookies, javascript, etc -- which are executed successfully in your browser, returned to the server & it then delivers the file. Your python request is not doing any of that.
4. Your browser (may) have existing state which your python does not. You say you can access the file using your browser, but it could be that works only because you've accessed other resources on the site, or logged in, or whatever. Your browser is communicating that information (perhaps a session_id via cookie?) with the server recognizes. Your python code states with no previous state, so the server forbids that.

Which is it in this case? You'll need to investigate. Can you get wget or curl to work? Debug your browser's access: what headers are being sent, what are you receiving in reply?

OMG!

Finally, I know what goes wrong.

I use mem_top() to see what takes up the memory.

Guess what?

It is for i in range(0, 38849402):

In the memory, there is a list [0, 1, 2, 3 ... 38849401], which takes up my memory.

I change it to :