django-zip-stream | Django extension to assemble ZIP | Runtime Evironment library
kandi X-RAY | django-zip-stream Summary
kandi X-RAY | django-zip-stream Summary
Django extension to assemble ZIP archives dynamically using Nginx.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Initialize a transfer response .
- Build file content .
- Finalize options .
- Runs the tests .
django-zip-stream Key Features
django-zip-stream Examples and Code Snippets
Community Discussions
Trending Discussions on django-zip-stream
QUESTION
I've got a proxy written in Django which receives requests for certain files. After deciding whether the user is allowed to see the file the proxy gets the file from a remote service and serves it to the user. There's a bit more to it but this is the gist.
This setup works great for single files, but there is a new requirement that the users want to download multiple files together as a zip. The files are sometimes small, but can also become really large (100MB plus) and it can be anywhere from 2 up to 1000 files simultaneously. This can become really large, and a burden to first get all those files, zip them and then serve them in the same request.
I read about the possibility to create "streaming zips"; a way to open a zip and then start sending the files in that zip until you close it. I found a couple php examples and in Python the django-zip-stream extension. They all assume locally stored files and the django extension also assumes the usages of nginx.
There are a couple things I wonder about in my situation:
- I don't have the files locally stored. I can get them with an async/await structure and serve them simultaneously. That would mean I always have two files in memory (the one I'm currently serving, and the next one I'm getting from the source server).
- Unfortunately I don't have control over the web servers which will serve this. I can of course put an nginx container in front of it, but I don't think nginx could serve from files I store in Python vars because I get them from the source server.
- Whether I'm doing this in Python or let it be zipped in nginx, I presume the needed CPU cycles for this would be substantial.
Does anybody know whether streaming zips are a good idea with my setup of very large remote files? I'm a bit afraid that many requests will easily DOS our servers because of CPU or memory limits.
I can also build a queue which zips the files and sends an email to the user, but if possible I'd like to keep the application as stateless as possible.
All tips are welcome!
...ANSWER
Answered 2020-Dec-11 at 00:43Ok this is tough one!
After the first request you could create and save the zipped file on the file servers. So the File Servers always deliver zipped files at the end. First time request will take longer because of creating the zip file but next times it will always deliver the zipped file as long as it will not be deleted.
a) You could deliver a single stream which could be at the end a tape archive aka tar file which includes all the zipped files.
-- or --
- b) HTTP/2 "One of the key advantages of the protocol is that it is multiplexed, meaning that multiple files can be transferred on a single connection." (sitepoint.com) There should not be any problems with the browsers (caniuse.com)
In case of an DOS Attack you could limit the amount of requests for file downloads. So if there are too many requests at the same time they will be bounced back and they have to try it later on.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install django-zip-stream
You can use django-zip-stream like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page