practices | Guides principles from the web team at Canonical | Natural Language Processing library

I'm leaving this question up in case anyone else ever gets stuck in my situation, but the code works as it is meant to, I just committed a folly.

I didn't account for height changes with asynchronous image loading and as such, the list would not be at its saved position upon composable navigation, due to the list state being smaller than the screen height on returning to the composable.

However, If the images were given static containers to load into to that don't change their size, then upon back navigation, the composable would correctly display the saved list state.

a) Even if obviously it could be better, is this solution is acceptable or not at all ?

I think it has some problems you want to address, but the general idea of using the filesystem and storing just filenames can be acceptable, depending on how many users you need to serve and what guarantees regarding data consistency and concurrent accesses you want to make.

I would consider the uploaded file temporary data that may be lost on system failure. If you want to provide any guarantees of not losing the data, you want to store it in a database instead of on the filesystem.

b) I pass 'uploaded_file' from one view to another using "request.session" is it a good practice? Is there another way to do it without using GET variables?

There are up- and downsides to using request.session.

• attackers can not change the filename and thus retrieve data of other users. This is also the reason why you should not use a GET parameter here: If you used one, attackers could simpy change that parameter and get access to files of other users.
• users can upload a file, go and do other stuff, and later come back to actually import the file, however:
• if users end their session, you lose the filename. Also, users can not upload the file on one device, change to another device, and then go on with the import, since the other device will have a different session.

The last point correlates with the leftover files problem: If you lose your information about which files are still needed, it makes cleaning up harder (although, in theory, you can retrieve which files are still needed from the session store).

If it is a problem that sessions might end or change because users clear their cookies or change devices, you could consider adding the filename to the UserProfile in the database. This way, it is not bound to sessions.

c) At first my wish was to avoid to save the csv-file. But I could not figure out how to do it? Reading all the file to request.session seems not a good idea for me. Is there some possibility to upload the file into memory in Django?

You want to store state. The go-to ways of storing state are the database or a session store. You could load the whole CSVFile and put it into the database as text. Whether this is acceptable depends on your databases ability to handle large, unstructured data. Traditional databases were not originally built for that, however, most of them can handle small binary files pretty well nowadays. A database could give you advantages like ACID guarantees where concurrent writes to the same file on the file system will likely break the file. See this discussion on the dba stackexchange

Your database likely has documentation on the topic, e.g. there is this page about binary data in postgres.

d) If I have to use the tmp-file. How should I handle the situation if user abandon upload at the middle (for example, he sees the confirmation page, but does not click "yes" and decide to re-write his file). How to remove the tmp-file?

Some ideas:

• Limit the count of uploaded files per user to one by design. Currently, your filename is based on a timestamp. This breaks if two users simultaneously decide to upload a file: They will both get the same timestamp, and the file on disk may be corrupted. If you instead use the user's primary key, this guarantees that you have at most one file per user. If they later upload another file, their old file will be overwritten. If your user count is small enough that you can store one leftover file per user, you don't need additional cleaning. However, if the same user simultaneusly uploads two files, this still breaks.
• Use a unique identifier, like a UUID, and delete the old stored file whenever the user uploads a new file. This requires you to still have the old filename, so session storage can not be used with this. You will still always have the last file of the user in the filesystem.
• Use a unique identifier for the filename and set some arbitrary maximum storage duration. Set up a cronjob or similar that regularly goes through the files and deletes all files that have been stored longer than your specified maximum duration. If a user uploads a file, but does not do the actual import soon enough, their data is deleted, and they would have to do the upload again. Here, your code has to handle the case that the file with the stored filename does not exist anymore (and may even be deleted while you are reading the file).

You probably want to limit your server to one file stored per user so that attackers can not fill your filesystem.

e) Small additional question : what kind of checks there are in Django about uploaded file? For example, how could I check that the file is at least a text-file? Should I do it?

You definitely want to set up some maximum file size for the file, as described e.g. here. You could limit the allowed file extensions, but that would only be a usability thing. Attackers could also give you garbage data with any accepted extension.

Keep in mind: If you only store the csv as text data that you load and parse everytime a certain view is accessed, this can be an easy way for attackers to exhaust your servers, giving them an easy DoS attack.

Overall, it depends on what guarantees you want to make, how many users you have and how trustworthy they are. If users might be malicious, you want to keep all possible kinds of data extraction and resource exhaustion attacks in mind. The filesystem will not scale out (at least not as easily as a database).

I know of a similar setup in a project where only a handful of priviliged users are allowed to upload stuff, and we can tolerate deletion of all temporary files on failure. Users will simply have to reupload their files. This works fine.

1. It is the main method, so there is no need to cleanup. When main returns, the program exits. If this wasn't the main, then you would be correct.

2. There is no best practice that fits all use cases. The code you show here is a very common pattern. The function creates a goroutine, and returns a channel so that others can communicate with that goroutine. There is no rule that governs how channels must be created. There is no way to terminate that goroutine though. One use case this pattern fits well is reading a large resultset from a database. The channel allows streaming data as it is read from the database. In that case usually there are other means of terminating the goroutine though, like passing a context.

3. Again, there are no hard rules on how channels should be created/closed. A channel can be left open, and it will be garbage collected when it is no longer used. If the use case demands so, the channel can be left open indefinitely, and the scenario you worry about will never happen.

Keeping the files in the same server comes with more risk, if your server crashes or region goes down your application stopped as well as your files not gonna render if you are using those files separately for different applications like mobile applications.

In this case, users will also face a high loading time for those media files if the users are not in the same zone as your application hosted.

The best practice to store the image/media files on some cloud storage like S3 or Azure Blob then connect it with some CDN like CloudFront or Azure CDN.

Now you can serve your media files via CDN which will act as a global caching system for your media files.

Honestly, after days of research, I'm trying to talk the team into the idea of creating an API dedicated to Docs, and as part of that effort would migrate documents to AWS S3 or azure blob storage. We will introduce Aspose for rendering and separate the Issue.

Why? because all the solutions I have seen to support doc and Docx in .net5 are really hacky and I can see lacks in terms of security. And I would not recommend anyone to try to do the approach I was looking to do at the begging of this question.

I leave here what experience cause this research was pain and I hope this question can still help someone.

Let's say we have 2 components

1. BooksHome for /books/
2. SingleBook for /books/:id

So when the user visits /books/ route, fire a thunk fetchBooks(..) which loads books and saves them in state.bookstore.books and display those books in BooksHome

And when the user visits /books/:id route, fire a thunk fetchSingleBook(..) which will find book item with :id in state.bookstore.books if that's present return it and display it in SingleBook and if it's not there, fetch it, add it to state.bookstore.books and return it and display it in SingleBook.

Explanation:

When the user visits /books/123 right away, state.bookstore.books is [] so no item is found, so we'll fetch it, store it and display it.

When the user visits /books/ (data fetched and stored) and then visits /books/123, state.bookstore.books is having an item with id:123 so the item is found, so we'll get it from state.bookstore.books and display it.

Must do things:

1. fetchBooks must be dispatched in BooksHome but not in App
2. Same way fetchSingleBook must be dispatched in SingleBook

Things optimised with this approach:

1. We need not fetch all books for displaying a single book.
2. We need not re-fetch book's data when it is already available in state.bookstore.books

Hope this helps🤝

It seems that export command isn't environment-proof.

Finally, I removed these lines:

I would add the celery user to the sudoers file with the only command allowed being the one needed. Use visudo and add these lines

Is there generic practices and patterns to succesfully switch from dynamic to static memory allocation without breaking the code ?

No, not really. You'll have to rewrite all such code in pretty radical ways.

You have to realize why all safety-related and embedded systems ban malloc. The main reason is that it is non-deterministic. Instead of allowing completely variable sizes, you have to specify a maximum size for each such item, to cover the worst case scenario of the application.

Also, the presence of things like pointer-to-pointers instead of 2D arrays is a pretty certain indication that the original programmer didn't quite know what they were doing in the first place.

Additionally you need to drop the default types of C for stdint.h ones. That's standard practice in all embedded systems.

In general, I'd strongly advise to drop those "NASA rules" and implement MISRA-C instead. It's a way more professional and in-depth document. Some of the "NASA rules" simply don't make sense and the rest can be summarized as "No s***t Sherlock" beginner-level stuff which we were already told during our first beginner-level C programming class back in school. If these rules come as a surprise to someone, they shouldn't be writing mission-critical firmware in the first place.