special | CSS Installer - | DevOps library

You can use special characters in key names by delimiting them with "":

Tesseract takes a lang variable that you can expand to include different languages if they're installed. I've used the UB Mannheim (https://github.com/UB-Mannheim/tesseract/wiki) installation which includes a ton of languages supported.

To get better and more accurate results, the best thing to do is to process the image before handing it to Tesseract. Set a white/black threshold so that you have black text on white background with no shading. I'm not sure how to do this in Node, but I've done it with Python's OpenCV library.

If that font doesn't get you decent results with the out of the box, then you'll want to train your own, yes. This blog post walks through the process in great detail: https://towardsdatascience.com/simple-ocr-with-tesseract-a4341e4564b6. It revolves around using the jTessBoxEditor to hand-label the objects detected in the images you're using.

Edit: In brief, the process to train your own:

1. Install jTessBoxEditor (https://sourceforge.net/projects/vietocr/files/jTessBoxEditor/). Requires Java Runtime installed as well.
2. Collect your training images. They want to be .tiffs. I found I got fairly accurate results with not a whole lot of images that had a good sample of all the characters I wanted to detect. Maybe 30/40 images. It's tedious, so you don't want to do TOO many, but need enough in order to get a good sampling.
3. Use jTessBoxEditor to merge all the images into a single .tiff
4. Create a training label file (.box)j. This is done with Tesseract itself. tesseract your_language.font.exp0.tif your_language.font.exp0 makebox
5. Now you can open the box file in jTessBoxEditor and you'll see how/where it detected the characters. Bounding boxes and what character it saw. The tedious part: Hand fix all the bounding boxes and characters to accurately represent what is in the images. Not joking, it's tedious. Slap some tv episodes up and just churn through it.
6. Train the tesseract model itself

• save a file: font_properties who's content is font 0 0 0 0 0
• run the following commands:

tesseract num.font.exp0.tif font_name.font.exp0 nobatch box.train

unicharset_extractor font_name.font.exp0.box

shapeclustering -F font_properties -U unicharset -O font_name.unicharset font_name.font.exp0.tr

mftraining -F font_properties -U unicharset -O font_name.unicharset font_name.font.exp0.tr

cntraining font_name.font.exp0.tr

You should, in there close to the end see some output that looks like this:

Master shape_table:Number of shapes = 10 max unichars = 1 number with multiple unichars = 0

That number of shapes should roughly be the number of characters present in all the image files you've provided.

If it went well, you should have 4 files created: inttemp normproto pffmtable shapetable. Rename them all with the prefix of your_language from before. So e.g. your_language.inttemp etc.

Then run:

combine_tessdata your_language

The file: your_language.traineddata is the model. Copy that into your Tesseract's data folder. On Windows, it'll be like: C:\Program Files x86\tesseract\4.0\tessdata and on Linux it's probably something like /usr/shared/tesseract/4.0/tessdata.

Then when you run Tesseract, you'll pass the lang=your_language. I found best results when I still passed an existing language as well, so like for my stuff it was still English I was grabbing, just funny fonts. So I still wanted the English as well, so I'd pass: lang=your_language+eng.

You should separate creating your menu from your content. Please review the following example. I decoupled your menu, component, and event logic into meaningful phases.

a) Even if obviously it could be better, is this solution is acceptable or not at all ?

I think it has some problems you want to address, but the general idea of using the filesystem and storing just filenames can be acceptable, depending on how many users you need to serve and what guarantees regarding data consistency and concurrent accesses you want to make.

I would consider the uploaded file temporary data that may be lost on system failure. If you want to provide any guarantees of not losing the data, you want to store it in a database instead of on the filesystem.

b) I pass 'uploaded_file' from one view to another using "request.session" is it a good practice? Is there another way to do it without using GET variables?

There are up- and downsides to using request.session.

• attackers can not change the filename and thus retrieve data of other users. This is also the reason why you should not use a GET parameter here: If you used one, attackers could simpy change that parameter and get access to files of other users.
• users can upload a file, go and do other stuff, and later come back to actually import the file, however:
• if users end their session, you lose the filename. Also, users can not upload the file on one device, change to another device, and then go on with the import, since the other device will have a different session.

The last point correlates with the leftover files problem: If you lose your information about which files are still needed, it makes cleaning up harder (although, in theory, you can retrieve which files are still needed from the session store).

If it is a problem that sessions might end or change because users clear their cookies or change devices, you could consider adding the filename to the UserProfile in the database. This way, it is not bound to sessions.

c) At first my wish was to avoid to save the csv-file. But I could not figure out how to do it? Reading all the file to request.session seems not a good idea for me. Is there some possibility to upload the file into memory in Django?

You want to store state. The go-to ways of storing state are the database or a session store. You could load the whole CSVFile and put it into the database as text. Whether this is acceptable depends on your databases ability to handle large, unstructured data. Traditional databases were not originally built for that, however, most of them can handle small binary files pretty well nowadays. A database could give you advantages like ACID guarantees where concurrent writes to the same file on the file system will likely break the file. See this discussion on the dba stackexchange

Your database likely has documentation on the topic, e.g. there is this page about binary data in postgres.

d) If I have to use the tmp-file. How should I handle the situation if user abandon upload at the middle (for example, he sees the confirmation page, but does not click "yes" and decide to re-write his file). How to remove the tmp-file?

Some ideas:

• Limit the count of uploaded files per user to one by design. Currently, your filename is based on a timestamp. This breaks if two users simultaneously decide to upload a file: They will both get the same timestamp, and the file on disk may be corrupted. If you instead use the user's primary key, this guarantees that you have at most one file per user. If they later upload another file, their old file will be overwritten. If your user count is small enough that you can store one leftover file per user, you don't need additional cleaning. However, if the same user simultaneusly uploads two files, this still breaks.
• Use a unique identifier, like a UUID, and delete the old stored file whenever the user uploads a new file. This requires you to still have the old filename, so session storage can not be used with this. You will still always have the last file of the user in the filesystem.
• Use a unique identifier for the filename and set some arbitrary maximum storage duration. Set up a cronjob or similar that regularly goes through the files and deletes all files that have been stored longer than your specified maximum duration. If a user uploads a file, but does not do the actual import soon enough, their data is deleted, and they would have to do the upload again. Here, your code has to handle the case that the file with the stored filename does not exist anymore (and may even be deleted while you are reading the file).

You probably want to limit your server to one file stored per user so that attackers can not fill your filesystem.

e) Small additional question : what kind of checks there are in Django about uploaded file? For example, how could I check that the file is at least a text-file? Should I do it?

You definitely want to set up some maximum file size for the file, as described e.g. here. You could limit the allowed file extensions, but that would only be a usability thing. Attackers could also give you garbage data with any accepted extension.

Keep in mind: If you only store the csv as text data that you load and parse everytime a certain view is accessed, this can be an easy way for attackers to exhaust your servers, giving them an easy DoS attack.

Overall, it depends on what guarantees you want to make, how many users you have and how trustworthy they are. If users might be malicious, you want to keep all possible kinds of data extraction and resource exhaustion attacks in mind. The filesystem will not scale out (at least not as easily as a database).

I know of a similar setup in a project where only a handful of priviliged users are allowed to upload stuff, and we can tolerate deletion of all temporary files on failure. Users will simply have to reupload their files. This works fine.

Those are stray bytes like *, %, etc. prefixed with some of the lines.

The stray bytes appear to be a custom stream multiplexing protocol, allowing STDOUT and STDERR to be sent down the same connection.

Using stdcopy.StdCopy() interprets these custom headers and those stray characters are avoided by removing the protocol header for each piece of data.

Refer: https://github.com/moby/moby/blob/master/pkg/stdcopy/stdcopy.go#L42

If you are interested and you do have access to Microsoft 365's dynamic arrays:

Formula in B1:

This can be done in C# by customizing the JsonMediaTypeFormatter (from the NuGet package Microsoft.AspNet.WebApi.Client) like so:

The site you mentioned links to Unicode in RTF:

If the character is between 255 and 32,768, express it as \uc1\unumber*. For example, , character number 21,487, is \uc1\u21487* in RTF.

If the character is between 32,768 and 65,535, subtract 65,536 from it, and use the resulting negative number. For example, is character 36,947, so we subtract 65,536 to get -28,589 and we have \uc1\u-28589* in RTF.

If the character is over 65,535, then we can’t express it in RTF

Looks like RTF doesn't know UTF-8 at all, only Unicode in general. Other answers for Java and C# just use the \u directly.

Interesting question. I think the first thing to note is that, as mentioned on this Github issue, rvest and xml2 use httr for the connections. As such, I'm going to introduce httr into this answer.

The following code chunk shows how to use httr to query a url using a proxy and extract the html content.