mojibake | Recover mojibake text using a reverse-mapping table

I've had a website running for almost 20 years, unfortunately I made the mistake of not aligning the HTML character set with the MySql character set, so all of my data seems to be double encoded (I think) or possibly mojibaked, or both. Perhaps one of you experts can clear this up for me.

Before I go on, you should know that I'm intending to upgrade to tomcat 9 HTML5 with UTF8 characters and emojis

In R, I have vectors like this:

Mysql table looks like this:

The actual contents are like this:

I tried these: https://stackoverflow.com/a/23179613/13865853

in model.php

I am trying to create a discord bot for my server with Python and I am trying to set my token, special variables, etc. So I set my variable in the .env file and called it. The message worked but it has a "Mojibake" problem that looks like this:

we're making keyboard with five sensor and we have problem with mojibake

if we click sensor 'a' five time, there should be five 'ㄱ'

but there are mojibake like attached picture.

we think this problem is related with unicode but can't find what the exact probelm is

please answer what is problem. thank you

I'm trying to decode the JSON you get from Facebook when you download your data. I'm using Node JS. The data has lots of weird unicode escapes that don't really make sense. Example:

How can I store emoji from TextMesh pro in a database using the WWW class to post the string containing the emoji as a parameter in the URL?

I'm having a problem getting the emoji from a TextMesh Pro text field into my database. When I try, the emoji data is stored in plain text like this: 😂 or like this � or like this □ depending on which encoding i try.

I have a php script that uses a sql statement to store the text in my mysql database. And when I manually type the url of the php script in my browser and add the emoji as a parameter it works perfectly fine, it properly stores the emoji (as I have already set the collation of my database to utf8mb4).

Here's the part I don't understand: if I take the string that contains the emoji through my c# code and access the php file with the string as a parameter, it doesn't work. It stores the emojis as mojibake. (😂 this sequence of characters should be this: ).

It stores the text just fine otherwise, so there's no problem with the code below. Here's how it looks:

While trying to read a UTF-16 encoded file with hints from this answer, I got the problem that, after reading few thousand characters, the getline-method starts to read in garbage mojibake.

Here is my main:

Using Python 3.6.1, Requests 2.13.0, I am getting strange encoding of the URL being requested. I have a URL with Chinese characters in the query string, for example huà 話 用, which should %-encode to hu%C3%A0%20%E8%A9%B1%20%E7%94%A8 or even hu%C3%A0+%E8%A9%B1+%E7%94%A8, but for some reason it is %-encoding to hu%C3%83%C2%A0%20%C3%A8%C2%A9%C2%B1%20%C3%A7%C2%94%C2%A8. This is not correct. I've been using http://r12a.github.io/apps/conversion/ page to help me work the encodings. I've also used JavaScript encodeURI and PHP urlencode and don't get anything near what I see the Requests library doing.

Am I doing something wrong such that the encoding is so far off?

UPDATE: I looked into Mojibake encoding and dug into the Requests library a little more and found out what the problem is, but I'm still not sure how to fix it.

I'm making a call against an internal server, using a simple .get(url) call. The call goes to the server and gets a redirect response. The redirect page has a meta charset="UTF-8" in the header and the URL listed in it is correct. The location header leaving the server is ok; it is encoded as UTF-8 and the Content-Type header has a charset=UTF-8 on it. However, when I debug the redirect response in Python I note that the header on the response object is incorrect, it doesn't seem to be decoded correctly. The headers property contains this in location: huÃ\xa0 話 ç\x94. As said above, it should be decoded as: huà 話 用. So, that strange URL query string get's % encoded by Requests and set back to the server, which then rejects that URL (obviously).

Is there something I can do to prevent Requests from messing this up? Or get it to correctly decode the location header? Web browsers don't seem to have trouble with this.