to-utf-8 | Detect input encoding and convert

I'm using redcarpet gem to render some markdown text to html, a portion of the markdown was user inserted, and they typed in a totally valid special character (£), but now when rendering it I get a: Encoding::UndefinedConversionError "\xC2" from ASCII-8BIT to UTF-8

I know it's the £ sign because if I replace it in the text to render then it all works. but they might be inserting other special characters.

I'm not sure how to deal with this, here's my code building the html:

On recent versions of Win10 it is possible to set the Active Code Page (ACP) to a UTF-8 code page. And as discussed here, it is possible to set the System Locale (used to map between the "A" version and "W" version of the Windows API) to use the UTF-8 code page.

How does a script detect if the UTF-8 code page is in use?

As discussed here and here, it is normally possible to use WMI to get the system code page ID:

Is there a way to convert a \x escaped string like "\\xe8\\xaa\\x9e\\xe8\\xa8\\x80" into readable form: "語言"?

I am successfully modifying multiple Excel files using library(RDCOMClient). However, setting a cell value to a non-ascii string results in å becoming Ã¥ etc. I also cannot pass an UTF-8 filename to Excel's Open() and Save() methods. Hopefully there is a single solution to both problems.

Here's a simple reproducible example using Save(): Creating an empty workbook and trying to save it as å.xlsx results in Ã¥.xlsx. The same operation works fine for a.xlsx.

I am building a web application in NodeJS version 12. I have data from an old MySQL database. There are several fields that contain characters that are not displaying properly due to an encoding issue with the old database. There are some similar questions already but none of them have solved my issue. After trying, I'm a little closer to a solution, but still need help on this.

Current value in database to convert:

Just like this question, I need to convert html entities (e.g. &) to UTF-8 (&) while ignoring other UTF-8 characters. The difference is that in my case, I need to do this via the bash command line.

I can use a tool like recode and run echo '&' | recode html..utf-8 which converts over to & just fine, however with UTF-8 characters in the string, like in

I have multiple gzfile in subfolders that I want to unzip in one folder. It works fine but there's a BOM signature at the beginning of each file that I would like to be removed. I have checked other questions like Removing BOM from gzip'ed CSV in Python or Convert UTF-8 with BOM to UTF-8 with no BOM in Python but it doesn't seem to work. I use Python 3.6 in Pycharm on Windows.

Here's first my code without attempt:

in a C++ software using the libmysqlclient, I am trying to insert a row into a table containing html-encoded characters such as é. The request looks like this:

This question is related to another one which went the perl way but found much difficulties due to Windows bugs. (see Perl or Powershell how to convert from UCS-2 little endian to utf-8 or do inline oneliner search replace regex on UCS-2 file )

I would like the POWERSHELL equivalent of simple perl regex on a little endian UCS-2 format file (UCS-2LE is same as UTF-16 Little Endian). ie: