shellex | hex converter , handy tool | Hacking library

I found the solution, but I did not use the thumbnail way.

First, I created a tempdrawing, and then I saved that tempdrawing as a preview.

The solution is adding to the COM object the empty value 'ContextMenuOptIn':

You might want to add the following two lines to ensure compiling your code as 32-bit:

I found a solution as follows. Instead of linking the .xyz extension to a ProgID for the preview handler (XYZPreview.xyzfile), I link directly to the CLSID from the extension and removed XYZPreview.xyzfile entirely. Perhaps having two ProgIDs for the extension was causing the conflict (XYZFile.xyz & XYZPreview.xyzfile). The preview handler entries were modified as follows and both the file association and preview handler now work.

The solution is to relinquish control of the tty to the other process group using tcsetpgrp, and when the child is done, take back the control of the tty using tcsetpgrp again. Note that tcsetpgrp sends SIGTTOU to its caller if the calling process belongs to a background process group, so SIGTTOU and SIGTTIN must be blocked.

I managed to make my shell extension to work. This is not really an answer, but at least a collection of things I have learnet, that may help others.

There were 2 main problems:

1. for part of development there was probably an error in registration of my file type or dll, so it was really not being recognized and called by shell.
2. when this was fixed, the code was called, but was crashing silently.
   Shell runs your code in isolated process, and it does not have access to file system (and probably much anything else either). If you'll try to access it, for example to write a debug log file, your code will crash silently.
   The same will happen if you'll try to use Trace listener that writes to log file (like System.Diagnostics.TextWriterTraceListener)!

So there isn't really an easy way to detect if your code was even run by shell, and verify if your registry registrations part is right.
I tried to attach Explorer process to VisualStudio, but my breakpoint was never hit, even though my extension code was run.
What you can do is opt out of "Process isolation" in your assembly registration. See my question for DisableProcessIsolation=1 param and context. Than your assembly will be loaded by Explorer process directly and you can see that in sysinternals Process Explorer utility. If your registrations are right.
This may require Explorer process restart to work.
If you do not opt out, your assembly will be loaded for only a brief time and than uloaded again, so you will not be able to spot it, or as I read elsewhere, it may be loaded by other process than Explorer.exe entirely.

Few other things I learned:

• try catch blocks in my code for some reason did not catch exceptions. The code still crashed and thumbnail provider did not work.
• no errors were reported in windows event logs
• you can compile your assembly as AnyCPU, and register it with both 32 and 64bit RegAsm. It does work at least for 64bit Explorer (Windows). I did not yet test if 32bit processes can run it too though.
• Some code that worked when my dll was called by test app did not work when same dll was called by shell. For example IStream wrapper.
• There is a problem with shell not releasing file locks when they are passed to managed shell extensions. You can read more about it in this stack question and this blog post. The blog mentions there is no official solution and managed shell extensions are officialy not supported by Microsoft.
  My dirty solution is loading whole file content to sparate MemoryStream (that keep no reference to the otiginal file or IStream) and calling Marshal.ReleaseComObject() in Initialize() method and not keeping any reference to the original IStream pointer.
  Simon Mourier mentions in his comment above that using Marshal.ReleaseComObject() is not a good idea, but that was the only solution I was able to find.
• Which ever way you solve this, do not get rid of your file data / stream access in first GetThumbnail() method call. Shell may keep your thumbnail provider instance and call GetThumbnail() more times, for different thumbnail sizes atc.

There is no guarantee that the ProgID for the .txt file extension is txtfile on every system. Many apps hijack .txt for their own purposes. You should read the actual ProgID from the HKEY_CLASSES_ROOT\.txt key, and then register your handler for that ProgID.

But, more importantly, you really should NOT be modifying HKEY_CLASSES_ROOT subkeys directly, modify the corresponding subkeys under HKEY_CURRENT_USER\Software\Classes and HKEY_LOCAL_MACHINE\Software\Classes instead. This is documented on MSDN:

Class registration and file name extension information is stored under both the HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER keys. The HKEY_LOCAL_MACHINE\Software\Classes key contains default settings that can apply to all users on the local computer. The HKEY_CURRENT_USER\Software\Classes key contains settings that apply only to the interactive user. The HKEY_CLASSES_ROOT key provides a view of the registry that merges the information from these two sources. HKEY_CLASSES_ROOT also provides this merged view for applications designed for previous versions of Windows.

The user-specific settings have priority over the default settings. For example, the default setting might specify a particular application to handle .doc files. But a user can override this setting by specifying a different application in the registry.

Registry functions such as RegOpenKeyEx or RegQueryValueEx allow you to specify the HKEY_CLASSES_ROOT key. When you call these functions from a process running in the interactive user account, the system merges the default settings in HKEY_LOCAL_MACHINE\Software\Classes with the interactive user's settings at HKEY_CURRENT_USER\Software\Classes. For more information on how these settings are merged, see Merged View of HKEY_CLASSES_ROOT.

To change the settings for the interactive user, store the changes under HKEY_CURRENT_USER\Software\Classes rather than HKEY_CLASSES_ROOT.

To change the default settings, store the changes under HKEY_LOCAL_MACHINE\Software\Classes. If you write keys to a key under HKEY_CLASSES_ROOT, the system stores the information under HKEY_LOCAL_MACHINE\Software\Classes. If you write values to a key under HKEY_CLASSES_ROOT, and the key already exists under HKEY_CURRENT_USER\Software\Classes, the system will store the information there instead of under HKEY_LOCAL_MACHINE\Software\Classes.