net-adapter | Net Adapter – AKA the “ Get IP/MAC address ” library | Networking library

Your hardware setup sounds fine. Note that you don't necessarily need a direct ethernet cable connection, you can use your normal office-/home-wide wired network, it's fine to have a switch (or even more than one) between debugger and target. The debugger machine can even be using wifi as long as it can reach the target's wired network connection that way, though it's not recommended. The Thunderbolt Ethernet interface need not be Apple branded, but its driver does need to include kernel debugging support. In practice, for example, I've successfully used the Ethernet port in a OWC brand Thunderbolt Dock as well. Using one of Apple's own adapters of course guarantees it will work.

If the boot-args on your target machine are the same as what you used in the VM, that's probably where your problem is coming from. You should specify the device to be used for kernel debugging, using the kdp_match_name= option. Use the ifconfig command in the Terminal to work out which of the listed devices is your Thunderbolt ethernet adapter (enX), possibly based on the IP address. en0 is typically wifi in Mac laptops, so you're usually looking for X > 0.

So you'll end up with something like kdp_match_name=en1 added to your boot-args. Make sure to reboot the machine cleanly before attempting a debugging session after updating the boot-args setting.

A few more comments:

kext-dev-mode=1

This option no longer has any effect. It only existed in OS X 10.10. You can remove it.

I didn't find the exact KDK version, but I don't think it should be the problem here

Having a KDK version that doesn't match the target's exact kernel binary UUID will not prevent a connection, but it will prevent you from doing any meaningful debugging as the memory layout of threads etc. will not be available.

My debugger is Big Sur, and my debuggee is Catalina

If you ever run Big Sur on your target, make sure to add wdt=-1 to your boot-args or the hung/crashed machine will reboot before you get a chance to connect with the debugger.