Control-Flow-Integrity | proposed hardware-based method | Reverse Engineering library

I think the idea for 32-bit CFI is to set a limit on DS/ES/SS, and put the shadow stack outside that, so it can only be corrupted by instructions with a gs: segment override prefix.

This only works in compat mode (32-bit user-space) on x86-64, because CS/DS/ES/SS have fixed base=0 and no limit when the CPU is in long mode.

FS and GS can have a non-zero base even in long mode, but even if you could set a limit on those segments, it's the opposite of what you need. (That would protect regular memory from instructions with a gs prefix, instead of the other way around.) Fun fact: @MichaelPetch says most CPUs don't support a limit on those segments in long mode anyway.

For the Linux x32 ABI (32-bit pointers in long mode), you could put the shadow stack outside the low 4GiB of virtual address space. Normal compiler-generated code always takes care to truncate pointers to 32-bit. The current default code-gen strategy (-maddress-mode=short) is to use address-size prefixes on every instruction with a memory operand, except rsp, which is assumed to always be zero-extended.

This leads to a lot of wasted address-size prefixes, but it means that code compiled for x32 by current gcc will definitely truncate pointers to 32-bit, even when there's UB, making it impossible for compiler-generated code to load/store outside the low 4GiB of address space.

I don't think long mode has any general way to make a region of virtual address space accessible only with some kind of special instruction, either a prefix or something else. So if you're defending against normal code using 64-bit operand-size, there are no options that replace segment tricks with baseline x86-64. You'd have to map / unmap the shadow stack, which would be unusably slow.

There are several other technologies for enforcing / checking control flow these days. Intel's CET (Control-flow Enforcement Technology) gives hardware support for various stuff, including a shadow stack. Grsecurity published a review / critique of it. Their conclusion starts with "In summary, Intel's CET is mainly a hardware implementation of Microsoft's weak CFI implementation with the addition of a shadow stack." IDK if this review is accurate. Grsecurity does produce a competing product (which they acknowledge), so there might be some bias. I'm sure CET is better than nothing.

See also https://github.com/huku-/research/wiki/Intel-CPU-security-features for a list of x86 (or maybe specifically Intel) CPU features that assist security in general (like MPX bounds checking), not just control-flow.