lineclip | A very fast JavaScript polyline and polygon clipping library | Learning library

If the movement that the user tries to make is [dx,dy] then consider the line segments from the corners of the selection to those points translated by [dx,dy] (the yellow lines in the example below). These lines may intersect with the rotated image boundaries (green lines) at certain points (the red dots). If there are no intersections, then the movement is legal. If there are one or more intersections, these will tell you up to which point the movement was legal; the intersection point which is closest to its original position (checking either horizontal or vertical distance is enough to establish this) determines the maximum movement (the bottom right corner in the example). You can then limit the translation to this point.

Depending on which quadrant the direction of the movement is in (towards top right in the example) you can skip checking one of the corners (the bottom left corner in the example); the other corners will always bump into the boundaries first.
You can also skip checking two of the boundaries (bottom and left in the example), by comparing the direction of the movement with the rotation angle of the image.
So you need to check for intersections of 3 line segments with 2 line segments. For line segment intersection code, see e.g. this question.

If the user is dragging only one side, and extending the rectangle instead of moving it, then you only have to check the two corners that are moving.