I'm pretty sure I just found the solution while thinking about it in the bathroom
Here's it in theory:
- Both sides only make knight moves (and king move/s of course). This is important because knights can't make tempo moves.
- The Bishops and Rooks and Kingside will be captured and the Kings will be on g1/g8
- If both sides castle that will be 2 halfmoves for the king to reach their destination square. If neither of them castle it will be 4 halfmoves. And now comes the keypoint: If only one of them castled it will be 3 halfmoves.
- Now remember that knights cant do tempo moves.
Conclusion: If in a given position you don't know who's turn it is, you won't be able to tell which side castled. You will only be able to tell (from the position of the knights towards each other) that the kings made 3 halfmoves, which means 1 side castled and 1 didn't.