It should work but I've noticed some recent versions Unity doesn't update the scene view and game view immediately in some cases. We've had to resort to some minor hacks just to get it to work somewhat, but if you're using an older version of the runtime, those workarounds may not be there.
Did you try playing and stopping the game and seeing if the skin change is applied?
