I thought you were talking about the “regular” rubbers, for pencil. Those rubbers are more “grippy” than paper, so the graphite particles will stick to it.

This doesn’t work for ink, of course, and those rough rubbers literally scratch the paper with ink off of page.