Maybe you should leave people alone to their personal time buddy? The introverted people of the world need that time to themselves in order to degauss from the constant interaction that our jobs often require. It really makes you realize how Squidward must have felt.
You insisting on getting all-up in your coworkers space and forcing him to engage with you during his only time to himself actually makes you the rude one. Maybe you should learn to read non-verbal cues from people and leave them alone when they're not engaging with you.
Is that such a hard concept to grasp?
If you're a regular Squidward and annoying coworkers are your thing check out this Employee's Coworker That Gets Mad That They Won't Work While On Vacation.