Ski Pylon Will not fit - 2006 Maristar SS

I bought this boat about a month ago and I went to put the removable ski pylon in the slot and its too tight of an interferance fit. Does that plastic sleeve need to come out? Do I possibly have a pylon that is just a bit to big around? I mostly wakeboard and wakesurf so I did not pay much attention to the pylon fitting when purchasing. If someone could shed some light on this it would be much appreciated.

Thanks!

Tim