I change mine (07 - X2) several times a year depending on the body of water I'm boating on and do it without messing with the rudder linkage. Now that I'm not in front of the boat I'm trying to remember which way I turn the rudder. I believe I push the blade of the rudder to the left. The rudder is slightly offset to the left which when biased in that direction will give you some additional room.
Let the prop slide all the way down the shaft so its off the splines and will turn freely, at that time you should be able to turn and rotate the prop enough to get it by the rudder. Don't try to force it, just turn it while moving it to a vertical position as you slide it off the bottom of the shaft and it should almost fall off. Take your time when you're turning it to find the location with the most clearance. You may also get just a little more clearance by moving the rudder blade back to the right a bit.
It takes a couple times to get the procedure down but believe me it will come off if turned and positioned correctly. After all someone put it on there and I can assure you they didn't remove the rudder to put it on. I've done mine so many times that I've actually done this while the boat was in the water after hitting a sunken island this summer in the Detroit river. The key is rotating the prop vertical once you find the location of maximum clearance. It's like on of those deals where you pull and pull then when you've given up and are about to walk away it falls off. You have to think in terms of Chinese finger cuffs.