It's normal to pull a bit to one side or the other - it actually makes it easier to drive straight if you're working against a slight pull. But the pull shouldn't be tiring or be something that you're fighting.
If the pull is too strong, do a visual inspection on the rudder - make sure it's straight (get 20 feet behind the boat on the trailer, and visually check it), and doesn't have damage ( a big nick or dent could cause a pull). If everything looks clean, then you could consider simply filing the trailing edge of the rudder - This
is a good thread.