Valheim's raft works a little differently than I am used to from other video games; these are some notes on how it works. The most important thing is to pay attention to the navigation wheel indicator. The other thing is to row unless the wind is going your direction.
When steering the raft from the rudder, you should pay attention to the little wheel indicator at the bottom of your screen. This wheel shows you two important things: Your "navigation speed," and your turn rate.
The first thing is the navigation speed, or arrows. If you hit forward or back (W or S in the default keyboard bindings) you can change your speed. You can go "1 arrow" backwards, which means you're rowing backwards (shows 1 down arrow, red), you can row forwards (1 up arrow, yellow), you can half-sail forwards (2 up arrows), or you can full-sail forwards (3 up arrows). There is also "no arrows" which means you aren't rowing and your sails are folded up – so you're only moving under momentum. The sails respond to the wind – more on that later – so you'll want to default to rowing.
The second thing is the turn rate indicator. There is a circle around the wheel indicator. When you turn left or right, a yellow bar fills in the radius of the circle. When the bar is pointing left, you are turning left. When it's pointing right, you're turning right. If the circle is empty, you aren't turning. Note that hitting left or right strafe (A or D in the default keyboard bindings) will adjust this turn rate. The turn rate does not automatically decay to "no turn" – the rudder stays where you left it – so you always have to steer back to "neutral/forwards" if that's where you want to head.
By default you'll want to row, because this works (albeit slowly) no matter the wind. Pay attention to the wind indicator on your minimap. That's the white arrow in the bottom left of the minimap. If your raft is pointing roughly the same direction, you can unfurl the sail to go faster. If it's not, better keep the sail up and row, because otherwise your raft will be pushed sideways or even backwards.