When it comes to swing speed, the only place you really need it is in your follow through. Most amateur golfers decelerate and the resulting shot is a chunk, a push or a wild slice. Try this: take a practice swing and “swoosh the shaft,” as you finish. Make the shaft sound LOUD, as you follow through! Then, when you are ready to hit your shot, swing aggressively, through your finish. Swing through with conviction! Swing speed should feel natural, not forced.
Remember to use soft grip pressure and turn your hips and shoulders in sequence. But ultimately, focus on a fast finish!