Generally yes, but technology thankfully keeps moving forward. I'm soon to upgrade the turbo in my WRX. It will be a bigger turbo, but with the billet compressor wheel, will give me similar spool up to what I have now. That is important to me because spool up and torque are more important than power not just off road, but on the road and the track. Horsepower figures are there to impress people- torque gets results
The turbo in the GT Forester is apparently really small- smaller even than the TD04 that was standard in most 2 litre WRX's. So if you can get a billet wheel for say a TD04 or similar sized turbo, you will have quicker spool up than a TD04- maybe the same as what you have now and also be able to run more boost. You could also look at twin scroll turbos and while they work well you do lose the boxer note which is a price I am not willing to pay for a road car.
Of course, then you may have to look at injectors but there is good news there as well. Newer technology and finer tolerances all make up to noticeably improved performance. A new fuel pump- say a Walbro and of course a re-tune would all be part of the deal. Cheap it won't be.
Finally, as always it will come down to a compromise between on road and off road performance. Having been on many trips with Damo and seeing how his car handles the conditions , it seems turbos will always be more of an issue than non turbos. It does not stop him normally- but it has been an issue which has surfaced from time to time.