If you keep the scope fairly low and your skill set is diverse enough, it can definitely be done. I wouldn’t say it’s “fun”, but it can be done.
Personally, I’m aiming for a level of complexity (the AA space) on my game I could never fully realize without a small team, but I’ve been working on a publisher greenlight demo and first public trailer (intended to be released at the same time) for about a year and a half – the last two months of which has been full-time. And despite about 11ish years of experience across a number of disciplines, it’s still hard as hell both in development, mentality, and overall optimism for the thing. Not having any team members on a project is just not that great.
But, that’s me, so that may or may not be the case for you.