This bit I agree with. But imagine we had a PR List electoral system
You still wouldn't be voting for a Prime Minister, you'd be voting for a Party unlike now where you vote for a specific MP of that party in your area
The exact same thing could and would happen (though it would be less likely I think)
Then look at the Presidential system (it doesn't have to be the American one), if you vote in a President for 4 or 5 years, you are stuck with him or her, regardless of how they are doing. Under our system, the party can act (as we've seen) to remove the PM. Even under the Presidential System, there is also usually a Prime Minister and the big question is how much power do you give to each. (that varies a lot)