Personally, I think that if you don't make the engine open-source what you're describing will happen. Open-source development allows many people to branch off development and create new variants of your software it also allows anyone to continue development even after the "main" development branch is done working on it. I think in terms of development processes you should take a look at how Unreal Engine or many other open-source games/projects function. Where you have a main development branch but other people can fork the code, submit new features and contributions, or just create a whole new thing from your engine.