What I have considered, though, is making parts of it open source, and keeping only the “secret sauce” proprietary. The open source parts would be stuff that could be used to build similar software for other niches of the same target industry, whereas the super specific niche stuff and all the regulation compliance stuff (much of which is just for that one niche anyway - other niches have different regulations) would be proprietary.

This seems perfectly reasonable and I wish you the best of luck. Just don’t expect anyone to provide the infrastructure for your proprietary secret sauce for free!