Not What I Would Have Done
Many of your reasons, to me, can be chalked up to one thing: fear. There’s the fear of judgment, which you mention, but more than just with your relationship. It seems to me that the reason you think so much about others and seek to build wealth is to please others or be pleasing to them.
Often, giving others the responsibility of making you happy is unfair to them. Your situation is the other side of that coin, where you have entered into a position that you seem to question — a life-long commitment, at that — mostly because you are afraid of what others would think.
It’s your actual life, and you only get one. If you can live with your decision, and at the end of the day you can call yourself happy, then you win, man. You win. If not…