Should I remove myself from a friend's credit card as an authorized user because he keeps a high balance?

Last year a friend of mine with excellent credit added me as an authorized user to his credit card to help me get my mortgage. His account was in good standing since '93 and his limit was close to $75k. Now, its almost a year later, I have added highly positive credit history, increased lines of credit, lower balances to all of my own accounts. He has been keeping a high balance of $20k+ on his card and when it goes up I notice my score go down a few points. I am wondering whether I am strong enough to stand on my own (no late payments in 6 years), my score is fragile as I had old bad marks against me, I keep my inquiries and accounts as low as possible, and have recently gone from secured Visa to unsecured $3000, then last week to $4500 unsecured. This made my TransUnion score go up 30 points. Account Pros: 2 years positive payment history, and a high line of credit, but Cons: a $20k balance. Should I remove myself from this account? Will it help me or hurt me?