There are times when the partner with 15 hcps (or more) should pass,and the person with 5 hcps should balance (or double). This is not one of them.
The issue is the bidder or doubler should be the one with shortness in the opponents' suit. Here you have length in the opponents' suit, which deters you from bidding. Partner also likes to bid with five card suits but here your heart suit is nothing special. Opposite say Qxx of hearts and a 12 point opener, you will go down, perhaps badly, at three hearts.
Partner should have been the one to act with 20s hcps AND the singlesingleton spade ace. The position cried out for a double from him. You would have made three of your best suit with 25 high card points between you.