I'm not at all sure what the prevailing agreement is by rubber players, but the Standard American Yellow Card is marked "RONF" (raise-only non-forcing), which I suggest you and your partner adopt. The SAYC booklet has this to say:
“RONF” on the card means “Raise Only Non-Force.” A new-suit response is
forcing one round and shows at least a five-card suit. Opener should raise a major
suit response with a three-card fit, or perhaps with a doubleton honor.
With no fit for responder’s suit, opener rebids:
With a minimum weak two-bid (5–8 points), rebid the suit at the lowest level.
With a maximum weak two-bid, name a new suit or bid notrump.
The reason for playing RONF is that it is very rare that you can profitably improve partner's preempt to a new suit when you hold a weak hand and a void in partner's suit (which is one situation in which many beginners try to bid 2H over 2D). On the other hand, it is not so uncommon to have the potential for a game across from partner's preempt with a hand that has its own suit (as your partner had here). What's more, the gain from finding those games is much higher than any loss that may be prevented by improving a partscore with a nonforcing bid.
In fact, your hand is not weakish. Your suit is not so good but you have an outside ace. I would put in 2NT with your hand. Now you should discuss further bids with partner, but I would suggest that responder's next bid should be taken as a concentration trying to determine whether 3NT is the correct contract (unless opener rebids the suit), so I think a reasonable auction is:
4H 5D (1)
(1) Partner may also consider passing given that you've shown a heart doubleton and denied a spade card; they can see two spade losers are likely but a club loser is not (would you have bid 2NT without a spade stopper and without the club ace?), but that a diamond loser is also possible. If hearts are 3-3 (36%), you'll (usually) take the same number of tricks in hearts and diamonds, meaning you can make 4H when 5D goes down. When hearts are 4-2 (48%), you'll likely make one fewer trick in hearts than diamonds (making 4H when 5D makes). It's only when hearts are 5-1 or worse (16%) that hearts is likely to be a a worse contract than diamonds.