In our family, we've always interpreted the rules as paying to the bank. We had friends over who had played by Warlord paying to the player whose buildings they got destroyed. To us, it seems like the latter weakened the Warlord special ability quite significantly. I could not find an FAQ that would explain where the money goes.
I have two reasons to conclude you pay to the bank:
The rulebook doesn't say that you have to pay to the bank when building a district. This is assumed because it wouldn't make sense otherwise. Similarly, if the rulebook doesn't say whom to pay to destroy, then I assume it must go to the bank.
The warlord can destroy one of it's own district if he wants to replace it. The warlord still has to pay the amount, which then obviously would not go to himself but to the bank.