According to the official rules, after completing the structure these things happen:
When you complete a structure, take its Site card and tuck it underneath the top edge of your Camp so that the gold coins are showing. The materials stay with the completed structure.
According to that, the materials used to build this structure need to stay below, for the whole game.
Anyway, the buildings stacked on the 2-3 materials, especially when you've put your cards into sleeves, are still moving a bit, and they make some chaos because of that stacking.
I'm wondering if they could be just put away from the game, after completing the building? I was looking for the others buildings rules, trying to search any situation, when materials of completed building could matter in any way.
Do you know any situation of this kind? Do you think I can remove them from the game safely?