Say that I have Desperate Castaways and Walking Ballista with 1 +1/+1 counter on the battlefield. This means that Desperate Castaways can attack at the moment.
What would happen if I declare Desperate Castaways as an attacker, and then, at the beginning of "declare blockers", remove the +1/+1 counter from the Walking Ballista? The Ballista dies, of course, but does that mean that Desperate Castaways will be removed from combat, or does the "can't attack unless X" restriction apply only to declaring it as an attacker (which would mean that it would remain as an active attacker)?