There is some difference in difficulty based on which color becomes faded; but shouldn't change things very much compared to other random chance elements in the game.
As a whole, Blue being faded makes for the most difficult games.
The biggest difference between the colors is that they each have a different number of connections to other colors. This increases the number of faded counters that will end up on other colors on average.
Based on this statistic, blue is a difficult color to get, because it has 13 total connections to other colors, compared to 9 for Black, 8 for Yellow, and 6 for Red.
There are some other factors such as the number of internal connections; but ultimately which random cards you draw and when both from the player card deck and the infection deck will have a bigger luck-based impact on how difficult it is compared to which of the 4 colors ends up faded.
For more details and analysis, see here (where I got my numbers from).