"Coastline" is used three times in the rules. The following is the first instance:
Roads and ships may not be built along the same coastline.
The very next sentence explains what this means.
There can only be one ship OR one road along any given coastal hex side
As such, the rules define "coastline" as "coastal hex side". And for any given coastal hex side, you can't use both a ship and and road.
The following are the other two instances:
If your shipping route reaches a coastline, you can then build a new settlement on that coast
[You] may elect to place either or both of their starting settlements on the coastline. If you do so, you have the option of placing a ship, instead of a road, adjacent to that settlement.
Since settlements can't be placed on hex sides, this is obviously a different usage of the term. The standard English definition applies here: The coast or coastline is the line that separates water from land.