which is why I am saying it is a server side, it may be producing higher pings as more people are joining in from outside the network. Moreover, there are chances that certain patch or update of your game would be messing with the latency as well. It does not make sense that if other servers (assuming they are on the same network as L4D server) are not giving out higher pings.
__________________
|