To start with what might help you is to use a traceroute program such as WhatRoute this will help you to determine where the IP packets are falling down.
Do this from the local machine out and also if possible from the remote machine in. It will show the Min, Avg and Max reponse times of each router/host that your IP packets pass through and help you determine where to start to look at least.
