From William’s description, my guess would be that the web server is not allowing partial-content downloads, and so the video player is not able to start downloading the video file part-way through. Hence, it has to download the whole preceding part of the video before the jump facility starts working.
If this is the problem, it may be fixed by some lines like the following in the relevant htaccess file:
Header set Access-Control-Allow-Origin “*”
Header set Access-Control-Expose-Headers “Content-Type”
Header set Access-Control-Expose-Headers “Accept-Encoding”
Header set Access-Control-Expose-Headers “Range”
… with the last line being the particularly important one. However, WordPress possibly has a switch somewhere for controlling this behaviour.
I vaguely recall seeing exactly this same problem on another website, many moons ago… possibly the previous BAA website, in fact!