Each Layer III frame holds about 26ms of audio data (at 44.1kHz) so you could see up to this much latency just to sync with the next frame.
If you are unlucky enough to jump past part of the bit reservoir holding bits for the next frame, the frame will not be able to be decoded and will be skipped, adding even more latency.
I know that this would require some work on the decoder, but I think that it should be possible to jump to a granule instead of a frame.
Of course you can do better than that. You can jump to any PCM sample position you like; you just have to find the right frame containing the position you want to jump to.
There is a caveat. In order to properly decode samples in one frame, it is often necessary to decode the previous frame so that the Layer III IMDCT overlap outputs are correct and the synthesis filterbank is in phase, since both carry state across frames.
Cheers, -rob