The reason for fixing bugs immediately is not only the cost of content switching later on when you have to come back, but it is even more the cumulative cost of fixing all the code you have built on top of the bug you introduced.
Think about it this way. Bug = Bad code. If you have a bug and continue development you are developing working code (good code) on top of that bug (bad code).
When you later on go back to fix the bad code, you will then have to fix all of the good code that is now broken because of the bug fix you introduced.
Software is build in stacks, the lower level affects directly the upper levels (many of them), if you have a bug lower in the stack and then fix it you will have to change all of the upper levels! And that is where the real cost is.
So, if you find a bug, fix it now. If you did not find a bug, think of ways you could find it faster -- before you build good code on top of bad code.