Try to imagine that knowledge, where our memories reside, is like an Infinite and Eternal Sea available for each of us to draw from.

Now imagine that each lifetime is like a bucket that is used to draw from that Sea our lifetime's knowledge which becomes part of our current lifetime's memories. Each lifetime we have, we are only ever able to go and draw one bucket full. At the end of each lifetime, the bucket is emptied back into the Infinite and Eternal Sea of Knowledge. The memories are still there but have returned to the Sea of knowledge from which they have come and to which they return.

Given the above example, it may be that there is a residual leftover in the bucket when our new life starts. Hence we may have glimpses of a previous life's memories. Also, as the bucket is only finite in its capacity to hold, you will never be able to draw beyond its capacity to hold even though you have access to the entire Sea of Eternal and Infinite knowledge!

The above is meant as a thought experiment and not meant to be true or perfect. I am sure that this idea probably has many holes but it's s starting point!