This seems like a very close issue as raised in Lucretius Book 3 when he points out that even if our atoms were rearranged later due to the effects of infinite time and space, that would still not be "us" because of the absence of continuous memory.
However I am not sure as I reread that whether Lucretius is making a specific assertion that continued memory is somehow necessarily impossible. He may be relying solely on the objection that we don't remember any past lives, which I gather he is taking as sufficient proof that these rearrangements have already happened. He may well be inferring from the fact that we have no such memories that this is sufficient proof that the break is a matter of fact regardless of the cause.
I tend to think that that is his reasoning and that given the implications of infinite universe/eternal time that the inference is sound.
3-843
And even if the nature of mind and the power of soul has feeling, after it has been rent asunder from our body, yet it is naught to us, who are made one by the mating and marriage of body and soul. Nor, if time should gather together our substance after our decease and bring it back again as it is now placed, if once more the light of life should be vouchsafed to us, yet, even were that done, it would not concern us at all, when once the remembrance of our former selves were snapped in twain. And even now we care not at all for the selves that we once were, not at all are we touched by any torturing pain for them. For when you look back over all the lapse of immeasurable time that now is gone, and think how manifold are the motions of matter, you could easily believe this too, that these same seeds, whereof we now are made, have often been placed in the same order as they are now; and yet we cannot recall that in our mind’s memory; for in between lies a break in life, and all the motions have wandered everywhere far astray from sense.
