Sum Omnia
Let's try abstracting this infinity journey in a way that can be visualized. Imagine we have a large room full of air molecules. Your egonaut is one of these molecules and your current existence is your position in this room. So your molecule moving from one place to another corresponds to you in two different lives.
The Second Law of Thermodynamics tells us that the entropy and disorder of this system is more likely to increase as time goes on. Therefore, you should expect that, given enough time, you will occupy all spaces within this room.
Given infinite time, you should expect that you will occupy all of these space an infinite number of times.
So, you should expect that you will live at some point as everyone and every thing capable of experiential existence.
Even if you take away the entropy room visual, the eternal throwing of the dice would get you into every position eventually and then continue on unabated.
Without a mechanism to ensure only one, assume more than one.
And if eternity, assume infinity.
So, when you see someone else, in a way you are seeing what has been and will again be you at a different point in your infinite journey.
Like with the one-electron universe, it may not literally be the case that they are you at a different time, but it might as well be the case. This is Sum Omnia, Latin for "I Am All," and it's implications for morality are vast beyond belief.