I certainly recognize what you're saying about the emotive nature of slavery as a concept, and the tendency to get mushy in our thinking about it.
I agree with you that just using the word 'moral' is a problem, and that there is no "outside moral standard"--what in philosophy is often called a Transcendental Moral 'Ought'. This relates to David Hume's famous formulation, afterward called "Hume's Guillotine"; There is no possible account of how things are that can tell us how things ought to be. There are no Transcendental Moral Oughts, but there are what we call "Rational Oughts"; this is usually an "if-then" statement. If you want a society that recognizes private property, then you ought to criminalize theft. That's not a moral argument, but a rational one.
So perhaps I would reformulate that sentence. An Epicurean can make a rational case against slavery by citing the non-willingness of the slave as an impediment to justice, by definition.
Could the thief make the same defense as the slave? "That may be your convention, but I didn't agree to it. My convention says I can take what I want." I don't believe so; the thief is pleading himself out of the pact, and therefore forfeits its protection. The slave is trying to plead himself into the pact, and is therefore worthy of it.
Another way to put it; if a man takes as his mantra that line from Achilles--that there are no pacts between lions (himself) and men (society)--then he can't very well complain when society treats him like they would a man-eating lion, can he? This is again a rational rather than a moral conclusion.
Good points! It always helps to clarify one's thinking.