Good question! Any bonus you use to increase your experience gain is based on the base experience value provided by the monster, this means that when using an XP prey with or without boost the additional gain of experiencie provided by the prey will be the same.
For example (considering green stamnia 150%):
If a monster give you 1000 points of experience, with boost an prey of XP 10 stars (40%) we have:
(1000 * 2,25) + (1000 * 0,4) = 2650 points of experience
Only with prey we have:
(1000 * 1,5) + (1000 * 0,40) = 1900 points of experience