I ran some tests myself since no answer could be provided.
For this tests, I had to use Bane & Archfoe bosses since there were no Nemesis spawned and my friends didn't want to risk a Dreamcourts boss loss, but we can assume they work the same.
To the question is there a time limit for a boss to count towards Bosstiary?
-Yes. The limit is 5 minutes. It is the same limit as a regular creature for it to count towards Bestiary, so if you are Nemesis-hunting and hit it, be aware your timer may run out.
Is there a time limit to get loot from a boss you have damaged?
-Yes. The limit is 30 minutes. Between 5 and 30 you don't receive a Bosstiary point but are eligible for loot.