Using UE4 template assets in marketplace demos?

Is it permissible to use material from a starter template in a marketplace package for demo maps? For example, I would like to include a stripped 3rd person template with the mannequin to allow buyers to run around and try their new goodies.

Yes, that is perfectly fine. You can use any of Epics free content in your submission, but it cant be the focus of the pack. In your case its not a problem, although I would make sure that you keep all template content separate from your own content so the customer can remove it without issues.