Kick off the scavenger hunt
You might pair up smaller children with an adult, or have older kids work in pairs. That way, if they wander out of view, you know everyone is safe.
But that brings up a key point - you must now set the boundaries for the hunt. Do they have to stay in your backyard? If you are at a park, what are the boundaries? If it is not super obvious, you might tie flags on trees or poles to remind them of the boundaries - to keep things safe and fair.
Also, tell them how much time they have, and what will be the signal to come back. Perhaps you will ring a bell or hit a gong!