'He's so small, I struggle to watch': Should kids' sports be grouped by size, not age?

Kids develop at different rates — so is it fair to simply group them into sports teams by age? Would focusing on their size or ability help level the playing field?