On earth, race is a social construct that tries to categorize different versions of the same species: homo sapiens. Different species, on the other hand, are animals that don’t share DNA, like dogs and cats.
You need to decide if your new world has various races and species, then you need to create them—quite a tall order, but one with which you can ultimately have fun.
If you decide to have different races in your story, they’ll share DNA. They’ll have superficial differences like different shaped eyes, noses, mouths, and more. But they’ll still be genetically the same, so consider this when creating races in your world.
Species in fantasy and sci-fi novels are entirely different from each other. You wouldn’t expect a dragon in your novel to give birth to a chicken any more than you would expect an elf with pointy ears to bear a human child.
It’s up to you to make sure readers understand if your world’s inhabitants are different races sharing similar DNA, or if they're an entirely different species like a humanoid character with gills who can live underwater. If it's new species you’re after, read on!
You can’t populate your sci-fi world with Vulcans and Klingons. Those species are proprietary. You must create your own.
Focus on creating a few species in depth instead of a host of species that only show up in a paragraph or two. For example, Vulcans, Klingons, and humans are standard in Star Trek, while other species crop up for a single episode or two. Those that crop up shouldn’t get too much of your effort; but you need those who are part of your spacecraft’s crew to be well-developed.
In fantasy tales, you have a ripe selection of species in the public domain like dragons, elves, dwarves, etc. Just stay away from Hobbits, which Tolkien owns. If you really want a Hobbit-like species in your fantasy story, change at least three of its aspects to create something entirely new.
Weighing the Risk
Whether you’re writing a sci-fi or fantasy novel, you run the risk when creating a new species that readers won’t warm to it. Weigh the pros and cons of using what’s already in the public domain. For example, writers have written about elves so often, there’s not much new you can say about them. If you’re craving the freedom of creating your own species, though, you can have fun with it.
Consider inventing a species to tell a specific story. Let’s say you create a bizarre species, but its characteristics are elemental to the story. Without its bizarre aspects, your story would be dead.
On the other hand, you could invent a new species in-depth and then think of ways to use it in a story. It really depends on the type of story you’re creating. For example, if you’re thinking "series," you’ll want to build a comprehensive species and an exciting new world in which they live. At the very minimum, you must determine in advance what they physically look like and their overall disposition.
Think of Chewbacca in Star Wars; you never see another Wookie, so you assume other Wookies are just like Chewy. Because he never actually speaks a word, Lucas didn’t have to go deep to create Chewbacca. He’s more of a straight man for Harrison Ford’s jokes.
Don’t waste your time writing a cast full of different characters who have little to no impact on your story.
To make your new species believable, you need to know where they’re from, its climate, what their physical appearance is like, and social elements like language, history, clothing, etc. If you’re writing sci-fi, you must pay particular attention to their technology.
It’s easy to go overboard, but sometimes it’s necessary to create depth. Write in the backstory and get your beta readers to tell you if any of it can be stripped out when editing.
For a series, do weigh characteristics, history, relationships, language, customs, even combat methods. Once you’ve written the first few books in your series, you can’t go back and retroactively assign new characteristics to a species you need in future stories. For example, a species can’t suddenly sprout wings in Book No. 4 because your main character needs to fly out of a pickle.
If your story includes humanoid species, you need not spend as much time on habitat because readers will expect something similar to earth. But if your species’ habitat is a frozen tundra on a far-off planet, they must have evolved to live in sub-zero temperatures.
Consider how a water habitat requires certain characteristics like gills and webbed fingers and toes. You wouldn’t put heavy armor on a water species because they would sink. But they might use their water world to develop specific weapons like using poisonous fish to develop poisoned darts.
Now consider how an underground species would behave differently from those in a water world. Would it be blind and use echolocation, or have hyper-developed sight to see in underground darkness?
What if you created a species that was amphibious—it lived both in water and on land? How would that affect its characteristics, physical appearance, clothing, technology, combat style, etc.? What if this new species was nomadic and built settlements around your story’s world? What kind of habitat would such a species need to survive and thrive?
Part of habitat means you know what materials your new species builds with. Do they use stones to shape spears and other weapons, or do they have sophisticated engineering skills that let them build huge skyscrapers? Maybe they’re squatters living in another species’ abandoned settlements. Consider how their technology affects warfare, travel, culture, buildings, etc.
How does the terrain in their habitat impede or help them? For example, a species that is hunted, stalked, and killed needs places to hide. They wouldn’t settle in an open plain-type terrain; they’d be sitting ducks. Rather, they might look for dense forests or mountainous terrain in which to live.
Climate also plays a role in creating a believable habitat for your species. Maybe one species is cold-blooded and needs a suitable habitat. What if you put a warm-blooded species in an inhospitable habitat? How would it adapt to its new climate?
When creating new species, you need to know their physical appearance, clothing, and accessories like tools and weapons. You should know characteristics, world view, societal norms, language, history, relationships, any supernatural or magical elements, technology, their combat style, and so much more. But only do this for the main characters in your sci-fi or fantasy novel. If you try to get this in-depth with every species in your new world, you’ll spend years working out the details and never write the book.