Are you trying to fit into a genre or sub-genre because it's popular right now? That's like trying to fit into a political party when your philosophy is somewhere in the middle. It's hard to find the right fit in either party, right?
Maybe it's time you created your own sub-genre or genre. Look at what Bridget Jones's Diary did for chick lit. And what The Hunger Games did for YA dystopian. And I'm still not sure how to categorize Jodi Picoult's novels. If you look up the genres of her books, you'll find "Genre: Fiction + Literature; Sub-Genre: Literary or Contemporary." Huh? Nonetheless, she's created her own space on the best seller list.
The bad news is publishers are leery of taking on books that aren't easily categorized and don't have a well-defined audience. But when they do take a chance, you can get something like The Time Traveler's Wife, Cloud Atlas, or Infinite Jest that breaks down genre barriers.
Genres and sub-genres
Writer's Digest have a list of genres and their sub-genres that cover almost everything published. You can find it here. It's pretty comprehensive.
The reason we're pointing out what sub-genres exist is so you can do something new and unique: mash up sub-genres.
Think about how Anne Rice mashed up romance with a vampire story in her Interview With a Vampire. Her vampire stories led to mainstream best sellers like the Twilight series and other vampire romance novels. But Rice was the first one who took a sympathetic look at vampires.
Read over the Writer's Digest list of genres and sub-genres with a critical eye. How can you take two sub-genres and mash them together to create something new?
How to create your own genre/sub-genre
Let's look at an example of an author who mainstreamed a new sub-genre: Tom Clancy and the techno-military thriller. And he was prolific in his new sub-genre with several on the best sellers list. He may not have been the best writer, but he created something new that readers loved. Without Clancy breaking down the barriers, other authors like W.E.B. Griffin may not have hit the best seller lists with their own books.
So how do you do it? It's not as simple as thinking of two sub-genres and mashing them together. You need to find a common link between two sub-genres that's unique.
Consider combining crime fiction with a spy novel. How would that work? Spy novels usually have someone like a terrorist trying to kill a country's president or the Queen, or trying to steal a list of the nation's undercover agents. With a crime fiction twist, a crime spy story would be driven by personal greed, lust, revenge, or hate, but not by the need to save the world or avoid war. So instead, you'd have characters committing crimes using skills and tactics normally reserved for elite spies and Special Forces. Does this sound like an interesting twist? You could make it so.
Take your normal genre that you usually write. What other sub-genres have you always wanted to try or have you been drawn to? Think of how Stephen King took his weird horror type sub-genre and married it to a western to create the Dark Tower series. Or consider taking a sub-genre's trope and making it a sympathetic character like Anne Rice did with vampires.
Finally, consider writing a specific genre and subverting the tropes and expectations. Can you write a crime story without a crime? See if you can write a story that pushes a genre's structure so every choice you make is atypical, but still retains the atmosphere, mood, tone, and thrill.
Whatever you're working on, try to take it in a new direction that's entirely different from what you've written in the past. Not just different from your original choices, but choices that make sense when you subvert expectations and formulas. Create something entirely your own.