Overuse of adverbs is the hallmark of lazy, cluttered writing. Good dialogue should use strong verbs rather than '-ly' adverbs. Often the adverbs mean the same as the verb and become redundant, leading to messy prose.
The most common (over)use of adverbs is to modify the verb said, e.g. "I'm leaving," he said angrily.
By reducing these adverbs, the author allows the characters to convey the emotions of the dialogue themselves. Instead of telling the reader, they are able to show them. e.g. He thumped his hand on the desk. "I'm leaving."
As a general rule, writers should use no more than one adverb per 300 words.
You can either replace the '-ly' adverb with a stronger adjective or verb, or remove it if it is obsolete. It is not necessary to remove every adverb, but limiting them will help strengthen your writing.