All rodents and rabbits (technically rabbits are lagomorphs, not rodents) have teeth with open roots that grow throughout their lives. By chewing on hard objects they naturally wear the teeth down so they don't become too long. However, sometimes the teeth do become overgrown and this can cause serious problems.
Most people don't look at their pet's teeth, so early signs usually go unnoticed. It's also impossible to see their molars by simply looking at their mouth, so normally the incisors are the only ones visible. Therefore problems with the incisors can be noticed by an owner, but not problems with the molars. So what do you look for? Really, there are two main signs. The first is not eating. Overgrown teeth can rub against the tongue or the inside of the cheeks and cause painful sores. They can also make normal movement of the mouth difficult, preventing eating. The other symptom is hypersalivation, commonly called "slobbers". If either of these are happening, overgrown teeth can be the cause.
Here are some rather dramatic pictures of overgrown incisors in rabbits.
If your veterinarian diagnoses your pet with overgrown teeth, they will need to be trimmed. However, this needs to be done by a vet who knows how to do it and ideally has the proper equipment. Do NOT let a vet trim your pet's teeth with nail trimmers! Some vets are in the habit of doing so (and I did this before I learned better), but it is actually dangerous to do. Nail trimmers can cause the tooth to split along its length, causing even worse problems. It's also common that once teeth start to overgrow they will periodically need to be trimmed for the rest of the pet's life. But not doing so can lead to your pet's death.