The word doubtful matters according to dictionary it means those matters or things which is not known with certainty or surety. 
Narrated An-Nu`man bin Bashir: The Prophet (ﷺ) said "Both legal and illegal things are obvious, and in between them are (suspicious) doubtful matters. So whoever forsakes those doubtful things lest he may commit a sin, will definitely avoid what is clearly illegal; and whoever indulges in these (suspicious) doubtful things bravely, is likely to commit what is clearly illegal. Sahih al-Bukhari 2051 (Vol. 3, Book 34, Hadith 267), Jami` at-Tirmidhi 1205 and Sunan an-Nasa'i 5710
The Messenger of Allah [sallallaahu alayhi wa sallam] said, "Abstain from the forbidden and you will be the most devout worshippers amongst the people. Be content with what Allah allocates for you from provision and you will be the richest of people. Be good to your neighbor and you will be a believer. Love for the people what you love for yourself and you will be a Muslim and do not increase in laughter because too much laughter kills the heart. At-Tirmidhi 2305 in az-Zuhd classed as Hasan by Shaikh Albani, Ibn Maajah 4217 in az-Zuhd with this meaning - Al-Albaani says it is Hasan and Sahih at Targheeb 2567 
Explanation of the hadith
This hadîth sets down an important principle of conduct for Muslims to live by. It does so by dividing all matters with respect to their Islamic rulings into the following three categories:
1. Matters that are clearly permissible.
2. Matters that are clearly unlawful.
3. Matters wherein it is unclear whether or not they are permissible.
There are many examples that we can think of for things that are clearly permissible. It is permissible to eat wholesome foods. No one can doubt the permissibility of eating an apple. It is permissible to wear clothing made form pure substances like cotton and wool. It is permissible to get married to a virtuous Muslim. The list is endless. When something is clearly permissible, a Muslim knows that he can engage in it without any fear of falling into sin.
Likewise, there are many matters that are clearly unlawful. No one can doubt the unlawfulness of murder, fornication, adultery, incest, drinking, gambling, dealing in usury, and eating pork.
Then there are doubtful matters that a Muslim should avoid. There are basically two reasons for doubt to exist about something. The first is where there is evidence indicating that something is unlawful, but that evidence is uncertain. The second is where there is a clear ruling in Islamic law that something is unlawful, but it is unclear if that ruling applies to a certain matter.
Even when we have clear evidence that something is prohibited, it is not always clear that the ruling applies to a given situation. The world is complicated and full of extenuating circumstances and unclear situations. Sheikh al-`Uthaymeen cites as an example of this the ruling on smoking when cigarettes were first encountered by the Muslims. There is a clear ruling in Islamic Law that substances that are injurious to the health are prohibited. However, it was unclear in the old days whether or not this ruling should be applied to cigarettes. There were some indications that cigarettes were unhealthy, but nothing conclusive. For this reason, scholars had widely differing opinions about smoking; many of them gave the ruling that it was disliked.
This was a good example of a doubtful matter that it was best for a Muslim to avoid.
With the experience accumulated over time and the advancement of medical knowledge, it has been proven conclusively that cigarettes have severe negative effects upon a person’s health. There can no longer be any doubt that smoking is unlawful.
The Prophet (ﷺ) said "But between them are certain doubtful things which many people do not recognize. He who guards against doubtful things keeps his religion and his honor blameless, but he who falls into doubtful things falls into what is unlawful. "Classed as Sahih by Shaikh Albani in Sunan Abi Dawud 3330
There are two possible ways that this sentence can be understood:
1. It is unlawful to get involved in doubtful matters. Whoever does so is sinning simply for doing so.
2. Getting involved in doubtful matters is not necessarily unlawful in and of itself, but it can easily lead a person to fall into sin.
The example given by the Prophet (peace be upon him) clears up which meaning is intended. He mentions a man grazing his flock near a private pasture. In this example, the private pasture represents things that are clearly prohibited in Islamic Law. A person who grazes his sheep near a private pasture is not trespassing at the moment, but he will have a very difficult time keeping his sheep from straying into it and thus breaking the law.
The same thing applies to a person who engages in doubtful matters. It is very difficult for such a person to prevent himself from falling into sin. By staying far away from the doubtful matters, he protects himself from falling into sin, just like a shepherd who keeps his flock far away from the private pasture does not have to worry about his flock straying into it.
He who guards against doubtful things keeps his religion and his honor blameless. 
Halal; Sins; Forgiveness; Dietary Laws in Islam; Smoking;