Harm to others should be the dividing line between that which is legal and that which is not.

There are actions which are clearly harmful to others and these should be felonies.

Then there is the grey area where harm to others is debatable and these should be at most misdemeanors.