Its not that its toxic, just that if you are breathing 100% nitrogen you are not getting any oxygen and so will soon be dead...
As the air is already mostly nitrogen our bodies don't notice a few more % so it sneaks up on you before you know it.

Even if you are not convinced it would wise to treat nitrogen with caution just in case!