If it stresses you out, but doesn’t have to be done, don’t do it.
You could probably lower your tax bill if you spent more time learning the tax code, but it might also be tedious and stressful. You might be better off spending that time with your family or working on your next project.
Everything has a cost, in terms of money, time, stress, etc. Choose wisely.