I've always felt that if you back down from your fear, the ghost of that fear never goes away. It diminishes people. So I've always said 'yes' to the thing I am most scared about.
It is strange looking back on my life that my failures don't haunt me. The things I should have tried but didn't out of fear do haunt me. Don't make that mistake if you can help it.