Siogmoid and Tanh are mostly used non-linear funtions. Their derivatives can be represented by themself, which simplify the back propagation. This blog will show how to calculate their derivatives and the connection between them.
\[\begin{align} sigmoid(x) &= \frac{e^x}{e^x+1} \tag{1} \\ tanh(x) &= \frac{e^{2x} - 1}{e^{2x} + 1} \tag{2} \\ \end{align}\]
Sigmoid
\[\begin{align} \sigma'(x) &= \frac{e^x(e^x+1) - e^{2x}}{(e^x+1)^2}\\ &= \frac{e^x}{(e^x+1)} \cdot \frac{1}{(e^x+1)} \\ &= \sigma(x) \cdot (1 - \sigma(x)) \\ \end{align}\]
Tanh
\[\begin{align} \tanh'(x) &= \frac{2e^{2x}(e^{2x} + 1) - 2e^{2x}(e^{2x} - 1)}{(e^{2x}+1)^2}\\ &= \frac{4e^{2x}}{(e^{2x}+1)^2} \\ &= \frac{(e^{2x}+1)^2 - (e^{2x}-1)^2}{(e^{2x}+1)^2} \\ &= 1 - \frac{(e^{2x}-1)^2}{(e^{2x}+1)^2} \\ &= 1 - tanh^2(x) \\ \end{align}\]
Connection
The conclusion goes first: They have linear relationship. \[\begin{align} 1 - 2\sigma'(x) &= \frac{1 - e^x}{e^x + 1} \\ &= - \frac{e^x - 1}{e^x + 1} \\ &= - tanh(\frac{x}{2}) \end{align}\]