AI Grounds
Open AI Grounds on a desktop
These interactive lessons need a larger screen. Please continue on a desktop or laptop computer.
AI Grounds
These interactive lessons need a larger screen. Please continue on a desktop or laptop computer.
Guided discovery
Change the input. Compare three activation rules.
One scalar input x passes through a fixed activation, an input-output rule. ReLU means rectified linear unit; tanh means hyperbolic tangent. All three rules use the same input. Choosing Activation keeps x fixed, while editing Input value recomputes all outputs. No learned weights, bias or training is supplied.
2
ReLU
max(0,2)
2
A shared scalar input in tenths. All rules transform the same x; the selected rule determines the primary output. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.
ReLU(2) = 2 · Zero for nonpositive inputs, unchanged positive inputs; no global upper bound.
Actual input or function changes clear stale explanations and transfer answers. Unchanged controls preserve them. Display values round to six decimals; calculations use full precision. A bounded sigmoid output by itself is not evidence of a trained or calibrated class probability.
Zero for nonpositive inputs, unchanged positive inputs; no global upper bound.
2
max(0,x)
Fixed input 2: 2. Current output minus that reference: 0.
Strictly between 0 and 1 for finite real inputs; at zero it is 0.5.
0.880797
1 / (1 + exp(−x))
Fixed input 2: 0.880797. Current output minus that reference: 0.
Strictly between −1 and 1 for finite real inputs; negative inputs give negative outputs.
0.964028
(exp(x) − exp(−x)) / (exp(x) + exp(−x))
Fixed input 2: 0.964028. Current output minus that reference: 0.
exp(x) is the exponential e raised to x, where e is about 2.71828. ReLU is max(0,x). Sigmoid is 1/(1+exp(−x)); tanh is (exp(x)−exp(−x))/(exp(x)+exp(−x)). At input 0, exp(0)=1 gives sigmoid 0.5 and tanh 0. Saturation means approaching a bound with smaller output changes, not becoming exactly constant at a finite input here.
| Activation | Current x | Current output | Fixed x=2 output | Change from x=2 |
|---|---|---|---|---|
| ReLU | 2 | 2 | 2 | 0 |
| Sigmoid | 2 | 0.880797 | 0.880797 | 0 |
| Tanh | 2 | 0.964028 | 0.964028 | 0 |
All curves share input −6..6 and output −1..6. Circle/solid line=ReLU; square/dashed=Sigmoid; diamond/dotted=Tanh. A navy outline highlights the selected output at the shared input guide. Shapes, patterns, labels and the exact table accompany color; coincident markers draw the selected one last.
Curves connect the 121 supported input positions in steps of 0.1. Values and marker positions are exact at those positions up to floating arithmetic; straight segments between them are illustrative. The editor’s maximum6 is not a mathematical upper bound on ReLU. Sigmoid and tanh approach their ideal bounds without reaching them at any supported input.
The input is encoded as an integer−60..60 divided by 10, with no weights, bias, network layers or dataset. ReLU uses max(0,x), sigmoid uses1/(1+exp(−x)), and tanh uses the hyperbolic tangent. These fixed scalar functions can also be applied element by element in a tensor, but tensor behavior is outside this lesson.
For finite real inputs, sigmoid lies strictly between 0 and1 and tanh strictly between −1 and1. At the supported endpoints, sigmoid(6)≈0.997527 and tanh(6)≈0.999988; neither is exactly1. ReLU is zero at and below 0 and has no global upper bound. No derivatives, gradient at the ReLU kink, optimization, learned representation, calibrated probability or recommendation of an activation for a task is claimed.
Reference input 2 is fixed for each activation. Changing function preserves input; changing input recomputes all three functions. Reset/prediction restores Positive input 2/ReLU for the current experiment; free Reset begins Experiment 1. Ordinary display rounding is not used for subsequent calculations or mastery checks. The optional common-axis curves have121 vertices and preserve this same numeric model.
PyTorch · ReLU definition