Automatic Differentiation
Compiler-generated adjoints with @differentiable, @backward, @implicit, grad, newton_solve, and custom VJPs.
Vyne ships a source-to-source reverse-mode AD surface with a working implementation for straight-line Float64 functions and Array<Float64> reductions. There is no runtime tape interpreter and no per-operation dispatch: the compiler emits a primal function and a companion adjoint.
| Form | Surface | Lowers to |
|---|---|---|
| Function | @differentiable fn f(x :: Float64) -> Float64 { ... } | primal fn_f plus fn_f_grad |
| Region | @backward region step { ... backward(l); ... } | a tape scope inside the region; .grad reads |
| Region | @implicit region solve { z := newton_solve(F, x, z0); backward(z); } | Newton primal plus an Implicit Function Theorem adjoint |
| Expression | grad(f, x) | a seeded call to fn_f_grad |
@backward is a predicate, not a memory policy. It composes with @bounded, @recompute, and @mixed:
Expression form
grad is compiled-backend functionality. The AST interpreter does not evaluate grad / backward.
Region form
The tape is a set of ordinary C locals inside the region block. It dies at the region rewind, and @bounded accounts for it like any other region memory.
Implicit regions
@implicit differentiates through an equation, not through a solver. The body calls newton_solve(F, x..., z0) to find a root z of the scalar residual F(x..., z) = 0; the reverse pass differentiates F at that root and applies the Implicit Function Theorem:
No Newton iteration is stored or replayed, so reverse-mode memory is independent of the solver's iteration count.
The residual signature is (x..., z) -> Float64: the state z is the last parameter. newton_solve receives the residual, the parameter values in that same order, and the initial guess. backward(root) seeds dL/dz = 1.0; backward(root, seed) chains an explicit upstream sensitivity.
@implicit implies the @backward tape scope, so it composes with the memory policies exactly like @backward:
Rules and limits:
- The residual must be a
@differentiablescalarFloat64function whose last parameter is the state. A custom<fn>_vjpcannot supply thedF/dzthat the theorem needs, so it is rejected. - The parameter arguments to
newton_solvemust be plain variables;backward(root)publishes.gradon those names. - The root binding must be a fresh
:=declaration directly assigned fromnewton_solve, because the compiler needs a stable name forbackward(root). newton_solveandimplicit_solveare intercepted only inside an@implicitregion. Outside, the names resolve as ordinary functions.- Newton uses tolerance
1e-12, fails after the iteration ceiling (64 by default), and rejects a singulardF/dzwithVNE-128. - This slice is scalar and desktop-only. Vector residuals, Jacobian solves, and the AVR target are deferred.
The table below summarizes the reserved intrinsics:
| Intrinsic | Where | Meaning |
|---|---|---|
newton_solve(F, x..., z0) | @implicit region | Solve F(x..., z) = 0 and record the root for IFT backward |
implicit_solve(F, x..., z0) | @implicit region | Alias for newton_solve |
backward(root) | @implicit region | Apply IFT with dL/dz = 1.0; populate parameter .grad slots |
backward(root, seed) | @implicit region | Apply IFT with an explicit upstream sensitivity |
Array gradients
grad(f, w) returns a fresh Array<Float64> when w is an Array<Float64> parameter. The adjoint lowers each reduction VJP to a loop over the native array storage.
Recognized reductions and their VJPs (d is the incoming scalar gradient, n the array length):
| Primitive | Result | VJP |
|---|---|---|
mse(a,b) / k_mse_sum(a,b,n) | mean / sum of squared error | d_a += 2(a-b)/n * d, d_b -= ... |
mae(a,b) / k_mae_sum(a,b,n) | mean / sum absolute error | d_a += sign(a-b)*d/n, d_b -= ... |
dot(a,b) / k_dot(a,b,n) | dot product | d_a += b*d, d_b += a*d |
sum(a,n) / mean(a,n) | sum / mean | d_a += d / d_a += d/n |
norm_sq(a,n) / sum_sq(a,n) | squared norm / sum of squares | d_a += 2a*d |
abs_sum(a,n) | sum of absolute values | d_a += sign(a)*d |
Array arguments must be plain variables (parameters or read-only globals). Array locals and element-wise array intermediates are deferred. backward(...) on array parameters is deferred; use grad(f, x).
Custom adjoints
A function named <fn>_vjp registers itself as the reverse pass for <fn>. This is the escape hatch for functions outside the scalar/reduction subset, such as a convolution loss that carries tensor locals.
Rules:
- The name must be exactly
<primal>_vjp; module-qualified names are supported. - The adjoint's parameters must match the primal's parameter list.
- There is no
d_outparameter:backward(loss)seeds with1.0. - The compiler does not validate the primal body when a custom adjoint exists.
- Gradient storage lives wherever the adjoint puts it (globals, a
Gradsobject, caller state).w.gradscalar slots are not extended to array/tensor parameters.
The SRCNN showcase uses this convention: sr_loss is @differentiable, and sr_loss_vjp fills a global Grads using the explicit vconv backward kernels.
What the compiler emits
For:
the backend emits the primal unchanged and a companion of the form:
The transformation recomputes primal intermediates, seeds the returned expression, walks assignments in reverse, and publishes input gradients through out-parameters.
Supported subset
- Parameters are
Float64scalars orArray<Float64>. The return type isFloat64. - The body is a sequence of fresh scalar declarations (
:=) followed by onereturn <expr>;. - Scalar expressions use
+ - * / **andsin cos exp log sqrt tanh abs relu pow. - Array values participate through the reduction primitives above.
- Scalar bodies must close over their parameters. Reading a scalar global inside a scalar
@differentiablebody is rejected because it is not part of the adjoint signature. Array reductions may read read-only array globals.
Everything else is rejected with a stable diagnostic instead of producing a wrong gradient:
| Code | Meaning |
|---|---|
VNE-120 | function outside the supported differentiable subset |
VNE-121 | unsupported statement or mutation inside @differentiable |
VNE-122–VNE-125 | unsupported expression or grad() misuse |
VNE-126 | backward() could not resolve the recorded differentiable call |
VNE-127 | malformed newton_solve() call or residual outside the implicit scalar subset |
VNE-128 | runtime implicit solve failure (non-convergence or singular dF/dz) |
Deferred
- Array locals, element-wise array intermediates, matrix/matmul VJPs;
backward(...)on array parameters (usegrad(f, x));- branches,
through/whileloops, loop-carried state; - vector-valued implicit roots, dense Jacobian assembly, and linear solves for
@implicit; - the AVR target for
@implicit; @recomputecheckpointing and@mixedprecision lowering;- interpreter execution of
grad/backward; - cross-module adjoint discovery beyond the defining translation unit.