pyhgf.model.transformer.MultiHeadAttention

class pyhgf.model.transformer.MultiHeadAttention(wq=None, wk=None, wv=None, wo=None, n_heads=1, *, wqkv=None)

Causal multi-head self-attention as a composite pipeline part.

Holds four single-input parts for the Q, K, V, and O weight tables (any mix of frozen and learning parts) around the weight-free mixing step. The query, key, and value projections read the same input, so they can instead be one fused part wqkv whose output stacks the three streams on the feature axis, [q | k | v]: the same three linear maps computed as one matrix product, with the three routed errors concatenated on the way back. Operates on (batch, seq, features) arrays — attention is the one part that needs the sequence axis explicit, because it moves information between positions; everywhere else each position is independent.

Forward: every position emits a query, a key, and a value through the Q/K/V parts; each position’s attention scores against all earlier positions (causal mask) become percentages through a softmax; the values are blended accordingly and pass through the O part.

Backward: the error at the output goes back through O, is re-routed across positions by the mixing formula (an error at position t flows back to the positions t attended to, and to the query/key pair that set those percentages), and the three resulting messages return through Q, K, and V — whose input errors add, since all three read the same input.

Parameters:

• wq (Optional[PCModule])

• wk (Optional[PCModule])

• wv (Optional[PCModule])

• wo (Optional[PCModule])

• n_heads (int)

• wqkv (Optional[PCModule])

__init__(wq=None, wk=None, wv=None, wo=None, n_heads=1, *, wqkv=None)

Parameters:

• wq (PCModule | None)

• wk (PCModule | None)

• wv (PCModule | None)

• wo (PCModule | None)

• n_heads (int)

• wqkv (PCModule | None)

Methods

__init__([wq, wk, wv, wo, n_heads, wqkv])
init_state()	Return the weight tables' state tuple.