Matrix Multiplication Rules

The general rule: element Z[i,j] = dot(row i of Aᵀ, column j of W). This is all you need to compute any element of a matrix product.

Worked example: A is (2×3), W is (2×4):

• Aᵀ is (3×2) — rows: a₁ᵀ, a₂ᵀ, a₃ᵀ
• W columns: w₁, w₂, w₃, w₄
• Z = AᵀW is (3×4). Z[2,3] = a₂ᵀ · w₃ — row 2 of Aᵀ dotted with column 3 of W

Colour intuition: each row of Aᵀ (one colour) influences an entire row of Z. Each column of W (another colour) influences an entire column of Z. The intersection element at (row i, col j) comes from row i of Aᵀ and column j of W.

Why this matters for neural networks: A_in has shape (batch, n_in). W has shape (n_in, n_units). Z = A_in × W gives (batch, n_units) — all pre-activations for all examples at once. The rule above explains every element of that output.

Matching dimensions as taking dot products: the inner dimension (n_in) is the length of both vectors being dot-producted. That's why dimensions must match — you can't dot-product vectors of different lengths.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• The general formula for AᵀW — computing every element systematically from rows and columns.
• Here's the matrix A, which is a 2 by 3 matrix because it has two rows and three columns.
• The rule above explains every element of that output.
• Because the numbers are the same shade of blue, are the ones that are grouped together to form the vectors w1, w 2, or w3 or w4.
• Z[2,3] = a₂ᵀ · w₃ — row 2 of Aᵀ dotted with column 3 of W Colour intuition: each row of Aᵀ (one colour) influences an entire row of Z.
• Z = A_in × W gives (batch, n_units) — all pre-activations for all examples at once.
• Matching dimensions as taking dot products: the inner dimension (n_in) is the length of both vectors being dot-producted.
• More expressive models improve fit but can reduce interpretability and raise overfitting risk.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.