Attention Lab · 3D Neural Architecture Atlas

—

STAGE 01 · INPUT EMBEDDINGS

Input Embeddings

Token sequences enter the network and project into continuous latent representations through the embedding lookup table.

LOGICAL SHAPE [30522 × 768]

PARAM COUNT 23.44M

SUBSPACE / RANK d=768

DELTA / STORAGE 93.8 MB

HEAD-TO-HEAD COMPARISON

Model: DistilBERT

Dense Base: 66.36M

Compressed: 29.20M

Savings: 2.27x (-56.0%)

Memory Saved: 148.6 MB

Active Node: input_embeddings

Dense Baseline ($O(N^2)$) Compressed ($U \times V$, MoE, MLA) Residual Skip Highway (+)

LOGICAL SHAPE —

PARAMETERS —

EST. STORAGE —

COMPRESSION RATIO —

MATRIX DIMENSIONS [— × —]

ASPECT RATIO —

3D TENSOR VOLUME —

HEAD / RANK ARCH —

PARAM COMPARISON -0%

Dense Baseline: —

SVD Low-Rank: —

STRUCTURAL TRANSFORMATION (BEFORE vs AFTER) DECOMPOSITION

ATTRIBUTE	BASELINE	COMPRESSED	DELTA

4-MODEL ARCHITECTURE MATRIX

Click to switch 3D model

TOKEN LIFECYCLE & DATAFLOW

7-STAGE FLOW

Tracks what enters, what gets added (+), and what exits at each transformer stage:

ARCHITECTURAL COMPONENT PIPELINE

0 BLOCKS

MATHEMATICAL FORMULATION

Module Description & Implementation

Click any 3D node or choose from the tree above to inspect rank decomposition, storage memory, and tensor dimensions.

📌 Pinned vs Active Comparison

Pinned: —

💡 Token Funneling: In SVD layers, watch how tokens squeeze horizontally through rank dimension $r=64$ in Factor $V$ before expanding back to full output dimension in Factor $U$.

HIERARCHICAL LAYER EXPLORER

0 modules