Module neuralops.tests.test_cross_architecture_injection

Exhaustive cross-architecture injection tests across DistilBERT, RoBERTa, GPT-2, and Qwen. Verifies forward pass shape, finite outputs, LoRA parameter trainability, and RoPE/Causal handling.

Classes

class MockGPT2Attn (hidden_dim: int = 64)

Expand source code

class MockGPT2Attn(nn.Module):
    def __init__(self, hidden_dim: int = 64):
        super().__init__()
        self.c_attn = nn.Linear(hidden_dim, 3 * hidden_dim)
        self.c_proj = nn.Linear(hidden_dim, hidden_dim)
        self.split_size = hidden_dim

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Ancestors

• torch.nn.modules.module.Module

class MockMHA (hidden_dim: int = 64)

Expand source code

class MockMHA(nn.Module):
    def __init__(self, hidden_dim: int = 64):
        super().__init__()
        self.q_lin = nn.Linear(hidden_dim, hidden_dim)
        self.k_lin = nn.Linear(hidden_dim, hidden_dim)
        self.v_lin = nn.Linear(hidden_dim, hidden_dim)
        self.out_lin = nn.Linear(hidden_dim, hidden_dim)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Ancestors

• torch.nn.modules.module.Module

class MockQwenAttn (hidden_dim: int = 64)

Expand source code

class MockQwenAttn(nn.Module):
    def __init__(self, hidden_dim: int = 64):
        super().__init__()
        self.q_proj = nn.Linear(hidden_dim, hidden_dim)
        self.k_proj = nn.Linear(hidden_dim, hidden_dim)
        self.v_proj = nn.Linear(hidden_dim, hidden_dim)
        self.o_proj = nn.Linear(hidden_dim, hidden_dim)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Ancestors

• torch.nn.modules.module.Module

class RobertaSelfAttention (hidden_dim: int = 64)

Expand source code

class RobertaSelfAttention(nn.Module):
    def __init__(self, hidden_dim: int = 64):
        super().__init__()
        self.query = nn.Linear(hidden_dim, hidden_dim)
        self.key = nn.Linear(hidden_dim, hidden_dim)
        self.value = nn.Linear(hidden_dim, hidden_dim)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Ancestors

• torch.nn.modules.module.Module

class TestCrossArchitectureInjection (methodName='runTest')

Expand source code

class TestCrossArchitectureInjection(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    def test_injection_distilbert(self):
        """DistilBERT linear projections (q_lin, k_lin, v_lin, out_lin) are extracted and injected."""
        mock = MockMHA(64).to(self.device)
        net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=False).to(self.device)
        net.copy_weights(mock)
        x = torch.randn(2, 16, 64, device=self.device)
        out, _ = net(x)
        self.assertEqual(out.shape, (2, 16, 64))
        self.assertTrue(torch.isfinite(out).all())

    def test_injection_roberta(self):
        """RoBERTa self-attention (query, key, value) is extracted and nested mode enabled."""
        mock = RobertaSelfAttention(64).to(self.device)
        net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=False).to(self.device)
        net.copy_weights(mock)
        self.assertTrue(net.is_nested_encoder)
        x = torch.randn(2, 16, 64, device=self.device)
        out, _ = net(x)
        self.assertEqual(out.shape, (2, 16, 64))
        self.assertTrue(torch.isfinite(out).all())

    def test_injection_gpt2(self):
        """GPT-2 fused c_attn is split properly and causal mask applied."""
        mock = MockGPT2Attn(64).to(self.device)
        net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=True).to(self.device)
        net.copy_weights(mock)
        self.assertTrue(net.is_decoder)
        x = torch.randn(2, 16, 64, device=self.device)
        out, _ = net(x)
        self.assertEqual(out.shape, (2, 16, 64))
        self.assertTrue(torch.isfinite(out).all())

    def test_injection_qwen3_5(self):
        """Qwen projections (q_proj, k_proj, v_proj, o_proj) copied and forward works."""
        mock = MockQwenAttn(64).to(self.device)
        net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=True).to(self.device)
        net.copy_weights(mock)
        x = torch.randn(2, 16, 64, device=self.device)
        out, _ = net(x)
        self.assertEqual(out.shape, (2, 16, 64))
        self.assertTrue(torch.isfinite(out).all())

A class whose instances are single test cases.

By default, the test code itself should be placed in a method named 'runTest'.

If the fixture may be used for many test cases, create as many test methods as are needed. When instantiating such a TestCase subclass, specify in the constructor arguments the name of the test method that the instance is to execute.

Test authors should subclass TestCase for their own tests. Construction and deconstruction of the test's environment ('fixture') can be implemented by overriding the 'setUp' and 'tearDown' methods respectively.

If it is necessary to override the init method, the base class init method must always be called. It is important that subclasses should not change the signature of their init method, since instances of the classes are instantiated automatically by parts of the framework in order to be run.

When subclassing TestCase, you can set these attributes: * failureException: determines which exception will be raised when the instance's assertion methods fail; test methods raising this exception will be deemed to have 'failed' rather than 'errored'. * longMessage: determines whether long messages (including repr of objects used in assert methods) will be printed on failure in addition to any explicit message passed. * maxDiff: sets the maximum length of a diff in failure messages by assert methods using difflib. It is looked up as an instance attribute so can be configured by individual tests if required.

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

Ancestors

• unittest.case.TestCase

Static methods

def setUpClass()

Hook method for setting up class fixture before running tests in the class.

Methods

def test_injection_distilbert(self)

Expand source code

def test_injection_distilbert(self):
    """DistilBERT linear projections (q_lin, k_lin, v_lin, out_lin) are extracted and injected."""
    mock = MockMHA(64).to(self.device)
    net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=False).to(self.device)
    net.copy_weights(mock)
    x = torch.randn(2, 16, 64, device=self.device)
    out, _ = net(x)
    self.assertEqual(out.shape, (2, 16, 64))
    self.assertTrue(torch.isfinite(out).all())

DistilBERT linear projections (q_lin, k_lin, v_lin, out_lin) are extracted and injected.

def test_injection_gpt2(self)

Expand source code

def test_injection_gpt2(self):
    """GPT-2 fused c_attn is split properly and causal mask applied."""
    mock = MockGPT2Attn(64).to(self.device)
    net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=True).to(self.device)
    net.copy_weights(mock)
    self.assertTrue(net.is_decoder)
    x = torch.randn(2, 16, 64, device=self.device)
    out, _ = net(x)
    self.assertEqual(out.shape, (2, 16, 64))
    self.assertTrue(torch.isfinite(out).all())

GPT-2 fused c_attn is split properly and causal mask applied.

def test_injection_qwen3_5(self)

Expand source code

def test_injection_qwen3_5(self):
    """Qwen projections (q_proj, k_proj, v_proj, o_proj) copied and forward works."""
    mock = MockQwenAttn(64).to(self.device)
    net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=True).to(self.device)
    net.copy_weights(mock)
    x = torch.randn(2, 16, 64, device=self.device)
    out, _ = net(x)
    self.assertEqual(out.shape, (2, 16, 64))
    self.assertTrue(torch.isfinite(out).all())

Qwen projections (q_proj, k_proj, v_proj, o_proj) copied and forward works.

def test_injection_roberta(self)

Expand source code

def test_injection_roberta(self):
    """RoBERTa self-attention (query, key, value) is extracted and nested mode enabled."""
    mock = RobertaSelfAttention(64).to(self.device)
    net = GatedDeltaNet(hidden_size=64, num_heads=4, is_decoder=False).to(self.device)
    net.copy_weights(mock)
    self.assertTrue(net.is_nested_encoder)
    x = torch.randn(2, 16, 64, device=self.device)
    out, _ = net(x)
    self.assertEqual(out.shape, (2, 16, 64))
    self.assertTrue(torch.isfinite(out).all())

RoBERTa self-attention (query, key, value) is extracted and nested mode enabled.