trying to reproduce classification results on CIFAR10, but facing some issues

[GeJulia]

First, thanks a lot for sharing the code and generally really nice work!

I'm currently trying to reproduce the classification results on CIFAR10 with akorn_conv. When trying to train solely on CIFAR10 I end up with a clean accuracy of 78.89% which is ~10% below the reported numbers in Table 4.

After going trough the appendix I found that you "conduct pre-training on the Tiny-imagenet (Le & Yang, 2015) dataset with the SimCLR loss for 50 epochs with batchsize of 512". which could explain ~4% difference in accuracy ("We observe that this pre-training
is effective for AKOrN and improves the CIFAR10 clean accuracy compared to training from scratch
(from 87% to 91%).")
However, I'm still missing 6% difference, can you help me figuring out what could be the issue?

I'm using the your python script train_classification.py --exp_name 1 --epochs 400 --criterion ce --T 3 --J conv --ksizes [9, 7, 5].

[takerum]

Hi!

The current code should reproduce the model's performance, achieving 84-85% accuracy with EMA on clean images, simply by running python train_classification.py <exp_name> (without any other arguments). If you run it and cannot replicate that accuracy, the issue might be with your environment rather than the code. For reference, I used torch 2.9.0 with CUDA 12 for this open-source version

[takerum]

oh sorry, I forgot that I used self-ensmebling technique to compute the final accuracy, where I average predictions computed from different random initial oscillators. You may need to do this to get higher acc. The code for self-ensembling is below. You can replace the adversarial image eval part with clean image to evaluate the model on the original cifar10 images.

import torch
import torchvision
from torchvision import transforms

_transforms = transforms.Compose(
    [transforms.ToTensor()]
)

shuffle = False
N_EVAL = 10000
BATCH_SIZE = 100

# load testset
testset = torchvision.datasets.CIFAR10(
    root="./data", train=False, download=True, transform=_transforms,
)
testloader = torch.utils.data.DataLoader(
    testset, batch_size=BATCH_SIZE, shuffle=shuffle, num_workers=4,
)

class LogWrapper(torch.nn.Module):

    def __init__(self, net):
        super().__init__()
        self.net = net

    def forward(self, x):
        out = self.net(x)
        return torch.log(out)
    
class EnsembleWrapper(torch.nn.Module):

    def __init__(self, nets, apply_softmax=False):
        super().__init__()
        self.nets = torch.nn.ModuleList(nets)
        self.apply_softmax = apply_softmax

    def forward(self, x):
        out = 0
        for net in self.nets:
            pred = net(x)
            if self.apply_softmax:
                pred = torch.softmax(pred, dim=1)
            out += pred
        out /= len(self.nets)
        return out

eps=8/255
net = None # load model here 
net = net.eval()
net = LogWrapper(EnsembleWrapper([net]*8, apply_softmax=True))

_loader = iter(testloader) 

images = []
labels = []

for i in range(N_EVAL//BATCH_SIZE):
    image, label = next(_loader)
    images.append(image)
    labels.append(label)

images = torch.cat(images, 0)
labels = torch.cat(labels, 0)

from source.evals.classification.adv_attacks import autoattack
version = 'rand'
advimg = autoattack(net, images.to('cuda'), labels.to('cuda'), epsilon=eps, version=version, bs=100)

[GeJulia]

Thanks a lot for the quick response! :)
I just checked with using the ensemble and it boosts accuracy:
for the EMA model I get now 82.60% and the standard model achieves 80.46%.
However this is still below 84-85%.
I'm running on pytorch 2.9.1 and cuda 12.8.

[takerum]

Hmm I see. Sorry that you could not reproduce the results. One thing you can do is to use larger batchsize (e.g. 512 instead of 128), which should improve the acc a bit. I got ~84% with EMA and without ensembling at the 200th epoch.

[GeJulia]

but you still trained for 400 epochs in total? or just reduced the number of epochs down to 200?
I now also tested the AKOrN_attn model and this one achieves higher accuracy.
This are the numbers I end up with when no simclr pretraining on Tiny-ImageNet is applied:

Connectivity	standard Acc	EMA standard Acc	ensemble Acc	EMA ensemble Acc
Convolution	76.87 +/- 0.08	80.61 +/- 0.11	78.95 +/- 0.28	82.44 +/- 0.12
Attention	89.22 +/- 0.18	90.91 +/- 0.14	89.54 +/- 0.07	91.27 +/- 0.03

Numbers are reported over 5 testing runs.

[takerum]

but you still trained for 400 epochs in total? or just reduced the number of epochs down to 200?

I just meant that with a larger batchsize, the model already achieves 84% acc in the earlier steps. It gets slightly better accuracy (~84.5) at the 400th epoch.