Data Loading Issues¶

Common data loading problems and solutions.

Corrupted or Missing Images¶

import autotimm as at  # recommended alias
# Enable validation to skip corrupted images
from autotimm import ImageDataModule

data = ImageDataModule(
    data_dir="./data",
    dataset_name="custom",
    validate_images=True,  # Skip corrupted images
)

Dataset Not Found¶

import os

# Verify data directory structure
print("Data directory contents:")
print(os.listdir("./data"))

# For COCO format detection
# Expected structure:
# data/
#   ├── images/
#   │   ├── train/
#   │   └── val/
#   └── annotations/
#       ├── instances_train.json
#       └── instances_val.json

Annotation Format Issues¶

Symptoms: KeyError, ValueError when loading annotations

Solutions:

# For COCO format, verify structure
import json

with open("data/annotations/instances_train.json") as f:
    coco_data = json.load(f)

# Check required keys
required_keys = ["images", "annotations", "categories"]
for key in required_keys:
    assert key in coco_data, f"Missing key: {key}"

# Verify image IDs match
image_ids = {img["id"] for img in coco_data["images"]}
ann_image_ids = {ann["image_id"] for ann in coco_data["annotations"]}
print(f"Images: {len(image_ids)}, Annotated: {len(ann_image_ids)}")

Class Imbalance Warnings¶

from collections import Counter

# Check class distribution
def check_class_distribution(datamodule):
    train_labels = []
    for batch in datamodule.train_dataloader():
        labels = batch["labels"] if isinstance(batch, dict) else batch[1]
        train_labels.extend(labels.tolist())

    counts = Counter(train_labels)
    print("Class distribution:", counts)

    # If imbalanced, use weighted loss
    class_weights = [1.0 / count for count in counts.values()]
    return class_weights

# Apply weighted loss
from autotimm import ImageClassifier
class_weights = check_class_distribution(data)
model = ImageClassifier(
    backbone="resnet50",
    num_classes=10,
    loss_fn="crossentropyloss",
    loss_kwargs={"weight": torch.tensor(class_weights)},
)

Segmentation Masks Not Loading¶

Symptoms: Empty masks, incorrect pixel values, shape mismatches

Common Causes:

# Check mask format
from PIL import Image
import numpy as np

mask = Image.open("path/to/mask.png")
print(f"Mode: {mask.mode}")  # Should be 'L' (grayscale)
print(f"Unique values: {np.unique(np.array(mask))}")  # Should be [0, 1, ..., num_classes-1, 255]

Solutions:

1. Masks are RGB instead of single-channel:

   # Convert RGB masks to single-channel
   mask_rgb = Image.open("mask.png")
   mask_gray = mask_rgb.convert('L')
   mask_gray.save("mask_fixed.png")
   

2. Pixel values out of range:

   # Verify pixel values are [0, num_classes-1] and 255
   mask_array = np.array(Image.open("mask.png"))
   assert mask_array.max() <= num_classes or mask_array.max() == 255
   assert mask_array.min() >= 0
   

3. Filename mismatch:

   # Verify images and masks have matching names
   import os
   from pathlib import Path
   
   image_dir = Path("data/train/images")
   mask_dir = Path("data/train/masks")
   
   image_stems = {p.stem for p in image_dir.glob("*")}
   mask_stems = {p.stem for p in mask_dir.glob("*")}
   
   missing_masks = image_stems - mask_stems
   print(f"Images without masks: {missing_masks}")
   

Slow Segmentation Data Loading¶

Symptoms: Long wait times between epochs, low GPU utilization

Solutions:

1. Increase num_workers:

   import os
   data = SegmentationDataModule(
       data_dir="./data",
       num_workers=min(os.cpu_count(), 8),  # Up to CPU cores
   )
   

2. Reduce image size:

   data = SegmentationDataModule(
       data_dir="./data",
       image_size=512,  # Try 384 or 256 if still slow
   )
   

3. Use SSD storage:

4. Move dataset to SSD instead of HDD

5. Use faster storage for better I/O performance

6. Enable persistent workers:

   data = SegmentationDataModule(
       data_dir="./data",
       num_workers=4,
       persistent_workers=True,  # Keep workers alive between epochs
   )
   

Segmentation Transform Errors¶

Symptoms: Shape mismatches, type errors, mask corruption

Solutions:

1. For albumentations:

   import albumentations as A
   from albumentations.pytorch import ToTensorV2
   
   # Correct order: transforms → normalize → ToTensorV2
   train_transforms = A.Compose([
       A.RandomCrop(512, 512),
       A.HorizontalFlip(p=0.5),
       A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
       ToTensorV2(),  # Must be last
   ])
   

2. Don't mix backends:

   # WRONG: mixing torchvision and albumentations
   import torchvision.transforms as T
   import albumentations as A
   
   # Do NOT do this:
   transforms = T.Compose([...])  # torchvision
   transforms = A.Compose([...])  # albumentations
   
   # RIGHT: pick one backend
   transforms = A.Compose([...])  # Use albumentations for segmentation
   

3. Verify mask transforms:

   # Masks should be transformed with nearest-neighbor interpolation
   # Albumentations handles this automatically, just verify:
   transforms = A.Compose([
       A.Resize(512, 512),  # Automatically uses cv2.INTER_NEAREST for masks
       ToTensorV2(),
   ])
   

Related Issues¶

• Augmentation - Transform and augmentation errors
• Slow Training - Data loading bottlenecks
• Profiling - Performance analysis