Overview

Storage Overview

The Odin HPC cluster provides multiple storage systems optimized for different use cases.

Storage Architecture

graph TB
    subgraph "Compute Nodes"
        LN[Login Nodes]
        HN[Head Node]
        CN[Compute Nodes]
    end
    
    subgraph "Data Management"
        DML[Data Manager Linux]
        DMW[Data Manager Windows]
    end
    
    subgraph "Storage Systems"
        FSX1["/mnt/odin<br/>12TB FSx Lustre"]
        FSX2["/mnt/qcs<br/>1.2TB FSx Lustre"]
        FSX3["/mnt/gxp<br/>1.2TB FSx Lustre"]
        FSX4["/mnt/ingest<br/>9.6TB FSx Lustre"]
        EFS["/home<br/>EFS"]
        EBS["/mnt/shared<br/>EBS"]
    end
    
    subgraph "Cloud Storage"
        S3[(S3 Buckets)]
    end
    
    LN --> FSX1
    LN --> FSX2
    LN --> EFS
    LN --> EBS
    HN --> FSX1
    HN --> FSX2
    CN --> FSX1
    CN --> FSX2
    
    DML --> FSX1
    DML --> FSX2
    DML --> FSX3
    DML --> FSX4
    DML -->|Samba| DMW
    
    FSX1 <-->|Auto Sync| S3
    FSX2 <-->|Auto Sync| S3
    FSX3 <-->|Auto Sync| S3
    FSX4 <-->|Auto Sync| S3

Storage Summary

Mount Point Type Size Access Use Case
/mnt/odin FSx Lustre 12TB All nodes Legacy data, Kamino output
/mnt/qcs FSx Lustre 1.2TB All nodes Navify ingest/output
/mnt/gxp FSx Lustre 1.2TB Data managers only GxP-compliant storage
/mnt/ingest FSx Lustre 9.6TB Data managers only Kamino migration data
/home EFS Elastic All nodes User home directories
/mnt/shared EBS Variable All nodes Shared temporary storage

FSx Lustre with S3 Integration

All FSx Lustre filesystems are configured with Data Repository Associations (DRAs) that provide:

  • Auto-import: Files added to S3 automatically appear in FSx
  • Auto-export: Files written to FSx automatically sync to S3
  • Lazy loading: File data is fetched on-demand when accessed

This means you can:

  1. Upload data directly to S3 buckets
  2. Access it immediately from FSx mount points
  3. Write results to FSx and have them sync to S3

Access Patterns

For Compute Jobs

Use FSx mounts (/mnt/odin, /mnt/qcs) for:

  • High-performance data access during jobs
  • Reading input data from S3-backed directories
  • Writing output that needs to sync to S3

For Data Management

Use Data Manager instances for:

  • GxP-compliant workflows
  • Windows access via SMB shares
  • Large data transfers and migrations

For User Data

Use EFS (/home) for:

  • Personal files and scripts
  • Job submission scripts
  • Small datasets

Next Steps