Refactor backend data schema for clearer splits and modalities (frontend filtering support)

[Weiyu-Kong]

Background

The current backend data format does not clearly distinguish between different dataset splits or modalities. This makes it hard for the frontend to:

• Filter data by split (e.g., train / test / eval)
• Distinguish different modalities in contrastive learning tasks (e.g., code vs. natural language)

---

Requirements

1. Expose dataset splits explicitly

   • Clearly label and organize data by split:

     • train
     • test
     • eval (or val, depending on naming convention)

   • Ensure the frontend can query or filter data based on these split labels.

2. Expose modalities for contrastive learning tasks

   • For contrastive tasks, explicitly separate or label:

     • code (modality 1)
     • nl (natural language, modality 2)

   • Make it easy for the frontend to:

     • Filter by modality
     • Pair corresponding code–nl examples when needed

3. Provide a consistent, documented schema

   • Define a stable data schema (e.g., JSON structure or API response format) that includes:

     • Split information (e.g., split: "train" | "test" | "eval")
     • Modality information (e.g., modality: "code" | "nl" or separate fields)

   • Add minimal documentation or comments so frontend developers know how to consume the new structure.

---

Acceptance Criteria

• Frontend can:

  • Filter data by train / test / eval using backend-provided fields.
  • Filter and distinguish code vs. nl examples for contrastive learning views.

• The new data schema is consistent across endpoints and/or datasets.

• No breaking changes for existing consumers, or a clear migration path is provided if breaking changes are necessary.