Next Generation AI Competitions and Benchmarks Platform

Built for organizations that can't compromise on data privacy, scientific rigor, or real-world impact, with full control over your data, submissions, and compute.

Get in Touch Download AVALIA Proposal

A project by

The Problem

Today's platforms aren't built for high-stakes AI

Existing benchmarking platforms like Kaggle, CodaLab, and Codabench have done a lot to bring people together around AI challenges, but they still fall short on the basics that matter most in high-stakes settings: statistical rigor, data privacy, and whether the winning solution actually works in production.

  • 01

    Methodological fragility

    Most leaderboards prioritize raw performance scores over statistical significance. Without confidence intervals or control over the baseline, marginal gains become indistinguishable from noise — and nothing stops repeated submissions from exhausting the test set.

  • 02

    Constraints on sensitive data

    Current benchmarking platforms typically require data to be hosted on a central server. This excludes sectors governed by strict privacy rules — healthcare, finance, defense — since they can't risk exposing sensitive datasets.

  • 03

    Narrow evaluation metrics

    There's a disconnect between leaderboard performance and production readiness. Platforms focus almost exclusively on accuracy, F1-score, or RMSE, ignoring computational efficiency, inference latency, and energy consumption.

The Solution

AVALIA: a rigorous, sovereign, production-aware framework

AVALIA is designed and built from scratch, not bolted onto an existing tool, so that methodological rigor and strong data protection are part of the platform's foundation rather than an afterthought.

  • 01

    Rigorous methodology

    AVALIA treats benchmarking as a statistical experiment rather than a static list of scores: principled ranking functions, hidden evaluation sets to avoid overfitting, baseline comparisons, and uncertainty quantification through bootstrap resampling.

  • 02

    Handling sensitive data

    Data is stored in a decentralized way with restricted access to compute. By leveraging containerized environments, organizers can evaluate third-party models on local, private servers without the sensitive raw data ever leaving the host's infrastructure.

  • 03

    Evaluating real-world applicability

    AVALIA expands the definition of performance to include the hidden costs of AI. Every submission is profiled across inference latency, peak memory usage, and estimated carbon footprint — not just accuracy on paper.

Architecture

An architecture built around your control

Technology Stack & Components

Built on React for the frontend, Django for the backend, S3-compatible storage, and VMs for compute — each layer can be swapped out or self-hosted independently.

Frontend · React

A portal for admins, organizers, and participants — profile, team, competition, and submission management in one interface.

Backend · Django

Handles core logic, API coordination, the database, user permissions, the submission queue, and compute connections.

Data Storage · S3

Only public metadata and sanitized scores live on the host hub; datasets, ground truth, and submission code stay on the client's local storage or S3-compatible buckets.

Compute · VMs

Evaluation tasks run in isolated containers on virtual machines; external compute resources can be attached for heavier workloads.

Submission Flow

A participant's model is submitted as code through AVALIA's front-end, which exchanges input and output with the organizer's own infrastructure for storage and compute.

Submission Execution Flow

Once received, a submission moves through a configurable pipeline — optional pre- and post-execution checks — before results are published back to participants.

Deployment Levels

Choose how much of the stack AVALIA hosts on your behalf — from a fully managed cloud instance to a completely self-hosted deployment.

Level 1

AVALIA Cloud

AVALIA hosts everything on your behalf — the fastest way to get started. Best for public benchmarks without sensitive data.

Frontend
Backend
Storage
Compute
Recommended Level 2

Hybrid

AVALIA hosts the frontend and backend; your data and compute never leave your infrastructure. The right balance of ease and control for most sensitive use cases.

Frontend
Backend
Storage
Compute
Level 3

Self-Hosted

Every component is deployed fully within your own infrastructure. Maximum privacy and control — ideal for defense and highly regulated environments.

Frontend
Backend
Storage
Compute
AVALIA-hosted Your infrastructure

Prominent Features

Security and reproducibility, by design

  • Git-Integrated Workflows

    Manage training and evaluation logic through GitHub or GitLab-integrated workflows.

  • Containerized Execution

    Every submission runs in an isolated container, for fairness, reproducibility, and long-term sustainability.

  • Automated Re-Evaluation

    Results are automatically re-evaluated when organizers update a task; outdated results are flagged if re-evaluation is disabled.

  • Pre-Execution Checks

    File type and size limits, expected entry points, per-participant quotas, and rate limits are enforced before a submission ever runs.

  • Submission Scanning

    Optional scanning via predefined templates, custom scanner code, or a locally-running LLM — combinable with human-in-the-loop review before execution.

  • Network Isolation

    Execution is network-isolated by default, for security and for clean, uninfluenced resource measurements. Access can be explicitly enabled and is flagged when required (e.g. API-based LLM evaluation).

  • Decentralized Storage

    Files are routed directly to the client's linked storage (AWS S3, GCS, Azure Blob, or local hardware) — never stored centrally on AVALIA's infrastructure.

Where We Are

From first idea to production release, here's where things stand.

  • Project idea Done
  • Project proposal Done
  • Design and architecture In progress
  • Implementation Next
  • V1 release Next

Who It's For

Built for organizations that can't compromise on data

  • Industrial actors

    Organize internal benchmarks across problems like electronic optimization, material handling, and predictive maintenance of equipment and vehicles.

  • Research institutes

    In sensitive domains such as health, education, and energy: install once, then run multiple public or internal benchmarks (imaging, automatic diagnosis, and more).

  • Defense & public bodies

    License with maintenance, with full internal control and no external visibility into internally organized benchmarks.

How to Get AVALIA

Flexible engagement, your infrastructure, your rules

Who's behind AVALIA

Built by the people who built Codabench

AVALIA is developed by MLChallenges, a French consulting company for AI competitions and benchmarks, founded by Adrien Pavão and Ihsan Ullah. The team are core developers of Codabench, an open-source competition platform, and have organized large-scale challenges for institutions including Lawrence Berkeley National Laboratory, Université Paris-Saclay, the University of Washington, and the U.S. Department of Energy.

Learn More About Us Browse Our Projects
Adrien Pavão

Adrien Pavão

Founder, Competition Architect

PhD in Machine Learning, Université Paris-Saclay — core developer of CodaLab Competitions and Codabench.

Ihsan Ullah

Ihsan Ullah

Co-Founder, Research Software Engineer

MSc in Artificial Intelligence, Université Paris-Saclay — core contributor to Codabench, with 3+ years organizing scientific challenges.

Trusted By

Institutions running high-stakes AI research

These institutions have placed their trust in the MLChallenges team behind AVALIA to run their benchmarks and competitions.

AVALIA is being built by people who developed and maintained , the most popular open-source competitions and benchmarks platform.

Get in Touch

Let's talk

Interested in AVALIA, or have a question for us? We'd love to hear from you.