WP7: Software, Benchmarking & Co-Design
Work Package 7 focuses on software development, benchmarking, co-design activities, and training material to deliver production-ready exascale software following the CI/CD framework proposed by ExaDIP.
Objectives
Software Development
From basic to advanced testing, including benchmarking, to verify the capabilities of exascale computing and address identified challenges.
Co-Design Coordination
Working closely with ExaDIP project to ensure effective communication and synergy for advancements in exascale computing.
Showroom
Platform to present and highlight outcomes and achievements of the Exa-MA project to the community.
Training Material
Educational resources and materials based on project results, facilitating knowledge transfer and dissemination.
Testing Process
WP7 relies on the principles of non-regression, verification, and validation. All developments undergo rigorous testing before integration into demonstrators.
Non-Regression
Guarantee that results remain identical (bit-by-bit) for options unaffected by software evolutions. Focus on elementary tests.
Verification
Measure absence of drift in consolidated solutions. Define acceptable relative variation thresholds.
Validation
Evaluate algorithm performance in real-life configurations with representative test cases.
Interpretable Benefits
Establish simple metrics that demonstrate positive impact compared to previous approaches.
Software Development Workflow
The workflow builds on the CI/CD framework with automated builds, testing, and deployment.
Demonstrator Levels
Level 1: Mini-App
Covers one or two Exa-MA Work Packages. Focuses on specific objectives within those WPs.
Level 2: Multi-WP Mini-App
Covers two or more Work Packages. Demonstrates cross-collaboration and integration.
Level 3: Proxy-App
Covers at least three Work Packages. Representative workload for evaluating HPC system performance.
| Work Package | Core Hours Estimate (Level 1) |
|---|---|
Discretization |
500,000 |
Model Order Reduction and ML |
1,000,000 |
Solvers |
750,000 |
Inverse Problems and Data Assim. |
1,500,000 |
Optimization |
600,000 |
Uncertainty Quantification |
900,000 |
Total |
4,250,000 |
A computing resources form is available here for gathering requests.
CI/CD for HPC
Continuous Integration/Continuous Delivery strategies adapted for high-performance computing enable streamlined development, improved quality, and enhanced efficiency.
Version Control & Builds
Git-based source management with automated build processes ensuring consistent, reproducible compilation across environments.
Testing & Validation
Automated test suites covering unit tests, integration tests, and performance tests as part of the CI/CD pipeline.
Artifact Management
Central repository for build artifacts (executables, libraries) facilitating deployment and distribution.
Continuous Deployment
Automated deployment to target HPC systems (supercomputers, clusters) with rollback capabilities.
Containerization
Docker and Singularity provide portable, isolated containers for consistent and reproducible execution across different computing systems.
Docker
Reproducible environments, dependency management, and versioning. Docker Hub for sharing containerized applications.
Singularity
Designed for HPC with seamless integration with resource managers (Slurm), GPU passthrough, and unprivileged user execution.
Key Benefits:
-
Reproducible Environments — Encapsulate entire application stack
-
Portability — Run consistently across different HPC systems
-
Isolation & Security — Sandbox applications from host system
-
Versioning — Easy rollback to previous versions
HPC Concepts
Mini-Apps and Proxy-Apps
Mini-Apps are small-scale programs focusing on specific computational patterns of larger applications. Used for benchmarking, performance analysis, and optimization.
Proxy-Apps mimic behavior and computational patterns of real-world scientific applications. Provide lightweight representation of computational requirements and communication patterns.
Performance Bottlenecks
CPU Bound
Performance limited by processing power. Address with parallelization, code optimization, and performance profiling.
Memory Bound
Limited by memory speed/capacity. Optimize data locality, cache utilization, and reduce data movement.
Latency Bound
Limited by data transfer delays. Optimize network, disk I/O, and use asynchronous operations.