Resources
Article
How Does Our HybridSolver Compare to State-of-the-Art Solvers?
We are frequently asked how our optimization solver, the HybridSolver, stands in comparison to other state-of-the-art solvers. While we always recommend testing solvers on your specific problems, we understand the importance of providing benchmarks to help you gauge performance.
The Challenge of Comparison
Comparing solvers isn't straightforward. We often recommend testing solvers on your specific problems because performance can vary significantly depending on the use case. Widely recognized benchmarks like MIPLIB2017 aren't always ideal. Many solvers have extensively tuned their algorithms for these benchmark problems, which may not reflect real-world performance on untuned, unique challenges.
Moreover, well-known benchmarks like those by Hans Mittelmann do not include cloud-based solvers, and many solver providers are withdrawing from these benchmarks for various reasons, making direct comparisons even more challenging.
Our Benchmarking Approach
Despite these challenges, we've made a sincere effort to provide a meaningful comparison to help position our HybridSolver in the landscape of optimization solvers. To this end, we used a Mini-ITX PC with exactly the same CPU as Hans Mittelmann used in his previous benchmark: an Intel i7-11700K, 8 cores and 8 threads, 64GB RAM, running at 3.6 GHz. With this setup, we solved the 240 benchmark instances of MIPLIB2017 with our HybridSolver. In accordance with Hans Mittelmann's methodology, we applied a 2-hour time limit to each instance. If an instance reached this limit, its run time was counted as the full 2 hours (7200 seconds). Consistent with his benchmarks, we calculated the shifted geometric mean (SGM) with a shift of 10 seconds, which is widely used for comparing optimization solvers.
Our unscaled SGM was 136.3 seconds. In comparison, Gurobi 11 achieved an SGM of 72.1 seconds in the July 14, 2024 benchmark. This results in a scaled factor of 1.89 for our HybridSolver relative to Gurobi 11.
We also compared our HybridSolver to open-source alternatives. With unscaled SGMs of 894 seconds for SCIP and 720 seconds for HiGHS in the July 14, 2024 benchmark, our HybridSolver has scaled factors of 0.15 and 0.19, respectively. This means that the HybridSolver is approximately 5 to 6 times faster than these open-source alternatives.
Understanding Hardware Factors
When evaluating solver performance, it's important to consider the hardware used. Our benchmark tests were conducted on a high-end consumer PC with an Intel i7-11700K processor running at 3.6 GHz. In contrast, our cloud platform operates on Google Cloud VMs with Intel Xeon Gold 6253CL processors at 3.10 GHz. Cloud VMs may be slower than on-premises machines due to factors like virtualization overhead, resource sharing, and hardware optimized for scalability rather than peak performance. Based on a thorough analysis, we determined that these cloud VMs are approximately 40–50% slower than the consumer PC used in our benchmarks. This difference underscores the trade-off between the convenience and scalability of cloud solutions and the raw performance of dedicated hardware.
What This Means for You
The scaled factor of 1.89 compared to Gurobi 11 is based on the MIPLIB2017—a benchmark that has been extensively optimized by many solvers over the years. We are already comparable to leading solvers, and we believe that with additional tuning, we can further improve our performance across a wide range of problems.
Test Our HybridSolver Yourself
We invite you to test our HybridSolver with your own models to see how it performs on your specific optimization challenges. For most problems, we can achieve even better performance through targeted tuning. Your feedback helps us continue to improve and tailor our solutions to meet your needs.
More Than Just a Solver
As we discussed in our previous blog post We Are Not a Solver Company, our mission extends beyond providing a powerful solver. We focus on delivering a comprehensive optimization solution that includes the right model formulation, algorithms, hardware, and intuitive results interpretation.
Get Started Today
Ready to explore the capabilities of our HybridSolver? Send us your models, put us to the test, and discover how we can help solve your optimization problems. You can create a test account in just 30 seconds here. Feel free to use our web UI or API client to get started.
