Machine Learning with C++ in 2026 — LibTorch, ONNX Runtime, and When C++ Wins

The usual claim is that C++ is faster than Python for machine learning. Measured on the same model, through the same runtime, the honest answer is narrower and more useful than that.

On a tiny model, the C++ API ran inference in 1.99 µs against Python’s 4.80 µs — 2.4× faster. On a model with real computation in it, the same comparison was 14.48 µs against 18.29 µs — only 1.26×. Both numbers come from programs compiled and run for this article, and the gap between them is the whole point: C++ is not doing faster mathematics. It is avoiding about three microseconds of per-call overhead, which dominates a small model and vanishes into a large one.

That single distinction decides most real C++-versus-Python questions in ML, and it is what this guide is built around — along with a working ONNX Runtime inference program you can compile.

Table of Contents

• Training Versus Inference: The Distinction That Matters
• The Measured Difference
• A Working C++ Inference Program
• The C++ ML Ecosystem in 2026
• When C++ Actually Wins
• The Practical Workflow
• Key Takeaways
• Frequently Asked Questions
• Conclusion

Training Versus Inference: The Distinction That Matters

Almost every confused discussion of “ML in C++” collapses two different jobs.

Training is exploratory. You change the model, rerun, look at a graph, change it again. The bottleneck is how fast you can iterate, and the actual arithmetic already happens in optimised C++ or CUDA kernels regardless of which language calls them. Python wins here, decisively, and there is no serious argument otherwise.

Inference is production. The model is fixed, the code runs millions of times, and the constraints are latency, memory, binary size and what the target hardware will run at all. This is where C++ earns its place — not because the maths is faster, but because you remove an interpreter, a GC, a runtime, and a large dependency tree from the deployment.

The practical pattern in almost every serious ML system:

Train in Python  →  Export the model (ONNX, TorchScript)  →  Serve in C++

If you are asking “should I write ML in C++”, the honest first question is which of those two jobs you mean.

The Measured Difference

Both bars use the same ONNX Runtime, the same model file, and a single intra-op thread. The only variable is which language’s API drives it.

Tiny model  (3-element linear layer)
  C++    : 199 ms total, 1.99 us per inference
  Python : 480 ms total, 4.80 us per inference

Larger model (512 x 512 matrix multiply)
  C++    : 290 ms total, 14.48 us per inference
  Python : 366 ms total, 18.29 us per inference

The absolute saving is roughly constant — about 3 µs per call. As a proportion, it collapses from 141% to 26% as soon as there is real computation to do.

What follows from that:

• C++ wins clearly on small models at high request rates, on embedded and edge devices, and anywhere per-request latency budgets are measured in microseconds.
• C++ wins marginally on large models, where you are paying implementation cost for a percentage-point improvement.
• C++ wins for reasons other than speed far more often than people expect: no Python runtime to ship, a single binary, predictable memory, and deployment onto hardware where CPython is not an option.

A caveat worth stating plainly: this was measured on one machine, single-threaded, with a synthetic model. It demonstrates where the difference comes from, not what your production numbers will be. Measure your own model.

A Working C++ Inference Program

Here is the complete thing — ONNX Runtime, C++ API, compiled and run.

// Minimal ONNX Runtime inference in C++.
// Build: g++ -std=c++17 -I include infer.cpp -lonnxruntime -L lib -o infer
#include <iostream>
#include <vector>
#include <onnxruntime_cxx_api.h>

int main() {
    // 1. One environment per process; it owns logging and thread pools.
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "example");

    // 2. Session options: this is where you tune threading and optimisation.
    Ort::SessionOptions opts;
    opts.SetIntraOpNumThreads(1);
    opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

    // 3. Load the model once, reuse the session for every inference.
    Ort::Session session(env, "linear.onnx", opts);

    // 4. Describe the input tensor.
    std::vector<float>   input{2.0f, 3.0f, 4.0f};
    std::vector<int64_t> shape{1, 3};

    auto mem = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
    Ort::Value tensor = Ort::Value::CreateTensor<float>(
        mem, input.data(), input.size(), shape.data(), shape.size());

    // 5. Run. Names must match the model's declared inputs and outputs.
    const char* input_names[]  = {"input"};
    const char* output_names[] = {"output"};

    auto outputs = session.Run(Ort::RunOptions{nullptr},
                               input_names,  &tensor, 1,
                               output_names, 1);

    // 6. Read the result.
    const float* result = outputs[0].GetTensorData<float>();
    std::cout << "input:  [" << input[0] << ", " << input[1] << ", " << input[2] << "]\n";
    std::cout << "output: " << result[0] << '\n';
    return 0;
}

input:  [2, 3, 4]
output: 5

The model is a linear layer with weights [2.5, -1.0, 0.5] and bias 1.0, so 2.5(2) − 1(3) + 0.5(4) + 1 = 5. The Python runtime returns 5.0 for the same input — worth checking, because a mismatch between your Python and C++ paths is the most common bug in this workflow.

Five things in that program generalise to every ONNX Runtime deployment:

1. One Ort::Env per process. It owns thread pools; creating several wastes both.
2. Load the session once. Model loading is expensive; inference is cheap. Never construct a session per request.
3. SetIntraOpNumThreads is the first knob to turn. The default may oversubscribe a container’s CPU quota.
4. Input and output names must match the model exactly. Inspect them with session.GetInputNameAllocated(...) rather than assuming.
5. The tensor does not own its buffer. CreateTensor wraps memory you supply, so it must outlive the call.

The C++ ML Ecosystem in 2026

ONNX Runtime is the sensible default for inference. It runs models exported from PyTorch, TensorFlow, scikit-learn and others, it is genuinely cross-platform, and it does not tie you to the framework you trained in.

LibTorch is the right choice when you are already a PyTorch shop and want your C++ code to stay close to the Python. It is the only mainstream option here that supports training in C++ as a first-class activity, and its tensor API deliberately mirrors PyTorch’s. The cost is size — LibTorch adds hundreds of megabytes, which rules it out for many embedded targets.

TensorRT is the fastest path on NVIDIA hardware and the least portable. Reach for it when you have already decided the deployment target is NVIDIA and inference latency is the product.

mlpack and dlib cover the case people forget: not all machine learning is deep learning. If you need gradient boosting, SVMs, k-means or a random forest in a C++ binary, these are mature, small and well documented, and neither requires an ONNX pipeline.

When C++ Actually Wins

Six situations where the answer is genuinely C++, in rough order of how often they come up:

1. Embedded and edge deployment. A microcontroller, an ECU, a camera SoC. There is no Python runtime, and often no operating system worth the name.
2. Latency budgets in microseconds. High-frequency trading, real-time control, ad auctions. The per-call overhead measured above is not a rounding error at these scales.
3. Integration into an existing C++ system. A game engine, a CAD package, a trading platform. Embedding CPython to call a model is worse than linking a C++ runtime.
4. Small models at very high request rates. Where the fixed per-call cost dominates, C++’s advantage is proportionally largest — exactly the tiny-model case above.
5. Deployment simplicity. One statically-linked binary against a Python environment, its interpreter version, and a dependency tree that must be reproduced on every host.
6. Memory-constrained or GC-hostile environments where predictable allocation matters more than throughput.

And the honest inverse — use Python when: you are experimenting, the model changes weekly, you need the data-science tooling, or the model is large enough that a 26% call-overhead saving does not repay the engineering cost.

The Practical Workflow

1. Train in Python.  PyTorch or TensorFlow, whichever your team knows.
2. Export.           torch.onnx.export(...) or tf2onnx.
3. Verify.           Run the exported model in Python. Compare outputs
                     to the original model before going further.
4. Serve in C++.     Load with ONNX Runtime, wrap in your service.
5. Verify again.     Same input, same output, C++ against Python.

Step 3 and step 5 are the ones people skip, and they are where the bugs live. Export can silently change behaviour — an unsupported operator gets approximated, a dynamic shape gets frozen, a preprocessing step that lived in Python never makes it into the graph. The failure mode is not a crash; it is a model that is subtly wrong in production.

The single most common cause of a C++/Python mismatch is preprocessing, not the model. Image normalisation, tokenisation and feature scaling usually live outside the exported graph, and reimplementing them in C++ is where the discrepancy creeps in. Either export preprocessing into the graph where you can, or test it independently.

Key Takeaways

• C++ does not do faster maths. Both APIs call the same optimised kernels. C++ saves per-call overhead — measured at roughly 3 µs.
• That saving is 2.4× on a tiny model and 1.26× on a larger one. The advantage shrinks as compute grows.
• Train in Python, serve in C++. Training is exploratory and Python wins; inference is production and C++ has real advantages.
• ONNX Runtime is the default for deployment; LibTorch when you are a PyTorch shop or need training in C++; TensorRT when the target is NVIDIA and latency is the product.
• Not all ML is deep learning. mlpack and dlib cover classical methods without an ONNX pipeline.
• Verify the exported model twice — in Python after export, and in C++ after loading. Preprocessing is the usual culprit when they disagree.

Frequently Asked Questions

Conclusion

The C++-versus-Python argument in machine learning is usually conducted as though one language must be better. The measurements above suggest a more boring and more useful conclusion: they are fast at different things, and the difference is smaller and more specific than the rhetoric on either side.

What C++ actually buys you is not arithmetic speed — that comes from the same kernels either way — but control over everything around the model. Predictable latency, one binary, no interpreter, and a deployment target that can be an ECU or a camera rather than a server. If those matter to your problem, C++ is the right tool and the ecosystem is now mature enough to make it a routine choice rather than a heroic one. If they do not, Python’s iteration speed is worth more than three microseconds a call. The C++ section covers the language features these libraries lean on most heavily.