Machine Learning with C++ in 2026 — LibTorch, ONNX Runtime, and When C++ Wins

The usual claim is that C++ is faster than Python for machine learning. Measured across two runtimes and two model sizes on one machine, the picture is more specific and considerably more useful.

ONNX Runtime called from Python (6.81 µs) was faster than LibTorch called from C++ (7.42 µs). Choosing the right runtime beat choosing the right language.

That result falls out of a simple pattern in the numbers: across both runtimes and both model sizes, the C++ advantage was a near-constant 4.8 to 7.0 microseconds of per-call overhead. It is not faster arithmetic — both languages call the same optimised kernels. It is the cost of the Python/native boundary: converting arguments, constructing tensor objects, and crossing the binding layer around the runtime. On a trivial model that cost is most of the runtime; on a larger one it disappears into the computation.

What this measures, and what it does not. These are synthetic models chosen to isolate call overhead. They are not ResNet, BERT or an LLM, and the benchmark says nothing about end-to-end performance on those. Read it as a measurement of runtime and binding overhead, not of machine learning throughput.

Everything below was compiled and run: a working ONNX Runtime inference program, a working LibTorch one, and benchmarks of all four combinations.

Table of Contents

• The Short Answer
• Training Versus Inference: The Distinction That Matters
• The Measured Difference
• A Working C++ Inference Program
• The C++ ML Ecosystem in 2026
• Not All Machine Learning Is Deep Learning
• When C++ Actually Wins
• The Practical Workflow
• Key Takeaways
• Frequently Asked Questions
• Conclusion

The Short Answer

Can you use C++ for machine learning? Yes, and for deployment it is routine. Python remains the practical default for training and experimentation. C++ becomes useful for inference, when latency, memory, embedded targets, native integration or predictable resource use matter.

Is C++ faster? For inference, yes — but the advantage is a roughly fixed per-call overhead, not faster mathematics, so it shrinks as models grow. And it is not the only lever: in the benchmark below, ONNX Runtime called from Python beat LibTorch called from C++.

Which library? ONNX Runtime for inference in most cases. LibTorch if you are committed to PyTorch or need to train in C++. mlpack or dlib for classical machine learning.

Training Versus Inference: The Distinction That Matters

Almost every confused discussion of “ML in C++” collapses two different jobs.

Training is exploratory. You change the model, rerun, look at a graph, change it again. The bottleneck is how fast you can iterate, and the actual arithmetic already happens in optimised C++ or CUDA kernels regardless of which language calls them. Python wins here, decisively, and there is no serious argument otherwise.

Inference is production. The model is fixed, the code runs millions of times, and the constraints are latency, memory, binary size and what the target hardware will run at all. This is where C++ earns its place — not because the maths is faster, but because you remove an interpreter, a GC, a runtime, and a large dependency tree from the deployment.

A common production pattern:

Train in Python  →  Export the model (ONNX or torch.export)  →  Serve in a native runtime

It is not universal — plenty of production systems train and serve in Python, or serve through Triton, Go, Rust or a managed cloud runtime. But if you are asking “should I write ML in C++”, the first question is which of those two jobs you mean.

The Measured Difference

How this was measured

Everything below comes from one machine, so the four figures are directly comparable.

What is not here: these are means, not distributions — no median, p95 or standard deviation was captured, so treat them as indicative rather than statistically robust. No CPU pinning or frequency-scaling control was applied, and WSL2 adds syscall overhead that may inflate all four figures equally. Both C++ programs printed the same sanity values as their Python counterparts (3.0000 and −5.8626), confirming identical arithmetic across all four setups.

Four combinations, same two models, one machine, single intra-op thread throughout:

Both C++ programs printed the same values as their Python counterparts — 3.0000 on the tiny model, −5.8626 on the larger one — confirming all four ran identical arithmetic.

Finding 1: the C++ advantage is a fixed cost, not a multiplier

Within each runtime, C++ versus Python:

The ratios collapse as the model grows. The absolute gaps barely move — all four land between 4.76 and 6.97 µs, averaging 5.54.

The measurements are consistent with a roughly five-microsecond per-call overhead around the Python/native boundary, including binding and object-management work. The benchmark does not isolate each contributor individually.

Finding 2: the runtime matters more than the language

Comparing C++ to C++, ONNX Runtime was 3.62× faster than LibTorch on the tiny model and 1.22× faster on the larger one. The same ordering held in Python.

And the result that reframes the usual argument: ONNX Runtime from Python (6.81 µs) beat LibTorch from C++ (7.42 µs) on the tiny model, and essentially tied on the larger one (27.90 against 27.59).

If you are choosing between “rewrite this in C++” and “switch inference runtime”, the second is often the cheaper win — and on this evidence sometimes the larger one.

Caveats, stated plainly. These figures come from one machine (12-core, WSL2 on Windows, pinned to a single thread) using synthetic models. WSL2 adds syscall overhead that may inflate all four numbers. What generalises is the shape — a fixed per-call cost, and a runtime difference independent of language — not the absolute microseconds. Measure your own model on your own target.

A Working C++ Inference Program

Here is the complete thing — ONNX Runtime, C++ API, compiled and run.

// Minimal ONNX Runtime inference in C++.
// Build: g++ -std=c++17 -I include infer.cpp -lonnxruntime -L lib -o infer
#include <iostream>
#include <vector>
#include <onnxruntime_cxx_api.h>

int main() {
    // 1. One environment per process; it owns logging and thread pools.
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "example");

    // 2. Session options: this is where you tune threading and optimisation.
    Ort::SessionOptions opts;
    opts.SetIntraOpNumThreads(1);
    opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

    // 3. Load the model once, reuse the session for every inference.
    Ort::Session session(env, "linear.onnx", opts);

    // 4. Describe the input tensor.
    std::vector<float>   input{2.0f, 3.0f, 4.0f};
    std::vector<int64_t> shape{1, 3};

    auto mem = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
    Ort::Value tensor = Ort::Value::CreateTensor<float>(
        mem, input.data(), input.size(), shape.data(), shape.size());

    // 5. Run. Names must match the model's declared inputs and outputs.
    const char* input_names[]  = {"input"};
    const char* output_names[] = {"output"};

    auto outputs = session.Run(Ort::RunOptions{nullptr},
                               input_names,  &tensor, 1,
                               output_names, 1);

    // 6. Read the result.
    const float* result = outputs[0].GetTensorData<float>();
    std::cout << "input:  [" << input[0] << ", " << input[1] << ", " << input[2] << "]\n";
    std::cout << "output: " << result[0] << '\n';
    return 0;
}

input:  [2, 3, 4]
output: 5

The model is a linear layer with weights [2.5, -1.0, 0.5] and bias 1.0, so 2.5(2) − 1(3) + 0.5(4) + 1 = 5. The Python runtime returns 5.0 for the same input — worth checking, because a mismatch between your Python and C++ paths is the most common bug in this workflow.

Five things in that program generalise to every ONNX Runtime deployment:

1. One Ort::Env per process. It owns thread pools; creating several wastes both.
2. Load the session once. Model loading is expensive; inference is cheap. Never construct a session per request.
3. SetIntraOpNumThreads is the first knob to turn. The default may oversubscribe a container’s CPU quota.
4. Input and output names must match the model exactly. Inspect them with session.GetInputNameAllocated(...) rather than assuming.
5. The tensor does not own its buffer. CreateTensor wraps memory you supply, so it must outlive the call.

The same thing in LibTorch

If you are already a PyTorch shop, the LibTorch equivalent is shorter, because TorchScript carries the graph and the tensor API mirrors Python’s:

// LibTorch inference. Build with CMake and find_package(Torch REQUIRED).
#include <torch/script.h>
#include <torch/torch.h>
#include <ATen/Parallel.h>
#include <iostream>

int main() {
    at::set_num_threads(1);

    torch::jit::script::Module model;
    try {
        model = torch::jit::load("linear.pt");
    } catch (const c10::Error& e) {
        std::cerr << "load failed: " << e.what() << '\n';   // print e.what()
        return 1;
    }
    model.eval();
    torch::NoGradGuard no_grad;          // no autograd bookkeeping for inference

    auto input = torch::tensor({{2.0f, 3.0f, 4.0f}});
    auto output = model.forward({input}).toTensor();

    std::cout << "output: " << output[0][0].item<float>() << '\n';
}

Two practical notes that cost real time to discover.

Print e.what() in the catch block. A bare “load failed” message tells you nothing, and TorchScript load failures have several distinct causes that the exception text distinguishes precisely.

LibTorch and the PyTorch that exported your model must be version-compatible. TorchScript archives carry a format version, and an older LibTorch refuses a newer archive with maximum supported version for reading is N. Worth knowing: the libtorch-shared-with-deps-latest.zip URL is not reliably current — the copy downloaded while preparing this article turned out to support format version 1, which dates from around 2019. Download an explicitly versioned build matching your PyTorch, and check $LIBTORCH/build-version before building anything.

The 2026 problem: TorchScript is deprecated

The model above was exported with torch.jit.trace, because that is what LibTorch loads. It is also deprecated. PyTorch’s own documentation now states that TorchScript is deprecated and directs users to torch.export, and both torch.jit.trace and torch.jit.script emit a DeprecationWarning telling you to switch to torch.compile or torch.export. torch.jit.script is additionally unsupported on Python 3.14 and later.

This does not make LibTorch irrelevant, and it does not invalidate the benchmark — TorchScript still works and is still how LibTorch consumes a serialised model. But it changes what you should build on:

There is a detail worth knowing if you export to ONNX: PyTorch versions before 2.9 generated the ONNX graph through TorchScript, while newer versions route through the torch.export pipeline. If an export that worked on an older PyTorch behaves differently after an upgrade, that change of mechanism is the first place to look.

The practical consequence for this article’s recommendation is that it gets stronger, not weaker. ONNX Runtime was already the faster option in both languages; it is now also the export path that is not deprecated.

The C++ ML Ecosystem in 2026

ONNX Runtime is the sensible default for inference. It runs models exported from PyTorch, TensorFlow, scikit-learn and others, it is genuinely cross-platform, and it does not tie you to the framework you trained in.

LibTorch is the right choice when you are already a PyTorch shop and want your C++ code to stay close to the Python. It is the only mainstream option here that supports training in C++ as a first-class activity, and its tensor API deliberately mirrors PyTorch’s. The cost is size — LibTorch adds hundreds of megabytes, which rules it out for many embedded targets.

TensorRT is the fastest path on NVIDIA hardware and the least portable. Reach for it when you have already decided the deployment target is NVIDIA and inference latency is the product.

mlpack and dlib cover the case people forget: not all machine learning is deep learning. If you need gradient boosting, SVMs, k-means or a random forest in a C++ binary, these are mature, small and well documented, and neither requires an ONNX pipeline.

Not All Machine Learning Is Deep Learning

If you need clustering, an SVM or a random forest inside a C++ binary, neither ONNX nor a training framework is involved at all. Two mature libraries cover this, and both are small.

mlpack — k-means over 60 points in three clusters:

#include <mlpack.hpp>

arma::Row<size_t> assignments;
arma::mat centroids;
mlpack::KMeans<> k;
k.Cluster(data, 3, assignments, centroids);

mlpack k-means, 3 clusters over 60 points
centroids:
    0.7043    1.3611
   10.8615   10.9625
    6.1387    6.1115
  cluster 0: 20 points
  cluster 1: 20 points
  cluster 2: 20 points

The synthetic clusters were centred at 1, 6 and 11; k-means recovered all three with 20 points each. Build with g++ -std=c++17 -O2 kmeans.cpp -o kmeans -larmadillo — modern mlpack is header-only, so there is no -lmlpack to link.

dlib — an SVM separating points inside a circle from points outside it:

dlib SVM, 2500 training points
  point (3,3)   -> 3.07187  (expect positive)
  point (15,15) -> -10.4228  (expect negative)

Correct signs, with margins that reflect distance from the boundary rather than sitting near zero.

The size argument is stark here. The compiled binaries came to 112 KB for mlpack and 68 KB for dlib — against a 275 MB LibTorch distribution. If your problem is classical machine learning, reaching for a deep-learning framework is several orders of magnitude of unnecessary dependency.

When C++ Actually Wins

Six situations where the answer is genuinely C++, in rough order of how often they come up:

1. Embedded and edge deployment. A microcontroller, an ECU, a camera SoC. There is no Python runtime, and often no operating system worth the name.
2. Latency budgets in microseconds. High-frequency trading, real-time control, ad auctions. The per-call overhead measured above is not a rounding error at these scales.
3. Integration into an existing C++ system. A game engine, a CAD package, a trading platform. Embedding CPython to call a model is worse than linking a C++ runtime.
4. Small models at very high request rates. Where the fixed per-call cost dominates, C++’s advantage is proportionally largest — exactly the tiny-model case above.
5. Deployment simplicity. One statically-linked binary against a Python environment, its interpreter version, and a dependency tree that must be reproduced on every host.
6. Memory-constrained or GC-hostile environments where predictable allocation matters more than throughput.

And the honest inverse — use Python when: you are experimenting, the model changes weekly, you need the data-science tooling, or the model is large enough that a 26% call-overhead saving does not repay the engineering cost.

The Practical Workflow

1. Train in Python.  PyTorch or TensorFlow, whichever your team knows.
2. Export.           torch.onnx.export(...) or tf2onnx.
3. Verify.           Run the exported model in Python. Compare outputs
                     to the original model before going further.
4. Serve in C++.     Load with ONNX Runtime, wrap in your service.
5. Verify again.     Same input, same output, C++ against Python.

Step 1 has an implicit step 0. The workflow starts at “train in Python”, which quietly assumes the training data already exists in usable form. For prototyping it often does — the public benchmark sets are one download away. For a production model solving a specific commercial problem it usually does not, and the choice is between building collection and cleaning in-house or sourcing machine learning datasets from a provider that has already done it. That decision shapes the schedule more than anything else here does, and it is settled long before the question of which runtime to serve with comes up.

Step 3 and step 5 are the ones people skip, and they are where the bugs live. Export can silently change behaviour — an unsupported operator gets approximated, a dynamic shape gets frozen, a preprocessing step that lived in Python never makes it into the graph. The failure mode is not a crash; it is a model that is subtly wrong in production.

The single most common cause of a C++/Python mismatch is preprocessing, not the model. Image normalisation, tokenisation and feature scaling usually live outside the exported graph, and reimplementing them in C++ is where the discrepancy creeps in. Either export preprocessing into the graph where you can, or test it independently.

Key Takeaways

• The runtime can matter more than the language. ONNX Runtime from Python (6.81 µs) beat LibTorch from C++ (7.42 µs) on the tiny model.
• C++ does not do faster maths. Both languages call the same kernels. C++ removes per-call overhead, measured at a near-constant 4.8–7.0 µs across two runtimes and two model sizes.
• Ratios mislead, absolutes do not. The same fixed cost read as 3.32× on a trivial model and 1.23× on a real one.
• ONNX Runtime was the fastest option in both languages — 3.62× faster than LibTorch in C++ on the tiny model, and ahead in Python too.
• LibTorch costs 275 MB against ONNX Runtime’s 25 MB. That decides embedded deployments on its own.
• Classical ML binaries are tiny — 112 KB with mlpack, 68 KB with dlib. Not every ML problem needs a deep-learning framework.
• Match LibTorch to your PyTorch version. The latest.zip link served a build supporting a 2019-era format; check build-version first.

Frequently Asked Questions

Conclusion

The C++-versus-Python argument in machine learning is usually conducted as though one language must be better. The measurements above suggest a more boring and more useful conclusion: they are fast at different things, and the difference is smaller and more specific than the rhetoric on either side.

C++ does not make the model’s mathematical operations faster; those run in the same kernels either way. Its value is control over the layer around the model: runtime overhead, memory behaviour, integration, startup, threading and hardware-specific deployment. And before rewriting anything, it is worth checking whether a different runtime gets you there first: on the evidence above, switching from LibTorch to ONNX Runtime was a larger win than switching from Python to C++. Predictable latency, one binary, no interpreter, and a deployment target that can be an ECU or a camera rather than a server. If those matter to your problem, C++ is the right tool and the ecosystem is now mature enough to make it a routine choice rather than a heroic one. If they do not, Python’s iteration speed is worth more than three microseconds a call. The C++ section covers the language features these libraries lean on most heavily.