GPU memory demand includes the data and weights in use, intermediate results, runtime workspaces and workload-dependent caches. Measure peak usage at the intended precision, batch size and concurrency; the saved model size is only one input.
Choose the workload before choosing the GPU
A GPU specification becomes useful when you know the software and the work it will perform. Rendering, image processing, training and inference have different memory and compute patterns. Check the application's supported GPU architecture, drivers and runtime before comparing capacity. A card with enough memory can still be unsuitable if the required software path or feature is not supported.
Describe the maximum input size, precision, batch size and simultaneous work you intend to allow. For inference, include the longest supported request and concurrent sessions. For rendering, include the representative scene and textures. A successful tiny demonstration is evidence that the stack can start; it is not evidence that the full production workload fits.
Separate persistent data from working memory
Weights or input assets may remain on the device while intermediate results and temporary buffers appear during execution. Training can additionally require gradients and optimiser state. Inference workloads may maintain caches whose size changes with the request. The exact arrangement depends on the model, framework and implementation. Do not apply one universal multiplier to every task and treat it as a capacity guarantee.
As a simple arithmetic example, a hypothetical seven-billion-parameter model represented with two bytes per parameter needs about fourteen billion bytes for those parameters alone. That calculation excludes runtime allocations, activations, caches and format overhead. It is a lower-level planning example, not a statement that such a workload fits on a particular SwissOctet configuration.
| Memory component | What changes it | How to investigate |
|---|---|---|
| Weights or scene assets | Model size, representation and loaded assets | Inspect the deployed format and actual loaded allocation |
| Intermediate results | Input shape and batch size | Measure a full representative operation |
| Caches | Request length and concurrency | Exercise the supported maximum workload |
| Workspaces and runtime | Libraries, kernels and execution strategy | Observe peak device and framework usage |
Measure peak usage through the right view
Framework metrics and device-level metrics can describe different things. PyTorch distinguishes memory allocated to tensors from memory reserved by its caching allocator, and provides peak measures. Its memory visualisation documentation also explains that allocations made outside its allocator may not appear in that view. Compare the framework's report with the wider device state rather than assuming one number covers every allocation.
Warm up the actual operation, then measure the full task and realistic overlap between tasks. Record software versions and the exact input conditions. An out-of-memory event may occur at a temporary peak, so a quiet final snapshot can miss the problem. Keep diagnostic captures private when they might contain model details or information about your application's inputs.
Treat batch size and precision as design choices
Larger batches can change both utilisation and memory demand. NVIDIA's performance guidance discusses the interaction between operation shape, arithmetic intensity and hardware capabilities. The useful batch size depends on the workload's latency target as well as throughput. A batch policy that maximises completed work per second may make individual users wait longer than the application permits.
Reduced precision or quantisation can reduce some memory requirements, but compatibility and output quality need testing. Keep an evaluation dataset and compare the relevant results after a format change. Do not change a model's representation solely to make it fit without checking the behaviour the project depends on. Memory optimisation should preserve the required outcome, not merely avoid an allocation error.
Keep the rest of the machine in the budget
The CPU, host RAM and storage path prepare and feed GPU work. Slow input decoding or insufficient host memory can leave the accelerator waiting. Large model files and datasets also need enough disk space for installation, temporary conversions and updates. Evaluate the complete pipeline instead of purchasing a GPU and expecting every preceding step to become faster.
Multiple GPUs do not automatically behave as one pooled memory device. The application must use an appropriate parallel strategy, and communication overhead can matter. Confirm how many devices are provided, the memory of each and the software's supported arrangement. A list of GPU upgrade names is not a verified description of the memory available to one unmodified process.
Use a representative acceptance test
Define a test with the chosen software stack, expected maximum input and allowed concurrency. Measure peak memory, completion time, errors and the quality of the output. Keep some headroom for normal runtime variation and maintenance, with the amount justified by observations. If the workload does not fit, decide whether to reduce concurrency, change the implementation or choose different hardware.
Tensor GPU presents GPU-related choices in the SwissOctet catalogue. Confirm the exact device, per-device VRAM, driver compatibility and selected configuration before relying on it. The site does not provide an application-specific fit guarantee. A documented acceptance test connects those hardware details with the work you need the machine to perform.
Your practical checklist.
- Confirm supported GPU architecture and runtime.
- Budget weights, intermediates, caches and workspaces.
- Measure peaks at maximum intended input and concurrency.
- Compare framework and device-level memory views.
- Validate output after precision or format changes.
- Confirm exact per-device VRAM before ordering.
The details that matter.
Does the saved model size equal VRAM demand?+
No. Execution adds intermediate results, workspaces, caches and other allocations, while storage formats can differ from in-memory representations.
Will adding a second GPU automatically double usable memory?+
No. The application needs an appropriate parallel strategy. Per-device limits and communication behaviour still matter.
Can reduced precision always solve an out-of-memory problem?+
No. It may reduce some allocations, but support, remaining buffers and acceptable output quality must be tested.
Go to the source.
These references support the technical concepts in this guide. Follow the current documentation before changing a production system.
- NVIDIA: Get Started With Deep Learning Performance docs.nvidia.com
- PyTorch: Understanding CUDA Memory Usage docs.pytorch.org
- PyTorch: CUDA memory management docs.pytorch.org
Editorial guidance by SwissOctet. Examples are illustrative. Service specifications, available options and contractual conditions must be verified separately.