IEEE

Toward Cost-Efficient LLM Serving: A System-Level Memory Optimization Approach

Abstract: Serving large-scale language models (LLMs) requires significant system resources, where GPU memory limits, system bottlenecks, and I/O delays collectively ...
GitHub

samixyzdev/llm-inference-bench

A GPU benchmarking toolkit for measuring Large Language Model (LLM) inference performance. This tool evaluates throughput, latency, and memory usage across different models, quantization levels, and ...
IEEE

LLM-on-the-Palm: Mobile LLM Inference with PIM-Enhanced NAND Flash Memory

Abstract: Large Language Model (LLM) inference on edge devices is crucial for democratizing AI and addressing privacy and security concerns associated with cloud services. However, the large parameter ...