Google has announced a new compression algorithm called TurboQuant, which could fundamentally change one of the biggest bottlenecks in artificial intelligence models: memory consumption and processing cost.
Google has announced a new technology that could fundamentally change one of the biggest bottlenecks in artificial intelligence models: memory consumption and processing cost. The compression algorithm, named “TurboQuant,” aims to make modern AI systems, especially large language models, much smaller, faster, and more efficient.
According to information shared on the company's research blog, TurboQuant can significantly reduce model size while remarkably operating without loss of accuracy. This feature directly addresses one of the biggest problems encountered in AI optimization to date.
TurboQuant focuses on the “key-value cache.” Google likens this to a kind of digital cheat sheet where the model stores important information to avoid recalculating. This is because large language models don't actually 'know' information; instead, they operate through vectors that represent meanings. These vectors digitize text to establish semantic relationships. The closer two vectors are, the more conceptually similar they are.
However, these vectors are very high-dimensional and can contain hundreds or even thousands of parameters. This leads to both high memory consumption and reduced performance. To solve this problem, developers typically process data with lower precision using methods called “quantization.” The disadvantage of this, however, is a reduction in the quality of model outputs. According to Google's initial tests, TurboQuant can achieve 6 times less memory usage and 8 times higher performance without compromising quality.
How does TurboQuant work?
TurboQuant consists of a two-stage process. In the first step, a system called “PolarQuant” comes into play. While AI vectors are normally expressed with XYZ (Cartesian) coordinates, PolarQuant converts them into polar coordinates. Thus, each vector is represented by only two pieces of information: radius (the strength of the data) and angle (the semantic direction of the data).
Google illustrates this as follows: While the traditional method is like saying “go 3 blocks east, 4 blocks north,” the new method offers a shorter and more efficient expression like “go 5 blocks at a 37-degree angle.” This reduces both data size and computational load.
In the second stage, minor errors that may occur are corrected. PolarQuant can create some deviations when compressing data. To correct this, the “Quantized Johnson-Lindenstrauss (QJL)” method is used. This technique adds an error correction layer by representing each vector with only a single bit (+1 or -1), preserving important relationships. As a result, the model's attention calculations become more accurate.
Impressive performance figures
Google states that it has tested TurboQuant on open models like Gemma and Mistral. TurboQuant reportedly achieved flawless results while maintaining output quality in all tests, successfully reducing key-value cache memory usage by 6 times. The algorithm can compress the cache to just a 3-bit level without requiring additional training, meaning it can be directly applied to existing models. Furthermore, with 4-bit TurboQuant, attention calculations can be performed 8 times faster on an Nvidia H100 compared to 32-bit uncompressed keys.
Technologies like TurboQuant can reduce the operational cost of AI models and enable the creation of more powerful systems with less memory. This development is particularly important for mobile devices. Considering the hardware limitations of smartphones, compression techniques like TurboQuant could allow for higher-quality AI outputs directly on the device without sending data to the cloud.
0 Comments: