Introduction
Python's execution speed often leaves much to be desired. Some people abandon Python for this very reason, but there are several ways to optimize Python code, both in terms of time and memory usage.I'd like to share a few methods that help with real-world problems. I'm using Windows 10 x64.
Saving Memory with Python
Let's look at a very real-world example. Let's say we have a store with a product list. We need to work with these products. The best case scenario is when all the products are stored in the database, but suddenly something goes wrong, and we decide to load them all into memory for processing. A reasonable question arises: will we have enough memory to handle such a large number of products?Let's first create a class responsible for our store. It will have only two fields: name and listGoods, which represent the store name and the product list, respectively.
class ShopClass:
def __init__(self, name=""):
self.name = name
self.listGoods = []
Now we want to populate the store with products (specifically, fill the listGoods field). To do this, we'll create a class responsible for information about a single product (I use dataclasses for examples like this).
# если ругается на dataclass, то делайте
# pip install dataclasses
# затем в коде вызывайте импорт
# from dataclasses import dataclass
@dataclass
class DataGoods:
name:str
price:int
unit:str
Next, we need to fill our store with products. For the sake of clarity, I'll create 200 identical products in three categories:
shop = ShopClass("MyShop")
for _ in range(200):
shop.listGoods.extend([
DataGoods("телефон", 20000, "RUB"),
DataGoods("телевизор", 45000, "RUB"),
DataGoods("тостер", 2000, "RUB")
])
Now it's time to measure the memory our store takes up in RAM (I use the pympler module to measure memory):
from pympler import asizeof
print("Размер магазина:", asizeof.asizeof(shop))
>>> Размер магазина: 106648
So, our store took up almost 106 KB of RAM. While that's not much, considering I only stored 600 products, filling in only the name, price, and currency information, a real-world scenario would require storing several times that many fields. For example, you might need to store the product's SKU, manufacturer, quantity in stock, country of origin, model color, weight, and many other parameters. All this data could bloat your store from a few kilobytes to several hundred megabytes (and that's before you even begin processing the data).
Now let's move on to solving this problem. Python creates a new object in a way that allocates a lot of information for it, information we don't even realize is there. It's important to understand that Python creates a __dict__ object within a class so that new attributes can be added and existing ones removed without much effort or consequences. Let's look at how to dynamically add new attributes to a class.
shop = ShopClass("MyShop")
print(shop.__dict__)
>>> {'name': 'MyShop', 'listGoods': []}
shop.city = "Москва"
print(shop.__dict__)
>>> {'name': 'MyShop', 'listGoods': [], 'city': 'Москва'}
However, in our example, this doesn't matter at all. We already know in advance which attributes we need. Python has a magical attribute, __slots__, which allows us to eliminate the need for __dict__. Eliminating __dict__ will prevent new classes from creating a dictionary with all the attributes and the data they store, which should ultimately reduce their memory footprint. Let's make some minor changes to our classes:
class ShopClass:
__slots__ = ("name", "listGoods")
def __init__(self, name=""):
self.name = name
self.listGoods = []
@dataclass
class DataGoods:
__slots__ = ("name", "price", "unit")
name:str
price:int
unit:str
And we will test our store from memory.
from pympler import asizeof
print("Размер магазина:", asizeof.asizeof(shop))
>>> Размер магазина: 43904
As you can see, the memory footprint of the store has been reduced by almost 2.4 times (this value may vary depending on the operating system, Python version, values, and other factors). We were able to optimize the memory footprint by adding just a couple of lines of code. However, this approach has its drawbacks. For example, if you try to add a new attribute, you'll get an error:
shop = ShopClass("MyShop")
shop.city = "Москва"
>>> AttributeError: 'ShopClass' object has no attribute 'city'
The benefits of using slots don't end there. By eliminating the __dict__ attribute, Ptyhon no longer needs to populate each class's dictionary, which also impacts the algorithm's performance. Let's test our code using the timeit module. We'll first test our code with __slots__ disabled (__dict__ enabled):
import timeit
code = """
class ShopClass:
#__slots__ = ("name", "listGoods")
def __init__(self, name=""):
self.name = name
self.listGoods = []
@dataclass
class DataGoods:
#__slots__ = ("name", "price", "unit")
name:str
price:int
unit:str
shop = ShopClass("MyShop")
for _ in range(200):
shop.listGoods.extend([
DataGoods("телефон", 20000, "RUB"),
DataGoods("телевизор", 45000, "RUB"),
DataGoods("тостер", 2000, "RUB")
])
"""
print(timeit.timeit(code, number=60000))
>>> 33.4812513
Теперь включим __slots__ (#__slots__ = ("name", "price", "unit") -> __slots__ = ("name", "price", "unit") и # __slots__ = ("name", "listGoods") -> __slots__ = ("name", "listGoods")):
# включили __slots__ в коде выше
print(timeit.timeit(code, number=60000))
>>> 28.535005599999998
The result was more than satisfactory, it was possible to speed up the code by about 15% (testing was carried out several times, the result was always approximately the same).
Thus, we were able to not only reduce the amount of memory occupied by the program, but also speed up our code.
We're trying to speed up the code.
There are several ways to speed up Python, ranging from using built-in language features (for example, those described in the previous chapter) to writing extensions in C/C++ and other languages.I'll only tell you about those methods that won't take you much time to learn and will allow you to start using this functionality in a short time.
Cython
In my opinion, Cython is an excellent solution if you want to write code in Python but are concerned about execution speed. Let's implement code to calculate the total cost of all TVs, phones, and toasters in pure Python and calculate the elapsed time (we'll be creating 20,000,000 items):import time
class ShopClass:
__slots__ = ("name", "listGoods")
def __init__(self, name=""):
self.name = name
self.listGoods = []
@dataclass
class DataGoods:
__slots__ = ("name", "price", "unit")
name: str
price: int
unit: str
shop = ShopClass("MyShop")
t = time.time()
for _ in range(200*100000):
shop.listGoods.extend([
DataGoods("телефон", 20000, "RUB"),
DataGoods("телевизор", 45000, "RUB"),
DataGoods("тостер", 2000, "RUB")
])
print("СОЗДАЕМ ТОВАРЫ НА PYTHON:", time.time()-t)
>>> СОЗДАЕМ ТОВАРЫ НА PYTHON: 44.49887752532959
telephoneSum, televizorSum, tosterSum = 0, 0, 0
t = time.time()
for goods in shop.listGoods:
if goods.name == "телефон":
telephoneSum += goods.price
elif goods.name == "телевизор":
televizorSum += goods.price
elif goods.name == "тостер":
tosterSum += goods.price
print("ВРЕМЯ НА ПОДСЧЁТ СУММ PYTHON:", time.time() - t)
>>> ВРЕМЯ НА ПОДСЧЁТ СУММ PYTHON: 13.135360717773438
As we can see, the processing time is quite disappointing. Now let's start using Cython. First, install the cython_npm library (see the official GitHub page ): pip install cython-npm . Now let's create a new folder in our project, name it cython_code , and create a file called cython_data.pyx in it (Cython programs are written with the .pyx extension).
Let's rewrite the store class for Cython:
cdef class CythonShopClass:
cdef str name
cdef list listGoods
def __init__(self, str name):
self.name = name
self.listGoods = []
In Cython, you must strictly type every variable you use in your code (this isn't required, but not doing so won't reduce your execution time). To do this, you need to write cdef <data type> <variable name> in each class or method. We'll implement the rest of the code in Cython. We'll implement the my_def() function without cdef, using the familiar def, since we'll be calling it from the main Python file. We also need to specify the language version at the beginning of our .pyx file (# cython: language_level=3).
# cython: language_level=3
# на забывает вставить код класса магазина
cdef class CythonDataGoods:
cdef str name
cdef int price
cdef str unit
def __init__(self, str name, int price, str unit):
self.name = name
self.price = price
self.unit = unit
cdef int c_testFunc():
cdef CythonShopClass shop
cdef CythonDataGoods goods
cdef int i, t, telephoneSum, televizorSum, tosterSum
size, i, telephoneSum, televizorSum, tosterSum = 0, 0, 0, 0, 0
shop = CythonShopClass("MyShop")
t = time.time()
for i in range(200*100000):
shop.listGoods.extend([
CythonDataGoods("телефон", 20000, "RUB"),
CythonDataGoods("телевизор", 45000, "RUB"),
CythonDataGoods("тостер", 2000, "RUB")
])
print("СОЗДАЕМ ТОВАРЫ НА CYTHON:", time.time()-t)
t = time.time()
for goods in shop.listGoods:
if goods.name == "телефон":
telephoneSum += goods.price
elif goods.name == "телевизор":
televizorSum += goods.price
elif goods.name == "тостер":
tosterSum += goods.price
print("ВРЕМЯ НА ПОДСЧЁТ СУММ CYTHON:", time.time() - t)
return 0
def my_def():
data = c_testFunc()
return data
Now, in our project's main.py , we'll call the Cython code. To do this, first import all installed libraries:
from cython_npm.cythoncompile import export
from cython_npm.cythoncompile import install
import time
And we immediately compile our Cython and import it into the main Python code
export('cython_code/cython_data.pyx')
import cython_code.cython_data as cython_data
Now we need to call the cython code
if __name__ == "__main__":
a = cython_data.my_def()
Let's run it. Let's take a look at the console output. In Cython, where we were printing the product creation time, we got:
>>> СОЗДАЕМ ТОВАРЫ НА CYTHON: 4.082242012023926
And where there was a conclusion after calculating the amounts, we received:
>>> ВРЕМЯ НА ПОДСЧЁТ СУММ CYTHON: 1.0513946056365967
As we can see, the product creation time has dropped from 44 seconds to 4 seconds, meaning we've sped up this part of the code by almost 11 times. The time for calculating amounts has dropped from 13 seconds to 1 second, a reduction of approximately 13 times.
Thus, using Cython is one of the easiest ways to speed up your program several times over, and it's also suitable for those who adhere to data typing in their code. It's also worth noting that the speedup time depends on the task; for some tasks, Cython can speed up your code by up to 100 times.
The Magic of Python
While using third-party add-ons or modules to speed things up is certainly a good idea, it's also worth optimizing your algorithms. For example, let's speed up the part of the code that adds new products to the store list. To do this, we'll write a lambda function that will return a list of parameters needed for the new product. We'll also use a list generator:shop = ShopClass("MyShop")
t = time.time()
getGoods = lambda index: {0: ("телефон", 20000, "RUB"),
1: ("телевизор", 45000, "RUB"),
2
"тостер", 2000, "RUB")}.get(index) shop.listGoods = [DataGoods(*getGoods(i%3)) for i in range(200*100000)]
print("СОЗДАЕМ ТОВАРЫ НА PYTHON:", time.time()-t)
>>> СОЗДАЕМ ТОВАРЫ НА PYTHON: 19.719463109970093
Speed increased by approximately 2 times, and we did this by leveraging Python's own capabilities. Generators in Python are a very convenient feature; they not only allow you to speed up your code but also optimize its memory usage.
PyPy
Sometimes it's impossible to rewrite code in Cython or another language because you already have a large codebase (or for some other reason), but you want to improve the program's execution speed. Consider the code from the previous example, where we used lambda functions and a list comprehension. PyPy, a Python interpreter with a JIT compiler, can help here. However, PyPy doesn't support all third-party libraries; if you use them in your code, consult the documentation for more details. Running Python code with PyPy is very easy.First, download PyPy from the official website . Unzip it to any folder, open a command prompt, and navigate to the folder where pypy3.exe now resides. We'll also place our program code in this folder. Now, enter the following command in the command prompt:
Thus, we were able to reduce the 19 seconds of Python code from the previous example to 4.5 seconds without rewriting the code at all, that is, almost 4 times.
Conclusion
We explored several options for optimizing code for time and memory. Despite all the haters who say Python is slow, we were able to achieve dozens of times faster code execution.Not all possible code acceleration options have been covered. In some cases, Numba, NumPy, Nim, or multiprocessing can be used. It all depends on the task you're solving. Some tasks will be easier to solve in other languages, as Python can't handle everything.
Before choosing code optimization features, it's essential to perform internal optimization of your pure Python code, eliminating as many loops within loops within loops as possible, manually clearing memory, and removing unnecessary elements as the code runs. Don't expect that rewriting your code in another language will solve all your problems; learn to find bottlenecks in your code and optimize them algorithmically or using the language's own features.
