• Python Slot Machine Code
• Python Slots Memory Drive

Introduction

Prevent Signal/Slot memory leaks in python. By Jon Boyce On November 15, 2018. When using Qt in python, any QObject with a connection to at least one of its signals is not eligible for garbage collection. This makes it very easy to have memory leaks when using dynamically created QWidgets. In order to not leak memory, you must disconnect any slots from all signals on the QObject before dereferencing. When dealing with thousands of instances, memory consumption can be a problem. Because of the underlying implementation of a hash table, creating a dictionary for each instance takes a lot of memory. Hopefully, Python provides a way to disable per-instance dictionary by defining slots attribute. Here is how slots are usully defined.

In Python, every object instance comes pre-built with standard functions and attributes. For example, Python uses a dictionary to store an object's instance attributes. This has many benefits, like allowing us to add new attributes at runtime. However, this convenience comes at a cost.

Dictionaries can consume a fair chunk of memory, especially if we have many instance objects with a large number of attributes. If the performance and memory efficiency of code are critical, we can trade the convenience of dictionaries for __slots__.

1. In Python, there is no default functionality to allocate a static amount of memory while creating the object to store all its attributes. Usage of slots reduce the wastage of space and speed up the program by allocating space for a fixed amount of attributes. Example of python object with slots.
2. Python does not support single-precision floating point numbers; the savings in processor and memory usage that are usually the reason for using these are dwarfed by the overhead of using objects in Python, so there is no reason to complicate the language with two kinds of floating point numbers.

In this tutorial, we will look at how what __slots__ are and how to use them in Python. We'll also discuss the tradeoffs for using __slots__, and look at their performance when compared to typical classes that store their instance attributes with dictionaries.

What Are _slots_ and How to Use Them?

Slots are class variables that can be assigned a string, an iterable, or a sequence of strings of the instance variable names. When using slots, you name an object's instance variables up front, losing the ability to add them dynamically.

An object instance using slots does not have a built-in dictionary. As a result, more space is saved and accessing attributes is faster.

Let's see it in action. Consider this regular class:

In the above snippet:

• organization is a class variable
• name and location are instance variables (note the keyword self in front of them)

While every object instance of the class is created, a dynamic dictionary is allocated under the attribute name as __dict__ which includes all of an object's writable attributes. The output of the above code snippet is:

This can be pictorially represented as:

Now, let's see how we can implement this class using slots:

In the above snippet:

• organization is a class variable
• name and location are instance variables
• The keyword __slots__ is a class variable holding the list of the instance variables (name and location)

Running that code will give us this error:

That's right! Object instances of classes with slots do not have a __dict__ attribute. Behind the scenes, instead of storing the instance variables in a dictionary, the values are mapped with the index locations as shown in the figure below:

While there's no __dict__ attribute, you still access the object properties as you would typically do:

Slots were created purely for performance improvements as stated by Guido in his authoritative blog post.

Let's see if they outperform standard classes.

Efficiency and Velocity of Slots

We are going to compare objects instantiated with slots to objects instantiated with dictionaries with two tests. Our first test will look at how they allocate memory. Our second test will look at their runtimes.

This benchmarking for memory and runtime is done on Python 3.8.5 using the modules tracemalloc for memory allocation tracing and timeit for the runtime evaluation.

Results may vary on your personal computer:

In the above snippet, the calculate_memory() function determines the allocated memory, and the calculate_runtime() function determines the runtime evaluation of the class with slots vs the class without slots.

The results will look something along these lines:

It's evident that using __slots__ gives an edge over using dictionaries in size and speed. While the speed difference is not particularly noticeable, the size difference is significant.

Gotchas Using Slots

Before you jump into using slots in all of your classes, there are a few caveats to be mindful of:

1. It can only store attributes defined in the __slots__ class variable. For example, in the following snippet when we try to set an attribute for an instance that is not present in the __slots__ variable, we get an AttributeError:

Output:

With slots, you need to know all the attributes present in the class and define them in the __slots__ variable.

2. Sub-classes will not follow the __slots__ assignment in the superclass. Let's say your base class has the __slots__ attribute assigned and this is inherited to a subclass, the subclass will have a __dict__ attribute by default.

Consider the following snippet where the object of the subclass is checked if its directory contains the __dict__ attribute and the output turns out to be True:

Output:

This can be averted by declaring the __slots__ variable one more time for the subclass for all the instance variables present in the subclass. Although this seems redundant, the effort can be weighed against the amount of memory saved:

Output:

Conclusion

In this article, we have learned the basics about the __slots__ attribute, and how classes with slots differ from classes with dictionaries. We also benchmarked those two classes with slots being significantly more memory efficient. Finally, we discussed some known caveats with using slots in classes.

If used in the right places, __slots__ can boost performance and optimize the code in making it more memory efficient.

Python has two similar sequence types such as tuples and lists. The most well-known difference between them is that tuples are immutable, that is, you cannot change their size as well as their immutable objects.

You can't changes items in a tuple:

But you can change mutable objects:

Internally, both lists and tuples are implemented as a list of pointers to the Python objects (items). When you remove an item from a list, the reference to an item gets destroyed. Keep in mind, that removed item can stay alive if there are other references in your program to it.

Tuples

Despite the fact that tuples are less popular than lists, it is a fundamental data type, which is used a lot internally.

You may not notice, but you are using tuples when:

• working with arguments and parameters
• returning 2 or more items from a function
• iterating over dictionary's key-value pairs
• using string formatting

Typically, a running program has thousands of allocated tuples.

Empty lists vs. empty tuples

Empty tuple acts as a singleton, that is, there is always only one tuple with a length of zero. When creating an empty tuple Python points to already preallocated one, in such way that any empty tuple has the same address in the memory. This is possible because tuples are immutable and sometimes saves a lot of memory.

But this doesn't apply to lists since they can be modified.

Allocation optimization for small tuples

To reduce memory fragmentation and speed up allocations, Python reuses old tuples. If a tuple no longer needed and has less than 20 items instead of deleting it permanently Python moves it to a free list.

A free list is divided into 20 groups, where each group represents a list of tuples of length n between 0 and 20. Each group can store up to 2 000 tuples. The first (zero) group contains only 1 element and represents an empty tuple.

In the example above we can see that a and b have the same id. That is because we immediately occupied a destroyed tuple which was on the free list.

Allocation optimization for lists

Since lists can be modified, Python does not use the same optimization as in tuples. However, Python lists also have a free list, but it is used only for empty objects. If an empty list is deleted or collected by GC, it can be reused later.

List resizing

To avoid the cost of resizing, Python does not resize a list every time you need to add or remove an item. Instead, every list has a number of empty slots which are hidden from a user but can be used for new items. If the slots are completely consumed Python over-allocates additional space for them. The number of additional slots is chosen based on the current size of the list.

Developer documentation describes it as follows:

This over-allocates proportional to the list size, making room for additional growth. The over-allocation is mild but is enough to give linear-time amortized behavior over a long sequence of appends() in the presence of a poorly-performing system realloc().

The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...

Note: new_allocated won't overflow because the largest possible value is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.

For example, if you want to append an item to a list of length 8, Python will resize it to16 slots and add the 9th item. The rest of the slots will be hidden and reserved for new items.

The growing factor looks as follows:

Performance

If you are interested in speed comparison, there is a good summary about the overall performance by Raymond Hettinger.

Python Slot Machine Code

Want a monthly digest of these blog posts?

Python Slots Memory Drive

• AutoSniper 2 years, 6 months ago (from disqus) #

  This was educational. I encourage using generators and lazy evaluation whenever possible; it is preferred over working with tuples and lists.