Python Optimization Guide: Make Your Code Run 5X Faster

A journey into the art and science of making your Python code run faster, smarter and more efficiently.

Python Code Optimization

In the world of programming languages, Python stands tall as one of the most versatile languages that offer simplicity and readability. Python has become popular among developers due to its easy-to-read syntax, object-oriented nature, community support and large pool of libraries. It can be used in fields like Data analysis, Artificial Intelligence, Web Development, Game Development and so on.

However, as with any programming language, Python’s strengths come with certain trade-offs. One of them being performance optimization. Due to its interpreted nature there are concerns about its speed and performance. This is where code optimization comes into play.

What is Code Optimization?

Python is an interpreted language and this means it may not run as fast as compiled languages like C or C++. However, there are certain techniques and strategies you can take advantage of to optimize your Python code and improve its performance.

Code optimization involves making your code run faster, use less resources and execute more smoothly, hence increasing its performance and efficiency.

This article is a journey into the art and science of making your Python code run faster, smarter and more efficiently. You will learn various techniques, strategies and best practices that enables developers harness Python’s full potential while overcoming its certain performance limitations.

We would be using Python’s timeit module for benchmarking and comparing execution times between traditional Python codes and optimized Python codes. Note that the timeit module runs the function a million times - by default.

Without further ado, let’s get right into the optimization techniques and strategies.

Use Comprehensions and Generators

In versions 2.7 and 3.0, Python, among other features, released the features: List comprehension, Dictionary comprehension and the closely related Set comprehension. These features made it easier to generate lists, dictionaries and sets, in a cleaner, concise and more efficient way.

Create a function and generate a list using the traditional “looping and appending” method:

>>> def do_1():
...     list_object = []
...     for i in range(100):
...         list_object.append(i)

Import Python’s in-built timeit module to see how long this function runs:

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do_1', stmt='do_1()')
>>> print(t.timeit())

9.681053700041957

The output above shows that it takes the function approximately 9.68 seconds to run.

Now, use comprehension to generate this list and see how long it takes:

>>> def do():
...     [i for i in range(100)]

>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()

7.292758799972944

As you can see from the code above, it took this function approximately 7.29 seconds to run as opposed to 9.68 seconds from the previous function - without comprehension — which is a 2.39 seconds difference.

In addition to comprehensions being more concise and easier to read, it is also faster. Which makes it the preferred method for generating lists and looping in general.

I wrote a more in-depth article on list, dictionary and sets comprehensions here.

Avoid String Concatenation

String concatenation using the += operator is commonly used by developers to concatenate strings. However, within a loop, it can be slow due to the immutable nature of strings.

Instead, use the str.join() method for efficient concatenation.

Concatenate strings using the += operator and see its execution time:

>>> def do():
...     obj = ["hello", "my", "name", "is", "Delight", "!"]
...     s = ""
...     for elem in obj:
...         s += elem

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()

0.9870554000372067

It takes this function — which implements string concatenation using += operator — approximately 0.98 seconds to complete.

Using join() instead for effective concatenation:

>>> def do():
...     s = ["hello", "my", "name", "is", "Delight", "!"]
...     "".join(s)

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()

0.38973289995919913

Using join() reduces the execution time of the function from 0.98s to 0.38s which makes it a faster and preferable method of concatenating strings.

Loops

In Python, a for loop is a control structure that allows you to iterate over a sequence of items, such as elements in a list, characters in a string, or elements in other iterable objects. The for loop repeatedly executes a block of code for each item in the sequence, iterating from the first item to the last.

However, in most cases, for loops could be replaced by a more efficient function known as map(). Themap() function is a built-in higher-order function that allows you to apply a given function to each item in an iterable (such as a list, tuple, or string) and generate a new iterable containing the results of applying that function to each item. The key advantage of using map() is that it provides a concise and efficient way of transforming data without the need for explicit loops.

Compare the execution time of functions implementing for loops and map():

def do():
    obj = ["hello", "my", "name", "is", "Delight", "!"]
    new = []
    for i in obj:
        new.append(i.upper())

Get the execution time of this function:

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()

1.042804399970919

Implement this same function using the in-built map() function and get the execution time:

>>> def square(x):
...     return x.upper()

>>> def do():
...     obj = ["hello", "my", "name", "is", "Delight", "!"]
...     map(square, obj)

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()

0.37273399997502565

In the code above, using map() instead of a for loop made the function run about 3 times faster.

Python’s in-built functions makes our code run faster mainly because they are written and compiled in C language.

Choosing the right Data Structure

Choosing the right data structure can have a significant impact on the speed and efficiency of your Python code. Different data structures are optimized for specific types of operations, and selecting the appropriate one can lead to faster lookups, insertions, deletions, and overall improved performance.

For example, using sets for membership tests is much faster than using lists:

>>> def do():
...     fruits_list = ['apple', 'banana', 'orange', 'grape', 'pear']
...     'banana' in fruits_list
...     'kiwi' in fruits_list

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()
0.48580530006438494

>>> def do():
...     fruits_list = {'apple', 'banana', 'orange', 'grape', 'pear'}
...     'banana' in fruits_list
...     'kiwi' in fruits_list

>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import do', stmt='do()')
>>> t.timeit()
0.34570479998365045

Avoid Global Variables

Global variables play an important role in sharing data across a program. However, they should be used careful and only when necessary. Accessing global variables is slower than accessing local variables. Take note to always minimize the use of global variables, especially within loops.

Vectorization

Vectorization in Python refers to the practice of applying operations to entire arrays or sequences of data instead of using explicit loops to iterate over individual elements. It leverages specialized libraries like NumPy to efficiently perform element-wise operations on arrays, taking advantage of hardware-level optimizations and reducing the need for explicit looping constructs.

If you’re performing numerical computations, consider using libraries like NumPy which provides optimized array operations that can be significantly faster than performing element-wise operations in standard Python lists.

Vectorization is a fundamental concept in scientific computing and numerical analysis, and it plays a crucial role in making Python a powerful language for data analysis, machine learning, and other computational tasks involving large datasets.

Avoid Unnecessary Function Calls

Avoiding unnecessary function calls in Python is important for improving the efficiency and performance of your code. Unnecessary function calls can introduce overhead, consume memory, and slow down the execution of your program. Try to combine operations where possible.

Avoid Unnecessary Import Statement

import statements can be executed just about anywhere. It’s often useful to place them inside functions to restrict their visibility and/or reduce initial startup time. — Python.org

Avoiding unnecessary import statements in Python is essential for maintaining clean, efficient, and readable code. Unnecessary imports can sometimes lead to circular dependencies between modules. This could cause issues during runtime and making it harder to refactor your code.

Although, modern text editors now make it easy for you to identify unused code in your program, adding unused imports can confuse these editors.

Compile Your Code

If performance is a critical factor in the execution of your program, consider using tools like Cython which compiles your code to C or C++. This could significantly improve the performance of your program.

By using tools like Cython, you can take advantage of the expressiveness of Python and the performance of a compiled language like C or C++.

Profile Your Code

The first step to speeding up your program is learning where the bottlenecks lie. - Python.org

Profiling in Python is measuring and analysing the execution time of your code and its resource usage in order to find bottlenecks in your code and opportunities for optimization.

Python provides various built-in tools for profiling like profile and timeit (which we used in this article).

The following is an example of profiling a function named do() using the profile module:

import profile
profile.run('do()')

Stay up-to-date with Python Releases

Newer Python releases often include performance enhancements, bug fixes, security updates, and so on.

Python is constantly evolving, and new releases often include performance improvements. Keeping your interpreter up to date can help you benefit from these enhancements.

However, it’s important to strike a balance between staying current and considering the impact of upgrading on your existing projects. If you have a large codebase or depend on third-party libraries that aren’t yet compatible with the latest release, a cautious approach to upgrading may be necessary.

Conclusion

In this article, we have covered various techniques, strategies, and best practices that enable developers to harness Python’s full potential while overcoming its certain performance limitations.

It is important to note that optimization is not just about shaving off milliseconds or microseconds from your code’s execution time; it’s about writing readable, scalable, and maintainable code.

Thanks for reading.

Connect with me on GitHub, Twitter, and LinkedIn.

In Plain English

Thank you for being a part of our community! Before you go:

• Be sure to clap and follow the writer! 👏
• You can find even more content at PlainEnglish.io 🚀
• Sign up for our free weekly newsletter. 🗞️
• Follow us on Twitter, LinkedIn, YouTube, and Discord.