Celery is the default choice whenever there is a need to introduce a worker queue into Python projects. Not not exactly ideal, but it’s feature-rich, gets the job done, and can be reliable when being used properly. Over the years of working with Celery, I’ve developed certain rules that I would recommend one to follow for building reliable background pipelines. Prefer late acknowledgement In the context of background job queues, acknowledgment is the process of notifying the queue that a particular job has been completed. In most cases it means the job can now be removed from the queue. By default, Celery uses early acknowledgment, meaning the queue is acknowledged as soon as the child worker starts the job. What is the problem here? Well, the worker can simply crash (by receiving a SIGKILL for example) and the task will now be lost without any option to rerun it. ...

When I first started using Github Copilot, I felt quite uplifted about it. I set up the plugin in my editor (I use Neovim with copilot.vim plugin), and started enjoying the incredible amount of additional productivity I now possess. I no longer need to type the whole block of code that I came up with in my brain, all I need is a little hint to the model and everything is magically there. Incredible! How did I even live without it before? Well, after a year or so of using it both for professional work and my personal projects, I’m now making a conscious decision to remove it from my workflow (although I’m leaving the manual triggering option). Why would someone do such a weird thing in the AI-driven development era? Well, I took some time to seriously analyze the benefits it brings and the costs I incur (the actual financial cost being the least of significance), and I concluded that the latter greatly outweighs the former. Here is why: ...

Task prioritization can be an effective technique to ensure the most critical tasks are processed in time. There are other ways to achieve this, like using dedicated workers, but prioritizing is arguably the simplest, and in many cases, the most cost-effective approach. Celery supports task prioritization natively, but it works differently depending on the broker you use. If you use RabbitMQ, for example, prioritizing will be handled on the broker side, meaning that the broker will decide which message to deliver first. However, RabbitMQ might be an overkill for a lot of projects, and Redis is often preferred due to its simplicity. It turns out, one can still prioritize tasks with Redis, but it works in a slightly different fashion. ...

Mocks are a powerful concept in testing. They are one of several types of test doubles, which are objects that can be used in place of real objects in your tests. Mocks are used to isolate the code under test from the rest of the system, and to verify that the code under test interacts with its dependencies correctly. However, if not used properly, mocks can lead to false positives in your tests. One common pitfall is that mocks can be too permissive, allowing you to call methods that don’t exist on the real object. This can lead to tests that pass even when the code under test is broken. ...

I graduated from a university with a degree in Finance and immediately started my career in investment banking. After about 7 years of working in the industry, the realization came to me that it wasn’t for me. So in my late 20s I decided to change my career path completely and become a software engineer. I did not have much time nor motivation to obtain another college degree, so I decided to take self learning path. And the transitioning process was much harder than I could imagine, it was probably the hardest thing I’ve ever done in my life. ...

If you follow the Test-driven Development practice in your Python projects, you need to run your test suite often. Having to run it manually can become tedious. You can configure handy shortcuts in your favorite IDE to make the process easier. But there is even better way using pytest-watcher. What is pytest-watcher? pytest-watcher is a continuous test runner for Python projects that reruns your tests whenever you change a *.py file inside your project. ...

When building asynchronous applications, oftentimes you need to limit the number of simultaneous connections to a shared resource. It can be your internal server, or an API that has usage limits. asyncio library provides a dedicated synchronization primitive Semaphore created exactly for this purpose. However, let’s first try to solve this problem without using it, in order to fully appreciate the value of this mechanism. We can limit the number of simultaneous connections by using a counter variable that will be incremented whenever we start making a request and decremented when we receive our response. Let’s look at the example code: ...

I recently discovered the Advent of Code project for myself and I absolutely loved it. At first glance, it seemed similar to solving Leetcode problems. However, after trying it for a few days now I realized that I like the experience much more. In this short blog post I’ll try to break down main advantages that I see over Leetcode: No online editor You don’t have to use the poor built-in online editor. I can freely use my favorite Neovim for this ...

I recently started applying to Big Tech companies, and as a result, I ended up solving a lot of LeetCode problems (284 problems at the time of writing this blog post). During the preparation process, it became clear to me why so many developers argue about the broken state of the tech interviewing process. Solving LeetCode problems takes a significant amount of time and energy that could otherwise be spent on building projects. However, I believe that the main issue lies not with the tech interviews themselves, but rather with the preparation process. ...

If you are working on a large Django project, you probably have lots of automated tests running as part of your CI/CD process. As long as tests run fast, everyone is happy. But as your application grows in complexity, your tests start to take more and more time to run and eventually become a real bottleneck. In this post, I will share some ideas that can help you optimize runtime of your test suite. I assume you are using pytest, but recommendations described in this post should be easily applicable to other runners as well. ...