Interview with Anastasia Abrashitova, Head of Repository Tools at Yandex

Anastasia Abrashitova began her career as a C# developer and is now the Head of Repository Tools at Yandex, leading six teams. She is responsible for the Arcadia mono-repo system, which contains code from all Yandex projects. In an interview with the WebMind portal, Anastasia talks about the benefits and challenges that come with using a mono-repo system and reveals how Yandex manages teams and code that operate within this extensive codebase.

Interview with Anastasia Abrashitova

• Can you describe the complexity of the codebase and the challenges you face in managing the “Arcadia” mono-repo system?

Arcadia represents a mono-repo system, a large codebase containing independent projects. As Yandex is an extremely developed company that has been operating for 26 years, the codebase we manage on a daily basis processes more than 10TB of data in the database, or 240GB of data in the current revision. There are no open-source or commercial tools capable of handling such a massive codebase. Typically, large companies develop their own tools, which is precisely what we have done. We have our own control system based on a virtual file system called Arc. Additionally, we have a web portal for developers to review code, access CI/CD results, and search through the codebase. Moreover, we are continuously developing add-ons for popular IDEs to allow our developers to use them seamlessly with the Arcadia system.

• What were the main reasons for adopting the mono-repo architecture? What benefits have you noticed from using a unified codebase?

Mono-repo provides you with three main advantages:

1. Code reusability – We aim to avoid our developers writing code that has already been written. That’s precisely why we strive to contribute to technological advancement through knowledge sharing and constant application of new information.
2. Knowledge sharing – We have exceptional programmers, and we want them to become even better through experience exchange. Arcadia enables everyone to review codes from other projects, contribute to shared components, and search for specific code segments. Every day, 58 percent of our programmers use code search on the web.
3. Continuous code updates and prevention of unauthorized changes – Since our projects are complex and often involve reusing existing code, it is crucial to ensure that further development is based on the latest version of the code. The mono-repo system allows us to achieve this. Additionally, within the Arcadia system, we use Continuous Integration (CI) programs to ensure that each code update is valid and correct.

• How do you manage the different codes and teams that work within the Arcadia system?

As previously mentioned, code reuse is crucial for us. Therefore, mutual understanding and good collaboration between different teams are essential. To achieve this, we have three rules:

1. Within each project, there is code testing. It is essential for developers to rely on tests that give them a sense of security when all results are labeled green. If an error occurs, and all project tests show success, it is necessary to work on improving the code. That’s why our Continuous Integration (CI) for data storage is truly massive, as we perform over a billion tests daily.
2. Everyone can access the codebase as long as their tests are given the green light. This way, when you contribute to the shared codebase, you can be confident that everything is under control. If some tests fail, it is the responsibility of the programmer to notice and fix the code, thus contributing to the further development of the system.
3. Teams can provide code reviews whenever necessary. It is okay if you want to evaluate the changes your colleagues make to the code you are responsible for, but it is essential to respond fast for a more efficient process.

• Can you give examples of how Arcadia has facilitated code reuse and improved collaboration among developers?

First and foremost, we have a significant amount of shared code that is developed internally. These can be utility functions, open-source components, libraries, and clients for interacting with our infrastructure systems like YDB or YTSaurus. All of them are shared and widely reused, with 44 percent of the code in our projects being shared code.

Furthermore, we promote code sharing between projects. Sometimes, our teams have highly sophisticated code that is not widely needed in projects, so it shouldn’t go into shared components. However, other colleagues may still reuse it, whether through planners or complex encryption, thereby saving time.

• Maintaining a stable and reliable system like a mono-repo is crucial. How do you approach testing and ensuring quality to preserve the integrity of the codebase?

We make a conscious effort to thoroughly test our storage tools before releasing them, as failing to do so could risk losing our developers’ code and even potentially corrupt the database. We have unit and end-to-end tests, as well as stable environments where new versions are tested before being released. Additionally, we conduct internal beta testing to ensure a stable product. We possess mechanisms that allow us to roll back to previous releases and maintain backup copies of the entire repository and tool states in case anything goes wrong. Fortunately, we haven’t had to use them in recent years.

Most importantly, we strive to implement the best Site Reliability Engineering (SRE) practices. Our backends are geographically distributed and horizontally scalable, with traffic frequently passing through load balancers. Data is stored in highly reliable YDB (Yandex Database) storage.

• Are there any specific lessons or best practices you would like to share with other programmers considering working with large-scale mono-repo systems?

There are two golden rules for successful work on a mono-repo system: it is essential to use tools that fit their scale and to maintain constant collaboration with other teams. Sometimes, infrastructure programmers invest all their efforts in tools and forget about the organizational culture. This can result in a large repository of disconnected code instead of a well-functioning mono-repo system. Therefore, it is crucial for both individual programmers and the company as a whole to promote code reuse, knowledge sharing, and mutual trust.