neuralgia.

I am about 3 months into a machine learning project that is somewhat complex, and some recurring, foundational problems are cropping up. Even though hindsight is perfect, I think it is still worth exploring what I could have done better upon examining the situation.

1 – Establish expectations and team culture as soon as possible.

I think this would be considered part of team onboarding, but having a common ground for expectations within the team is necessary right from the start. Such expectations would include working arrangements, standard of work, communication styles, best practices (that are not engineering related) and also assigning responsibilities (and power to enforce it).

I only realized this after the 1st sprint ended and I sounded out what I was unhappy about internally with my team. An experienced teammate then did a very useful session to set expectations across the whole team. While it did not resolve all the problems we had with communication and working together, it did improve the working situation drastically.

I also think it is a worthwhile exercise to understand what are some of the non-negotiables for myself when it comes to team expectations, and what I can compromise on. This is useful for avoiding team cultures that are potentially toxic, and choosing the right people to work with.

2 – Identify key features that need to be built first.

The key features that will be used commonly throughout the project have to be built first. Even if they cannot be clearly defined at the start of the project, starting with a boilerplate and fleshing out the details as the project progresses is still important as there is a possibility that such features will be forgotten over time, and the complexity of building it will outweigh the cost past a certain stage in the project.

Some examples of features like this in the context of a machine learning project would be common evaluation frameworks, data processing utilities and format converters and samplers. In some cases it would make sense to share a common code base for implementation classes, since both implementations were meant to utilize the same code in different pipelines.

The drawbacks of not doing this can be noticed and measured quite acutely. Not having a common evaluation framework, for example, makes model evaluation highly subjective and prone to further scrutiny. In such cases especially, I find the trade-off of having to verify if the evaluation metrics conform to a set standard (without it being built in the first place) to be more costly in terms of time and energy, both for the engineer and the product owner.

Sometimes this approach can backfire, if the key features are not designed properly, or if too much time is spent building a key feature that is only required at the end of the project. More importantly it takes intuition, foresight and good project sense to know what, how and when to build such key features.

3 – Review code regularly, get people to review each other’s code.

One big criticism I have for the way code review is conducted with other teams is the idea of getting a group of 5 people (along with a mentor who is often times not software-engineering-trained) to sit together in a meeting room and to take turns going through each other’s code, often only after a sprint has ended.

Fundamentally I do not believe this to work as it can cause misinformation as to what the best practice really is. This method only works with a very experienced software engineer onboard directing the code review and handling the bulk of the technical disagreements that will inevitably come up. However I would still regard it to be largely a waste of time for everyone involved.

In most situations, I believe the best arrangement is still 1-to-1 code reviews conducted between peers and have code review partners rotate regularly. This cuts the time wasted by any single member of the team and allow disagreements or decision points to be noted and surfaced to a more senior engineer. The latter should ideally be held in a separate discussion, else be formalized and brought forth in sprint retro for bigger, foundational points.

4 – Pair programming works, but only with software engineers.

There is a dark side to pair programming which I think is not discussed enough, namely with regards to pair programming with people who are not software engineers. The idea of pair programming is simple but also clearly defined for the sole reason of driving technical work forward.

In scenarios where a software engineer does pair programming with someone who is non-technical, there is a likelihood of a teacher-student dynamic than a driver-navigator dynamic. Pair programming has its uses in bringing a more junior software engineer up to speed, but it cannot be used as a substitute for training non-technical people to become software engineers.